Semantic2Graph: graph-based multi-modal feature fusion for action segmentation in videos

Junbin Zhang, Pei Hsuan Tsai*, Meng Hsun Tsai

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review


Video action segmentation have been widely applied in many fields. Most previous studies employed video-based vision models for this purpose. However, they often rely on a large receptive field, LSTM or Transformer methods to capture long-term dependencies within videos, leading to significant computational resource requirements. To address this challenge, graph-based model was proposed. However, previous graph-based models are less accurate. Hence, this study introduces a graph-structured approach named Semantic2Graph, to model long-term dependencies in videos, thereby reducing computational costs and raise the accuracy. We construct a graph structure of video at the frame-level. Temporal edges are utilized to model the temporal relations and action order within videos. Additionally, we have designed positive and negative semantic edges, accompanied by corresponding edge weights, to capture both long-term and short-term semantic relationships in video actions. Node attributes encompass a rich set of multi-modal features extracted from video content, graph structures, and label text, encompassing visual, structural, and semantic cues. To synthesize this multi-modal information effectively, we employ a graph neural network (GNN) model to fuse multi-modal features for node action label classification. Experimental results demonstrate that Semantic2Graph outperforms state-of-the-art methods in terms of performance, particularly on benchmark datasets such as GTEA and 50Salads. Multiple ablation experiments further validate the effectiveness of semantic features in enhancing model performance. Notably, the inclusion of semantic edges in Semantic2Graph allows for the cost-effective capture of long-term dependencies, affirming its utility in addressing the challenges posed by computational resource constraints in video-based vision models.

Original languageEnglish
Pages (from-to)2084-2099
Number of pages16
JournalApplied Intelligence
Issue number2
StatePublished - Jan 2024


  • Computer vision
  • Graph neural networks
  • Multi-modal fusion
  • Semantic features
  • Video action segmentation


Dive into the research topics of 'Semantic2Graph: graph-based multi-modal feature fusion for action segmentation in videos'. Together they form a unique fingerprint.

Cite this