GNN-BERT Fusion
A multimodal music classification system that combines temporal and similarity graphs from audio with BERT-based contextual representations.
01
Problem & Motivation
Standard audio classification pipelines often rely solely on flat spectrograms or 1D convolutions, failing to capture complex non-local acoustic relationships and temporal transitions. Furthermore, combining audio representations with textual context via naive concatenation (early fusion) loses modality-specific structural nuances.
A research-oriented multimodal classification system combining Graph Neural Networks for acoustic structure with BERT-based representations for textual context.
02
System Architecture
Multimodal Graph & Cross-Attention Pipeline
Audio Graph Construction
Constructed temporal and similarity graphs from audio segment feature spaces to model intra-track structural transitions.
Graph Representation Learning
Applied GraphSAGE via PyTorch Geometric to aggregate neighborhood feature information across graph nodes into dense acoustic embeddings.
Contextual Text Encoding
Passed track metadata, lyrics, and contextual textual descriptors through BERT to extract contextual token embeddings.
Cross-Attention Fusion
Fused acoustic graph representations and textual BERT representations using a multi-head cross-attention mechanism before classification.
03
Technical Decisions & Trade-Offs
GraphSAGE over standard GCN
GraphSAGE supports inductive node embedding generation and efficient neighborhood sampling, scaling effectively across variable-length audio graph topologies.
Cross-Attention over Naive Concat / Early Fusion
Cross-attention dynamically weights acoustic features against contextual textual tokens, resolving misalignment between disparate modality representations.
04
Data & Experimental Evaluation
Dataset Specifications
- Dataset:Free Music Archive (FMA Small)
- Size:8,000 tracks across 8 balanced genres
- Split:6,400 training / 800 validation / 800 test
- Protocol:
Strict separation between training and evaluation splits with zero track leakage.
Evaluated Baselines & Ablations
- Unimodal BERT Baseline
- Unimodal GraphSAGE GNN Baseline
- CNN Audio Baseline (Spectrograms)
- Early Fusion (Feature Concatenation)
- Cross-Attention Multimodal Fusion (Proposed)
06
Technologies & Tools
07
Key Takeaways & Findings
Demonstrated that graph-structured audio representations capture structural transitions that standard spectrogram CNNs miss.
Cross-attention systematically outperformed early concatenation across all evaluated metrics on the 8,000-track dataset.