Multimodal Sentiment and Personality Perception Under Speech: A Comparison of Transformer-based Architectures

Ádám Fodor, Rachid R. Saboundji, Julio C. S. Jacques Junior,
Sergio Escalera, David Gallardo-Pujol, András Lőrincz

Paper Live demo Code
PyPI Docs Model weights Dataset

Abstract

Human-machine, human-robot interaction, and collaboration appear in diverse fields, from homecare to Cyber-Physical Systems. Technological development is fast, whereas real-time methods for social communication analysis that can measure small changes in sentiment and personality states, including visual, acoustic and language modalities are lagging, particularly when the goal is to build robust, appearance invariant, and fair methods. We study and compare methods capable of fusing modalities while satisfying real-time and invariant appearance conditions. We compare state-of-the-art transformer architectures in sentiment estimation and introduce them in the much less explored field of personality perception. We show that the architectures perform differently on automatic sentiment and personality perception, suggesting that each task may be better captured/modeled by a particular method. Our work calls attention to the attractive properties of the linear versions of the transformer architectures. In particular, we show that the best results are achieved by fusing the different architectures’ preprocessing methods. However, they pose extreme conditions in computation power and energy consumption for real-time computations for quadratic transformers due to their memory requirements. In turn, linear transformers pave the way for quantifying small changes in sentiment estimation and personality perception for real-time social communications for machines and robots.

Task
  • Develop and compare transformer-based architectures for real-time multimodal sentiment analysis and personality perception under speech.
  • Focus on fusing audio, visual, and textual modalities in a way that balances accuracy, efficiency, and fairness.
Challenges
  • Managing complex temporal dynamics and interdependencies in multimodal data during speech.
  • Handling appearance-related bias by using appearance-invariant features such as Action Units.
  • Addressing the computational and memory limitations of quadratic attention in transformers for real-time processing.
  • Navigating the regression-to-mean problem in personality trait prediction due to unbalanced label distributions.
Proposed Method
  • Evaluate and compare three architectures: Multimodal Transformer (MulT) with quadratic attention, Linear Multimodal Transformer (LinMulT) with linear attention, and Self-Supervised Embeddings Transformer (SSE-FT).
  • Use modality-specific preprocessing: handcrafted features (e.g., eGeMAPS, Action Units, BERT) and self-supervised embeddings (e.g., Wav2Vec, Fab-Net, RoBERTa).
  • Fuse modalities through cross-modal attention and self-attention layers operating on temporal embedded sequences.
  • Train models on large-scale datasets for sentiment (CMU-MOSI, CMU-MOSEI) and personality perception (ChaLearn First Impressions), employing loss functions addressing label imbalance.
Main Results
  • On CMU-MOSI and CMU-MOSEI sentiment datasets, SSE-FT (with self-supervised embeddings) achieved the best overall performance.
  • MulT outperformed LinMulT on mean absolute error but LinMulT showed improved precision/recall balance and better scalability to larger datasets.
  • For personality perception on the First Impressions dataset, all architectures yielded similar results due to label centralization; however, LinMulT with combined features and trait-wise training outperformed others in overall accuracy.
  • Cross-modal ablation showed no dominant modality for prediction, with varying effectiveness depending on the target modality and architecture.
  • LinMulT offered significant efficiency gains in memory and speed, making it suitable for real-time applications despite slightly lower accuracy compared to quadratic transformers.
  • Reported results achieve performance competitive with state-of-the-art, while promoting reduced bias through use of appearance-invariant features.

From paper to production

Two promoted champions ship from this repository — wavlm_best-ccc (acoustic only) and avt_best-ccc (acoustic + visual + textual fusion). Both are wired end to end through the public API, ONNX export, container, and hosted demo.

Model lifecycle

  • Every run tracked in MLflow, with the Model Registry as the promotion boundary.
  • Alias-based promotion (@champion, not deprecated stages), so a specific trained run becomes “the” deployable model through a deliberate, auditable step.
  • Optuna hyperparameter optimization and systematic feature-ablation matrices across seven features, trained with PyTorch Lightning across multiple GPUs (DDP).

Deployment

  • A Docker image serving the ONNX path, running identically on localhost and on Google Cloud Run.
  • Cloud Run with true scale-to-zero, so an idle demo bills nothing.
  • A Hugging Face Space with a live PyTorch-vs-ONNX backend switch, comparing both runtimes on the same input.

Public inference API

  • pip install personalitylinmult, then PersonalityModel.from_pretrained(...) — no MLflow and no repo clone needed.
  • Checkpoints are self-describing, so one class serves every architecture; you never track LinT vs. LinMulT yourself.
  • An optional ONNX extra gives a full raw-audio-to-prediction pipeline with no PyTorch or Lightning installed.

Reproducibility & metrics

  • Training reads a single HDF5 artifact and nothing else — that file is the reproducibility boundary.
  • It embeds the builder config verbatim, the commit SHA, and the label hash, so a built file always traces back to what produced it.
  • Reported alongside 1-MAE: Pearson r, CCC, R², and std_ratio, which catches regression-to-the-mean that 1-MAE alone hides.

Visualization

(a) Standard Multi-head attention unit.
(b) Multimodal transformer: source modalities i and j are transformed to target modality k. Such two units are combined by another transformer network that utilizes self-attention to fuse the information pieces to form a branch within the multimodal network before outputting the predicted score(s).

BibTex

If you found our research helpful or influential please consider citing:

@InProceedings{pmlr-v173-fodor22a,
   title = {Multimodal Sentiment and Personality Perception Under Speech: A Comparison of Transformer-based Architectures},
   author = {Fodor, {\'A}d{\'a}m and Saboundji, Rachid R. and Jacques Junior, Julio C. S. and Escalera, Sergio and Gallardo-Pujol, David and L{\H{o}}rincz, Andr{\'a}s},
   booktitle = {Understanding Social Behavior in Dyadic and Small Group Interactions},
   pages = {218--241},
   year = {2022},
   editor = {Palmero, Cristina and Jacques Junior, Julio C. S. and Clapés, Albert and Guyon, Isabelle and Tu, Wei-Wei and Moeslund, Thomas B. and Escalera, Sergio},
   volume = {173},
   series = {Proceedings of Machine Learning Research},
   month = {16 Oct},
   publisher = {PMLR},
   pdf = {https://proceedings.mlr.press/v173/fodor22a/fodor22a.pdf},
   url = {https://proceedings.mlr.press/v173/fodor22a.html}
}
Details at the Top: Paper