Motivation

This repository presents LinMulT, a general-purpose Multimodal Transformer with Linear complexity attention mechanism, offering a versatile solution for various multimodal learning tasks. It seamlessly handles multiple modalities, be it vectors or sequences, and is trainable for both classification tasks and sequential predictions.

Setup

Install package from PyPI

pip install linmult

Install package for development

git clone https://github.com/fodorad/LinMulT
cd LinMulT
pip install -e .
pip install -U -r requirements.txt
python -m unittest

Similar projects using LinMulT

(2025) EmotionLinMulT

LinMulT is trained for categorical and dimensional emotion estimation, emotion intensity, and sentiment estimation tasks. The network is trained on 4 tasks, using 10 public benchmark databases.

• paper: EmotionLinMulT: Transformer-based Emotion Sensing
• code: coming soon

(2023) BlinkLinMulT

LinMulT is trained for blink presence detection and eye state recognition tasks. Our results demonstrate comparable or superior performance compared to state-of-the-art models on 2 tasks, using 7 public benchmark databases.

• paper: BlinkLinMulT: Transformer-based Eye Blink Detection
• code: https://github.com/fodorad/BlinkLinMulT

(2022) PersonalityLinMulT

LinMulT is trained for Big Five personality trait estimation using the First Impressions V2 dataset and sentiment estimation using the MOSI and MOSEI datasets.

• paper: Multimodal Sentiment and Personality Perception Under Speech: A Comparison of Transformer-based Architectures
• code: https://github.com/fodorad/PersonalityLinMulT

BibTex

If you found our research helpful or influential please consider citing:

(2023) LinMulT for blink presence detection and eye state recognition

@Article{fodor2023blinklinmult,
   AUTHOR = {Fodor, {\'A}d{\'a}m and Fenech, Kristian and L{\H{o}}rincz, Andr{\'a}s},
   TITLE = {BlinkLinMulT: Transformer-Based Eye Blink Detection},
   JOURNAL = {Journal of Imaging},
   VOLUME = {9},
   YEAR = {2023},
   NUMBER = {10},
   ARTICLE-NUMBER = {196},
   URL = {https://www.mdpi.com/2313-433X/9/10/196},
   PubMedID = {37888303},
   ISSN = {2313-433X},
   DOI = {10.3390/jimaging9100196}
}

(2022) LinMulT for personality trait and sentiment estimation

@InProceedings{pmlr-v173-fodor22a,
   title = {Multimodal Sentiment and Personality Perception Under Speech: A Comparison of Transformer-based Architectures},
   author = {Fodor, {\'A}d{\'a}m and Saboundji, Rachid R. and Jacques Junior, Julio C. S. and Escalera, Sergio and Gallardo-Pujol, David and L{\H{o}}rincz, Andr{\'a}s},
   booktitle = {Understanding Social Behavior in Dyadic and Small Group Interactions},
   pages = {218--241},
   year = {2022},
   editor = {Palmero, Cristina and Jacques Junior, Julio C. S. and Clapés, Albert and Guyon, Isabelle and Tu, Wei-Wei and Moeslund, Thomas B. and Escalera, Sergio},
   volume = {173},
   series = {Proceedings of Machine Learning Research},
   month = {16 Oct},
   publisher = {PMLR},
   pdf = {https://proceedings.mlr.press/v173/fodor22a/fodor22a.pdf},
   url = {https://proceedings.mlr.press/v173/fodor22a.html}
}

Acknowledgement

The code is inspired by the following two materials:

Multimodal Transformer

• paper: Multimodal Transformer for Unaligned Multimodal Language Sequences (1906.00295)
• code: https://github.com/yaohungt/Multimodal-Transformer

Linear Attention

• paper: Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention (2006.16236)
• code: https://github.com/idiap/fast-transformers