Motivation

This repository offers a comprehensive collection of preprocessing functions and deep learning techniques. It supports various features including audio processing, video analysis, text feature extraction, utility functions, and visualization tools, making it a versatile toolkit for multimodal data processing and analysis.

Supported Features

Audio	Video	Text	Utils
frequently used io for audio files openSMILE feature extraction spectrogram calculation Wav2Vec feature extraction CLAP feature extraction WavLM feature extraction	frequently used io for videos and frames bounding box manipulation methods face detection with RetinaFace face landmarks and head pose with 3DDFA_V2 body pose estimation with max-human-pose-estimator categorical and dimensional emotion estimation with EmoNet iris and pupil landmark estimation with MediaPipe Iris fine eye landmark estimation with MediaPipe FaceMesh action unit estimation with OpenGraphAU eye gaze vector estimation with L2CS-Net video background removal with RVM tracking using IoU and DeepFace FAb-Net feature extraction OpenFace feature extraction R2+1D feature extraction CLIP feature extraction	BERT feature extraction RoBERTa feature extraction multilingual support with XLM RoBERTa speech-to-text with Whisper V3	parallel processing io decorators loss functions normalization padding

Setup

Install package with all base and optional dependencies from PyPI

pip install exordium[all]

Install package with base dependencies from PyPI

pip install exordium

Install optional dependencies for specific modules

The following extras will install the base and specific dependencies for using TDDFA_V2.

pip install exordium[tddfa]

You can install multiple optional dependencies as well.

pip install exordium[tddfa,audio]

Extras Tag	Description
audio	dependencies to process audio data
text	dependency to process textual data
tddfa	dependencies of TDDFA_V2 for landmark and headpose estimation, or related transformations
detection	dependencies for automatic face detection and tracking in videos
video	dependencies for various video feature extraction methods
all	all previously described extras will be installed

Note: If you are not sure which tag should be used, just go with the all-mighty "all".

Install package for development

git clone https://github.com/fodorad/exordium
cd exordium
pip install -e .[all]
pip install -U -r requirements.txt
python -m unittest discover -s test

Projects using exordium

(2025) EmotionLinMulT

LinMulT is trained for categorical and dimensional emotion estimation, emotion intensity, and sentiment estimation tasks. The network is trained on 4 tasks, using 10 public benchmark databases.

• paper: EmotionLinMulT: Transformer-based Emotion Sensing
• code: coming soon

(2023) BlinkLinMulT

LinMulT is trained for blink presence detection and eye state recognition tasks. Our results demonstrate comparable or superior performance compared to state-of-the-art models on 2 tasks, using 7 public benchmark databases.

• paper: BlinkLinMulT: Transformer-based Eye Blink Detection
• code: https://github.com/fodorad/BlinkLinMulT

(2022) PersonalityLinMulT

LinMulT is trained for Big Five personality trait estimation using the First Impressions V2 dataset and sentiment estimation using the MOSI and MOSEI datasets.

• paper: Multimodal Sentiment and Personality Perception Under Speech: A Comparison of Transformer-based Architectures
• code: https://github.com/fodorad/PersonalityLinMulT