Exordium

Collection of preprocessing functions and deep learning methods for multimodal feature extraction across audio, video, and text modalities.

Contents

• API Reference
  • exordium

Features

Audio

• I/O utilities — load, save, resample waveforms

• Spectrograms — MFCC and Mel-spectrogram (with pre-emphasis)

• OpenSMILE — eGeMAPSv02 low-level descriptors

• CLAP — contrastive language–audio pre-training embeddings (512-d)

• Wav2Vec2 — self-supervised speech representations (base-960h / emotion-iemocap, 768-d)

• WavLM — masked speech modelling, layer-wise hidden states (768-d)

• emotion2vec+ — speech emotion features at ~50 Hz (768-d)

Video

• Face detection — YOLOv8-Face, YOLO11-pose, BlazeFace

• Facial landmarks — YOLO11 5-pt coarse keypoints, FaceMesh 478-pt dense mesh

• Head pose — SixDRepNet (yaw, pitch, roll in degrees)

• Gaze direction — L2CS-Net (ResNet-50), roll correction

• Iris landmarks — MediaPipe Iris 71 eye pts + 5 iris pts, EAR, iris diameters

• Action units — OpenGraphAU 41-dim intensity vector

• Deep visual features — Swin Transformer (768-d), AdaFace IResNet-18/50/101 (512-d identity embeddings), FAb-Net (256-d), CLIP ViT-H/14 (1024-d), DINOv2 (384 / 768 / 1024 / 1536-d), EmotiEffNet (1280 / 1408-d), MARLIN (384 / 768 / 1024-d, 16-frame clips)

• Tracking — IoU-based multi-face tracker; face-ID tracker (AdaFace embeddings + IoU gating) with occlusion recovery and identity-aware merge

Text

• Whisper — speech-to-text transcription (OpenAI Whisper)

• Word-level timestamps — Whisper ASR + forced alignment (torchaudio MMS_FA or whisperX), returns per-word (start, end, score)

• Forced alignment — align a known transcript to audio; interchangeable TorchaudioForcedAligner / WhisperxForcedAligner backends

• Transcript re-alignment — dataset-agnostic fuzzy search (rapidfuzz) of a known transcript in the word stream to recover true segment (start, end) and a coverage score

• Annotation checks — raw + Whisper-normalized (standard EnglishTextNormalizer) WER / CER, rapidfuzz string similarity, and XLM-RoBERTa semantic cosine, with a heuristic is_acceptable verdict; standalone TranscriptEvaluator (two strings) or via the SpeechAlignmentPipeline facade

• Segment validation — per-segment accept / recut / drop decisions with recovered start/end times

• BERT — token-level and sentence-level embeddings (768-d)

• RoBERTa — token-level and sentence-level embeddings (1024-d)

• XML-RoBERTa — multilingual sentence embeddings (768-d)

Utilities

• Concurrent processing — thread- and process-pool helpers

• Decorators — @load_or_create caching, retry, timing

• Normalization — global, per-feature, sliding-window

• Padding — fixed-length sequence padding and masking

• Loss functions — MSE, CCC, combined losses

Installation

pip install exordium          # base only
pip install exordium[all]     # all optional dependencies
pip install exordium[audio]   # audio extras only
pip install exordium[video]   # video extras only
pip install exordium[text]    # text extras only

Extras

Extra	Dependencies
audio	OpenSMILE, torchaudio — audio feature extraction
text	transformers, torchaudio, rapidfuzz, whisperX — text, speech, and alignment
video	MediaPipe, Ultralytics, timm — face & video models
all	all previously described extras

Example Notebooks

Interactive demos for every module are in the examples/ directory. All notebooks use fixture files from tests/fixtures/ — model weights are downloaded automatically on first run.

Notebook	Description
demo_video_io.ipynb	Video loading, frame iteration, batch access, FPS resampling
demo_video_deep.ipynb	Deep visual features: SwinT (768-d), AdaFace (512-d identity), FabNet (256-d), CLIP ViT-H/14 (1024-d), DINOv2 (384–1536-d), EmotiEffNet (1280/1408-d), MARLIN (384–1024-d, 16-frame clips)
demo_video_face_bb.ipynb	Face detection: YOLOv8 vs YOLO11 on easy and hard (extreme-pose) images
demo_video_face_landmarks.ipynb	Landmarks: YOLO11 5-pt keypoints + FaceMesh 478-pt dense mesh
demo_video_face_headpose.ipynb	Head pose (SixDRepNet) — yaw/pitch/roll with axis and cube overlays
demo_video_face_gaze.ipynb	Gaze direction: L2CS-Net (ResNet-50), roll correction
demo_video_face_iris.ipynb	Iris landmarks: 71 eye pts + 5 iris pts, EAR, iris diameters (YOLO11 + FaceMesh pipeline)
demo_video_face_action_units.ipynb	Facial action units with OpenGraphAU — 41-dim AU intensity vector
demo_video_tracking.ipynb	Multi-face tracking: IoU tracker and face-ID tracker (AdaFace) with occlusion recovery
demo_audio.ipynb	Audio features: spectrogram, OpenSMILE, CLAP, Wav2Vec2, WavLM, emotion2vec+
demo_text.ipynb	Text features: Whisper transcription, BERT, RoBERTa, XML-RoBERTa
demo_text_alignment.ipynb	Word-level timestamps, transcript re-alignment, and annotation validation via SpeechAlignmentPipeline (torchaudio + whisperX backends)

Development

git clone https://github.com/fodorad/exordium
cd exordium
uv pip install -e ".[all,dev]"
make check   # lint + type-check + test + docs

Related Projects

EmotionLinMulT (202X)

Efficient, transformer-based, multi-task emotion detection system.

• Paper: not published yet

• Code

BlinkLinMulT (2023)

Transformer-based eye blink detection and eye state recognition across 7 public benchmark databases.

• Paper

• Code

PersonalityLinMulT (2022)

LinMulT trained for Big Five personality trait estimation and sentiment analysis.

• Paper

• Code

LinMulT

General-purpose multimodal transformer with linear-complexity attention mechanisms.

• Website

• Code

Contact

Ádám Fodor — adamfodor.com · fodorad201@gmail.com