Multimodal Analytics
Computational frameworks for gesture, speech, and visual data in collaborative settings. Co-speech gesture detection and kinematic analysis from video.
Design Scientist
Babajide
Alamu Owoyele
Design scientist building reliable cartographic frameworks that help Social Sciences and Humanities researchers integrate multimodal data capture and analysis into their existing workflows. By capturing, integrating, and analyzing data across diverse sources — gesture, speech, and audio-visual recordings — these tools empower new research designs while complying with state-of-the-art standards for responsible AI: maximizing analytic utility and minimizing privacy risks.
Current Position
Design Scientist
Centre for Language Studies, Radboud University
PI: Mark Dingemanse
Affiliate
Affiliate Researcher
AI & Intelligent Systems, Hasso Plattner Institute
PI: Gerard de Melo
Location
Nijmegen, Netherlands
Potsdam, Germany
Research Focus
Multimodal Analytics
Design Science · Systems Level Transformations
Research Areas
What I work on
Design Science Research
Creating innovative artifacts that bridge academic theory with practical solutions for complex social systems and sustainability challenges.
Systems Level Transformations
Computational methods to understand how actors and concepts enact social systems that accelerate or decelerate sustainability at scale.
AI Ethics & Privacy
MaskingOPS — operational layer over MaskAnyone and MaskBench — embeds dashboards and archiving steps into the de-identification workflow, maximising collaboration potential while protecting participant privacy.
Community Building
Co-leading collaborative platforms that democratize access to multimodal research tools and foster international scientific communities.
Actor Role Cartography
Mapping role plurality and interconnectedness in multi-actor constellations using network analysis and computational text methods.
Funded Research
Projects / Grants
SYNAPSIS
NWO Funded · €250K
PI
2025–2027
Privacy-Preserving Audiovisual Data Sharing for Research
TDCC-SSH Challenge Project enabling researchers to share audiovisual data safely. Built around MaskingOPS — the operational layer that leverages MaskAnyone and MaskBench to maximise the potential for collaboration, embedding dashboards and archiving steps directly into the research workflow. SURF cloud infrastructure and researcher training make the full pipeline accessible across Dutch universities.
Capabilities
- Face & voice masking for GDPR compliance
- Body kinematics extraction
- SURF cloud integration
- Masking School training workshops
Impact
- 100+ trainees across Dutch universities
- 7 institutional partners
- DANS long-term data archiving
- Open science sustainability model
Privacy Tech
Open Science
Training
GDPR
Partners: VU Amsterdam, UvA, EUR, University of Groningen, Tilburg University, SURF, DANS
SMASH
NWO Funded · €50K
2024–Jun 2026
Concluded
Synthesis and Multimodal Analytics System for Humanities
Fully open-source tool that analyzes non-verbal communication — body language, facial expressions, and speech patterns. Integrates privacy-preserving data archiving with behavioral analysis, making multimodal communication research accessible to non-programmers.
Capabilities
- Hand gesture analysis & diversity metrics
- Facial expression & posture tracking
- Voice tone variation analysis
- Automatic speech transcription
Impact
- Cross-cultural speaker databases
- Clinical research applications
- Social sciences & humanities tool
- Privacy-first data archiving
Multimodal AI
Computer Vision
NLP
Privacy Tech
Open Source
Collaboration: Radboud University, Hasso Plattner Institute · Co-investigators: Gerard de Melo, Wim Pouw · Radboud project page
SMASHFIRE
NWO Impact Explorer · €30K
2026
Multimodal Reconstruction for Emergency Operations Analysis
Brandweer Rotterdam-Rijnmond currently deletes operational video every 28 days — a privacy necessity that also discards an enormous amount of analytic utility. SMASHFIRE builds the de-identification and archiving infrastructure that makes retention possible: maximising the analytic utility of footage that would otherwise be permanently deleted, by reconstructing speech, movement, drone feeds, and situational data streams into a multi-source evidential record. As extreme weather makes large-scale fires and floods a recurring reality across Europe, the approach is designed to transpose — with active interest from partners in Germany and beyond.
Analysis Capabilities
- Speech & communication pattern reconstruction
- Fire spread detection via computer vision
- Missing persons detection from UAV/drone footage
- GRIP-1 command protocol compliance detection
- Stress-consistent movement signatures
Privacy & Wellbeing Layer
- MaskAnyone masking of sensitive scenes in operational footage
- Protects UAV/drone operators from PTSD exposure
- Map rather than measurement device — partial, auditable, revisable
- Built on SYNAPSIS FAIR data infrastructure
Emergency Services
Multimodal Reconstruction
Crisis Informatics
Privacy-Preserving
Climate Resilience
Partner: Brandweer Rotterdam-Rijnmond · Co-PI: Wim Pouw (Tilburg University)
REDES
TSHD Seed Money · €20K
2026
Surveillance Sandbox
Seed project exploring privacy-preserving analysis of video from safety-critical environments. Develops community standards for responsible masking in high-stakes surveillance contexts, bridging technical capabilities with ethical frameworks for operational video data.
Privacy
AI Ethics
Policy
Surveillance
Collaboration: Tilburg University · TSHD · PI: Wim Pouw
EnvisionBOX Object Annotator
Donders Cross-Centre · €10K
2026
Object Annotation for Interaction Video Corpora
Cross-Centre Collaboration Fund project developing an open annotation tool for objects in multimodal interaction recordings. Enables researchers to label and track objects across gesture, action, and communication studies — filling a gap in the EnvisionBOX ecosystem between raw video capture and kinematic analysis.
Computer Vision
Annotation
Open Source
Multimodal
Collaboration: Donders Institute · Co-PI: Wim Pouw (Tilburg / Donders) · GitHub · Project page
EnvisionBOX
Co-Lead
Ongoing
Community Platform for Multimodal Data Analysis
Community-driven platform for sharing multimodal data analysis solutions. Co-leading development of open-source tools that lower entry barriers for newcomers in multimodal signal processing.
Mission
- Increase research transparency & reproducibility
- Lower entry barriers for newcomers
- Foster international collaboration
- Democratize multimodal research tools
Community Impact
- Teaching modules & dynamic dashboards
- Full body tracking with MediaPipe
- International research collaboration
- Educational platform serving globally
Community
Open Science
Education
Platform
Team: Cognitive scientists, AI researchers, linguists across Europe · Partners: Tilburg University, University of Amsterdam, Leibniz Centre Berlin · envisionbox.org
Open Source
Tools / Workflows
EnvisionHGdetector Framework
2025Computational framework for co-speech gesture detection and kinematic analysis in multimodal communication research.
Python
Computer Vision
Machine Learning
Preprint
MaskAnyone Toolkit
2024Four masking strategies — each a different point on the privacy–analytic utility spectrum. The key finding: masking is not only a tradeoff. In cluttered scenes, contour and solid fill overlays reduce visual noise enough that pose estimation and scene reconstruction actually improve — privacy and analytic quality as complements, not competitors. Blur and pixelation protect identity most aggressively at higher utility cost; contour overlays preserve the kinematic and gestural structure that behavioural analysis depends on.
Privacy
Data Ethics
Research Tools
GitHub
MaskBench
2025Benchmark framework that quantifies how masking affects analytic utility — pose estimation accuracy, kinematic fidelity, gesture timing. The evidence shows that certain strategies don't merely preserve utility: they improve it, as contour and solid fill overlays reduce visual clutter that otherwise degrades pose estimators and scene reconstruction. Gives researchers an evidence base for strategy selection. With Tim Riedel, Zainab Zafari, Sharjeel Shaikh, Wim Pouw, Gerard de Melo.
Benchmarking
Computer Vision
Privacy
GitHub
InterPerDyn Pipeline
2025Open-source standardized pipeline for equitable observations of interactive behavioral dynamics. Processes video recordings through YOLOv8 pose estimation, signal processing, movement smoothness extraction, and non-linear dynamical systems analysis.
Python
Pose Estimation
Time Series
R
GitHub
Deep Narratives Africa (DNA)
StealthSocio-semantic analysis of development actor narratives across African institutions using natural language processing.
NLP
Text Mining
Social Analysis
Academic Output
Publications
Published & In Press
MaskingOPS: A Tutorial for Navigating Video De-identification and Analytic Utility Workflows in Behavioral Research
Behavior Research Methods · 2026 · In press · with Martin Schilling, Tim Riedel, Sharjeel Shaikh, Zainab Zafari, Wim Pouw, Gerard de Melo · Tutorial Special Collection BR-TSC-26-051 · Preprint PDF · Zenodo
MaskingOPS is the operational layer that leverages MaskAnyone and MaskBench to maximise the potential for collaboration — embedding project dashboards and archiving steps directly into the de-identification workflow. The tutorial covers the full pipeline from masking strategy selection through analytic utility verification to FAIR-compliant data archiving.
An Open-Source Standardized Pipeline for Equitable Observations of Interactive Behavioral Dynamics: Theory-driven Measurement, Analysis, and Masking
American Psychologist · 2025 · with Wim Pouw, James Trujillo, Fred Hasselman, Arkadiusz Białek, Natalia Siekiera, Joanna Raczaszek-Leonardi, Travis J. Wiltshire · DOI: 10.31234/osf.io/ajks6_v2
Open-source standardized pipeline for equitable observations of interactive behavioral dynamics, combining pose estimation, signal processing, and non-linear dynamical systems analysis.
MaskAnyone Toolkit: Offering Strategies for Minimizing Privacy Risks and Maximizing Utility in Audio-Visual Data Archiving
ICIS 2024 · arXiv · DOI: 10.48550/arXiv.2408.03185
Privacy-preserving toolkit for de-identifying individuals in video and audio while balancing open science with personal data protection.
StudyU: A Platform for Designing and Conducting Innovative Digital N-of-1 Trials
Journal of Medical Internet Research · 2022 · DOI: 10.2196/35884
Open source platform for conducting personalized N-of-1 trials with innovative user-centric elements.
StudyMe: A New Mobile App for User-Centric N-of-1 Trials
Trials · 2022 · with Alexander M Zenner, Stefan Konigorski · DOI: 10.1186/s13063-022-06893-7
Mobile application supporting non-experts in conducting multi-crossover self-experiments to evaluate intervention effects on personal health goals.
Masked-Piper: Masking Personal Identities in Visual Recordings While Preserving Multimodal Information
SoftwareX · 2022 · with James Trujillo, Gerard de Melo, Wim Pouw · DOI: 10.1016/j.softx.2022.101236
Privacy-preserving tool for masking personal identities in audio-visual recordings while retaining multimodal signals for research use.
Designing for Value Creation: Principles, Methods, and Case Insights from Embedding Designing-as-Performance in Digital Health Education and Research
Springer Design Thinking Research · 2022 · DOI: 10.1007/978-3-031-09297-8_10
Exploration of value creation principles in healthcare through stakeholder engagement in academia and industry.
Designing as Performance: Bridging the Gap Between Research and Practice in Design Thinking Education
Springer Design Thinking Research · 2021 · DOI: 10.1007/978-3-030-62037-0_3
Research-based training materials for high-performing design teams, bridging design thinking research and practice in education.
Preprints
MaskBench: A Comprehensive Benchmark Framework for 2D Pose Estimation and Video De-Identification
Preprint · 2025 · with Tim Riedel, Zainab Zafari, Sharjeel Shaikh, Wim Pouw · Project site
Modular benchmark evaluating how privacy-preserving masking strategies (blurring, pixelation, contour overlays) impact the performance of 2D pose estimators across multiple kinematic metrics.
EnvisionHGdetector: A Computational Framework for Co-Speech Gesture Detection, Kinematic Analysis, and Interactive Visualization
PsyArXiv Preprint · 2025 · with Wim Pouw, Bosco Yung, Sharjeel Ahmed Shaikh · DOI: 10.31234/osf.io/psg5f_v1
Computational framework for kinematic analysis of automatically detected co-speech gestures from single-person videos using TensorFlow and five open datasets.
The Utility Triad: A Framework for Analytic, Reflexive, and Generative Utility in Multimodal Data Infrastructure
Preprint · 2026 · with Wim Pouw, Gerard de Melo · Preprint PDF
Proposes the AU/RU/GU utility triad as a principled framework for diagnosing and designing multimodal research infrastructure — naming three dimensions of utility that are often conflated, and arguing that responsible infrastructure must operationalise all three explicitly.
Beyond ELAN: Toward Open, Modular Infrastructure for Multimodal Annotation
Preprint · 2026 · Preprint PDF
Argues that ELAN's monolithic desktop architecture cannot accommodate modern requirements — scale, AI pre-annotation, GDPR — and proposes an open, layered annotation stack with privacy, AI integration, and collaboration as first-class design primitives. Introduces the Masking@Home initiative for community-driven corpus unlocking.
Building a DSR Ecosystem for Privacy-Preserving Behavioral Research: The SYNAPSIS Experience
Preprint · 2026 · Preprint PDF
Design science research report on the SYNAPSIS platform development cycle — building privacy-preserving infrastructure for multimodal behavioral research through iterative artifact construction and community evaluation.
Exploring Digital Health Solutions: Personalised Medicine and N-of-1 Trials in Ghana — A Scoping Review
medRxiv Preprint · 2024 · with Godsway Sackey, Frank Kwesi Baiden, Stefan Konigorski · DOI: 10.1101/2024.12.29.24319752
Scoping review of digital health applications and personalized medicine approaches in Ghana, examining N-of-1 trial implementations.
Exploring Ecosystem Orchestration Patterns and Systemic Intermediaries in Social Media
SSRN · 2022 · with Arash Hajikhani · DOI: 10.2139/ssrn.4042351
Analysis of social media ecosystems using digital trace data to map ecosystem orchestration and communication patterns in European Innovation Systems.
In Progress
Colabograph: Interactive Cartography for Open Exchange in Science Ecosystems
MaskBench Challenge: An EnvisionBOX Benchmark for Co-Speech Gesture Detection
with Wim Pouw, Gerard de Melo, James Trujillo, Sharjeel Shaikh · Challenge repo · GitHub
Infant-Caregiver Object-Related Paired Actions as Communication Development
with Arkadiusz Białek, Wim Pouw, Anna Filip · OSF Data
Forbidden Triads in the Making: A Computationally Intensive Approach to the Sociomateriality of Team Creativity
Brokering position rotates through team members, driven by artifact circulation across 14 design triads.
Multimodal Cartography of Social-Impact Fields
Towards a Cartography of Datacentering: A Multimodal Research Agenda
A Multi-Modal Mixture-of-Experts Framework for Predicting Green Tech Venture Scaling
Get in Touch
Contact
Open to research collaborations, speaking opportunities, and innovative projects in multimodal analytics and computational sustainability.
Radboud University
babajide.owoyele@ru.nlHasso Plattner Institute
babajide.owoyele@hpi.deProfiles