EMMA: Extracting Multiple physical parameters from Multimodal Data

CVPR 2026

Farhat Shaikh, Ayan Banerjee, Sandeep K. S. Gupta

IMPACT Lab, School of Computing & Augmented Intelligence (SCAI), Arizona State University

Project page · Demo video

---

Overview

EMMA is a physics-informed multimodal framework that recovers all identifiable dynamical parameters of a system directly from raw video, audio, and image-based time-series observations. Unlike prior video-only approaches that struggle with occluded states, hidden actuation inputs, and assumptions about known initial conditions, EMMA performs joint inference of explicit parameters, implicit dynamical components, and calibration invariants within a unified continuous-time model.

The user supplies the parametric structure of the governing ODE; EMMA solves the inverse problem of recovering its parameters, along with any latent forcing and invariants, from multimodal observations.

Key contributions

• Multi-modal dynamical parameter extraction from video, audio, and time-series reconstructed from visual charts.
• Recovery under unobserved forcing inputs by inferring latent actuation (e.g. wheel speed) from audio.
• Estimation of implicit dynamics associated with unmeasured physical effects (e.g. frictional drag).
• Invariant calibration from raw video, eliminating assumptions about known initial conditions or coordinate frames.
• Extensive validation on 100+ scenarios: Delfys benchmark (75 videos), real-world rover and quadrotor, and simulation charts.

Architecture

EMMA follows a three-step pipeline: Sense · Learn · Verify.

1. Sense. Video, audio, and chart images are converted into time-aligned signals through modality-specific pipelines.
2. Learn. A Liquid Time-Constant (LTC) network models the system's latent dynamics in continuous time.
3. Verify. A differentiable ODE solver simulates the recovered parameters and checks them against the observations under a physics-informed loss.

Results

EMMA delivers accurate multi-parameter recovery across diverse physical systems. Full tables and ablations are in the paper.

Compared against PAIG, NIRPI, and Delfys on the video benchmarks and PySINDy on the chart-based simulations.

Supported systems

Installation

Tested with Python 3.10+ on macOS and Linux.

git clone https://github.com/ImpactLabASU/EMMA-CVPR2026.git
cd EMMA-CVPR2026
python3 -m venv .venv && source .venv/bin/activate   # optional but recommended
pip install -r requirements.txt

System tools

• FFmpeg on your PATH (MoviePy uses it for audio extraction): brew install ffmpeg (macOS) or sudo apt install ffmpeg (Ubuntu).
• YOLO weights (default yolo11m.pt): pip install ultralytics then yolo download model=yolo11m.pt, or download from the Ultralytics releases page.
• A CUDA GPU is optional; every script falls back to CPU automatically.

Repository layout

Data

• Baseline datasets come from the Delfys "Physical Parameter Prediction" set on Kaggle (https://www.kaggle.com/datasets/jaswar/physical-parameter-prediction). Download it and copy the experiment folders into Baseline/; the scripts discover the data automatically.
• Sample rover and drone videos are available here: Dropbox. Place them under Rover/ and Drone/.

Usage

Baseline pipelines

Each baseline follows the same recipe:

1. cd Baseline/<Experiment>/
2. Edit the configuration block inside main():
   • video_path: path to the source video; leave empty to reuse existing data files.
   • weights_path: YOLO weights (yolo11m.pt by default).
   • pixel_to_meter (Free Fall, Torricelli, Sliding Block): set from your calibration grid.
   • output_folder: a unique run directory (e.g. run_01); the script creates output/ and data/ under it.
3. Run python3 <script>.py.
4. Optional: python3 <script>.py --simulation-only skips retraining and reuses the latest *_coefficients.csv and *_emma_final_model.pth (Free Fall, LED, Pendulum).

PySINDy baselines. Each experiment folder has pysindy_results/pysindy.py; run it from that folder (after the main pipeline has written the EMMA-formatted CSVs) for sparse-regression baselines.

Ablations. From Baseline/: python3 architecture_ablation.py and python3 run_additional_ablations.py (require pendulum datasets under Baseline/Pendulum-EMMA/<angle>_v*/data/).

Rover

cd Rover
# set video_path and weights_path in run.py (see the CONFIGURATION SECTION)
python3 run.py

Outputs: rover_coefficients.csv, rover_EMMA_final_model.pth, plots, GIF. Ablations: python3 rover-ablation.py, python3 rover_multimodal_ablation.py, or bash run_rover_ablation.sh (edit variables first). If you already have processed data/*.txt, set video_path = "" to skip detection.

Drone

cd Drone
EMMA_RUN_ORCHESTRATOR=1 python3 new_run.py --video /path/to/DroneVideo.mp4 --weights /path/to/yolo11m.pt

Note: Full orchestration also needs an external Dronepipeline/ folder containing droneExtract.py, droneExtractAudio.py, and EMMA_drone_torch_ltc_optimized.py. These are not bundled here; without them, new_run.py falls back to the local vision-only pipeline.

CGM chart digitizer

cd CGM
python3 extract_cgm_data.py   # reads CGMData.png, writes cgm_data.txt + a visualization

Troubleshooting

• Module not found: re-run pip install -r requirements.txt in the active virtual environment. For torch/torchvision, use the PyTorch selector.
• YOLO weights missing: download yolo11m.pt and point weights_path to it.
• FFmpeg errors: install FFmpeg (brew install ffmpeg / sudo apt install ffmpeg).

Citation

@InProceedings{Shaikh_2026_CVPR,
    author    = {Shaikh, Farhat and Banerjee, Ayan and Gupta, Sandeep},
    title     = {EMMA: Extracting Multiple physical parameters from Multimodal Data},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2026},
    pages     = {1716-1725}
}

Also on arXiv.