Sound-from-Video via Multiscale Spatial-Gradient Filtering of Intra-Frame Motion

This paper proposes a vision-based approach for speech acquisition, where speech is inferred by extracting motion from video recordings of sound-induced object vibrations. The method is based on a spatial gradient approach that combines steerable pyramid decomposition and smoothened gradient filtering to yield a single vibration value per frame. The approach relies on the fact that the spatial-gradient magnitude varies with the amount of motion occurring during the exposure and enables the recovery of speech-induced vibrations without segmentation or phase unwrapping. Experimental results show that the proposed method can recover intelligible speech and works reliably on different object surfaces and lighting conditions, maintaining low computational cost. These findings position intra-motion as a useful cue for visual microphones and acoustic sensing devices.