SonicDiffusion: Audio-Driven Image Generation and Editing with Pretrained Diffusion Models


Abstract

We are witnessing a revolution in conditional image synthesis with the recent success of large scale text-to-image generation methods. This success also opens up new opportunities in controlling the generation and editing process using multi-modal input. While spatial control using cues such as depth, sketch, and other images has attracted a lot of research, we argue that another equally effective modality is audio since sound and sight are two main components of human perception. Hence, we propose a method to enable audio-conditioning in large scale image diffusion models. Our method first maps features obtained from audio clips to tokens that can be injected into the diffusion model in a fashion similar to text tokens. We introduce additional audio-image cross attention layers which we finetune while freezing the weights of the original layers of the diffusion model. In addition to audio conditioned image generation, our method can also be utilized in conjunction with diffusion based editing methods to enable audio conditioned image editing. We demonstrate our method on a wide range of audio and image datasets. We perform extensive comparisons with recent methods and show favorable performance.

1. Model Architecture

 SonicDiffusion utilizes a dual branch audio projector to predict embeddings in Stable Diffusion 3 text-embedding space, and trains light-weight audio-condition layers in SD3 to enable audio to image generation. Given a source image and target audio, we first encode the image into the SD3 latent space and map CLAP audio features into SD3's conditioning space via the Audio Projector. During inference, we apply a FlowEdit-style update in latent space by composing the unconditional and audio-conditioned velocity fields of SD3 with audio adapters, yielding a modified trajectory that moves toward the target audio manifold while preserving the structure of the source image. The final latent state is decoded by the SD3 VAE to produce the edited image, whose appearance reflects the semantics of the target sound.

2. Image Generation

 SonicDiffusion can be used to generate images from audio clips. We show some examples below.

Landscape + Into the Wild


  Squishing water  

  Fire crackling  

  Snow  

  Waterfall burbling  

  Forest  

  Wind  

Greatest Hits


  Ceramic  

  Wood  

  Paper  

  Metal  

  Leaf  

  Water  

RAVDESS


  Angry  

  Sad  

  Happy  

3. Multi-Modal Image Generation

 SonicDiffusion can generate images from audio clips and text descriptions. We show some examples below.


Chinese Painting

Ghibli Style

Low Polygon

Mosaic Art

Papercraft

Vangogh Style Painting

4. Image Manipulation

 SonicDiffusion can be used to edit images using audio clips. We show some examples below. The image on the left is the original image. The image on the right is the edited image. You can change the slider to see the effect of the audio clip on the image.

Landscape + Into the Wild

Greatest Hits

5. Sound Interpolation

By the interpolation of the embeddings of two audio clips,  SonicDiffusion can generate a sequence of images that smoothly interpolates from one image to another. We show some examples below.


6. Volume Adjustment

 SonicDiffusion can generate images according to the intensity of the input volume in the audio clip. We show some examples below.


Update (June 2026): We recently upgraded our backbone from U-Net (SD2.1) to DiT (SD3). If you need the old weights, you can still find them on GitHub.