AudioX

Overview

AudioX takes text images video even partial sound and gives you back smooth audio or songs. This thing doesn’t care what you throw at it it just finds a way to turn it into sound

Built by a team from Hong Kong University of Science and Technology plus one solo dev it uses a wild setup called Diffusion Transformer. That’s where it gets the brainpower to catch complex sound patterns and mix them with whatever input you’ve got.

Most AI audio stuff only works with one input. AudioX works with anything. You can go from a sentence to a soundtrack from a video to a full soundscape. You want to fill a gap in an audio clip? Add music to a silent video? Done

What helps it do this is a smart input masking trick and some huge datasets:

• VGGSound-Caps. Around 190K audio clips with text
• V2M-Caps. Over 6 million music-caption pairs

Those helped it learn across formats and catch the rhythm.

Even though it's still a research model AudioX scored better or about the same as the top dogs in tests like AudioCaps and V2M. It’s doing great in things like:

• Text-to-music
• Video-to-sound
• Audio inpainting
• Sound effect making
• Music completion

It even outscored tools like AudioLDM-2 and Stable Audio on some jobs. MusicGen fans might wanna keep an eye out.

Supported Languages