LatentSync

Overview

LatentSync is an innovative tool for creating lifelike lip-synced videos directly from audio input. Unlike old-school methods, it skips complex intermediate steps like 3D modeling or facial landmarks. Instead, it uses latent diffusion models, which focus on delivering high-quality, time-based consistency in every frame.

The system was developed by ByteDance, the company behind TikTok, in partnership with researchers at Beijing Jiaotong University. 

The tool is available for download at GitHUB, for testing at FAL.ai and within ComfyUI (see wrappers links below).

The first tests are looking quite promising.

LatentSync now also comes with Gradio UI.

You can try adjusting the following inference parameters to achieve better results:

• inference_steps [20-50]: A higher value improves visual quality but slows down the generation speed.
• guidance_scale [1.0-3.0]: A higher value improves lip-sync accuracy but may cause the video distortion or jitter.