Embedding audio in Depthkit assets

If you have recorded audio from an external device, you can now sync it to your Depthkit combined-per-pixel video before taking your clip into the game engine. Since Depthkit exports a standard video file, this can easily be done by laying the audio file back in any video editing program.

❗️

Simultaneous playback in Unity

In the past, we have mentioned applying the audio file as a component in Unity. We do not recommend this, as it will often fall out of sync with the Depthkit clip.

You'll Need

• a combined-per-pixel video exported from Depthkit
• a corresponding audio file

Embedding audio in Depthkit Core/Cinema Combined-per-Pixel videos using Premiere

• Open Adobe Premiere, and create a new project.
• Import your Depthkit combined-per-pixel video and audio file into the project.
• Place the Depthkit combined-per-pixel video on a timeline, ensuring that the timeline's resolution and framerate match the Depthkit combined-per-pixel video precisely. This can be quickly done by right-clicking the combined-per-pixel and selecting "New Sequence from Clip".

• Add the audio clip to the timeline.
• If necessary, trim and move the audio clip along the timeline until it is in sync with the video. Using a clap or slate as a sync reference is useful for this.

• Export the all or part of the timeline, again preserving the same resolution and framerate as the Depthkit combined-per-pixel video. Reducing the bitrate too much can impact the quality of the asset in Unity, so keep this setting above 5Mbps.

Embedding audio in Depthkit Studio Combined-per-Pixel videos using FFMPEG

Depthkit Multiperspective Combined-per-Pixel videos are particularly sensitive to subtle changes in color, so syncing your Depthkit clips to audio in a non-linear editor like Adobe Premiere can produce visual artifacts.

For this reason, FFMPEG is a great tool for embedding audio into a video file.

🚧

Trimming the audio file

Before you embed the audio into the video, they must be trimmed to have the same starting point. Use the slate/clap in both the video and audio recording to trim the audio to the same start point.

Muxing

• -i audio_file.wav - specifies the source of the audio file. The beginning of the audio file must be trimmed to match the start of the video, and the duration of the audio clip must at least as long as the video to ensure the CPP video doesn't get trimmed in the process.
• -map 0:v -map 1:a - map the first -i source to the video channel, and the second -i source to the audio channel. Be sure to arrange your input options to map to the channels properly.
• -c:v copy Use this in place of the -c:v <codec> option to copy a pre-encoded video source without re-encoding, preserving the quality of the video. This can only be used when muxing an existing video stream into a new container; This cannot be used when encoding an image sequence to video, or transcoding one video to another.
• -c:a copy Use this in place of the -c:a <codec> option to copy the audio source without re-encoding. Use only if your audio source is already compressed to the codec and bitrate you want. This option to embeds the audio stream within the video with no changes.

To mux audio into an existing video file, use the following example command:

ffmpeg -i video.mp4 -i audio.wav -map 0:v -map 1:a -c:v copy -c:a aac -b:a 320k -shortest output_wAudio.mp4

Audio codec options

If your audio file is in an uncompressed format like WAV, use the following options to compress and encode the audio into a format compatible with the video codec.

• -c:a aac sets the resulting audio codec to AAC, but this can be codec any supported by FFmpeg.
• -b:a 320k sets the audio bitrate to 320 kbps. You can adjust this based on your preference to balance file size and quality.
• -shortest stops the output at the end of the shorter source