Broadcasting Depthkit live via CDN

In this page

What You'll Need →
Setting up Depthkit →
Setting up AWS MediaStore & MediaLive →
Setting up OBS →
Adding audio to the stream →
Starting a broadcast →
Setting up a receiver in Unity →
Tips and troubleshooting →

When broadcasting a Depthkit Livestream over the internet to many recipient devices, methods of video transmission like P2P or WebRTC become unsuitable. Generic video livestreaming workflows use content delivery networks or content distribution networks (CDN) to reliably deliver these livestreams thousands of recipients or more. Depthkit can integrate into this video distribution infrastructure to enable the same number of recipients to remotely receive and reconstruct a Depthkit asset in real-time. This is useful in broadcast applications like volumetric musical performances, keynotes, and other forms of addressing large audiences remotely.

601601

There are currently many options for broadcasting tools and CDNs:

CDN ServicesBroadcasting Software
Amazon AWS MediaLive & MediaStoreOBS (requires 3rd-Party OBS-Spout plugin)
Microsoft Azure CDNvMix (requires commercial license)
CloudflareDerivative TouchDesigner (requires commercial license)
Google Cloud CDN

For this guide, we’ll be using OBS and AWS MediaLive.

You’ll Need

Setup Depthkit

  • Launch Depthkit and ensure you are logged in with a license which supports live streaming, like Depthkit Studio.
  • Create a new project.
  • If using multiple sensors with Depthkit Studio, create a Studio Calibration.
  • Configure the sensors’ options, including resolution, exposure, and white balance.
  • Start Streaming
    • For Depthkit Core livestreams using a single sensor, enter the Record context, and the sensor stream will begin automatically.
    • For Depthkit Studio livestreams using multiple sensors, enter the Studio Calibration & Recording context, and click Start Streaming.
  • Set Near and Far clipping planes to remove unwanted geometry and increase the fidelity of the depth data. Get these planes as close to your subject as possible without cropping away any part of your subject. These clipping plane positions will not affect any recordings you make, but they will affect the live stream.
  • In Edit > Preferences, check the Enable Record Context Live Stream box.
16001600

🚧

When you have Livestreaming enabled, you cannot edit the depth range in Depthkit.

To modify the depth range of any of the sensors, disable Livestreaming and make the adjustments, then re-enable it. You'll need to replace your metadata in Unity after modifying the depth range for correct rendering.

15941594

📘

Some broadcast software and CDN services only permit standard video resolutions like 1920x1080 and 3840x2160. In this case, set the Live Output Texture Maximum Width and Height to the highest values that your pipeline supports.

  • Start streaming again.
  • A Live Streaming indicator will now appear in Depthkit’s 3D Viewport, and you will find a livestream metadata file in your Depthkit project bin’s “_Export” folder.
621621 562562

Setup AWS Elemental MediaLive and MediaStore CDN

https://aws.amazon.com/blogs/media/part1-how-to-send-live-video-to-aws-elemental-mediastore/

  • In the AWS Elemental MediaStore Console, click Create Container. This will be the cloud storage container which serves the Depthkit Livestream HLS stream to all audience devices later.
14311431
  • In the Container Policy tab, click Edit Policy, and paste in the JSON policy below.
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "MediaStoreFullAccess",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::AWS_CUSTOMER_NUMBER:root"
      },
      "Action": "mediastore:*",
      "Resource": "arn:aws:mediastore:us-east-1:AWS_CUSTOMER_NUMBER:container/CONTAINER_NAME/*",
      "Condition": {
        "Bool": {
          "aws:SecureTransport": "true"
        }
      }
    },
    {
      "Sid": "PublicReadOverHttps",
      "Effect": "Allow",
      "Principal": "*",
      "Action": [
        "mediastore:GetObject",
        "mediastore:DescribeObject"
      ],
      "Resource": "arn:aws:mediastore:us-east-1:AWS_CUSTOMER_NUMBER:container/CONTAINER_NAME/*",
      "Condition": {
        "Bool": {
          "aws:SecureTransport": "true"
        }
      }
    }
  ]
}
  • Replace every instance of AWS_CUSTOMER_NUMBER with the account ID found in the account menu in the top right.
  • Replace every instance of CONTAINER_NAME with the name of your container.
  • Replace every instance of us-east-1 with the correct region for your AWS resource.
  • Save the container policy.
  • In the Info tab, copy the data endpoint URL (*e.g. https://CONTAINER_ID.data.mediastore.us-east-1.amazonaws.com*) and keep it somewhere handy. We will use it in a moment to specify the MediaLive channel destination.
  • In the AWS Elemental MediaLive Console > Inputs, click Create Input.
15251525
  • Give the input a unique name.
  • For Input Type, select RTMP (push).
  • For Input security group, use an existing policy to only accept streams from a particular IP address, or select Create, and enter the public IP address of your Depthkit PC followed by /32 (CIDR format) to create one. To accept streams from any IP address, enter 0.0.0.0/0
  • For input destinations, select SINGLE_INPUT.
    • Generate an application name and instance. Later when you configure OBS, the Application name will be the end of the server path, and the Application instance will be the stream key.
  • Click Create.
  • Under Endpoints, copy the RTMP URL (e.g. rtmp://55.5.55.55:5555/path/streamkey) and keep it handy for setting up OBS.
  • In the AWS Elemental MediaLive Console > Channels, click Create Channel.
15101510
  • Generate a unique channel name.
  • For IAM role, use an existing MediaLiveAccess role, or create one from the template.
  • For Channel Class, select SINGLE_PIPELINE.
  • For Input specifications, select:
    • Input codec: AVC
    • Input resolution: UHD
    • Maximum Input Bitrate: MAX_20_MBPS
  • Under HLS Settings > CDN Settings, select HLS Media Store
  • When configuring the channel input, select ‘UHD’ and ‘MAX_20_MBPS’
  • In the left panel, under Input Attachments, click Add.
  • For input, select the MediaLive input you set up earlier.
  • Click confirm.
  • In the left panel, under Output Groups, click Add.
  • Select HLS, and click Confirm.
  • In HLS Group Destination A, enter the endpoint URL you saved from the MediaStore container you saved earlier:
    • Change the protocol: Replace https with mediastoressl
    • Postpend the URL with a the directory and object name prefix that you would like this stream to generate in your MediaStore container: /folder/object
    • The modified URL should follow this format: *mediastoressl://CONTAINER_ID.data.mediastore.us-east-1.amazonaws.com/folder/object (See Amazon’s documentation https://docs.aws.amazon.com/medialive/latest/ug/hls-specify-destination-ems.html)*
  • Under HLS settings, optionally specify a name, and for CDN Settings, select HLS Media Store.
  • Under HLS Outputs, click Output 1’s Settings.
  • Under Stream Settings, enter the Width and Height of your Depthkit Livestream (found in the livestream_meta.txt).
    • If your broadcast software requires a standard resolution like 3840x2160, enter those values instead, and stretch the Depthkit Livestream once setup in OBS.
646646
  • In the left panel, click Create Channel.

Setup OBS

  • Install OBS and the OBS-Spout Plugin on the Depthkit PC.
  • Open OBS, and verify that the OBS-Spout Plugin is installed by confirming that Spout2 Capture is an available option for Sources.
12571257
  • In OBS’s Settings > Stream, service to Custom, and the Server and Stream Key from the RTMP URL saved earlier (e.g. rtmp://55.5.55.55:5555/path/streamkey).
    • Parse the path without the instance/stream key, and paste it in the Server field.
    • Parse just the instance from the end, and paste it in the Stream Key field.
10511051
  • In OBS’s Settings > Output, select Output Mode: Advanced.
    • Ensure Rescale output is unchecked.
    • Change the bitrate to 10000 to 15000 Kbps
10511051
  • In OBS’s Settings > Video, change the Base (Canvas) Resolution and Output (Scaled) Resolution to match the resolution of your MediaLive Channel output, and the framerate to 30.
10531053

📘

Some broadcast software and CDN services only permit standard video aspect ratios like 16:9. In this case, set the Base and Output resolutions to a resolution with the required aspect ratio.

  • Add Spout source, and select the Depthkit Spout sender. If the Depthkit Spout sender is not available, confirm that on the same PC, Depthkit displays 'Live streaming is enabled', and check that Depthkit and OBS are using the same graphics pipeline in the Troubleshooting section below.

  • You should now see a combined-per-pixel stream in the preview window.

822822

If the Depthkit Livestream’s resolution/aspect ratio doesn’t match your OBS Base resolution, stretch the source to fill the frame, then modify the metadata file’s textureWidth and textureHeight values to match the OBS Base resolution.

12571257 643643

Add Audio

325325

Depthkit does not pass audio from the sensors or any other audio source, so if you want to include audio in your live stream, embed it into the OBS broadcast.

❗️

Although the Azure Kinect Mic Array shows up as an available audio device, do not use it, as doing so may interfere with Depthkit's access to the sensors. Instead, use a separate audio source connected to the Depthkit PC, such as the computer's built-in audio interface and a microphone.

  • This audio will automatically be embedded in your Depthkit live stream all the way through the CDN and the player on the remote end.

Start Broadcast

  • In AWS Elemental MediaLive Console, navigate to your channel, and click Start.
10621062
  • Wait until the Channel State reports that the channel is running.
  • In OBS, click Start Streaming.
  • In AWS MediaStore Console, your container will populate with the folder and objects specified in the MediaLive channel destination URL specified earlier.
  • Locate the base M3U8 object, and open it.
13931393
  • The URL found under Object Name is the URL we will be connecting to our receiver in Unity. Copy it and save it somewhere handy.
14171417
  • To verify that the broadcast is working, open the stream in a media player like VLC by clicking Media > Open Network Stream… and pasting in the object URL.
11591159

Setup Livestream Receiver in Unity

  • Open a Unity project on the PC which will be receiving the stream. For best performance, this should be a separate PC from the Depthkit PC.
  • Add AVPro to the project.
  • Add the Depthkit Expansion Packages in the following order:
    • depthkit.core
    • depthkit.avpro
    • If using Depthkit Studio assets, add depthkit.studio or depthkit.studio.lite.
    • If using URP/HDRP, add Depthkit Core (and Studio) Shader Graph and/or VFX Graph plugins.
  • Create a Depthkit Core or Depthkit Studio object from scratch, or import a Depthkit Core or Depthkit Studio prefab from the Package Manager, and add it to your scene.
  • Setup the Depthkit Clip component.
    • Change Player to AVPro.
    • Add the livestream_metadata.txt file to your Unity project’s Assets folder, and link it to your Depthkit object’s Metadata field.
    • Under Advanced Settings, check the box to Disable Poster.
513513
  • Setup the AVPro Media Player component.
    • Under Settings > Source, change Media Source to Path. Change the next dropdown to Absolute Path or URL.
    • Paste in the URL from the HLS (M3U8) object in your MediaStore container.
    • Click Load.
492492

Depthkit asset should now be visible!

16381638
  • For Depthkit Studio assets, adjust bounding box and set surface buffer capacity.
  • Configure other mesh reconstruction and texture settings for your Depthkit Core or Depthkit Studio asset.

Tips and Troubleshooting

  • Dynamically loading metadata - If you ever change your sensor settings or update your Depthkit Studio calibration, you’ll need to provide updated metadata to Unity. In order to do this, you must modify the Depthkit Unity plugin to load the metadata dynamically from an external source rather than from the local file in the project.
  • Depthkit Livestream not visible in OBS - If OBS is not seeing the Spout stream from Depthkit, the two apps may be using different graphics hardware. Go to Windows Graphics settings, and under Graphics Performance Preference, add both OBS and Depthkit items, and set both to High Performance, then restart both programs.
  • AVPro image frozen - AVPro may stop updating the video image if Unity’s game view is not visible in the editor.