Local livestreaming

In this page

What You'll Need →
Setting up Depthkit →
Setting up Unity →
Tips and troubleshooting →

Local live streaming refers to sending a Depthkit stream to another app (for example Unity) running on the same computer via Spout. This is the foundation of all forms of Depthkit live streaming, and can be augmented to send the live stream over a local network, or over the internet via peer-to-peer connections or broadcast infrastructure.

This local method of live streaming has some benefits and trade-offs:

  • Ultra-low latency - The delay between Depthkit and the receiving app is nearly imperceptible.
  • No network infrastructure needed - A local live stream all runs on one computer, so items like network routers, switches, and internet connections are not needed to transport the live stream.
  • High-performance PC required - Because one computer handles both the capture of Depthkit sensor data as well as the 3D reconstruction and rendering, a high-performance computer with a powerful CPU and GPU is required. To divide these tasks between multiple computers, consider live streaming over a local network.

You’ll Need

Setup Depthkit

The first part of this process is to set up Depthkit to send out the live stream.

  • Launch Depthkit and ensure you are logged in with a license which supports live streaming, like Depthkit Studio.
  • Create a new project.
29442944 29442944
  • For single-sensor live streaming, navigate to the Record workspace tab or View menu and select Record.
  • For multi-sensor live streaming, enter the Multicam workspace, create a multicam calibration, then click the Start Streaming button.
29442944
  • Configure the sensors’ options, including resolution, exposure, and white balance.
  • Set the Near and Far clipping planes to remove unwanted geometry and increase the fidelity of the depth data. Get these planes as close to your subject as possible without cropping away any part of your subject. These clipping plane positions will not affect any recordings you make, but they will affect the live stream.
  • Open Depthkit's preferences pane by selecting Edit → Preferences.
29442944
  • Under Live streaming settings, select Enable Record Context Live Stream. Make note of the Discoverable Live Stream Name, which by default is set to "Depthkit".
34533453

🚧

When you have Livestreaming enabled, you cannot edit the depth range in Depthkit.

To modify the depth range of any of the sensors, disable Livestreaming and make the adjustments, then re-enable it. You'll need to replace your metadata in Unity after modifying the depth range for correct rendering.

  • Set the Live Output Texture Maximum Width and Height. This is not necessary for local livestreaming, but may improve performance on the rendering portion of the pipeline.

🚧

Please note that Body Index in the live stream is currently only available for single-sensor live streaming with the Kinect for Windows v2.

  • Save the preferences.
  • Begin streaming again. Now there will be indicator in the upper left corner of the 3D viewport confirming that live streaming in enabled.
29442944 38403840
  • In your Depthkit project, go to your Exports folder and locate the livestream_meta.txt file. You will need this file for Unity. For multi-sensor live streams, this file is named livestream_multicam_meta.txt.
29442944

Configuring a Unity project for Livestreaming

  • Launch Unity and create a new project.
29442944
  • Add Keijiro's Spout plugin to Unity by navigating to Edit → Project Settings → Package Manager, expanding Scope Registries, and adding a new registry with the Name: Keijiro, URL: https://registry.npmjs.com, and Scopes: jp.keijiro.
  • Save the registry.
29442944

Add the Depthkit Expansion Packages.

  • In the Package Manager, click the add icon and select Add package from disk.
29442944 29442944

Setup the Depthkit Clip component.

  • In the Hierarchy, select your Depthkit clip, then in the Inspector, under the Depthkit Clip component, select the video player dropdown and select Depthkit Live Player.
29442944
  • Add the livestream_metadata.txt file to your Unity project’s Assets folder, and link it to your Depthkit object’s Metadata field.
29442944
  • Under Advanced Settings, check the box to Disable Poster.

Once linked, the Depthkit Clip component will report that your clip is setup.

29442944
  • In the Spout Receiver Component, select the Source Name Depthkit, or whatever custom stream name is specified in Depthkit's preferences.
29442944

Press Play and you will see the live stream from Depthkit into your unity scene.

Tips and Troubleshooting

  • Livestream not appearing in Unity, or only rendering a few triangles - If the surface buffer displays default values of 0 or 200000, then the surface buffer needs to be reset. With the livestream running and Unity in Edit mode, have someone step into the volume to give the renderer an approximate number of triangles to sample, and click 'Set Surface Buffer Capacity' in the Depthkit Studio Mesh Source component. You may need to momentarily task switch to Depthkit to get a steady stream of data into Unity, and then immediately click the 'Set Surface Buffer Capacity' button.
  • Live stream from Depthkit has low FPS - Depthkit will slow down significantly if not in-focus in Windows. After entering Play mode in Unity, task switch to Depthkit to bring it into Windows focus and increase the framerate. To divide these tasks between multiple computers, consider live streaming over a local network.
  • Dynamically loading metadata - If you ever change your sensor settings or update your Depthkit Studio calibration, you’ll need to provide updated metadata to Unity. In order to do this, you must exit Play Mode and bring the new metadata into your project, or modify the Depthkit Unity plugin to load the metadata dynamically from an external source rather than from the local file in the project.