Welcome to the Depthkit Documentation Portal!

Depthkit is the first volumetric filmmaking tool designed to empower creators of all experience levels to participate in the cutting edge of immersive storytelling.


Computer Requirements

Depth Sensors

Getting Started

Edit & Export

Edit Window

Once your take is recorded, you are ready to enter the Export Window! You can get here by double clicking your take or by clicking the Edit & Export button. Now you can play your recordings, set in and out points, reset the depth range - depth range - The depth near and far planes represented as hue encoded ranges, red as the near plane and pink as the farthest plane before clipping. , enhance your depth/color quality, and export your desired clip. Simply select a take from your Library to view in the 3D scene and edit in the timeline.

The Edit & Export window are divided into several panels for processing you clip. Explore the isolate, refine, and export panels below.

Isolate

The isolate panel allows you to remove excess or background data from your footage. You can isolate your scene in two ways: editing your depth range and applying a mask.

Segment: Depth Range

Just like in the Record window, depth segmentation allows you to adjust your depth range - depth range - The depth near and far planes represented as hue encoded ranges, red as the near plane and pink as the farthest plane before clipping. as needed. You might want to adjust this when you make a select and you want to make the range more narrow. This will increase the spatial resolution of your clip. An example of a high resolution is to set see dark orange at the near plane as seen in the 3D preview. The far plane should be represented as pink or dark blue.

Mask

If you would like to remove the background from your clips, continue to the Mask section under Segment. You will have two options; Add Refinement Mask and Apply automatic Kinect Mask.

  • Add Refinement Mask: will enable the enhancement algorithm to refine your clips. Once selected, you will be able to link a refinement mask, crop, and enhance your clip.
  • Apply automatic Kinect Mask: represents the body index segmentation data. You can enable this to remove the background when shooting in any location.

*Depthkit Pro is required in order to export with a refinement mask.

Once the refinement mask is enabled, you can apply a custom mask in the selection box under the Refinement Mask button.

Link Refinement Mask

Link Refinement Mask

Notes on the Refinement Mask

Make sure to export your mask with the same duration and aspect ratio as your source color video. You will also need to match the source codec.

What is the benefit of a refinement mask?
In addition to removing the background of your scene, a custom mask provides access to the Refine parameters. We call this the enhancement workflow and this gives you the power to:

  • Export with a resolution that matches your source color video.
  • Repair holes in your depth data.
  • Reduce depth noise - depth noise - Noise in the depth data caused by the way in which a sensor detects depth, by projecting an infrared pattern onto a surface. .
  • Export enhanced Depthkit footage.

How do I make a refinement mask?
This workflow requires that you can pull a key and generate a matte from your color footage. What this will do is generate clean edges while providing a canvas in which your depth can then be enhanced. Learn how to create a quick refinement mask.

Sensor Video (left), Refinement Mask (right)

Sensor Video (left), Refinement Mask (right)

Crop

Once Enable Refinement Mask is selected, a horizontal and vertical crop will appear. This allows you to crop the clip in order to remove excess data. When enhancing data, the crops lets you take advantage of every depth pixel to maximize quality.

*Note that the crop will impact the final export resolution.

Refine

Refine Parameters are used to refine your footage. **These are only available for export with Depthkit Pro.

Enhancement parameters include:

  • Filter Size: You can think of this as the pixel size of the enhanced depth. This parameter will fill holes in your depth data and reduce depth noise - depth noise - Noise in the depth data caused by the way in which a sensor detects depth, by projecting an infrared pattern onto a surface. . Start with a moderate value around 2-4. Increase if you have a lot of depth noise - depth noise - Noise in the depth data caused by the way in which a sensor detects depth, by projecting an infrared pattern onto a surface. . Decrease if you are losing depth details, most noticeable around facial details.

  • Color Contribution: The percentage of how much the color video with influence the depth data. This is particularly noticable when you have edges well defined in your color video that are not clear in the depth data alone. Increasing the color contribution in this case will allow the refinement algorithm to pull more cues from the color in order to modify the depth. In many cases, this creates a smoothing of your depth data and can almost act as a Gaussian blur.

Please note that when you have a color input with high contrast, increasing the color contribution may be too strong of a cue for the depth and may results in inaccurate depth details. For example, below I have a capture of a black and white checkerboard. By increasing the color contribution to 100%, I am creating a color influence from the black/white contrast that should not actually be represented in the depth.

  • Depth Contribution: The percentage of how much the depth data influences the refinement algorithm. At 100% contribution, your data will reflect the look of raw depth data. Decreasing the value will soften your data, putting more weight onto the other enhancement parameters.

Advanced Settings

  • Fill Amount: Complements the Filter Size by providing a secondary fill value. Leave at the default value of 4 unless you are dealing with holes or abrupt clipping in your depth. Decrease the value to remove these artifacts.

Apply automatic Kinect Mask: allows you to remove any background data that still may be available in your custom mask.

What Enhancement Parameters are best for me?

Sometimes it is tricky to determine the ideal settings, especially when depth data can vary based on lighting conditions, materials, etc. When getting started stick with the moderate default values. When you get to adjusting these values, keep in mind that the parameters work with each other. For example, in order to reduce depth noise - depth noise - Noise in the depth data caused by the way in which a sensor detects depth, by projecting an infrared pattern onto a surface. , you may notice that it is more effective to increase your Filter Size and Color Contribution, but only if your Depth contribution is reduced as well.

Tool Tip

Get a close up on the Enhancement Settings in action by "zooming in" with the crop sliders. Just remember to set the back to the frame crop before exporting!

Export

Export Formats

Depthkit exports a a video format called combined-per-pixel - combined-per-pixel - The export format optimized for Unity playback that is made up of the color video (top) and depth data (bottom). This format provides a performance friendly playback of your volumetric data. . This format is optimized for Unity and includes both the depth and color data.

Depthkit Pro subscribers can export this same format in png sequences as well as an mp4. This is helpful if you are taking Depthkit footage into another video editing tool while avoiding extra compression artifacts.

Ready to Export

When ready to export your clip, simply click the Export button within the Export panel and watch the blue progress bar in the timeline. This will only take a few moments.

Once exported, three files will populate your _Export folder or your otherwise specified file path.

  • combined-per-pixel - combined-per-pixel - The export format optimized for Unity playback that is made up of the color video (top) and depth data (bottom). This format provides a performance friendly playback of your volumetric data. video
  • poster image - poster image - a preview image pulled from the first frame of the Depthkit video export that provides a still image representation of a clip in a 3D scene when the clip is not playing.
  • metadata file - metadata file - A text file that holds the camera data of your capture. This file is required to play your recording in Unity.

The combined-per-pixel - combined-per-pixel - The export format optimized for Unity playback that is made up of the color video (top) and depth data (bottom). This format provides a performance friendly playback of your volumetric data. video export represents your color (top) and depth (bottom) streams as a video file optimized for Unity for performance friendly playback. The poster image is pulled from the first frame of your clip. When applied with the video export in Unity, it allows you to view your asset in the game engine when the clip is not playing. The metadata file - metadata file - A text file that holds the camera data of your capture. This file is required to play your recording in Unity. holds your depth and capture data and is needed with the video clip for playback in the game engine.

Each file will be labeled with the specified take prefix, the capture timestamp, and the export timestamp.

Your asset is now ready to lay back your external audio recording or to import right into Unity.

Auto Save

Depthkit automatically saves in/out points and the depth range values, so there is no need to save a take before closing the application or recording a new take.

Edit & Export


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.