Calibration

Get started with multi-sensor calibration. Explore the video tutorial for a complete step-by-step guide using a 3-sensor front-facing configuration. We recommend starting with a 3-sensor front-facing configuration, as it is the simplest and quickest configuration to construct and calibrate.

What you'll need

How to calibrate sensors with Depthkit Studio

Depthkit uses a series of samples to calculate sensor alignment in two steps: sampling, and filtering. Sampling adds reference points for Depthkit to factor into the calibration, and filtering removes the low-quality samples based on certain conditions to get the calibration as precise as possible.

❗️

One Calibration per Project/Sensor Configuration

A Depthkit project contains one multi-sensor calibration per project. If you want to create a backup calibration or need to recalibrate because one or more of your sensors changes position, create a new Depthkit project and calibrate/capture in that new project.

Start with a stable sensor configuration

A good calibration begins with a robust sensor configuration. In this example, we are using a 3-sensor front-facing configuration, as it is the simplest and quickest configuration to construct and calibrate.

Once you have calibrated for the first time with 3 sensors, add more devices based on your project needs.

  • Secure the sensor support equipment into a stable position. Sensors which sag or move even the smallest amount will invalidate the calibration.
  • Plug in all cables prior to starting the calibration. Even the movement of plugging in a cable can move a sensor enough to endanger the calibration.

Create your Project

  1. Launch Depthkit and ensure your license reads Depthkit Pilot. If your license was recently activated, you may need to click the refresh icon.
    Create a new project.
  1. The application will launch in the Pair Camera workspace. This context allows you to use the Depthkit Cinema workflow to pair a video camera to your depth sensor. For a complete guide on camera pairing, visit the Depthkit Cinema documentation.

❗️

Adding the Depthkit Cinema workflow to Depthkit Studio makes the process more complex.

  • The Cinema camera must be rigged to the depth sensor and calibrated in the Pair Camera workspace before creating a multi-sensor calibration.
  • Touching the cameras can jeopardize both the Cinema and Studio calibrations, so it is best to record to an external recorder, like an Atomos Shogun.

Click the Multicam tab to begin calibrating your sensors.

You can also navigate there by going to the View menu and selecting Multicam.

  1. Under Sensor Configuration, you will see all connected depth sensors. If you do not see all sensors, make sure all of your USB and power connections are secure, and click the refresh icon next to the number of sensors connected.

  2. Expand this sensor toggle to reveal the stream configuration of each connected sensor. Here you can adjust the stream configuration, sensor video settings, and advanced settings. Set the following Stream Configuration for best results in your calibration.

  • Set the Color Resolution to 720p for the best performance during calibration. This setting is for calibration only, and can be changed to a different resolution without needing to recalibrate once it's time to record.
  • Select the Depth Mode that is appropriate for your volume. Generally, 640x576 Narrow Raw is more accurate for larger volumes, but 1024x1024 can be used for smaller volumes where the sensors are within 3 feet of the subject. This setting can also be changed without the need to recalibrate once it's time to record.
    Optionally click Apply Settings to All Sensors to copy the settings from one sensor to all connected sensors.
  • You do not need to change the Sync Mode or Sync Delay. Once your synchronization cables are connected, the devices will be synchronized automatically during calibration. Do not touch these settings unless you run into synchronization issues.

The Calibration workflow

Floor detection

First we will establish the 3D origin and orientation of our scene. This is not mandatory, and doesn't affect sensor alignment, but it does ensure that the scene will be right-side-up and facing the proper direction when we later bring it into a 3D workspace.

  1. Under Controls, click Start Calibrating.
  1. Place the chart flat on the floor in view of at least one sensor.

📘

The floor sample can be determined by even just one camera perspective.

For the Floor Sample, you don't need the camera to be seeing all perspectives, which can be challenging in five or ten camera arrays. Choose a placement for the sample that is the actual floor and center point of your action, and as long as it's visible to some of the cameras your floor sample will be successful.

  1. Click 'Begin Floor Detection'. Wait five to ten seconds, and then click stop.
  2. Double-click to reset the view of the scene to the 'front' side of the origin point, and zoom out to see the whole volume.

Sensor alignment sampling

When capturing alignment samples, focus on two sensors at a time. Be sure that the chosen sensors are adjacent to each other, and have overlapping fields of view. This is necessary because Depthkit calculates 'links' between pairs of sensors, and each link adds up to make a daisy-chain of these links.

The best calibrations include many samples from within the common volume of each pair of sensors.

Capturing samples

When sampling, make sure the chart is stationary. This includes letting the stand the chart is attached to come fully to rest after moving it.

  1. Choose a pair of sensors, and imagine a point directly between the two sensors and always face the chart directly towards that point. If the chart is pointed in any direction other than directly in between, the quality of the sample may suffer. Don't worry if other sensors outside of this pair can see the calibration chart. This will be filtered out by the calibration algorithm.
  1. Start as close to the sensors as possible while keeping the chart in view of both sensors. If the chart is in view of the sensors, you will see the corners of each marker highlighted with a circle.
  1. Click Start Sampling. Sample for 5 seconds, then click Stop Sampling.
  1. Once a sample is captured, clusters of dots, rings, and lines will appear. The dots and rings represent each sensor's view of a marker, and the lines represent which way each sensor perceives the chart to be facing.
    Capture samples at a couple of heights, adjusting the tilt of the chart so it faces the mid-point between the sensors during each sample.
  1. Next, move the chart back toward the center of the volume and capture samples at a few different heights.
    Throughout the sampling process, the goal is to capture samples at a few different heights, and also from sides of the frame, as well any other part of the volume where the performance will take place. Don't forget to move the calibration chart a bit beyond the center of the volume as well.
    Repeat this sequence for each pair of adjacent sensors. A 3-sensor configuration will only need enough samples for two links (~40-50 samples total), but a 10-sensor configuration will need samples for nine links (~175-200 samples total). There's no proper number of samples per link, but most successful calibration contain around 20-30 samples per link.

Clearing a Calibration

The option to clear calibration is available if you want to delete your sensor calibration and start again.

❗️

Clearing the calibration will delete all links and samples.

Calibrations should not be cleared if you have existing clips that will rely on this calibration, as this takes will no longer be aligned once the calibration is cleared. *Once a calibration is deleted, it cannot be recovered.

Calibration metrics & indicators

  • Sampled Volume displays the total volume sampled, which is calculated by adding up the number of unique sample positions captured during calibration. Filtering the samples closest to the sensors (often via lower Angle of Incidence) or farthest from the sensors (often via a lower Sample Deviation Threshold) will reduce the size of the volume. If this Sampled Volume is too small, the calibration will not align all areas of your capture volume well, so keep the Sampled Volume as high as possible.
  • Mean Square Error (MSE) displays the average of the distances (squared) between each respective sensor's view of a each sample. A higher number means that at least some portion of the included samples are not well aligned. Ideally this number is as low as possible.
  • Quality is based on the MSE divided by the sampled volume. This is to indicate the balance that must be stricken between a low MSE and a high sampled volume. Ideally this number is as close to 100% as possible. This is the main metric to look at to determine if your calibration is good or bad.
  • Calibration Inliers displays how many total samples have been removed by the filtering tools.
  • Filtering Breakdown displays how many samples each filter is excluding. This is useful to determine the range where a specific filter starts to become effective, allowing you to more quickly hone in on the most effective filter positions.
  • Calibration Complete/Incomplete indicates if all of the connected sensors have been linked together with the current calibration, or if any of the sensors are orphaned. Even when a calibration is marked 'Complete', the calibration may still need additional samples or adjustment to the filters to get the sensors to align well.
  • Per-Link Metrics provide this information for each link. If one particular link has a low sampled volume, you can capture more samples in view of those two specific sensors to fill out that link.

Refinement tools

  • Sample Filtering can increase calibration accuracy by filtering out select samples. Adjust this slider to reduce your Mean Square Error. Shift towards 'Precision' to filter out samples that have the longest lines between each dot in a cluster. Try to get this as close to the precision side as possible before the reported error begins to rise, or one of the sensors goes out of alignment.
  • The Angle of Incidence Threshold filters out samples where a sensor sees the chart at a sheer angle, which is less accurate than when the sensor sees the chart from straight on. Generally, lower is better, but it might need to be brought up to include some of the samples which are nearer to the sensors, or if sensors have a wide angle between them. By default this slider is set to filter out samples with an incident angle greater than 50°.
  • Sample Deviation Threshold filters out samples where the marker detection was inconsistent over the duration of the sample capture - for example, if the chart was bumped during capture, or was captured too far away from the sensor, resulting in additional depth noise. Lower is better here as well, but pushing it too low will unlink some of the sensors.
  • Sensor Link Enable/Disable ☑️ If one sensor is particularly out of alignment, you toggle specific sensor links to see if rebuilding the link tree brings the sensor into better alignment.
  • Sample Enable/Disable ☑️ If one of your samples is causing issues (e.g. There was more than one chart in view of the sensors when the sample was captured), you can manually exclude that sample by expanding the link(s) that contains it, and disabling it.

Refining a Calibration

Refining often takes multiple passes of adjusting each filter and finding a combination of slider positions that results in the best metrics, as well as good visual alignment in the 3D viewport.

  • If you have a Low Error and a Small Volume you should relax the Angle of Incidence and Sample Deviation filters to include more samples. Capture more samples to fill out the coverage and increase the sampled volume.
  • If you have a Large Volume and a High Error note that this could be due to a number of factors, such as a calibration chart that is small, or poorly detected by the sensors, or unstable/oblique when captured.

To analyze your samples, zoom into the 3D viewport. It may be helpful to rotate the viewport so you are viewing samples from above to better understand the alignment.

Each set of samples is color coded based on the sensor perspective. In this tutorial, you can see red, green, and blue used for the sensor color coding.
Aligned samples are represented by solid overlapping circles.

The hollow circles are outliers in your sample set. These are samples that have been filtered out.

Validating a Calibration

Evaluate in Depthkit

There are many factors which dictate how precise the calibration needs to be. The calibration might perfectly align the sensors in one part of the volume, but not in another part, so be sure to check all areas of the volume that will be captured.

With the sensors still connected, have someone stand in, or place some stationary objects in the areas of the volume that will be captured, and begin streaming to see a point cloud approximation of the sensor alignment. Have the subject hold out their hands with a single finger pointed up to see how well small details line up.

This way of assessing the calibration is not as precise as assessing in Unity.

Preview live in Unity

Validating in a later stage of the pipeline like Unity is the best way to truly know that the calibration is up to the standards of the project. Click 'Stop Calibration', then 'Start Streaming' and Record or livestream a validation scene, then follow the Unity documentation and tutorials to setup the Depthkit Studio expansion package to visually assess alignment.

See Depthkit Livestreaming →

Validating a Calibration with OBJs of a static scene

You may find it easier to verify your calibration as a complete mesh. To take this verification stage one step further, check out this video walkthrough below to verify sensor calibration with OBJ Background Plates.