Get started with multi-sensor calibration. Explore the video tutorial for a complete step-by-step guide using a 3-sensor front-facing configuration. We recommend starting with a 3-sensor front-facing configuration, as it is the simplest and quickest configuration to construct and calibrate.
Older version used in tutorial
The Calibration interface has been updated since this tutorial was made. See below for updated explanations of the tools.
Getting a project ready for Calibration →
Floor detection →
Sensor alignment sampling →
Speeding up the Calibration sample capture →
Clearing a Calibration →
Metrics and indicators →
Refinement tools →
Refining a Calibration →
Validating a Calibration →
To follow along our step-by-step guide you will need:
- a PC matching these hardware specifications;
- Azure Kinects with USB & Power Cables;
- 3.5mm audio sync cables;
- Tripods, Stands, or other Grip Gear to hold the sensors;
- Aruco Calibration Chart with stand - see printing requirements.
Depthkit uses a series of samples to calculate sensor alignment in two steps: sampling, and filtering. Sampling adds reference points for Depthkit to factor into the calibration, and filtering removes the low-quality samples based on certain conditions to get the calibration as precise as possible.
One Calibration per Project/Sensor Configuration
A Depthkit project contains one multi-sensor calibration per project. If you want to create a backup calibration or need to recalibrate because one or more of your sensors changes position, create a new Depthkit project and calibrate/capture in that new project.
A good calibration begins with a robust sensor configuration. In this example, we are using a 3-sensor front-facing configuration, as it is the simplest and quickest configuration to construct and calibrate.
Once you have calibrated for the first time with 3 sensors, add more devices based on your project needs.
- Secure the sensor support equipment into a stable position. Sensors which sag or move even the smallest amount will invalidate the calibration.
- Plug in all cables prior to starting the calibration. Even the movement of plugging in a cable can move a sensor enough to endanger the calibration.
- Launch Depthkit and ensure your license reads Depthkit Pilot. If your license was recently activated, you may need to click the refresh icon.
Create a new project.
- The application will launch in the Pair Camera workspace. This context allows you to use the Depthkit Cinema workflow to pair a video camera to your depth sensor. For a complete guide on camera pairing, visit the Depthkit Cinema documentation.
Adding the Depthkit Cinema workflow to Depthkit Studio is experiemental.
- The Cinema camera must be rigged to the depth sensor and calibrated in the Pair Camera workspace before creating a multi-sensor calibration.
- As of Depthkit version 0.6.0, the sensors and Cinema cameras can now be temporally locked together using an external sync generator. Without locking the sensors to the Cinema cameras, even at 30.0fps, there's a possibility of temporal offset and drift, especially over long takes. This can somewhat be mitigated by adding a step of adjusting the speed of the Cinema clips (usually by a fraction of a percent), exporting them, and linking those new versions to your clips in Depthkit. If using multiple Cinema cameras, make sure they can be locked to each other using genlock, so if the footage must be retimed, the same speed adjustment can be applied to all clips from the same take.
- This workflow combines two different calibrations - the Cinema pairing between a sensor and camera, and the Studio pairing of each sensor. Any error in either calibration will be combined, so the calibrations must be precise, and validated using single-sensor Cinema exports and sensor-only Studio exports before proceeding.
- Touching the cameras can jeopardize both the Cinema and Studio calibrations, so it is best to record to an external recorder, like an Atomos Shogun.
Click the Studio Calibration & Recording tab to begin calibrating your sensors.
You can also navigate there by going to the View menu and selecting Studio Calibration & Recording.
Under Sensor Configurations, you will see all connected depth sensors. If you do not see all sensors, make sure all of your USB and power connections are secure, and click the refresh icon next to the number of sensors connected.
Here you can adjust the stream configuration, sensor video settings, and advanced settings. Set the following Stream Configuration for best results in your calibration. You can collapse the settings of each sensor to navigate easier.
- Set the Color Resolution to 720p for the best performance during calibration. This setting is for calibration only, and can be changed to a different resolution without needing to recalibrate once it's time to record.
- Select a Depth Mode of 1024x1024. This setting can also be changed without the need to recalibrate once it's time to record.
Optionally click Apply Settings to All Sensors to copy the settings from one sensor to all connected sensors.
- You do not need to change the Sync Mode or Sync Delay. Once your synchronization cables are connected, the devices will be synchronized automatically during calibration. Do not touch these settings unless you run into synchronization issues.
First we will establish the 3D origin and orientation of our scene. This is not mandatory, and doesn't affect sensor alignment, but it does ensure that the scene will be right-side-up and facing the proper direction when we later bring it into a 3D workspace.
- Under Controls, click Start Calibrating.
- Place the chart flat on the floor in view of at least one sensor.
The floor sample can be determined by even just one camera perspective.
For the Floor Sample, you don't need the camera to be seeing all perspectives, which can be challenging in five or ten camera arrays. Choose a placement for the sample that is the actual floor and center point of your action, and as long as it's visible to some of the cameras your floor sample will be successful.
- Click Start Floor Detection. Wait five to ten seconds, and then click stop.
- Double-click to reset the view of the scene to the 'front' side of the origin point, and zoom out to see the whole volume.
- Depthkit will automatically select one of the sensors to reference the floor sample. You can override this by clicking any of the other sensors in the 'Floor Samples' panel to make that the anchor sensor. To revert back to the default sensor, click the 'Reset to default' button.
When capturing alignment samples, focus on two sensors at a time. Be sure that the chosen sensors are adjacent to each other, and have overlapping fields of view. This is necessary because Depthkit calculates 'links' between pairs of sensors, and each link adds up to make a daisy-chain of these links.
The best calibrations include many samples from within the common volume of each pair of sensors.
When sampling, make sure the chart is stationary. This includes letting the stand the chart is attached to come fully to rest after moving it.
- Choose a pair of sensors, and imagine a point directly between the two sensors and always face the chart directly towards that point. If the chart is pointed in any direction other than directly in between, the quality of the sample may suffer. Don't worry if other sensors outside of this pair can see the calibration chart. This will be filtered out by the calibration algorithm.
- Start with the chart as close to the sensors as possible while keeping the chart in view of both sensors. If the chart is in view of the sensors, you will see the corners of each marker highlighted with a circle.
- Click Start Sampling. Sample for 5 seconds, then click Stop Sampling.
- Once a sample is captured, clusters of dots, rings, and lines will appear. The dots and rings represent each sensor's view of a marker, and the lines represent which way each sensor perceives the chart to be facing.
Capture samples at a couple of heights, adjusting the tilt of the chart so it faces the mid-point between the sensors during each sample.
- Next, move the chart back toward the center of the volume and capture samples at a few different heights.
Throughout the sampling process, the goal is to capture samples at a few different heights, and also from sides of the frame, as well any other part of the volume where the performance will take place. Don't forget to move the calibration chart a bit beyond the center of the volume as well.
Repeat this sequence for each pair of adjacent sensors. A 3-sensor configuration will only need enough samples for two links (~40-50 samples total), but a 10-sensor configuration will need samples for nine links (~175-200 samples total). There's no proper number of samples per link, but most successful calibration contain around 20-30 samples per link.
Different sensor configurations have different needs, but one way to speed up your calibration is to place the Aruco markers on a 3D object to calibrate multiple sensors in one sample.
Keep in mind:
- All of the markers captured a given sample must be unique.
- All markers must still be printed and affixed in a way which keeps them visible to the IR camera in the depth sensor.
- The most accurate samples are those in which the marker directly faces toward the sensor. Markers captured at oblique/glancing angles may generate samples with higher errors, which may be filtered out with the Angle of Incidence slider, so be sure to still get plenty of samples from different positions and rotations, even if your calibration object has many markers.
- Putting your calibration object on wheels makes it much faster and easier to move around, but ensure that the markers remain stationary while the sample is taken.
The option to clear your Calibration is available in the Controls panel. If you want to delete your sensor Calibration and start again, click the trash can icon.
Clearing the calibration will delete all links and samples.
Calibrations should not be cleared if you have existing clips that will rely on this calibration, as this takes will no longer be aligned once the calibration is cleared. *Once a calibration is deleted, it cannot be recovered.
Calibration Complete/Incomplete indicates if all of the connected sensors have been linked together with the current calibration, or if any of the sensors are orphaned. Even when a calibration is marked 'Complete', the calibration may still need additional samples or adjustment to the filters to get the sensors to align well.
Calibration Quality is based on the mean squared error (MSE) divided by the sampled volume. This is to indicate the balance that must be stricken between a low MSE and a high sampled volume. Ideally this number is as close to 100% as possible. This is the main metric to look at to determine if your calibration is good or bad.
Precision is calculated based on the average of the distances (squared) between each respective sensor's view of a each sample.
Coverage displays the total volume sampled, which is calculated by adding up the number of unique sample positions captured during calibration. Filtering the samples closest to the sensors (often via lower Angle of Incidence) or farthest from the sensors (often via a lower Sample Deviation Threshold) will reduce the size of the volume. If this Sampled Volume is too small, the calibration will not align all areas of your capture volume well, so keep the Sampled Volume as high as possible.
Total Sampled Markers displays the total number of markers captured. This is often the number of samples captured multiplied by the number of markers on your chart.
Included displays how many markers have not been removed by the filtering tools.
Excluded displays how many markers have been removed by the filtering tools.
Per-Link Metrics provide this information for each link. If one particular link has a low sampled volume, you can capture more samples in view of those two specific sensors to fill out that link.
Precision can increase calibration accuracy by filtering out select samples. Adjust this slider to reduce your Mean Square Error. Shift towards 'Precision' to filter out samples that have the longest lines between each dot in a cluster. Try to get this as close to the precision side as possible before the Precision metric begins to drop, or one of the sensors goes out of alignment.
Sheer Angle filters out samples where a sensor sees the chart at a sheer angle, which is less accurate than when the sensor sees the chart from straight on. Generally, lower is better, but it might need to be brought up to include some of the samples which are nearer to the sensors, or if sensors have a wide angle between them. By default this slider is set to filter out samples with an incident angle greater than 50°.
Temporal Stability filters out samples where the marker detection was inconsistent over the duration of the sample capture - for example, if the chart was bumped during capture, or was captured too far away from the sensor, resulting in additional depth noise. Lower is better here as well, but pushing it too low will unlink some of the sensors.
Sensor Links > Sensor Link Enable/Disable ☑️ If one sensor is particularly out of alignment, you toggle specific sensor links to see if rebuilding the link tree brings the sensor into better alignment.
Sensor Links > Samples > Sample Enable/Disable ☑️ If one of your samples is causing issues (e.g. There was more than one chart in view of the sensors when the sample was captured), you can manually exclude that sample by expanding the link(s) that contains it, and disabling it.
Refining often takes multiple passes of adjusting each filter and finding a combination of slider positions that results in the best metrics, as well as good visual alignment in the 3D viewport.
- If you have a Low Error and a Small Volume you should relax the Angle of Incidence and Sample Deviation filters to include more samples. Capture more samples to fill out the coverage and increase the sampled volume.
- If you have a Large Volume and a High Error note that this could be due to a number of factors, such as a calibration chart that is small, or poorly detected by the sensors, or unstable/oblique when captured.
To analyze your samples, zoom into the 3D viewport. It may be helpful to rotate the viewport so you are viewing samples from above to better understand the alignment.
Each set of samples is color coded based on the sensor perspective. In this tutorial, you can see red, green, and blue used for the sensor color coding.
Aligned samples are represented by solid overlapping circles.
The hollow circles are outliers in your sample set. These are samples that have been filtered out.
There are many factors which dictate how precise the calibration needs to be. The calibration might perfectly align the sensors in one part of the volume, but not in another part, so be sure to check all areas of the volume that will be captured.
With the sensors still connected, have someone stand in, or place some stationary objects in the areas of the volume that will be captured, and begin streaming to see a point cloud approximation of the sensor alignment. Have the subject hold out their hands with a single finger pointed up to see how well small details line up.
This way of assessing the calibration is not as precise as assessing in Unity.
Validating in a later stage of the pipeline like Unity is the best way to truly know that the calibration is up to the standards of the project. Click 'Stop Calibration', then 'Start Streaming' and recording or livestream a validation scene, then follow the Unity documentation and tutorials to setup the Depthkit Studio expansion package to visually assess alignment.
You may find it easier to verify your calibration as a complete mesh. To take this verification stage one step further, check out this video walkthrough below to verify sensor calibration with OBJ Background Plates.
Updated 2 months ago