Home | About | Journals | Submit | Contact Us | Français |

**|**Sensors (Basel)**|**v.16(6); 2016 June**|**PMC4934264

Formats

Article sections

- Abstract
- 1. Introduction
- 2. Preliminaries
- 3. The Principle of Global Calibration
- 4. Accuracy Analysis of Different Factors’ Effects
- 5. Experimental Results
- 6. Conclusions
- References

Authors

Related links

Sensors (Basel). 2016 June; 16(6): 838.

Published online 2016 June 8. doi: 10.3390/s16060838

PMCID: PMC4934264

Fabrizio Lamberti, Academic Editor

School of Automation Science and Electrical Engineering, Beihang University, Beijing 100191, China; Email: moc.anis@tsoow (S.W.); Email: nc.ude.aaub@gnixhz (Z.X.); Email: moc.361@706yfhl (X.J.)

Received 2016 April 7; Accepted 2016 June 1.

Copyright © 2016 by the authors; licensee MDPI, Basel, Switzerland.

This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC-BY) license (http://creativecommons.org/licenses/by/4.0/).

This paper presents a global calibration method for widely distributed vision sensors in ring-topologies. Planar target with two mutually orthogonal groups of parallel lines is needed for each camera. Firstly, the relative pose of each camera and its corresponding target is found from the vanishing points and lines. Next, an auxiliary camera is used to find the relative poses between neighboring pairs of calibration targets. Then the relative pose from each target to the reference target is initialized by the chain of transformations, followed by nonlinear optimization based on the constraint of ring-topologies. Lastly, the relative poses between the cameras are found from the relative poses of calibration targets. Synthetic data, simulation images and real experiments all demonstrate that the proposed method is reliable and accurate. The accumulated error due to multiple coordinate transformations can be adjusted effectively by the proposed method. In real experiment, eight targets are located in an area about 1200 mm × 1200 mm. The accuracy of the proposed method is about 0.465 mm when the times of coordinate transformations reach a maximum. The proposed method is simple and can be applied to different camera configurations.

Vision sensors have advantages of flexibility and high-precision. Distributed vision sensors (DVS) are always used due to their wider fields of view (FOVs). Calibration is an important step for most camera applications. Calibration of a DVS typically has two stages: the intrinsic calibration which can be done separately and the global calibration which calculates the relative poses of the camera frames and the global coordinate frame (GCF). Then the information extracted from each camera can be integrated into the GCF. Generally, the coordinate frame of the reference camera is selected as GCF. However, vision sensors are usually widely distributed to have a better coverage. As two adjacent cameras usually have small or no overlapping FOV, the global calibration of DVS becomes of prime importance.

DVS can be calibrated by high-precision 3D measurement equipment. Lu *et al.* [1] constructed a measurement system and achieved calibration of non-overlapping DVS using two theodolites and a planar target. Calibration methods without expensive equipment have also been investigated. Peng *et al.* [2] proposed an approach omitting translation vectors between cameras, due to the loss of depth information during the camera projection [3]. It assumes that all the cameras have approximately the same optical center. However, this assumption is not appropriate when the relative distances between cameras are not small in comparison with the distances to the captured scene. In addition, feature detection and matching, such as scale invariant feature transform (SIFT) [4] is not reliable in insufficient textural environments due to lack of enough distinctive feature points [2].

Global calibration methods for DVS with overlapping FOV cannot be applied in the case of non-overlapping FOV. Most of the global calibration methods for DVS with non-overlapping FOV are based on mirror reflections [5,6,7], rigidity constraints of calibration targets [8,9,10], movements of the platform [11] and the auxiliary camera [12]. For general distributed vision sensors, it is hard to make sure that each camera has a clear sight of the targets through mirror reflections, especially in complex environments. Liu *et al.* [9] proposed a global calibration method by placing multiple targets in front of the vision sensors at least four times. Bosch *et al.* [10] use a poster to determine the relative poses of multiple cameras in two steps. It requires that different parts of the poster are observed by at least two cameras at the same time, so that SIFT features can be utilized. However, it is not flexible to use a long one-dimensional target [8], rigidly-connected targets [9] or a large area poster [10] for the calibration of widely distributed cameras. Pagel [11] achieved extrinsic calibration of a multi-camera rig with non-overlapping FOV by moving the platform. However, it required that at least two adjacent targets be visible and two cameras can see the target at the same time. Sun *et al.* [12] used an auxiliary camera to observe all the sphere targets. However, all the targets can hardly been observed by one camera at the same time due to widely distribution of the vision sensors.

Structure from motion [13] solves similar problems as that in the global calibration of DVS. The difference is that the global calibration transforms local coordinate frames into GCF, while structure from motion estimates the locations of 3D points from multiple images [14]. Fitzagibbon *et al.* [15] recover the 3D scene structure together with 3D camera positions from a closed image sequence. Compared with open sequences, the closed image sequence contains additional constraints. Zhang *et al.* [16] propose an incremental motion estimation algorithm to deal with long image sequences.

Generally, one calibration target is selected as the reference target. Compared with employing the auxiliary camera to capture all the targets in one image, capturing neighboring pairs of targets is more suitable for widely distributed cameras. The relative poses between neighboring pairs of the targets can be solved separately. Then the relative poses between each target and the reference target can be achieved by chainwise coordinate transformations. However, the error accumulates with increasing times of transformations. When dealing with DVS that provides a vision of the surrounding scene just as [2,11], vision sensors are always configured in ring-topologies to have a better coverage of the surroundings. The first sensor adjoins the end to form a closed chain. Thus a closed image sequence of neighboring targets can be acquired by the auxiliary camera.

Line features are more stable than point features in detection and matching [17]. The principle of perspective projection indicates that an infinite scene line is mapped onto an image plane as a line terminating in a vanishing point. Vanishing points and vanishing lines are the distinguish features of perspective projection [18]. Xu *et al.* [19] proposed a pose estimation method based on vanishing lines of a T-shaped target. Wang [20] used a target with three equally spaced parallel lines to estimate the rotation matrix by moving the target into at least three different positions. Wei *et al.* [21,22] calibrated a line-structured vision sensor and a binocular vision sensor by a planar target with several parallel lines. Two mutually orthogonal groups of parallel lines are common in urban environments, such as crossroads, facades of architectures and so on. They can be used as the calibration targets. Even if they are absent in the scene, targets with two mutually orthogonal groups of parallel lines can be employed.

In this paper, we focus on the calibration of widely distributed cameras in ring-topologies. A planar target with two mutually orthogonal groups of parallel lines is allocated to each camera. The vanishing line of the target plane is obtained from two vanishing points. Then the relative pose between each camera and its corresponding target is initialized and refined based on vanishing features and the known line length. A closed image sequence of neighboring pairs of calibration targets is acquired by repeated operations of the auxiliary camera. Then the relative poses between two adjacent targets can be obtained and the transformation matrix from each target to the reference target is initialized in a chainwise manner. In order to adjust the accumulated error due to the chain of transformations, a global calibration method is proposed to optimize relative poses of the targets based on the constraint of the ring-type structure. Finally, using the targets as media, the optimal relative poses between each camera and the reference camera are obtained.

The rest of the paper is organized as follows: preliminary work is introduced in Section 2. The proposed global calibration method is described in Section 3. Accuracy analysis of different factors’ effects is given in Section 4. Synthetic data, simulation images and real data experiments are carried out in Section 5. The conclusions are given in Section 6.

In this paper, the camera coordinate frame is used as the vision sensor coordinate frame. Assuming DVS consists of *M* cameras, C* _{k}*CF (1 ≤

As shown in Figure 1, the target is constructed of two mutually orthogonal groups of parallel lines with a known length *L*_{1} and the distance *L*_{2}. ${P}_{m}^{k}$ and ${l}_{m}^{k}$ denote the *m*th corner point and the *m*th feature line of target *k*, 1 ≤ *m* ≤ 6. T* _{k}*CF (1 ≤

In this paper, a two-dimensional image point is denoted by $p={\left[u,v\right]}^{T}$, a three-dimensional spatial point by $P={\left[X,Y,Z\right]}^{T}$. $\tilde{p}$ and $\tilde{P}$ are the corresponding homogeneous points, $\tilde{p}={[{p}^{T},1]}^{T}$, $\tilde{P}={[{P}^{T},1]}^{T}.$ The projection of a spatial point in TCF onto the image plane is described as:

$$s\tilde{p}=\left[{\rm K}|{0}_{3\times 1}\right]T\tilde{P},\text{}K=\left[\begin{array}{ccc}{f}_{x}& 0& {u}_{0}\\ 0& {f}_{y}& {v}_{0}\\ 0& 0& 1\end{array}\right],\text{}T=\left[\begin{array}{cc}{R}_{3\times 3}& {t}_{3\times 1}\\ {0}_{1\times 3}& 1\end{array}\right],\text{}R=\left[\begin{array}{ccc}{R}_{11}& {R}_{12}& {R}_{13}\\ {R}_{21}& {R}_{22}& {R}_{23}\\ {R}_{31}& {R}_{32}& {R}_{33}\end{array}\right]$$

(1)

where ** K** is the intrinsic parameter matrix,

$$R\left(\mathsf{\phi},\mathsf{\theta},\mathsf{\varphi}\right)=\left[\begin{array}{ccc}cos\mathsf{\varphi}cos\mathsf{\phi}\text{}+\text{}sin\mathsf{\theta}sin\mathsf{\varphi}sin\mathsf{\phi}& cos\mathsf{\theta}sin\mathsf{\varphi}& cos\mathsf{\phi}sin\mathsf{\theta}sin\mathsf{\varphi}\text{}-\text{}cos\mathsf{\varphi}sin\mathsf{\phi}\\ cos\mathsf{\varphi}sin\mathsf{\theta}sin\mathsf{\phi}\text{}-\text{}cos\mathsf{\phi}sin\mathsf{\varphi}& cos\mathsf{\theta}cos\mathsf{\varphi}& sin\mathsf{\varphi}sin\mathsf{\phi}\text{}+\text{}cos\mathsf{\varphi}cos\mathsf{\phi}sin\mathsf{\theta}\\ cos\mathsf{\theta}sin\mathsf{\phi}& -sin\mathsf{\theta}& cos\mathsf{\theta}cos\mathsf{\phi}\end{array}\right]$$

(2)

In this paper, definitions of the transformation matrices are shown in Table 1.

The principle of global calibration is shown in Figure 2. In this paper, we choose camera 1 as the reference camera as well as target 1 as the reference target. The main process of the proposed global calibration method works as follows:

- Place the planar targets in each camera’s FOV. The symmetry axis of each target is set to approximately orient to its corresponding camera. Image
*I*denotes target_{k}*k*captured by camera*k*. An image sequence= {*I**I*|1 ≤_{k}*k*≤*M*} is obtained. - Use the auxiliary camera to capture neighboring pairs of the targets. As shown in Figure 2a, image ${\tilde{I}}_{i}$ denotes two adjacent targets (
*i*,*j*) captured by the auxiliary camera. A closed image sequence $\tilde{I}$ is acquired, $\tilde{I}=\left\{{\tilde{I}}_{i}|1\le i\le M\right\};$ 1 ≤*i*≤*M*;*j*=*i*+1 if*i*<*M*,*j*= 1 if*i*=*M*. - All the images are rectified to compensate for cameras’ distortion based on the intrinsic calibration results. The linear equation of parallel lines in the image plane can be obtained from the feature points extracted by Steger’s method [25].
- Compute the transformation matrix ${T}_{k}^{\mathrm{tc}}$ based on the undistorted image sequence
.*I* - Compute the transformation matrix ${T}_{ij}^{\mathrm{tt}}$ of two adjacent targets based on the undistorted image sequence $\tilde{I}.$
- Calculate the initial value of ${T}_{k1}^{\mathrm{tt}}$ (2 ≤
*k*≤*M*) by multiple coordinate transformations, as shown in Figure 2b. Then ${T}_{k1}^{\mathrm{tt}}$ (2 ≤*k*≤*M*) is refined by the global nonlinear optimization. - Compute the transformation matrix ${T}_{k1}^{\mathrm{cc}}$ (2 ≤
*k*≤*M*).The calibration is completed.

(**a**) Two adjacent targets (*i*, *j*) captured by the auxiliary camera; (**b**) The principle of the global calibration.

The feature points on a line can be extracted by Steger’s method [25], denoted by *p** _{i}* = [

Let $A={[\begin{array}{cccc}{p}_{1}& {p}_{2}& \dots & {p}_{s}\end{array}]}^{T},w={\left[\begin{array}{cccc}-1& -1& \dots & -1\end{array}\right]}^{T}$, $A$ is a $s\times 2$ matrix, $w$ is a $s\times 1$ vector. Thus the relation of $a,\text{}b$ and $c$ can be obtained by the least squares method:

$$\left[\begin{array}{c}a\\ b\end{array}\right]=c{\left[{A}^{T}A\right]}^{-1}\left[{A}^{T}w\right]$$

(3)

Thus, the linear equation of line ${l}_{m}^{k}$ can be found by the above method, where 1 ≤ *m* ≤ 6. Then the coordinates of ${\tilde{p}}_{m}^{k}$ are obtained from line intersections. As shown in Figure 1b, two vanishing points *v*_{1} and *v*_{2} can be found by lines ${l}_{1}^{k}$, ${l}_{3}^{k}$ and lines ${l}_{5}^{k}$, ${l}_{6}^{k}$, respectively. Then the linear equation of the vanishing line is obtained, denoted by:

$$\tilde{a}u+\tilde{b}v+\tilde{c}=0$$

(4)

As shown in Figure 1b, two groups of parallel lines converge at vanishing points *v*_{1} and *v*_{2} in the image plane, respectively. The line crossing *v*_{1} and *v*_{2} is the vanishing line. The equations of two non-parallel lines in T* _{k}*CF are:

$${a}_{i}x+{c}_{i}z+{d}_{i}=0,\text{}y=0,\left(i=1,2\right)$$

(5)

where ${a}_{1}{c}_{2}-{a}_{2}{c}_{1}\ne 0.$

Let ${\tilde{V}}_{1}\text{and}{\tilde{V}}_{2}$ be the infinite points in the two lines, ${\tilde{V}}_{i}={\left[-{c}_{i},0,{a}_{i},0\right]}^{T},\text{}i=1,2$ (the elaboration is given in Appendix A). We have:

$${s}_{i}{\tilde{v}}_{i}=\left[{\rm K}|{0}_{3\times 1}\right]{T}_{k}^{\text{tc}}{\tilde{V}}_{i}\text{}\left(i=1,2\right)$$

(6)

where ** K** denotes the intrinsic parameter matrix of camera

Combining Equations (1) and (6), we have:

$${\tilde{v}}_{i}={\left[{f}_{x}\frac{-{c}_{i}{R}_{11}+{a}_{i}{R}_{13}}{-{c}_{i}{R}_{31}+{a}_{i}{R}_{33}}+{u}_{0},{f}_{y}\frac{-{c}_{i}{R}_{21}+{a}_{i}{R}_{23}}{-{c}_{i}{R}_{31}+{a}_{i}{R}_{33}}+{v}_{0},1\right]}^{T}$$

(7)

Vanishing line ** l** can be computed by $l={\tilde{v}}_{1}\times {\tilde{v}}_{2}$. With Equation (7), we have:

$$l=\left[\begin{array}{l}{f}_{y}({R}_{23}{R}_{31}-{R}_{21}{R}_{33})\\ {f}_{x}({R}_{11}{R}_{33}-{R}_{13}{R}_{31})\\ {f}_{x}{f}_{y}\left({R}_{21}{R}_{13}-{R}_{11}{R}_{23}\right)+{f}_{x}{v}_{0}\left({R}_{13}{R}_{31}-{R}_{11}{R}_{33}\right)+{f}_{y}{u}_{0}\left({R}_{21}{R}_{33}-{R}_{23}{R}_{31}\right)\end{array}\right]$$

(8)

Combining Equations (2) and (8), the linear equation of the vanishing line is expressed as:

$$\frac{\mathrm{sin}\mathsf{\varphi}}{{f}_{x}}\left(u-{u}_{0}\right)+\frac{\mathrm{cos}\mathsf{\varphi}}{{f}_{y}}\left(v-{v}_{0}\right)-\mathrm{tan}\mathsf{\theta}=0$$

(9)

Combining Equations (4) and (9), the roll angle ϕ and pitch angle θ can be obtained:

$$\{\begin{array}{l}\mathsf{\varphi}={\mathrm{tan}}^{-1}\left[{f}_{x}\tilde{a}/\left({f}_{y}\tilde{b}\right)\right]\\ \mathsf{\theta}={\mathrm{tan}}^{-1}\left(-\frac{\tilde{c}}{\tilde{a}}\frac{\mathrm{sin}\mathsf{\varphi}}{{f}_{x}}-\frac{\mathrm{sin}\mathsf{\varphi}}{{f}_{x}}{u}_{0}-\frac{\mathrm{cos}\mathsf{\varphi}}{{f}_{y}}{v}_{0}\right)\end{array}$$

(10)

Vanishing points are determined by the directions of the parallel lines [18], we have:

$${\tilde{v}}_{i}=K{d}_{i}\left(i=1,2\right)$$

(11)

where *d** _{i}* is the 3 × 1 direction vector of the line in CCF.

${l}_{1}^{k}$ and ${l}_{6}^{k}$ coincide with the *z*-axis and the *x*-axis of T* _{k}*CF, respectively. Thus:

$$\{\begin{array}{l}{d}_{1}/\Vert {d}_{1}\Vert =R\cdot {\left[\begin{array}{ccc}0& 0& 1\end{array}\right]}^{T}\\ {d}_{2}/\Vert {d}_{2}\Vert =R\cdot {\left[\begin{array}{ccc}1& 0& 0\end{array}\right]}^{T}\end{array}$$

(12)

According to the orthogonal constraint of a Rotation matrix, the rotation matrix ** R** of ${T}_{k}^{\mathrm{tc}}$ can be obtained:

$$R=\left[\begin{array}{ccc}\frac{{d}_{2}}{\Vert {d}_{2}\Vert}& \frac{{d}_{1}\times {d}_{2}}{\Vert {d}_{1}\Vert \cdot \Vert {d}_{2}\Vert}& \frac{{d}_{1}}{\Vert {d}_{1}\Vert}\end{array}\right]$$

(13)

${P}_{7}^{k}$ is a virtual point in the target plane. As vector $\overrightarrow{{P}_{1}^{k}{P}_{2}^{k}}\perp {d}_{2},$ the projections of $\overrightarrow{{P}_{1}^{k}{P}_{7}^{k}}$ and $\overrightarrow{{P}_{2}^{k}{P}_{7}^{k}}$ onto the vector ${d}_{2}$ are equal. We have:

$${d}_{2}^{T}\overrightarrow{{P}_{1}^{k}{P}_{7}^{k}}={d}_{2}^{T}\overrightarrow{{P}_{2}^{k}{P}_{7}^{k}}$$

(14)

Combining Equations (1) and (14), we have:

$$\frac{{z}_{1}}{{z}_{2}}=\frac{{d}_{2}^{T}{K}^{-1}{\tilde{p}}_{2}^{k}}{{d}_{2}^{T}{K}^{-1}{\tilde{p}}_{1}^{k}}$$

(15)

where ${z}_{1}$ and ${z}_{2}$ are the z coordinates of ${P}_{1}^{k}$ and ${P}_{2}^{k}$ in ${\mathrm{C}}_{k}\text{CF}$, respectively. ${\tilde{p}}_{1}^{k}$ and ${\tilde{p}}_{2}^{k}$ are known coordinates of the corner points.

Besides, the length of ${P}_{1}^{1}{P}_{2}^{k}$ is known:

$$\Vert {z}_{1}{K}^{-1}{\tilde{p}}_{1}^{k}-{z}_{2}{K}^{-1}{\tilde{p}}_{2}^{k}\Vert ={L}_{1}$$

(16)

Combining Equations (15) and (16), ${z}_{1}$ and ${z}_{2}$ can be found. Thus, the coordinate of ${P}_{1}^{k}$ in ${\mathrm{C}}_{k}\text{CF}$ is obtained. In addition, ${P}_{1}^{k}$ is the origin of ${\mathrm{T}}_{k}\text{CF}$, the translation vector $t$ of ${T}_{k}^{\text{tc}}$ can be obtained:

$$t={z}_{1}{K}^{-1}{\tilde{p}}_{1}^{k}$$

(17)

Let ${\tilde{P}}_{m}^{k}\left(1\le m\le 6\right)$ be the homogeneous coordinate of ${P}_{m}^{k}$ in ${\mathrm{T}}_{k}\text{CF}$. Let ${\tilde{p}}_{m}^{k}$ be the corresponding coordinate in the image ${I}_{k}$. We have:

$${s}_{m}^{k}{\tilde{p}}_{m}^{k}=\left[{\rm K}|{0}_{3\times 1}\right]{T}_{k}^{\text{tc}}{\tilde{P}}_{m}^{k}$$

(18)

Assuming that image points are corrupted by independently and identically distributed Gaussian noise, the maximum likelihood estimation is obtained by minimizing the sum of squared distances between the observed feature lines and the re-projected corner points. ${T}_{k}^{\mathrm{tc}}$ (1 ≤ *k* ≤ *M*) are refined separately by minimizing the following function using Levenberg-Marquardt algorithm [26]:

$$f\left(\mathsf{\Omega}\right)={\displaystyle \sum _{m=1}^{6}\left[{d}^{2}\left({\tilde{p}}_{m}^{k},{l}_{m}^{k}\right)+{d}^{2}\left({\tilde{p}}_{m}^{k},{l}_{n}^{k}\right)\right]}$$

(19)

$$n=\{\begin{array}{l}m-1,\text{\hspace{1em}if}m\ge 2\\ 6,\text{\hspace{1em}\hspace{1em}\hspace{1em}if}m=1\end{array}$$

(20)

where Ω = ${T}_{k}^{\mathrm{tc}}$. ${l}_{m}^{k}$ and ${l}_{n}^{k}$ denotes the projections of ${l}_{m}^{k}$ and ${l}_{n}^{k}$ onto the image *I _{k}*, respectively.

Generally, target pair (*i*, *j*) is visible in the image ${\tilde{I}}_{i}$, that:

$$j=\{\begin{array}{l}i+1,\text{\hspace{1em}if}i\le M-1\\ 1,\text{\hspace{1em}\hspace{1em}if}i\text{=}M\end{array}$$

(21)

As shown in Figure 2a, ${T}_{ii}^{\mathrm{ta}}$ and ${T}_{ji}^{\mathrm{ta}}$ are the transformation matrices from target *i* and target *j* to the auxiliary camera, respectively. ${T}_{ii}^{\mathrm{ta}}$ and ${T}_{ji}^{\mathrm{ta}}$ can be initialized and refined separately by the methods described in Section 3.1. Then the initial value of ${T}_{ij}^{\mathrm{tt}}$ can be calculated by:

$${T}_{ij}^{\text{tt}}={\left({T}_{ji}^{\text{ta}}\right)}^{-1}{T}_{ii}^{\text{ta}}$$

(22)

The initial value of ${T}_{k1}^{\mathrm{tt}}$ (2 ≤ *k* ≤ *M*) can be obtained by the minimum times of chainwise coordinate transformations:

$${T}_{k1}^{\text{tt}}=\{\begin{array}{l}{\left[{T}_{k-1,k}^{\text{tt}}{T}_{k-2,k-1}^{\text{tt}}\cdots {T}_{2,3}^{\text{tt}}{T}_{1,2}^{\text{tt}}\right]}^{-1},\text{\hspace{1em}\hspace{1em}if}k\le \left(M/2\right)\\ {T}_{M,1}^{\text{tt}}{T}_{M-1,M}^{\text{tt}}\cdots {T}_{k+1,k+2}^{\text{tt}}{T}_{k,k+1}^{\text{tt}},\text{\hspace{1em}\hspace{1em}if}k\left(M/2\right)\end{array}$$

(23)

According to the camera model, we have:

$$\{\begin{array}{l}{s}_{m}^{ii}{\tilde{p}}_{m}^{ii}=\left[{\rm K}|{0}_{3\times 1}\right]{T}_{ji}^{\text{ta}}{\left({T}_{j1}^{\text{tt}}\right)}^{-1}{T}_{i1}^{\text{tt}}{\tilde{P}}_{m}^{i}\\ {s}_{m}^{ji}{\tilde{p}}_{m}^{ji}=\left[{\rm K}|{0}_{3\times 1}\right]{T}_{ii}^{\text{ta}}{\left({T}_{i1}^{\text{tt}}\right)}^{-1}{T}_{j1}^{\text{tt}}{\tilde{P}}_{m}^{j}\end{array}\text{}$$

(24)

where $K$denotes the intrinsic matrix of the auxiliary camera; ${\tilde{p}}_{m}^{ii}$ and ${\tilde{p}}_{m}^{ji}$ denote the reprojections of ${P}_{m}^{i}$ and ${P}_{m}^{j}$ onto the image ${\tilde{I}}_{i}$, respectively.

Assuming image points are corrupted by independent and identical Gaussian noise, ${T}_{k1}^{\mathrm{tt}}$ (2 ≤ *k* ≤ *M*) can be optimized by minimizing the following function using Levenberg-Marquardt algorithm [26]:

$$f\left(\mathsf{\Omega}\right)={\displaystyle \sum _{i=1}^{M}{\displaystyle \sum _{m=1}^{6}\left[{d}^{2}\left({\tilde{p}}_{m}^{ii},{l}_{m}^{ii}\right)+{d}^{2}\left({\tilde{p}}_{m}^{ii},{l}_{n}^{ii}\right)+{d}^{2}\left({\tilde{p}}_{m}^{ji},{l}_{m}^{ji}\right)+{d}^{2}\left({\tilde{p}}_{m}^{ji},{l}_{n}^{ji}\right)\right]}}$$

(25)

where $\mathsf{\Omega}=\left({T}_{2,1}^{tt},\dots ,{T}_{M1}^{tt}\right),{T}_{1,1}^{tt}={I}_{4\times 4}$ ${l}_{m}^{ii}$ and ${l}_{m}^{ji}$ denote the projections of line ${l}_{m}^{i}$ and ${l}_{m}^{j}$ onto the image ${\tilde{I}}_{i}$, respectively. $R\text{of}{T}_{k1}^{\text{tt}}\left(2\le k\le M\right)$ are parameterized by the Rodrigues formula. A good starting point of the optimization is provided by Equations (22) and (23). $\left(m,n\right)$ and $\left(i,j\right)$ subject to Equations (20) and (21), respectively.

After the global calibration of the targets, the transformation matrix from each camera to the reference camera can be found:

$${T}_{k1}^{\text{cc}}={T}_{1}^{\text{tc}}{T}_{k1}^{\text{tt}}{\left({T}_{k}^{\text{tc}}\right)}^{-1}\text{}\left(2\le k\le M\right)$$

(26)

where ${T}_{k}^{\mathrm{tc}}$ (1 ≤ *k* ≤ *M*) are the results of Equation (19), ${T}_{k1}^{\mathrm{tt}}$ (2 ≤ *k* ≤ *M*) are the optimization results of Equation (25).

In this section, analysis of several factors’ effects on the accuracy of the proposed method is performed by synthetic data experiments. The auxiliary camera’s intrinsic parameters are *f _{x}* =

The cameras’ positions are represented by the coordinates of the cameras’ origins in ECF. The cameras’ orientations are denoted by the Euler angles (, θ, ϕ) from ECF to CCF. Targets are placed on the ground for convenience. The targets’ positions are represented by the coordinates of the targets’ origins in ECF. The targets’ orientations are denoted by the yaw angle from positive z-axis of ECF to the symmetry axis of the target.

*dR* and *dt* denote the 2-norm of rotation vector and translation vector differences between the calculation results and the real data. The RMS errors of *dR*, *dt* are used to evaluate the accuracy. The number of points that emulate feature lines is equal to the line length in pixels. Gaussian noise with zero mean and different noise levels is added to the image coordinates of the points of feature lines. Analysis of the factors’ effects is shown as follows.

The image sequence $\tilde{I}$ is acquired by the auxiliary camera. The pitch angle of the auxiliary camera relative to the target plane is one of the factors affecting the calibration accuracy. In this experiment, two adjacent targets are captured by the auxiliary camera at different pitch angles. The targets’ positions in ECF are [−450, 0, 300]* ^{T}* and [450, 0, 300]

From Figure 3, the RMS errors of rotation and translation are roughly U-shape. When θ→−90°, the optical axis of the auxiliary camera is perpendicular to the target plane. The vanishing points approximate to infinity, which leads to higher errors. When θ→0°, the number of the extracted feature points decreases, also leads to higher errors. It is ideal to capture pair targets when θ = −40°.

In this experiment, denotes the yaw angle difference between the symmetry axes of two adjacent targets. varies according to the cameras’ distribution. We also use the error of ${T}_{1,2}^{\mathrm{tt}}$ calculated by Equation (22) to evaluate the effect of .

The positions of the two targets are same with those in Section 4.1, while varies from 0 to 85°. The two targets remain symmetric about the plane *y _{e}o_{e}z_{e}* and the auxiliary camera lies in the symmetry plane. Gaussian noise with σ = 0.2 pixel is added.

From Figure 4, both the error of rotation and translation rise with the increasing of . When > 80°, the errors increase sharply. This is because when →90°, a group of parallel lines of each target are parallel to the image plane, thus the vanishing points approximate to infinity, which leads to great errors, so it is necessary to avoid →90° during the calibration.

In this experiment, we also use the error of ${T}_{1,2}^{\mathrm{tt}}$ obtained by Equation (22) to evaluate the effect of the parallel line distances. The poses of the targets are same with those in Section 4.1. The pitch angle of the auxiliary camera relative to the target plane is set to −40°. L_{1} = 500 mm, L_{2} varies from 100 mm to 400 mm. Gaussian noise with different levels is added to the image points. For each distance level, 100 independent trials are performed.

From Figure 5, it can be seen that the error increases linearly with the noise level and decreases with the increasing distance of parallel lines. This is because the difference among slopes of intersecting lines goes up with the increasing distance of parallel lines. Calculation error of vanishing points is inversely proportional to the difference among the slopes of intersecting lines, which have been proven in [21].

Numerous situations require a system that provides a real-time view of the surroundings [28]. One of the typical cases is the operations on aerial vehicles. In this experiment, eight cameras are used to simulate a DVS mounted on an unmanned aerial vehicle (UAV), as shown in Figure 6. The proposed method is compared with other typical methods by both synthetic data and images simulated by 3ds Max software.

The intrinsic parameters of eight cameras are *f _{x}* =

There are many calibration methods for multiple cameras. Here five typical methods are described as follows and summarized in Table 3:

*Method 1*: This method is similar to the proposed method, except that corner points ${\tilde{p}}_{m}^{k}$ are extracted by the corner extraction engine of the J. Bouguet Camera Calibration Toolbox [23], rather than the intersections of feature lines.

*Method 2*: This method is similar to the proposed method, except that planar checkerboards with 12 × 12 grids are used as the calibration targets. The side length of each square is 50 mm.

*Method 3*: The calibration targets and the extraction of corner points ${\tilde{p}}_{m}^{k}$ are same with the proposed method. Instead of capturing neighboring target pairs, the auxiliary camera captures all the targets in one image frame, so the relative poses between targets can be computed directly.

*Method 4*: This method is similar to Method 3, except that corner points ${\tilde{p}}_{m}^{k}$ are obtained by the corner extraction method used in Method 1.

*Method 5*: This method is similar to Method 3, except that planar checkerboards of Method 2 are used as the calibration targets.

In order to illustrate the effect of the global calibration, there are two sub-methods called chainwise calibration method and global calibration method. The only difference between the two sub-methods is whether to use the global optimization in Section 3.3 or not. ${T}_{k1}^{\mathrm{tt}}$ (2 ≤ *k* ≤ *M*) of the chainwise method are obtained directly from multiple coordinate transformations by Equation (23), while the global calibration method is based on an additional global optimization by Equation (25).

In this experiment, the RMS error of ${T}_{k1}^{\mathrm{cc}}$ (2 ≤ *k* ≤ 8) is used to evaluate the accuracy. Gaussian noise with σ = 0.2 pixel is added. For each method, 100 independent trials are performed. From Figure 7, the proposed method outperforms Methods 1–5. Figure 7a,b show that the error accumulates with the coordinate transformations, and peaks at camera 5, due to the maximum times of transformations. The methods based on the constraint of ring-topologies can effectively reduce the accumulated error, especially for cameras which are far away from the reference.

Calibration error of each method. (**a**,**b**) RMS errors of rotation vector and translation vector of the proposed method, Method 1 and 2; (**c**,**d**) RMS errors of rotation vector and translation vector of Methods 3–5.

Methods 3–5 do not suffer from the accumulated error issue because all the targets are visible in one image frame. However, due to limitations of image resolutions, the accuracy of the pose estimation decreases with the increasing number of the targets observed in one image.

Compared with lines-feature algorithms, points-feature algorithms are more sensitive to the image noise. Figure 7 shows that Methods 1 and 4 are worse than other methods, respectively. Further discussion is given in Section 5.3.

In this experiment, the RMS error of ${T}_{k1}^{\mathrm{cc}}$ (2 ≤ *k* ≤ 8) is used to evaluate the effect of the noise level. Synthetic data is same with those in Section 5.1.2. Gaussian noise from 0.0 to 1.0 pixel is added. For each noise level, 100 independent trials are performed.

From Figure 8, the RMS error increases linearly with the noise level. It also shows that the proposed method is superior to other methods. If the noise level of the real DVS is less than 0.5 pixels, the RMS errors of rotation and translation of all the cameras are less than 0.05 deg and 1.1 mm, respectively, which is acceptable for common applications.

As shown in Figure 9, we use the 3ds Max software to simulate image sequences. The parameters of the cameras and the targets are same with those in Section 5.1.2. The feature lines are obtained based on feature points extracted by Steger’s method [25]. The error of ${T}_{k1}^{\mathrm{cc}}$ (2 ≤ *k* ≤ 8) is used to evaluate the accuracy.

Simulation images: (**a**) Target 1 captured by camera 1; (**b**) Target 2 and Target 3 captured by the auxiliary camera.

Figure 10 shows that errors of rotation and translation accumulate with the increasing times of coordinate transformations. The proposed method can reduce the accumulated error due to multiple coordinate transformations. Further discussion is given in Section 5.3.

As shown in Figure 11a, eight targets are located in an area about 1200 mm × 1200 mm. As the relative poses of the cameras with non-overlapping FOV are mainly determined by the relative poses of the targets, the RMS errors of point pair distances between eight targets are used as calibration errors in the real experiments.

Global calibration of eight targets: (**a**) Eight targets and the auxiliary camera in the real experiment; (**b**) The auxiliary camera captures eight checkerboards in one image frame.

The distance of point pair ${p}_{m}^{k}$ and ${p}_{m}^{l}$ is computed according to the calibration result, and is called measurement distances, *d _{m}*. The targets are also calibrated similarly by a calibrated Canon 60D digital camera. The distances of the same point pairs can be obtained in the same way and are used as the ground truth

Distance error can be computed by $\mathsf{\Delta}d={d}_{m}-{d}_{t}$. For the proposed method, Methods 1, 3 and 4, RMS error of $\mathsf{\Delta}d\left({P}_{1}^{1}{P}_{1}^{k}\right),\mathsf{\Delta}d\left({P}_{2}^{1}{P}_{2}^{k}\right)$ and $\mathsf{\Delta}d\left({P}_{6}^{1}{P}_{6}^{k}\right)$ is used to evaluate the accuracy. For Methods 2 and 5, five point pairs are randomly selected and the RMS error of distance error is computed as the calibration error.

The auxiliary camera is a 1/3-in Sony CCD image sensor (ICX673) with a 3.6 mm lens. The image resolution is 720 pixel × 432 pixel. Target parameters are ${L}_{1}=135\text{mm},{L}_{2}=70\text{mm}.$ The image resolution of the Canon device is 1920 pixel × 1280 pixel. The intrinsic parameters of the sensors are calibrated using Bouguet’s calibration toolbox [23], as shown in Table 4.

Figure 12 also shows that the proposed method achieves the best accuracy. The RMS error of point pair distances between target 5 and the reference target of the proposed method and Methods 1–5 are 0.465 mm, 0.828 mm, 3.94 mm, 3.83 mm, 1.92 mm and 21.6 mm, respectively. The proposed method is superior to other methods.

Due to the restriction of image resolutions, the accuracy of the pose estimation decreases with the increasing number of the targets observed in one image. There exists a trade-off between the available features of target projections and the accumulated error from the chain of transformations. The experimental results show that the accumulated error can be effectively adjusted from the constraint of ring-topologies. For the vision sensor such as the Sony CCD sensor, capturing all the targets is not a wise choice. The benefits of direct calculation of the relative poses of the targets are cancelled out by the rise of feature extraction errors.

Moreover, it is not convenient to capture all the targets in some applications, because the auxiliary camera should be far away from the widely distributed targets. As shown in Figure 11b, in order to capture all the checkerboards in one image frame, the targets are pasted on the wall.

The results of simulation images show that the accuracy of Methods 2 and 5 is close to or even better than the proposed method. However, Methods 2 and 5 achieve the worst accuracy in the real experiments. Figure 13 shows that simulation images are very sharp and clear, which greatly benefit the corner extraction of checkerboards. However, real images could not be so ideal.

Simulation image and real image. (**a**) Checkerboards simulated by software; (**b**) Checkerboards captured by the auxiliary camera.

These results indicate that the proposed method is accurate and robust, especially when dealing with real images. Methods 2 and 5 are not stable against the image quality. However, there is a gap among the results of synthetic data, simulation images and real experiments. There may be some reasons for this.

Firstly, there is a measurement error during the feature extraction. In our method, the line extraction algorithm is a common-used method with acceptable accuracy and good generality. Line extraction algorithms with higher accuracy contribute to improve the accuracy, which will be further studied in the future. Secondly, the targets used in real experiments are printed on paper. They may not be strictly planar, which also leads to measurement errors.

In addition, the Sony CCD vision sensor is not a professional high-precision vision sensor, which is usually used for security cameras and radio controlled vehicles. High-resolution vision sensors can be used to improve the accuracy.

In this article, we have developed a new global calibration method for vision sensors in ring-topologies. Line-based calibration targets are placed in each camera’s FOV. Firstly, the relative poses of cameras and targets are initialized and refined based on the principle of vanishing features and the known line length. Next, in order to overcome small or no overlapping FOV between adjacent cameras, an auxiliary camera is used to capture neighboring targets. The relative poses of the targets is initialized in a chainwise manner, followed by nonlinear optimization to minimizing the squared distances between the observed feature lines and the re-projected corner points. Then the transformation matrix between each camera and the reference camera is determined.

The factors that affect the calibration accuracy are analyzed by synthetic data experiments. Synthetic data, simulation images and real data experiments all demonstrate that the proposed method is accurate and robust to image noise. The accumulated error can be adjusted effectively based on the constraint of ring-topologies. Real data experiments indicates that the measurement accuracy of the farthest camera by the proposed method is about 0.465 mm in an area about 1200 mm × 1200 mm.

The poses of targets need not be known previously and can be adjusted according to the distribution of cameras. It does not need to place the targets into different positions, one placement is enough. Our method is simple and flexible and can be applied to different configurations of multiple cameras. It is well suited for the on-site calibration of widely distributed cameras.

In this paper, we focus on the calibration of DVS in ring-topologies, which contains additional constraint. When dealing with DVS in open-topologies, accumulated errors cannot be adjusted. In addition, vanishing points approximate to infinity when feature lines are parallel to the image plane, which leads to higher errors, so the angle between parallel lines and the image plane should be in a certain range to avoid vanishing points approximate to infinity.

Restricted by hardware conditions, experiments using eight sensors mounted on an UAV are temporarily lacking. We plan to apply our method for the calibration of multiple vision sensors mounted on the vehicle in the future. Methods based on the feature lines in both indoor and outdoor environments instead of planar targets will also be investigated.

This work is supported by the Industrial Technology Development Program under Grant B1120131046.

The following abbreviations are used in this manuscript:

DVS | Distributed vision Sensors |

FOV | Field of view |

GCF | Global coordinate frame |

SIFT | Scale invariant feature transform |

CCF | Camera coordinate frame |

ACF | Auxiliary camera coordinate frame |

ICF | Image coordinate frame |

TCF | Target coordinate frame |

ECF | Ground coordinate frame |

RMS | root mean square |

CCD | Charge coupled device |

A plane in a 3D space can be represented by an equation $ax+by+cz+d=0$. Thus, a plane may be represented by the vector $p={\left[a,b,c,d\right]}^{T}$. A 3D spatial point with homogeneous coordinates $x={\left[{x}_{1},{x}_{2},{x}_{3},{x}_{4}\right]}^{T}$ lies in the plane $p$ if and only if ${x}^{T}p=0$.

Homogeneous vectors ${\left[{x}_{1},{x}_{2},{x}_{3},{x}_{4}\right]}^{T}$ such that ${x}_{4}\ne 0$ corresponds to finite points in ${\mathbb{R}}^{3}$. The points with last coordinate ${x}_{4}=0$ are known as points at infinity. The set of points at infinity can be written as ${x}_{\infty}={\left[{x}_{1},{x}_{2},{x}_{3},0\right]}^{T}$. Note that ${x}_{\infty}$ lies in the plane at infinity, denoted by the vector ${p}_{\infty}={\left[0,0,0,1\right]}^{T}$, because ${x}_{\infty}^{T}{p}_{\infty}=0$.

From Equation (5), note that $\left[-{c}_{i},0,{a}_{i},0\right]{\left[{a}_{i},0,{c}_{i},{d}_{i}\right]}^{T}=0,\text{}\left[-{c}_{i},0,{a}_{i},0\right]{\left[0,1,0,0\right]}^{T}=0$, thus the line ${a}_{i}x+{c}_{i}z+{d}_{i}=0,y=0$ intersects the infinite plane in the infinite point ${\left[-{c}_{i},0,{a}_{i},0\right]}^{T}$.

Author Contributions

The work presented in this paper has been done in collaboration of all authors. Xiaolong Wu conceived the method, designed the experiments and wrote the paper. Sentang Wu was the project leader and in charge of the direction and supervision. Zhihui Xing and Xiang Jia performed the experiments and analyzed the data. All authors discussed the results together and reviewed the manuscript.

Conflicts of Interest

The authors declare no conflict of interest.

1. Lu R.S., Li Y.F. A global calibration method for large-scale multi-sensor visual measurement systems. Sens. Actuators A Phys. 2004;116:384–393. doi: 10.1016/j.sna.2004.05.019. [Cross Ref]

2. Peng X.M., Bennamoun M., Wang Q.B., Ma Q., Xu Z.Y. A low-cost implementation of a 360 degrees vision distributed aperture system. IEEE Trans. Circuits Syst. Video Technol. 2015;25:225–238. doi: 10.1109/TCSVT.2014.2335832. [Cross Ref]

3. Bazargani H., Laganiere R. Camera calibration and pose estimation from planes. IEEE Instrum. Measur. Mag. 2015;18:20–27. doi: 10.1109/MIM.2015.7335834. [Cross Ref]

4. Lowe D.G. Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 2004;60:91–110. doi: 10.1023/B:VISI.0000029664.99615.94. [Cross Ref]

5. Kumar R.K., Ilie A., Frahm J.M., Pollefeys M. Simple calibration of non-overlapping cameras with a mirror; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; Anchorage, AK, USA. 23–28 June 2008; pp. 1–7.

6. Hesch J.A., Mourikis A.I., Roumeliotis S.I. Algorithmic Foundation of Robotics VIII. Springer; Berlin, Germany: 2009. Mirror-based extrinsic camera calibration; pp. 285–299.

7. Takahashi K., Nobuhara S., Matsuyama T. A new mirror-based extrinsic camera calibration using an orthogonality constraint; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR); Providence, RI, USA. 16–21 June 2012; pp. 1051–1058.

8. Liu Z., Zhang G.J., Wei Z.Z., Sun J.H. Novel calibration method for non-overlapping multiple vision sensors based on 1D target. Opt. Lasers Eng. 2011;49:570–577. doi: 10.1016/j.optlaseng.2010.11.002. [Cross Ref]

9. Liu Z., Zhang G.J., Wei Z.Z., Sun J.H. A global calibration method for multiple vision sensors based on multiple targets. Measur. Sci. Technol. 2011;22:125102. doi: 10.1088/0957-0233/22/12/125102. [Cross Ref]

10. Bosch J., Gracias N., Ridao P., Ribas D. Omnidirectional underwater camera design and calibration. Sensors. 2015;15:6033–6065. doi: 10.3390/s150306033. [PMC free article] [PubMed] [Cross Ref]

11. Pagel F. Calibration of non-overlapping cameras in vehicles; Proceedings of the 2010 IEEE Intelligent Vehicles Symposium (IV); San Diego, CA, USA. 21–24 June 2010; pp. 1178–1183.

12. Sun J.H., He H.B., Zeng D.B. Global calibration of multiple cameras based on sphere targets. Sensors. 2016;16:14. doi: 10.3390/s16010077. [PMC free article] [PubMed] [Cross Ref]

13. Ullman S. The interpretation of structure from motion. Proc. R. Soc. Lond. B Biol. Sci. 1979;203:405–426. doi: 10.1098/rspb.1979.0006. [PubMed] [Cross Ref]

14. Szeliski R. Computer Vision: Algorithms and Applications. Springer Science & Business Media; London, UK: 2010.

15. Fitzgibbon A.W., Zisserman A. Computer Vision—ECCV'98. Springer; Berlin, Germany: 1998. Automatic camera recovery for closed or open image sequences; pp. 311–326.

16. Zhang Z., Shan Y. Incremental motion estimation through modified bundle adjustment; Proceedings of the 2003 International Conference on Image Processing; Barcelona, Spain. 14–17 September 2003.

17. Ly D.S., Demonceaux C., Vasseur P., Pégard C. Extrinsic calibration of heterogeneous cameras by line images. Mach. Vis. Appl. 2014;25:1601–1614. doi: 10.1007/s00138-014-0624-3. [Cross Ref]

18. Hartley R., Zisserman A. Multiple View Geometry in Computer Vision. 2nd ed. Cambridge University Press; Cambridge, UK/New York, NY, USA: 2003.

19. Xu G.L., Qi X.P., Zeng Q.H., Tian Y.P., Guo R.P., Wang B.A. Use of land’s cooperative object to estimate UAV’s pose for autonomous landing. Chin. J. Aeronaut. 2013;26:1498–1505. doi: 10.1016/j.cja.2013.07.049. [Cross Ref]

20. Wang X.L. Novel calibration method for the multi-camera measurement system. J. Opt. Soc. Korea. 2014;18:746–752. doi: 10.3807/JOSK.2014.18.6.746. [Cross Ref]

21. Wei Z.Z., Shao M.W., Zhang G.J., Wang Y.L. Parallel-based calibration method for line-structured light vision sensor. Opt. Eng. 2014;53:033101. doi: 10.1117/1.OE.53.3.033101. [Cross Ref]

22. Wei Z., Liu X. Vanishing feature constraints calibration method for binocular vision sensor. Opt. Express. 2015;23:18897–18914. doi: 10.1364/OE.23.018897. [PubMed] [Cross Ref]

23. Bouguet J.-Y. Camera Calibration Toolbox for Matlab. [(accessed on 28 March 2016)]. Available online: http://www.vision.caltech.edu/bouguetj/calib_doc/index.html.

24. Zhang Z.Y. A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell. 2000;22:1330–1334. doi: 10.1109/34.888718. [Cross Ref]

25. Steger C. An unbiased detector of curvilinear structures. IEEE Trans. Pattern Anal. Mach. Intell. 1998;20:113–125. doi: 10.1109/34.659930. [Cross Ref]

26. Moré J.J. Numerical Analysis. Springer; Berlin, Germany: 1978. The levenberg-marquardt algorithm: Implementation and theory; pp. 105–116.

27. Diebel J. Representing attitude: Euler angles, unit quaternions, and rotation vectors. Matrix. 2006;58:1–35.

28. Rose M.K., Chamberlain J., LaValley D. Real-Time 360° Imaging System for Situational Awareness. SPIE; Orlando, FL, USA: 2009.

Articles from Sensors (Basel, Switzerland) are provided here courtesy of **Multidisciplinary Digital Publishing Institute (MDPI)**

PubMed Central Canada is a service of the Canadian Institutes of Health Research (CIHR) working in partnership with the National Research Council's national science library in cooperation with the National Center for Biotechnology Information at the U.S. National Library of Medicine(NCBI/NLM). It includes content provided to the PubMed Central International archive by participating publishers. |