Augmented reality in sport scenarios using cameras mounted on a tripod

CTIM Technical Report ISSN 2254-2353 Augmented Reality in Sport Scenarios Using Cameras Mounted on a Tripod Miguel Alem´an Flores, Luis ´ Alvarez Le´on, Pedro Henr´ıquez Castellano and Agust´ın Trujillo Pino No. 2 Las Palmas de Gran Canaria 13 September 2012 Augmented Reality in Sport Scenarios Using Cameras Mounted on a Tripod Miguel Alem´an-Flores Luis Alvarez Pedro Henriquez Agust´ın Trujillo 26/07/12 Abstract In this paper we address the problem of inserting virtual content in a video sequence. The method we propose uses just image information. We perform primitive tracking, camera calibration, real and virtual camera synchronisation and finally rendering to insert the virtual content in the real video sequence. To simplify the calibration step we assume that cameras are mounted on a tripod (which is a common situation in practise). The primitive tracking procedure, which uses lines and circles as primitives, is performed by means of a CART (Classification and Regression Tree). Finally, the virtual and real camera synchronisation and rendering is performed using functions of OpenGL (Open Graphic Library). We have applied the method proposed to sport event scenarios, specifically, soccer matches. In order to illustrate its performance, it has been applied to real HD (High Definition) video sequences. The quality of the proposed method is validated by inserting virtual elements in such HD video sequence. Keywords: Augmented Reality, Graphic Insertion, Camera Calibration 1 Introduction The broadcasting of sport events increasingly introduces the processing of video sequences for the insertion of virtual objects. On the one hand, these objects can be for a better understanding of the scene such as country flags in swimming, yellow down line in football, offside line in soccer and puck path in hockey. At the other hand, them can be for adding advertisements in different places during the event without disturbing the viewers. Most methods of virtual objects insertion in a video consist of two main stages, which are camera calibration and virtual content insertion. Camera calibration consists of several stages, which usually include initialisation, calibration estimation, primitive tracking and calibration refining. Moreover, virtual content insertion is divided in two steps, camera synchronisation and rendering. In this paper, we analyse this problem in real application scenarios, 1 where we deal with some additional problems, such as the small number of visible primitives which are usually visible and are needed for camera calibration, or the large size of HD videos. The main assumptions we make are, firstly, that the video sequences have been acquired using a camera mounted on a tripod (which is a common situation in practise) and fixed in location it can freely rotate and change their intrinsic parameters by zooming. Secondly, that there exists a certain contrast between all primitives of interest for calibration (lines and circles) and the background (grass). Usually, in sport event scenarios, primitives are well contrasted with respect to the background (the green of the grass in the case of soccer matches). In our implementation, there are several stages as is shown in Figure 1. Initialisation stage consists of two different process, camera calibration initialisation and virtual objects configuration. Camera calibration initialisation is divided in three steps, which are only carried out on the first frame: load of previously calculated information (geometrical parameters of the tripod and training classes for the decision tree), primitive detection, and camera calibration for the first frame. Primitive detection is performed by means of a morphological method described in [7], whereas the camera calibration technique is explained in this paper. In virtual object configuration we have to define the objects appearance and their positions in the soccer stadium. For the following frames, we directly start at the calibration estimation and we consider the information obtained from the previous two frames. We assume that the changes between consecutive frames are not too large and, furthermore, as we are dealing with cameras placed on a tripod, the movements of the camera are restricted. Afterwards, we continue the process with the primitive tracking stage, which searches for the primitives in the image using a decision tree and the projection of the reference primitives. The latter are the lines and circles in a model of a soccer field with actual dimensions, which are projected using the homography estimated from the previous two frames. Finally, from the primitives detected and the geometry of the tripod, we refine the calibration for the current frame. When we have the camera calibrated in a frame, we can synchronise the real camera with a virtual camera and project the virtual objects onto the real image using OpenGL [2]. We use OpenGL because it provides functions to manage easily virtual camera and virtual objects. For example changing the viewpoint or the virtual camera position, adding textures and blending to the virtual objects, etc. This paper is structured as follows: In section 2, we summarise the state of the art. Section 3 explains the geometry and calibration process for cameras mounted on a tripod. In section 4, we introduce a method to perform primitive tracking using a decision tree. In section 5, we describe the camera synchronisation. Section 6 is the explanation of the rendering. In section 7, we show some experiments and results. Finally, in section 8, we present our main conclusions. 2 Figure 1: Stages in our implementation for inserting virtual content in a video sequence. 2 Related works Different approaches have been presented to tackle the problem of insert virtual content or advertisements in sport event videos. To achieve incorporate virtual graphics in a video, we have to calibrate the camera to get the camera parameters and be able to synchronise the real camera with the virtual camera. To perform a video camera calibration, the most common strategy consists of certain tasks which are applied on each video frame: feature extraction (e.g. primitives and background), camera calibration estimation (based on the previous frames), primitive tracking, improvement of camera calibration estimation. See for instance [3, 4, 5, 6, 9]. In our case, we deal with cameras mounted on a tripod, and therefore, we have to take into account the changes in the camera model, as explained in [8, 15]. With the calibration done, we can start the graphic insertion, which can be divided in two steps: camera synchronisation and rendering. Different techniques and libraries are used to perform these stages. For example in [10, 11] they project the virtual content with the projection matrix obtained in the camera calibration step, and paint the projected content pixel by pixel in the real image. To give a more realistic appearance mixing the virtual and real image, a blending technique is used in [13]. On the other hand, graphic libraries are used to improve the procedure efficiency as in [12]. A common issue in all the related works is the image segmentation for detect pixels which can be replaced, e.g. grass, and which not, e.g. players. 3 3 Geometry and calibration of cameras mounted on a tripod A tripod is defined by a centre of rotation ¯ X0= (X0, Y0, Z0)Tand two unitary rotation axes ¯e0=¯e0 x,¯e0 y,¯e0 zT, ¯e1=¯e1 x,¯e1 y,¯e1 zT. We call R¯ek, θk the matrix to rotate by an angle of θkabout axis ¯ek. In order to rotate a 3D point ¯ Xabout axis ¯ekusing the centre of rotation ¯ X0, the transformation turns into the following equation: ¯ X(θk) = ¯ X0+R¯ek, θk¯ X−¯ X0(1) The general motion of a tripod is the composition for two rotations of the above type. We assume that the centre of rotation ¯ X0is the same for both axes, which is equivalent to assume that the two axes about which the tripod rotates intersect at a point. This is a common situation and the points are then transformed according to the general equation for the motion of a tripod: ¯ X(θ0, θ1) = ¯ X0+R¯e0, θ0R¯e1, θ1¯ X−¯ X0(2) From now on, we use the following notation : R(θ0, θ1)≡R¯e0, θ0R¯e1, θ1(3) ¯ t(θ0, θ1) = ¯ X0−R(θ0, θ1)¯ X0(4) Therefore, Equation 2 can be written in the form: ¯ X(θ0, θ1) = R(θ0, θ1)¯ X+¯ t(θ0, θ1) (5) The general equation for the projection of a 3D point ¯ X= (X, Y, Z)T onto the image plane is as follows: s   x y 1  =A(f0)R0jId, −¯c0k     X Y Z 1      (6) where A(f0) =    f00xc 0rf0yc 0 0 1   (7) R0=   r0 00 r0 01 r0 02 r0 10 r0 11 r0 12 r0 20 r0 21 r0 22   (8) 4 hId, −¯c0i=   100−¯c0 x 010−¯c0 y 001−¯c0 z   (9) In Equation 6, we assume that the possible lens distortion has previously been corrected. The matrix P0≡A(f0)R0Id, −¯c0is called projection matrix. For each frame, the values of (fn, θn 0, θn 1) determine the projection matrix as follows: P(fn, θn 0, θn 1)≡A(f0)R0hId, −¯c0i R(θn 0, θn 1)¯ t(θn 0, θn 1) 0 1 !(10) Therefore, considering the following expression: Pn(fn, θn 0, θn 1)≡A(fn)R0R(θn 0, θn 1)jId, RT(θn 0, θn 1)¯ t(θn 0, θn 1)−¯c0k(11) we can deduce that the rotation and focus of the camera after the motion are: Rn≡R0R(θn 0, θn 1) (12) ¯cn=−RT(θn 0, θn 1)¯ t(θn 0, θn 1)−¯c0(13) We have to take into account that any view acquired with the tripod can be considered as a reference to move it, and when we change the initial reference camera, we are also modifying the rotation axes of the tripod. In practise, in order to estimate the geometry of the tripod, we previously calibrate some isolated frames from the video sequence using standard calibration techniques, and then we estimate the geometry of the tripod using a standard bundle adjustment technique. 4 Primitive tracking by means of a decision tree At the primitive tracking stage, we use a calibration estimation and a CART decision tree. For the calibration estimation at frame n, we use the parameters (f, θ0, θ1) from frames n−1 and n−2, as explained above. A CART decision tree, as those described in [1], is used to detect the white primitives. To build the decision tree is necessary a learning stage based on a training set with information about different classes. For each video sequence, we read a classification data set, which contains information about two classes, primitives and background. Usually, in our soccer field scenarios, primitives are white and the background is green. In the data set, we have RGB values obtained from a manual segmentation of the first frame of the sequence. For the rest of the frames we perform the primitive tracking that is completely 5 explained in [14]. After the tracking stage, we proceed to the improvement of the calibration estimation. 5 Camera synchronisation We use OpenGL to create a 3D virtual world which will be mixed with the real world image. To be able to insert objects in the real image with the same perspective, we need to synchronise the virtual camera with the real camera. That is mean, we have to place the virtual camera at the same position of real camera and with the same rotation and zoom. The synchronisation is done by calculating virtual camera parameters from real camera parameters. The parameters which define a real camera are rotation, translation and clip plane, as we can see in Figure 2. The clip plane is defined by the focus, the centre and the intrinsic parameters of the camera. To perform the camera synchronisation, we have to configure the virtual camera with the real camera parameters. OpenGL has functions that implements this process, but needs some inputs which we have to calculate. These inputs are: camera centre, projection centre and a vector indicating the camera vertical axis direction (VUP). Moreover, we need to define the viewing volume. The viewing volume determines how a 3D object is projected onto a 2D image. For a perspective projection, the viewing volume is a frustum. Determining the frustum in OpenGL needs distances from projection centre to clipping planes (left,right,top,bottom) and distances from the camera to the near and far depth clipping planes, as shown in Figure 3. Figure 2: Real camera. Where Rnis rotation, ¯ tnis translation and C is image centre, the centre components are xcand ycfrom intrinsic parameters shown in expression 7 We use the euclidean camera calibration performed in the previous stage to obtain all the requested parameters. Firstly, we get the inverse projection matrix from the euclidean camera, being the projection matrix as is showed in 11. The inverse projection matrix is obtained as follows: P−1≡RT 0A−1(f0)jId, ¯c0k(14) Now, we can calculate the principal point multiplying this matrix by the image centre, Cv=P−1C, which belongs to the intrinsic parameters in 6 Figure 3: Virtual camera. Where VUP is the vector that indicates the camera vertical axis. width and height are the real image width and height. Cvis the principal point. The points top,left,right and bottom are the known points to define the clipping planes. the euclidean camera. Secondly, we have to define the frustum clipping planes as is shown in Figure 3. To calculate them, we obtain the distances from principal point to the sides of the near clip plane. We know that the dimensions of the near clip plane are the real images dimensions. Firstly, we obtain four points, one for each clipping plane. These points are top,right,left and bottom, as we can see in Figure 3. They are defined using the near clipping plane dimensions and coordinates of the image centre C= (xc, yc), where xcand ycare extracted from the intrinsic parameters, expression 7. Then, they are multiplied by the inverse projection matrix: top =P−1(xc, height −1,1,1)T(15) bottom =P−1(xc,0,1,1)T(16) right =P−1(width −1, yc,1,1)T(17) left =P−1(0, yc,1,1)T(18) Now we have to calculate the distances between the principal point and the points previously calculated to pass them to OpenGL as parameters to define the frustum. Finally, we obtain the VUP projecting the vector from principal point to up side of the near clip plane: V UP =P−1(xc,0,1,1)T−(2C−Cv) (19) As a result of the synchronisation, the virtual camera is able to obtain images with the same perspective of the real camera as we can see in Figure 4, which shows a virtual camera synchronised with a real camera. 7 Centro de Tecnolog´ıas de la Imagen Universidad de Las Palmas de Gran Canaria http://www.ctim.es 1