Automatic guidance of and assistant robot in laparoscopic surgery

Proceedings of the 1996 IEEE International Conference on Robotics and Automation Minneapolis, Minnesota - April 1996 Automatic Guidance of an Assistant Robot in1 Laparoscopic Surgery A. Casals and J. Amat Dep. of Automatic Control. Universitat Polit6cnica de Catalunya Pau Gargallo, 5 08028 Barcelona (SPAIN) phone: 34 3 4016974; fax:34 3 4017045 e-mail:[email protected] Abstract A robotic arm automatically guidedto handle the camera in laparoscopic surgery is presented. The goal of this work is to generate the adequate camera control strategies to track the working scene during a surgical procedure. The system is based on the computer vision analysis of the laparoscopic image that allows to identi& either a scene's relevant point, or the surgical instruments. 1 Introduction In the last years, significant research efforts have been done in the development of technology for minimally invasive surgery [I], and more specifically in laparoscopy [2, 31. The extended use of laparoscopic techniques has demonstrated that there are still important limitations in its practical application. Laparoscopic surgery requires the introduction into the patient abdomen of several instruments, the surgical tools, as well as the TV camera microoptics, the laparoscope, to provide the surgeon with the visualisation of the intraabdominal working space. A good centring of the camera field of view onto the interest zone is essential to operate efficiently. The movement of the camera probe to adequately track the interest zone can be done by the surgeon himself or by an assistant. In the latter case, a certain experience and a surgeon-assistant mutual understanding is necessary, although usually difficult to obtain. In the former case, the difficulty arises from the need for the surgeon to move the camera support while working. Consequently, some efforts have been done E. Laporte Centre Quirurgic Adria Vallmajd 34-36 0802 1 Barcelona (SPAIN) phone: 34 3 20161 11; fax: 34 3 4146128 towards the automation of these movements under the surgeon supervision. Some robotic manipulators under human control have been developed and tested. In [4] a six degrees of freedom manipulator controlled by the surgeon by means of a foot pedal is described. Other manipulators have already been designed and tested using different interfaces to receive the surgeon orders, either through oral commands or by means of head movements [2, 51 or teleoperated [6]. Additionally, a feasibility study in the use of image processing and other human-machine telerobotic command interfaces have been published [7]. This paper describes an experimental robotic system designed to automatically guide, by means of computer vision, the laparoscope during a surgical procedure. In this first experimentation phase a conventional industrial robot, Scara type, has been used, but in the next future we will develop a specific robot so as to reduce the system cost as well as to minimise the occupied space within the operating theatre. Further considering the robot space reduction and with the aim to release some more free space for the surgical team, the robot that holds the laparoscope has been moved away from the patient, using a link that transmits the robot end effector movements. 2 System Structure The experimental system consists of a camera probe held by a robot., the laparoscope, the vision system for scene analysis and the control system to generate the robot trajectories. The system configuration is shown in figure 1. The carnera movements are controlled from the 0-7803-2988-4/96 $4.00 0 1996 IEEE 895 error signals detected by the image processing system in the centring of the scene, according to the operation mode desired by the surgeon. The laparoscope movements are restricted to four degrees of freedom due to the constrains imposed by the operation through the trocar, T. This behaviour corresponds to that of a passive cylindrical joint located in a fixed point. Due to the elasticity of the patient's abdominal walls, the above mentioned cylindrical joint has additionally two passive degrees of freedom, thus providing the required four d.0.f. The resulting four degrees of freedom are three rotations cp, w and 0 and the prismatic movement p, as shown in figure 2. Behaviour parameters PFig. 2 Laparoscope degrees of freedom Although the laparoscope rotation 0 can be useful for a better centring of the image, it produces a rotation of the visualised image on the screen with respect to the surgeon reference working frame. This change demands a mental effort from the surgeon to keep his or her operating ability. For this reason many surgeons, among them those of the research team working in this project, impose the avoidance of this rotation. This constrain produces certain difficulties in the manual guidance of the laparoscope, but on the other way, simplifies mechanically the link between the robot and the laparoscope in the automatic mode. The robot thus requires only three degrees of freedom. 3 The Vision System The vision system developed is fruit of the works carried out in the department in the last years on real time image processing and tracking [8]. In this application the basic image processing requirements are the identification and the 3D tracking of the surgeon tools. The visualisation techniques currently utilised are either the classical 2D monitoring systems, or the 3D ones based on the use of laparoscopes with sets of double Hopkin lenses and polarised eyeglasses. The automatic guidance system developed can be applied not only to 3D monitoring systems, but also to 2D ones, provided that besides the identification and tracking of the surgeon tools, the 3D information is infered from the apparent tool size. This vision system is based on the extraction of a very elemental set of features obtained from the visible surgical tools, but effective enough to assure the minimum robustness required. This set of features must be invariant with respect to the tools position and orientation in the image, as well as to zooming. With this aim, the image characteristics selected have been the lines considered as straight. The image contour lines having a length greater than 8 pixels and aligned with a dispersion bounded to +1 pixel are detected as straight lines. To assure the visualisation of at least one straight line corresponding to a tool, in conditions of possible low contrast, the tools have been stamped with specific marks. In a first version, these marks consisted of some horizontal strips [9], that afterwards have been changed to two straight lines along the tool axis. In this way, at least two edges can be detected and from them the tool location and orientation can be determined with much more precision over the image plane. Additionally, a ring mark, located at the tool rod end (fig. 3), is utilised for the calculation of the tool 3D position from the measure of its diameter. The processing algorithm developed for the computer vision system consists of: Edges extraction. Since the images to be processed can present low contrast and the use of edge images facilitates the visualisation and location of the surgical tools, the processing algorithm operates from edges images. The implementation of this operation using the same computing facilities as the rest of the processing, adds a new processing time that increases the sampling period and consequently worsens the closed loop control response. For that reason we use a dedicated processor to implement this operation in real time. In this processing step, a filter using a 4 x 4 pixels matrix is also implemented. This filter 896 eliminates an appreciable amount of irrelevant information, thus shortening the posterior steps processing times. Straight lines extraction. The next processing step, straight lines detection, allows to extract some image features to detect the tool or tools appearing on the scene. The implementation of this processing has been done using an industrial PC for a good efficiencylcost ratio, as well as good reliability. The straight lines are detected as alignements of pixels using window operators of up to 8 pixels size. The results are quite satisfactory having a processing time less than 100 ms. Tool detection. Once the non aligned contours have been filtered, those that verify some defined conditions are considered as pertaining to tools. The algorithm to detect tools searches pairs of straight lines, candidates of being considered tools, that coming from the scene perimeter converge, within the limits of the perspective distorsion derived from the utilised optics, towards the center of the image. In order to ensure the system robustness, only the detections carried on close to the previous detection tool position are considered. Otherwise, the system validates the pairs of lines as tools, only after the verification of more severe geometric conditions for a given number of successive detections. This verification must be done after a sudden tool movement, when the first proximity condition does not apply. Measurement of the tool tip position, its size and its orientation. The tool tip position, orientation and size, allows to determine the 3D position of the image interest center point. The third dimension, when using only one camera, is calculated from the measure of the tool diameter, which is measured from a cylindrical mark, invariant to rotation, at the laparoscope tool handle. The obtained position and orientation is used both, to guide the robot trajectory and also to feedback this data to the detection module, to increase robustness. e From the tooll position obtained approximately every 200 ms it is possible to track the tools and to generate the adequate control orders with a sampling time short enough to guaranty the tracking of the scene zone of interest with enough reliability. 4 Control Strategy One of the main problems to solve, to adequately guide the robot during an intervention, is to establish the control strategy that dynamically determines the laparoscope position which is really efficient to the surgeon at every instant. Essentially, there are two problems to solve to adequately control the camera point of view. First, to determine the laparoscope x, y position in the image and second its distance (zoom). 4.1 Laparoscope Position Control Criteria After the observation of the movements that the assistant uses to perform under the surgeon requirements and for the different situations appearing in a diverse set of surgical interventions, we have established a series of rules. The significant situations considered are: a) existence over the scene of two moving surgical tools b) existence of a moving tool and another auxiliary tool relatively steady c) existence of a unique tool In the above three cases the centring of the scene that results more efficient to the surgeon has been shown to obey the same common criteria. These criteria have been used to define the system behaviour. a) tracking of a scene with two tools When the suirgeon operates with two tools over the working scene, the vision system detects and tracks them by means of the detection of the superimposed marks. In this case the scene centre point is considered to be the central point of t!he segment defined by the two tools end point. A filtering algorithm is required to stabilise the image, instead of tracking instantaneously the movements derived from the surgeon work that determine the scene zone of interest. b) tracking of a scene containing a working tool and an auxiliary steady one When the two marked tools appearing on the scene do not have the same mobility, but one of them moves meaningfully more than the other, the previous criterion does not apply. In this case, the image centre zone is also Fig. 3 Surgical marked tools 897 a point located in the segment defined by the two tools end point, but this point is closer to the tool that moves faster. The distance considered to this tool has been 113 of the defined segment. c) Tracking of a scene containing only one working tool When the observation area contains only one marked tool the point selected as image centre is not exactly the tool tip, but it is shifted a step forward. 4. 2 Criterion for the Selection of the Adequate Zoom Factor Besides the centring over the most interesting zone, an additional parameter that defines a good image of the working scene is the optical aperture angle; the zoom. The most adequate aperture angle varies widely depending on the habit and preferences of the surgeon. For that reason, the criterion taken to automate its control is to modify the zoom, so as to ensure that the tool does not disappear from the camera field but respecting the surgeon requirements. Thus, the increasing or decreasing of the zoom is done automatically from an initial zoom position given manually and taken as an order parameter. From this parameter, the final zoom value is the ratio between the visible tool or tools and the size of the whole scene. This ratio is taken at the intervention initial instant or at any moment in which it can be manually redefined. Either the position error function &,,(t) or the zoom E, (t) values must be filtered to generate the robot arm orders for the laparoscope guidance. 4.3 Filtering of the Error Function The tracking of the tool or tools end effector, allows to calculate the error functions E (t) corresponding to the difference between the 3D measured position of the image center point and the desired reference position. This last position is defined from the criteria defined in points 4. 1 and 4.2. This error function could generate directly the orders to the camera guidance robot, but the obtained movements are continuous and frequently too sudden. This option would cause tiredness to the surgeon. He preffers some momentary displacements of the attention center point in exchange for a greater stability and softness of movements. The implemented filtering, that has provided a good compromise efficiencylsoftness acceptable by the surgeon criterion, has been a nonlinear type with dead zone and hysteresis. The filter behaviour is as follows. After the dead zone is surpassed the system tends to cancel, following a proportional law, the mean error measured, only in case the instantaneous value does not overcomes some defined boundaries. Otherwise, the filtering is substituted by a tracking, in order not to loss the tracked tool or tools. This behaviour provides a sufficient stabilisation when, during the task execution the tool moves relatively slightly within the scene. On the other hand, the filter produces a quick enough dynamic response , and without too big a delay, when the tool suddenly changes the working point. Figure 4 shows a sequence of the this error E (t) refered to one axis, during the execution of a task. There, we can observe that the errors do not return suddenly to 0, but they fluctuate around this value but with a movement softer that that of the tool. E 1 i + ._ __._. - - .. -~ - -. . . r:, sec Fig. 4 Evolution of the laparoscope tip position during an intervention, a) measured, h) filtered - .. 1,: 5. The Robotic Handling System The laparoscope movements must be performed through the trocar, resulting in the cp, y and p, movements and maintaining invariant the rotation 8, over the laparoscope axis. These movements must provide the required and wAZ displacements of the optics (the laparoscope tip), with respect to the scene coordinates system (world frame). To achieve the adequate generation of these movements it has been believed more appropiate to place the robot not just holding the laparoscope handle, but to remove the arm from the working zone. A stiff link, that produces only a translation effect on the robot end effector reference frame, has been instaled in the robot wrist. This robot configuration lets more free space to the surgeon and his team, a real need in the surgical theater. To simplify the generation of the robot control orders, the strategy to follow could be to provide the robot control unit with the increments ‘AX, RAY and RAZ tied to the robot Cartesian frame. These movements of the robot end - effector are transmitted to the laparoscope through the added link that is rigidly fasten to the robot tip. The junction between this link and the laparoscope is a universal joint R. The three degrees of freedom of this joint allows the free movement of the laparoscope at this 898 point to fit the constrains imposed by the abdominal enter point, the trocar. With this configuration, the incremental movements ‘AX, ‘AY and ‘AZ produce at the laparoscope tip, the optics point of view, a transformed working space AV, Ay and Ap (fig. 5). Consequently, an added control difficulty due to this kynematic chain is this workspace transformation. Therefore, when the vision system detects an error position in one axis direction, for instance “AX, the vision control closed loop, using the above mentioned transformation, would produce an incremental movement of the robot wrist, ‘AX. In this case, the laparoscope tip, due to the trocar constrains would move a trajectory determined by two coupled movements‘Acp and “Ap, thus modifying bbth its position over the 2D image and also the zoom. Since the system operates in closed loop, this error would be corrected by the succesive movements, but producing an undesirable transitory. To avoid this resulting jerking it is required to perform a coordinates frame transform, in such a way that a “AX position error measured by the vision system, produces a movement RAX, “AY and ‘AZ so as to obtain a displacement of the laparoscope tip of only one degree of freedoinWAX. This Cartesian laparoscope movement (fig. 6), corresponds to the trajectory of the laparoscope end-effector, but not to its orientation, which is forced due to the trocar constrains. The control of this orientation would require the addition of two additional degrees of freedom in the laparoscope tip, as proposed in [IO]. The calculation of this frame transform, to control the robot with decoupled movements, is possible, but one of the variables, the position of the trocar with respect to the robot frame, is not fixed during an intervention. On the other hand, the robot extension link, due to its ligthness is not completely rigid and produces a flexion that results in a translation and a rotation of the reference frame R’ (fig. 7) that is also difficult to measure. For those reasons, the calculations of the different frames, required to minimize the effects of the coupled movements, and the change of the gain of the different joint movements, in accordance with the laparoscope tip position within the scene, are based on some parameters which are dinamically evaluated from the result of the control actions measured by system itself. 6 Safety In surgery the patient safety is an essential the vision issue, and different safety aspects have to be considered. In a first group we have considered all the safety aspects related to the required electrical protections. The lack of an efficient connection of an instrument to ground could produce a ROBOT ARM R -I Fig. 5 Gyect of a robot Cartesian movement on the laparoscope tip position ROBOT ARM Fig. 6 Robot movement necessary to produce a Cartesian translation of the laparoscope tip ROBOT ARM Fig. 7 Translation and rotation of the R‘ frame due to the linkflexibility Faraday effect with respect to other electrical instruments used. On the other hand it is necessary to verify the efficient protection of the equipment against EMI, specially when using electrical instruments. A second aspect considered has been that related to the possible physical damages, produced by undesirable movements. To !solve this safety problem two different kind of complementary protections have been taken: a passive mechanical protection and an active one. 899 The passive protection consists on the use of two screws to hold the ball-and socket joint which have been previously conveniently weakened to force them to break in case of a laparoscope collision with a patient organ during the surgical procedure. This screw weakening has been attained using nylon screws with their threads internaly drilled, so as to produce their breaking when they support a force higher than 350 grm. This effort value surpasses the laparoscope - trocar friction forces without producing damages to the patient. On the other hand, and with the aim to avoid this kind of breaking during an intervention, that would cause an interruption of the surgical procedure in order to replace the laparoscope holding device, an additional active protection system is used. This protection is based on the use of torque sensors, gauges, placed on the robot - laparoscope junctions link. The detection of overefforts generates protective reactive movements adressed to the guaranty the patient integrity. Furthermore, and trying to avoid undesirable reactive realising movements of the laparoscope, it is necessary to assure a clear disposition of the cables, video and fiber optics, leaving the laparoscope that can produce overefforts due to friction or torques that could generate undesirable control reactions. 7 Results The development of this project has followed two successive phases. In a first phase we have developed and set up the vision system which identifies and tracks with high reliability the surgeon tools. In this phase a specific image processing hardware has been developed to obtain the edges and to reach a sampling time short enough to implement a not syncopated control of the robotic arm. After the development of the visual tracking system and closing the loop with the robot, the next step has been to establish the control criteria according to the surgeons needs and preferences. These preferences are not always exactly the same for all the surgeons, this fact has forced us to design the system so as to be easily adaptable to every one of them. This personalisation consists basically on the availability to select the desired scene size (zoom) and the speed - stabilisation factor in the tracking of the scene interest point. To establish these parameters some experimentation has been done working with chickens until the system behaviour has been considered as very satisfactory by the surgeons of our working team. From this results we have started to work in the experimental installation of the surgical theatre in the Centre Quirurgic Adria to start in brief with clinical trials. Acknowledges We would like to acknowledge to Xavier Valls for his contribution in the implementation and set up of the system. This work is funded by the CIRIT, the Catalan Research Agency. References H. H. Rinisland, "Basics of Robotics and Manipulators in Endoscopic Surgery" Endoscopic Surgery and Allied Technologies, Vol. 1, n. 3 June A. Casals, J. Amat, E. Laporte, "Robotics Aids for Laparoscopic Surgery Problems" The 7th International Symposium on Robotics Research, Munich, Germany October 2 1-24 1995 (in press) M. H Omstein, "Robotics in Laparoscopy and Thorascopy", presented at the 1 st.. European Conference on Medical Robotics, Robomed'94 Barcelona, Spain. June 20-22, 1994. pp. 17-21. N. I. Narwell, D. R Weker and Y. Wang A "Force Controllable Macro-Micro Manipulator and its Application to medical robotics". JPL Computer Motion Inc. report 1994 P. A. Finlay, "Small is Beautiful". 1st. European Conference on Medical Robotics, Robomed'94, Barcelona June 20-22, 1994 Spain. pp,. 13-15. D. Hurteau et al. "Laparoscopic Surgery Assisted by a Robotic Cameraman: Concepts and experimental results, IEEE Intemational Conference on Robotics and Automation, 1994. pp. 2286-2289. "An overview of Computer-Assisted surgery at IBM. G. Robotics Research: IFRR, 1994. pp. 533544 J. Amat, J Aranda and A. Casals "Tracking capabilities in Computer Vision" Optical Engineering, vol. 32 n 11, November 1993 pp. A. Casals, J. Amat and E. Laporte "Vision Guided Robotic System for Laparoscopic Surgery" 7th International Conference on Advanced Robotics, ICAR95 Sant Feliu de Guixols, Spain. September 1993 pp. 154-159 2796-2804 20-22, 1995. pp 33-36 [ 101 R. D. Scraft and M. Wapler "Manipulators Make Minimally Invasive Surgery Easier" 7th International Conference on Advanced Robotics, ICAR95 Sant Feliu de Guixols, Spain. September 20-22, 1995. pp 37-42 900