IROS2019國際學(xué)術(shù)會議論文集 1992

A Handheld Master Controller for Robot Assisted Microsurgery Dandan Zhang Student Member IEEE Yao Guo Member IEEE Junhong Chen Jindong Liu Senior Member IEEE and Guang Zhong Yang Fellow IEEE Abstract Accurate master slave control is important for Robot Assisted Microsurgery RAMS This paper presents a handheld master controller for the operation and training of RAMS A 9 axis Inertial Measure Unit IMU and a micro camera are utilized to form the sensing system for the handheld controller A new hybrid marker pattern is designed to achieve reliable visual tracking which integrated QR codes Aruco markers and chessboard vertices Real time multi sensor fusion is implemented to further improve the tracking accuracy The proposed handheld controller has been verifi ed on an in house microsurgical robot to assess its usability and robustness User studies were conducted based on a trajectory following task which indicated that the proposed handheld controller had com parable performance with the Phantom Omni demonstrating its potential applications in microsurgical robot control and training I INTRODUCTION One of the main challenges faced by surgeons for micro surgery such as vitreo retinal and cochlear implant surgeries is the capability to perform complex tasks with a high level of dexterity accuracy and safety It has been shown that the perceptual accuracy of a naked human eye is within 10s of micrometers while the human hand can only reach a precision of about 100 micrometers under optimal conditions 1 This natural limitation makes the manipulating at micro scale diffi cult even for expert surgeons In the last decades RAMS is rapidly growing bringing signifi cant benefi ts to help realize the full potential of mi crosurgery in terms of dexterity accuracy and safety 2 3 The master slave paradigm has been extensively applied to the existing platforms with the surgeon manipulating the master controller to control slave manipulators Various ma nipulator designs for microsurgical tasks have been presented in 4 Among the existing systems grounded robotic masters for microsurgery have seen promising commercial opportunities and clinical success due to their appreciated advantages of stability and reliability Generally surgeons hands are tracked by wire and pulleys or mechanical linkages to gener ate commands for remote control of a slave robot However these available systems usually have a large footprint high cost and long setup time thus preventing their widespread application With the advantages of being compact and lightweight handheld master controllers are attractive alternatives for This work is supported by the Hamlyn Centre All authors are with the Hamlyn Centre for Robotic Surgery Imperial College London Lon don United Kingdom G Z Yang is also with the Institute of Medical Robotics Shanghai Jiao Tong University China Corresponding author email d zhang17 imperial ac uk their grounded counterparts 5 Without mechanical link ages the physical footprint can be signifi cantly reduced Besides intuitive ungrounded design can assist the surgeons to manipulate the slave robot through teleoperation in a more natural way without motion constraints Therefore it is benefi cial for reducing extensive surgical training before the adoption of robotic microsurgery 6 More importantly for clinical translation handheld master controllers can be executed as a user interface at the patient s bedside with out introducing a physical distance between surgeons and patients 7 which enhances the potential safety benefi ts For handheld controllers capturing the surgeons motion in a precise manner plays a critical role in generating adequate commands for microsurgical robot remote control In the past decades various sensing technologies have been explored to track surgeons hand movements Acoustic sys tems utilize the time of fl ight and intensity of audio signals to affi rm the marker locations while electromagnetic systems employ the magnetic fi eld intensity 8 These two modalities are convenient to use but are sensitive to noises and external disturbances which reduces their reliability and hamper their potential applications for RAMS Inertial Measurement Unit IMU is now common for motion tracking 9 The advantages of IMUs include high sampling rate and being invariant to the environment 9 though sensor drift and accumulative errors are their inherent limitations With recent advances in computer vision depth cameras can provide promising tracking of hands and fi ngers 10 Existing products include a hand tracking system namely KinectTM based 3Gear However these systems incorporat ing depth sensors are typical with large size and need to be carefully placed in a specifi c location which limits their fl exibility 11 Optitrack and Vicon are two commercial products for motion tracking However they are not suit able for clinical applications since the resolutions of both systems may not be high enough to track the gestures for microsurgical operation On the other hand various optical markers such as chess board vertices 12 QR code and Aruco marker 13 are developed to achieve high precision tracking for the moving camera However the existing marker patterns have their limitations while being utilized in the handheld controller for RAMS Comprehensive comparisons of motion tracking systems based on different modalities were summarized in 14 Con sidering that different modalities have pros and cons a hybrid tracking method is benefi cial for combining the advantages 2019 IEEE RSJ International Conference on Intelligent Robots and Systems IROS Macau China November 4 8 2019 978 1 7281 4003 2 19 31 00 2019 IEEE394 of different methods as well as compensating their drawbacks 15 Inertial sensors may suffer from drift when estimating position but they can deal with fast motion acquisition In spite of the low sampling rate of the visual sensors they have precise tracking results of position with well designed markers Moreover the visual cues are necessary to deal with the drift problems of the IMU Therefore hybrid motion tracking methods can have many advantages In 16 inertial data and 2D locations of optical markers in the stereo camera images were fused to track a handheld device However the complexity of the system setup may limit its adoption The optical tracking system is a lab based design which is diffi cult to be fi tted in a general clinical operation room Occlusion may limit the consistency of the optical tracking and thus hamper the safety for microsurgical operation Using an on board camera is more effective and practical than placing extra tracking devices such as a depth camera or a stereo camera at a specifi c place The additional setup procedures are not necessary while the infl uence originated from the illumination effects can be reduced Therefore an on board micro camera is considered which ensures the system is easy to use In this paper we develop a novel handheld master con troller for RAMS which consists of a 9 axis IMU and a micro stereo camera The rest of the paper is organized as follows Section II introduces the proposed hybrid marker pattern design and a multi sensor fusion framework for the motion tracking system while the evaluation of the motion tracking results is performed Section III presents the system integration of the microsurgical robot and the handheld controller User studies are conducted in Section IV Conclusions are drawn in Section V II METHODOLOGY In this section the hardware of the proposed handheld controller will be briefl y introduced A novel hybrid marker pattern is designed to realize precise marker based motion tracking for the handheld controller with the illustration of sensor fusion algorithm A Handheld Master Controller The handheld controller incorporates a miniature stereo camera Precision Robotics UK and a 9 axis IMU sensor Xsens Netherland The overview of the handheld controller can be found in Fig 1 After fusing the pose estimation infor mation from the micro camera and the orientation estimation results provided by the IMU the full 6 DoF tracking can be obtained B Pose Estimation from IMU A 9 axis IMU combines 3 axis magnetometer with in ertial sensors Acceleration information can be measured by the 3 axis accelerometer while angular rate information is obtained by the 3 axis gyroscope based on the Coriolis acceleration effect 17 The 3 axis magnetometer measures the magnetic fi eld strength in a given direction allowing the 9 Axis IMU Micro Cameras Fig 1 Details of the handheld controller north direction to be found In this way the yaw information can be gained 3 axis acceleration and 3 axis magnetic fi eld messages are used to estimate the static orientation The accelerometer measurements can be directly computed into Euler angles using trigonometry while the magnetometer coupled with the accelerometer can effectively calculate the heading angle by measuring the magnetic fi eld Angular velocity is the rate of change of angular displacement as measured by the 3 axis gyroscope Since the gyroscope sensor outputs angular rates the measurement can be simply integrated over a period of time the sampling rate and summed up to obtain the absolute angle In this way the dynamic update of the orientation estimation can be achieved XKF3i is a frequently used fusion algorithm for the 9 axis IMU which fuses the Information from inertial and magnetic sensors and combines their advantages to obtain accurate orientation estimation 18 Although the position value can be obtained by the double integral of the acceleration data provided by the IMU the accuracy of the position estimate of inertial sensors degrades with time due to the random errors C Hybrid QR Aruco Chessboard Pattern Design Marker based visual tracking method is much more ac curate and reliable than the markless ones Considering that safety and accuracy are fundamental for microsurgery marker based visual tracking is utilized A hybrid QR Aruco Chessboard QAC pattern is de signed as a computer vision friendly 2D pattern that has enough salient features for 6 DoFs pose estimation see Fig 2 which combines the advantages of QR codes 19 20 Aruco markers 13 and chessboard vertices 12 together Three tracking modules are used to provide real time pose estimation information concurrently while the information is merged along with the orientation estimation results to provide robust 6D pose estimation 395 Hybrid QR Chessboard QR Codes Chessboard ChAruco Aruco 8 Hybrid QR ChAruco Pattern Chessboard Fig 2 A hybrid QR Aruco Chessboard QAC pattern for marker based visual tracking D Pose Estimation from Chessboard In order to accelerate the speed of real time pose estima tion the chessboard is limited to a 3 3 non symmetrical pattern Hence the chessboard vertices can be detected quickly For each pixel in the fi ltered image a ring of 11 pixels around the pixel is sampled at a constant radius with equal angular spacing Given the tracked projections on an image of a cali brated camera and the corresponding spatial positions in the marker s local coordinate frame the pose estimation based on 2D 3D correspondences can be realized by a Perspective n Point PnP method 21 22 However when the camera is closed to the pattern the number of tracked chessboard vertices will not be enough to realize pose estimation via PnP method E Pose Estimation from Multi Aruco Markers Aruco markers 23 and their derivatives are frequently used in augmented reality and robotics Eight different Aruco markers with ID range from 1 to 8 are used to provide track ing information This can ensure a wider range of tracking Therefore a multi Aruco module is utilized to compensate for the limitation of the chessboard vertices based tracking when the micro camera approaches the hybrid QAC pattern However the precision of the multi Aruco tracking mod ule will degrade when the micro camera is far from the plane of the markers F Pose Estimation from chAruco Pattern To achieve higher tracking accuracy chAruco pattern can be generated by merging Aruco markers and the chessboard information as part of the hybrid QAC pattern The pose esti mation accuracy from the chAruco is higher than using only chessboard information or pure Aruco markers However the successful estimation depends on whether more than fi ve of the Aruco markers are within the view of the camera Therefore the ChAruco only plays an important role in providing accurate tracking information while enough markers are within the camera s view G Pose Estimation from QR Code Another pose information can be obtained based on the QR code detection where the pose estimation process is achieved from the location of the four QR code corners The 2D coordinates are extracted from the image The transform relationship between the 2D image and the pose in the 3D space can be obtained thanks to camera intrinsic parameters Eight QR codes are merged to the chessboard to form the fi nal hybrid QAC pattern as shown in Fig 2 One advantage of the QR code is that the tracking preci sion is high due to more feature information is available than the Aruco markers In practice due to the varying light conditions or fast movement of the handheld controller failures of QR codes detection may occur The successful rate of QR codes detection will decrease when the distance between the camera and the hybrid QR Chessboard pattern is too large H Multi Sensor Fusion Framework Construction The calibration process using Zhang s calibration method 24 is conducted fi rst to determine the intrinsic camera parameters and lens distortion coeffi cients Gaussian fi lter is applied to the greyscale image to remove speckles and noises before sending the real time images for pose estimation The visual tracking framework is constructed for pose es timation via three major modules i e i multi Aruco module ii chAruco module iii QR codes module Considering that the position estimation from IMU data suffers from drift problem while the orientation estimation from single Aruco is unstable the 3D orientation information from the IMU and the 3D position information from the multi Aruco module are combined to form a stable 6D pose vector As for the chAruco module and the QR code module the 6D pose information can be fully utilized Fig 3 shows the available micro camera view for online visual data process ing A status monitor is utilized to select the suitable infor mation for multi sensor fusion Visual tracking state vector V t a c q indicates the status of the tracking situation of the multi Aruco chAruco and QR code at time step t If the value of q c a is equal to one the tracking result of the corresponding mode is reliable Suppose that there are 396 Multi Aruco ModuleQR code ModulechAruco Module Fig 3 Camera tracking results of the three modules in the micro camera view k Aruco markers in total V j a t j 1 2 k identifi es whether the Aruco marker with the ID value of jthis within the view of the camera or not Qq t Qc t represent the pose estimation results from the QR code module and the chAruco module respectively at time step t P t q c a represents the position while O t q c a represent the orientation Pa t Pj k j 1P j a t V j a t Pj k j 1V j a t 1 where Pj a t j 1 2 k represents the position estima tion result based on the Multi Aruco tracking with ID value of j Suppose that Q t P t O t is the fi nal pose estima tion result calculated by the sensor fusion framework at time step t Position P t can be estimated as follows P t q t Pq t c t Pc t a t Pa t q t c t a t 2 Oi t represents the orientation estimation result based on IMU The visual tracking framework is applied to every image in the sequence by which the motion tracking of the handheld controller can be realized Orientation O t can be estimated as follows O t q t Oq t c t Oc t Oi t q t c t 1 3 The pose increment value Q t will be obtained and scaled down which will be sent to the microsurgical robot as command to realize pose incremental control The algorithm of the multi sensor fusion is shown as Algorithm 1 I Tracking Performance Analysis Reliable visual tracking is fundamental for the usability of the handheld controller The detection rate and the motion tracking accuracy are utilized to analyze the performance of the motion tracking technique The tracking status of the three modules can be monitored simultaneously Suppose that N frames were obtained online niis the number of frames that the marker detection can suc cessfully provide pose estimation information The detection rate can be defi ned as ni N Different visual tracking modules have different detection ranges depends on their inherent properties Algorithm 1 Multi Sensor Fusion Input Qq t Qc t Oi t Q t 1 V t q t c t a t V j a t V j a t 1 j 1 2 8 Output Pose Increment Value Q t 1 Use Extended Kalman Filter EKF to fi lter the original data 2 Estimate Pa t based on Equation 1 3 Estimate P t based on Equation 2 4 Estimate O t based on Equation 3 5 Obtain the pose vector Q t P t O t 6 Use EKF to fi lter Q 7 Obtain the pose increment value Q t Q t Q t 1 Orientation rad Position m Time s Time s Handheld Master Ground Truth Orientation ComparisonPosition Comparison Fig 4 Orientation and position tracking results comparison between the ground truth data and the handheld master controller The experiments were divided into three groups based on different distances from the pattern to the camera near 20 100 mm middle 100 180 mm and far 180 260 mm For each group we repeated three trials and recorded the data of the tracking results The quantitative results as shown in Table I confi rms that the tracking system for the proposed handheld master is superior to the single tracking module by achieving nearly 100 detection rates in various range TABLE I DETECTIONRATE OFDIFFERENTMODES NearMiddleFar 20 100mm 100 180mm 180 260mm Chessboard0 70 96 Multi Aruco100 100 97 QR Codes98 82 76 chAruco0 80 98 Hybrid Marker100 100 100 The handheld controller c