

ORIGINAL ARTICLE 

Year : 2017  Volume
: 3
 Issue : 1  Page : 1824 

Pointbased visuohaptic simulation of multiorgan for virtual surgery
Weixin Si, PhengAnn Heng
Department of Computer Science and Engineering, Chinese University of Hong Kong, Hong Kong, China
Date of Web Publication  19Jun2017 
Correspondence Address: Weixin Si Department of Computer Science and Engineering, Chinese University of Hong Kong, Rm903, SHB, CUHK, Shatin, N.T., Hong Kong China
Source of Support: None, Conflict of Interest: None  Check 
DOI: 10.4103/digm.digm_7_17
Background and Objectives: Realistically and efficiently simulating dynamic behavior of human organs under interactions is crucial for the immersive user experience of the surgical simulator. Conventional methods are timeconsuming to simulate this phenomenon due to topological modifications. Materials and Methods: This paper proposes a robust and efficient pointbased framework for surgical simulation, allowing realistically simulating mechanical response of human organs under interactions with visual and tactile feedback. Considering the inevitable topological modifications occurred in surgical simulation, we adopt sparse point cloud to model the mechanics of deformable bodies while employ surface mesh to represent morphological details of human organ, which can not only disconnect mechanical complexity from geometrical details, but also enable precise boundary conditions to be solved with surface mesh. Results: We validate our method on a variety of challenging surgical scenarios, and the results demonstrate that our method can realistically and efficiently provide the visuohaptic feedback for surgical simulation. Conclusions: Our method can well tackle the inefficiency limitation of meshbased methods related to topological modifications issue, and has great potential to be adopted in practical surgical simulators. Keywords: Frictional contact mechanics, hybrid geometric model, point.based method, virtual surgery, visuo.haptic feedback
How to cite this article: Si W, Heng PA. Pointbased visuohaptic simulation of multiorgan for virtual surgery. Digit Med 2017;3:1824 
Introduction   
Virtual realitybased surgical simulators can give the surgeon visual and haptic cues promising to be powerful aids for training medical personnel and monitoring their performance, which greatly improves patient care and reduces risks. The major concern of such simulators is to reproduce the biomechanical response of organs in the interactive environment.^{[1]} Biomechanical simulation with user interaction involves many challenges such as realtime computation of soft tissues deformation, collision detection, contact modeling, topological modification and haptic feedback (Payan ^{[2]} for a broad survey). To well handle the topological modifications, such as large deformation and cutting, this paper presents a meshless method based computational framework to achieve the fast and robust surgical simulation. Pointbased method, one of the most promising meshless methods, has been well developed in computer graphics.^{[3],[4],[5]} It uses the total Lagrangian formulation to calculate elastic forces with reference to the initial configuration, which makes it competitive compared to finite element method (FEM) due to its low computational cost, reasonable precision, and stability. However, the above advantages are not enough for the pointbased method being applied in surgical simulation; there are still some open topics for point based computational framework simulating the dynamic, complex, and unpredictable mechanical response of human organs in virtual surgery.
Considering the complex boundary conditions of deformable organ, the method should be able to handle frictional contact problem induced by tissuetissue and tooltissue interactions to obtain a realistic global behavior. Besides, haptic rendering is also required in such simulators since there is evidence that haptics interface can better aid the user in the learning process over simulation without force feedback.^{[6],[7]} We suggest readers knowing more haptic technology through the book,^{[8]} and recent reviews of haptics in surgery.^{[9],[10]} A common solution to deal with contact is to use the penalty method, however, the selection of penalty coefficient is problemdependent, and depends strongly on the stiffness ratio between the contacting objects. Consequently, the penalty method is limited for some applications, such as the interaction between deformable bodies. Moreover, Duriez et al.^{[11]} introduced Signorini model for contact handling in the field of haptic rendering. However, this method is currently only applied in meshbased methods. To solve above issues, a hybrid geometric model, including a highresolution triangular surface and internal points cloud, is proposed to adopt the highresolution triangular surface to resolve frictional contact problem precisely while not increase the computational complexity in solving mechanical response, allowing realtime visuohaptic feedback when human organ undergoes deformations.
Materials and Methods   
In this paper, we work on investigating a stepbystep approach to apply pointbased method in biomechanical simulation with complex boundary conditions, aiming at providing the robust, realtime, and realistic visuohaptic feedback.
Hybrid geometric model
Our framework makes use of a hybrid geometric model comprising both surface meshes and volumetric points cloud. The highresolution triangular surfaces are employed to represent the exterior structures of human organs in virtual surgery. Meanwhile, the interior structures of organs are constructed by arbitrary points clouds, which are also employed as physical model participating in dynamic simulation. The hybrid geometric model organization can be seen in [Figure 1]. This hybrid geometric model can make the mechanical resolution become independent from the rendering model, allowing to achieve realtime performance with certain mechanical resolution while preserve the detailed surface features of the object. The functional diagram of hybrid geometric model is illustrated in [Figure 2]. However, this leads to a dilemma; on the one hand, we resolve the frictional contact problem for mesh surface, rather than for mechanical model. On the other hand, the dynamic behavior of deformable bodies is solved on the internal point cloud, rather than the rendering model. Thus, we have to dynamically bridge these two geometries, allowing the displacement and force to be transferred in a light manner.{Figure 1}{Figure 2}
Here, we adopt the moving least square shape function ^{[12]} to connect the surface mesh and internal points cloud. In more details, each surface vertex can be represented by the internal points in the support domain. Suppose a particular surface vertex x_{i}, the support domain S of x_{i} contains n internal points, U_{s} is an nvector containing the value of the field variables at each node in the support domain U_{s}= (u_{1}, u_{2},…, u_{n})^{ T}, we have
Where Φ (x_{i}) is the shape function of x_{i} in the support domain.
Assuming the contact point λ is inside the triangle (Ψ_{1}, Ψ_{2}, Ψ_{3}), Ψ can be expressed as
Where is the barycentric coordinates for Ψ is the barycentric coordinates for Ψ.
Thus, we can easily transfer the contact forces on surface vertices to internal points
Where f_{s}= (f_{1}, f_{2},…, f_{n})^{ T}, and f_{c} is contact force, contact normal N is equal to the surface normal of triangle where contact point locates in. Besides, the displacement of internal points can also be reflected on surface vertices according to Equation 1.
Pointbased deformation
Following the work of Ller et al.,^{[3]} we approximate the deformation gradient F_{i} for each particle i by minimizing
Where the sum is taken over the neighbors j, and w_{ij} is a weighting kernel evaluated in rest space, u_{ij} is the distance in the reference state, while x_{ij} is the distance in the present state. We compute the minimizer by solving the following linear system,
Where we refer to the quantity as the particle i's basis matrix, A_{i}, which we invert to solve the system.
We describe the biomechanical property of human organ with NeoHookean hyperelastic model; here, stress tensor can be formulated as
Where μ, λ are the material parameters, J_{i} = det (F_{i}), I is the identity matrix, and C_{i} the right CauchyGreen deformation tensor .
After that, we can compute the symmetrized elastic forces
Where v_{i} is the volume of i. For more details, we suggest readers refer to.^{[5]}
In this paper, we model the dynamics of human organ using the following mathematical formulation:
Where t is the time step, M is a (constant) mass matrix mass, X are the current nodes' displacements. is the damping force, represented by the popular Rayleigh assumption:
, here ϑ and ξ are massproportional damping coefficient and stiffnessproportional damping coefficient, respectively. In addition, K (X) X represents elastic forces, and R is the current applied force field equivalently distributed to each node of the object's mesh.
Finally, it is worth noting that explicit time integration is used in this paper due to its efficiency. However, explicit time integration is only conditionally stable and requires an estimation of the maximum stable time step. Recently, Joldes et al.^{[13]} developed an effectively method of estimating the stable time step for element free Galerkin (EFG) methods and we find it is also suitable for our work.
Where, in which n is the number of influence points for integration, c is the dilatational wave speed.
Where ρ is the density of human organ.
Frictional contact mechanics
Tooltissue and tissuetissue interactions in the surgical simulation are frictional contact problems, which are rather complicated from both the theoretical and numerical points of view. They are characterized by a geometric and material discontinuity at the interface instead of the usual continuity property holding in solid mechanics. As a consequence, frictional contact problems are inherently nonlinear (even nonsmooth), involving variational inequalities and constrained minimizations. In this paper, we model frictional contact problems through Coulomb's friction law with Signorini's condition,
Where δ_{N} and δ_{T} are the normal and tangential gap in the contact space, and f_{N} and f_{T} are their corresponding forces. ς is the coefficient of kinetic friction.
Thanks to the hybrid geometric model, we can solve the frictional contact mechanics with surface mesh, and further transfer the boundary conditions for internal point cloud. We first solve the tissuetissue interaction with method proposed in.^{[11]} In addition, we resolve the tooltissue interaction with penalty method, where penalty force f_{c} in tooltissue interaction can be defined as,
Where δ is the interpenetration, and M, D, K, h are the mass, damping, stiffness matrix, and time step, respectively, which can be precomputed before simulation.
In addition, the refresh rate of visual feedback in surgical simulation range typically between 25 and 60 Hz (vision rendering rates), while haptic feedback should run at a faster rate (1000 Hz). However, the dynamic behavior of human organ with complex boundary conditions can be hardly simulated at haptic refresh rate, so that their simulation loop for visual and haptic rendering should be separated. Here, we introduce the multirate technique proposed in ^{[14]} to penalty method. In more detail, the dynamics of the human organ is computed in the simulation, while the feedback forces induced by tooltissue interaction are solved using penalty method, which is recomputed at a higher rate in the haptic loop based on an intermediate representation shared between two loops. The overall process of visuohaptic simulation is illustrated in [Figure 3].{Figure 3}
Results   
In this section, we present results of experiments demonstrating the performance of our method. A companion video is available illustrating the detailed experiment setup and results. The experimental platform is Intel (R) Xeon (R) CPU, 3.40 GHz, 4 GB memory, NVIDIA Quadro K600.
We produced a simulation with a liver (8597 mesh vertices and 10,450 internal points) and applied the NeoHookean material on the biomedical models. We recorded the force data of an instrument controlled through the Geomagic Touch X haptic interface. [Figure 4] illustrates the biomechanical deformation simulation of the liver model with force feedback, and the experimental results demonstrate the effects of our method in deformation simulation. We also simulate the tissuetissue interaction [Figure 5], which is a complex multiple frictional contact problem. The liver is contact with the stomach, and the interaction between will significantly affects the deformation of each other. The experimental results show our method can achieve continuous tissuetissue interactions. [Figure 6] illustrates the force feedback by our method, which can achieve a stable force feedback for the visuohaptic simulation. The dynamic procedure of visuohaptic simulation can be seen in the accompanied video.{Figure 4}{Figure 5}{Figure 6}
In addition, we demonstrate that our method is generic enough to address several kinds of simulations in a medical context. We use it to simulate the mechanical response of human organs in laparoscopy and thoracoscopy respectively involving haptic feedback with multicontacts and frictions. We show our method can fulfill the realtime requirement of such simulations and handle complex boundary conditions. The visual effects of visuohaptic medical simulation can be more obviously observed in the supplementary video.
Laparoscopic surgery
Laparoscopic surgery is a kind of minimally invasive surgery, which allows doctors using fine instruments to cut or trim tissues, perform biopsies, grasp organs, etc., inside the abdomen. Except for the specific operations, biomechanical response of target organ induced by complex tissuetissue interaction is important for the global behavior. We performed a simulation involving seven deformable organs (liver, spleen, pancreas, gallbladder, kidney, colon, and stomach) in contact with each other. Experiment results in [Figure 7] show that our method can simulate the visuohaptic feedback with physical fidelity.{Figure 7}
Videoassisted thoracoscopic surgery
Videoassisted thoracoscopic surgery is also a minimally invasive surgical technique used to diagnose and treat problems in your chest. Deformation, induced by tooltissue interaction, is vital for this virtual surgery. This is because doctors have to frequently operate the lung to find the legions, which is accompanied with the deformation. Here, we model two major organs in thoracoscopy, lung, and heart, through modeling the mechanical response under the interaction with medical device to demonstrate the effectiveness of our method [Figure 8].{Figure 8}
Discussion   
Note that FEM is the most commonly used in biomechanical simulation and the complex nonlinear behavior of human organ can be directly accounted for through constitutive relations and provide high biomechanical realism. However, the accuracy and robustness of the FEM rely heavily on the mesh that discretizes the geometry that severe limit its applications in surgical simulation. This is because complicated and irregular meshes are required to represent the morphological details of organs in complex surgical scenarios. Furthermore, topological modifications are often accompanied with tooltissue interactions, such as large deformation and cutting, the quality of elements could be deteriorated, leading to severely distorted elements.^{[15]} Further progress of simulation is meaningless or even impossible without taking care of those distorted or degenerated elements. After years of constant progress, meshing and remeshing remain difficult issues.^{[16],[17]}
With the latest developments, meshless methods ^{[18],[19],[20]} have been suggested as a possible alternative to solve the above difficulties. One advantage of meshless methods over FEM is that the domain of interest is totally discretized only by nodes, and hence, they are completely free from mesh dependence.^{[21]} They only utilize an unstructured cloud of nodes to discretize the geometry instead of elements, and the arrangement of these nodes is almost arbitrary. A further advantage of using a meshless method over FEM is their greater flexibility in constructing shape functions. In meshless methods, connectivity between nodes is not defined or loosely defined. Meshless methods are thus more attractive in dealing with geometric discontinuities, such as large deformation.^{[22]} Horton et al. compared the EFG methods using the total Lagrangian formulation with conventional FEM, and it was concluded that their methods were superior in the case of large deformations.^{[23],[24]} Dehghan et al.^{[25]} compared EFG method with conventional FEM imposing certain boundary conditions. The results show that their method can run faster than FEM since it does not require remeshing for large deformation. Besides, they also proved that for small deformations FEM and EFG can result in the same responses, while for large deformations EFG leads to a much smaller error compared with FEM.
In this regard, this paper employed a meshlessbased computational framework to achieve the visuohaptic interaction with human organ. Different from the FEM method, our method adopts sparse point cloud to model the mechanics of deformable bodies while employ surface mesh to represent morphological details of human organ, which can not only disconnect mechanical complexity from geometrical details, but also enable precise boundary conditions to be solved with surface mesh. In this way, our method can effectively release the limitation of the mesh dependency of FEM and take the advantages of the meshless method and simultaneously achieve the accurate continuous collision detection and realtime visuohaptic simulation.
Conclusions   
This paper has explored visuohaptic simulation of multiorgan interactions, which plays a core role in realistic surgical simulation. We propose a robust and efficient pointbased framework based on a hybrid geometric model, which is to adopt the pointbased model to simulate the mechanical response of deformable body, represent morphological details with a surface mesh and and achieve the bijective mapping between internal point cloud and exterior surface mesh with MLS shape function. The frictional contact mechanics is solved with the Coulomb's friction law with Signorini's condition and the visuohaptic feedback are controled by the multirate technique. The effectiveness of the proposed framework is demonstrated through a variety of challenging surgical scenarios. Our method can achieve efficient and realistic visuohaptic simulation with both tooltissue and tissuetissue interactions.
However, our present method cannot model the mechanical behavior of heterogeneous human organ, such as tumorized organ so that our immediate plan next is to model the heterogeneity of human organ. We are also interested in integrating the GPUbased preconditioner proposed in ^{[26]} for contact problems to our present implementation, to further optimize the time performance of our method. In the long run, we will adopt the continuous penalty force ^{[27]} to improve the robustness and stability of our method and extend this computational framework to handle more complex topological modifications, such as cutting.
Financial support and sponsorship
The work is supported by Hong Kong Research Grants Council under General Research Fund (Project No. CUHK 14202514, CUHK 14203115).
Conflicts of interest
There are no conflicts of interest.
References   
1.  Basdogan C, Sedef M, Harders M, Wesarg S. VRbased simulators for training in minimally invasive surgery. IEEE Comput Graph Appl 2007;27:5466. [ PUBMED] 
2.  Payan Y. Soft tissue biomechanical modeling for computer assisted surgery. Studies in Mechanobiology Tissue Engineering and Biomaterials. Berlin Heidelberg: Springer; 2012. 
3.  Ller M, Keiser R, Nealen A, Gross M. Point Based Animation of Elastic, Plastic and Melting Objects. ACM Siggraph/Eurographics Symposium on Computer Animation. Eurographics Association; 2004. p. 14151. 
4.  Gerszewski D, Bhattacharya H, Bargteil AW. A Pointbased Method for Animating Elastoplastic Solids. ACM Siggraph/Eurographics Symposium on Computer Animation, SCA; 2009. p. 1338. 
5.  Jones B, Ward S, Jallepalli A, Perenia J, Bargteil AW. Deformation embedding for pointbased elastoplastic simulation. ACM Trans Graph 2014;33:19. 
6.  Tholey G, Desai JP, Castellanos AE. Force feedback plays a significant role in minimally invasive surgery: Results and analysis. Ann Surg 2005;241:1029. [ PUBMED] 
7.  Panait L, Akkary E, Bell RL, Roberts KE, Dudrick SJ, Duffy AJ. The role of haptic feedback in laparoscopic simulation training. J Surg Res 2009;156:3126. [ PUBMED] 
8.  Lin MC, Otaduy M, Lin MC, Otaduy M. Haptic Rendering: Foundations, Algorithms and Applications. Boca Raton, Florida: CRC Press; 2008. 
9.  Westebringvan der Putten EP, Goossens RH, Jakimowicz JJ, Dankelman J. Haptics in minimally invasive surgery – A review. Minim Invasive Ther Allied Technol 2008;17:316. [ PUBMED] 
10.  Puangmali P, Althoefer K, Seneviratne LD, Murphy D. Stateoftheart in force and tactile sensing for minimally invasive surgery. IEEE Sens J 2008;8:37181. 
11.  Duriez C, Dubois F, Kheddar A, Andriot C. Realistic haptic rendering of interacting deformable objects in virtual environments. IEEE Trans Vis Comput Graph 2006;12:3647. 
12.  Lancaster P, Salkauskas K. Surfaces generated by moving least squares methods. Math Comput 1981;37:14158. 
13.  Joldes GR, Wittek A, Miller K. Stable time step estimates for meshfree particle methods. Int J Numer Method Eng 2012;91:4506. 
14.  Peterlik I, Nouicer M, Duriez C, Cotin S, Kheddar A. Constraintbased haptic rendering of multirate compliant mechanisms. IEEE Trans Haptics 2011;4:17587. 
15.  Luo Y. Dealing with extremely large deformation by nearestnodes FEM with algorithm for updating element connectivity. Int J Solids Struct 2008;45:507487. 
16.  Tournois J, Wormser C, Alliez P, Desbrun M. Interleaving Delaunay refinement and optimization for practical isotropic tetrahedron mesh generation. ACM Trans Graph 2009;28:75. 
17.  Aras R. Meshless elasticity model and contact mechanicsbased verification technique. MICCAI Computational Biomechanics for Medicine. Berlin Heidelberg: Springer; 2014. 
18.  Belytschko T, Krongauz Y, Organ D, Fleming M, Krysl P. Meshless methods: An overview and recent developments. Comput Methods Appl Mech Eng 1996;139:347. 
19.  Li S, Liu WK. Meshfree particle methods. Comput Mech 2000;25:99101. 
20.  Liu GR. Meshfree Methods: Moving Beyond the Finite Element Method, Second Edition. Boca Raton, Florida: CRC Press; 2009. 
21.  Liew KM, Ng TY, Wu YC. Meshfree method for large deformation analysis – A reproducing kernel particle approach. Eng Struct 2002;24:54351. 
22.  Luo Y. A nearestnodes finite element method with local multivariate Lagrange interpolation. Finite Elem Anal Des 2008;44:797803. 
23.  Horton A, Wittek A, Joldes G R, Miller K. A meshless total Lagrangian explicit dynamics algorithm for surgical simulation. Int J Numer Method Biomed Eng 2010;26:97798. 
24.  Zhang GY, Wittek A, Joldes GR, Jin X, Miller K. A threedimensional nonlinear meshfree algorithm for simulating mechanical responses of soft tissue. Eng Anal Bound Elem 2014;42:606. 
25.  Dehghan MR, Rahimi A, Talebi HA, Zareinejad M. A threedimensional large deformation model for soft tissue using meshless method. Int J Med Robot 2016;12:24153. 
26.  Courtecuisse H, Allard J, Kerfriden P, Bordas SP, Cotin S, Duriez C. Realtime simulation of contact and cutting of heterogeneous softtissues. Med Image Anal 2014;18:394410. 
27.  Tang M, Manocha D, Otaduy MA, Tong T. Continuous penalty forces. Allelopathy J 2012;31:19. 
[Figure 1], [Figure 2], [Figure 3], [Figure 4], [Figure 5], [Figure 6], [Figure 7], [Figure 8]
