Point-based visuo-haptic simulation of multi-organ for virtual surgery
Weixin Si, Pheng-Ann Heng
Department of Computer Science and Engineering, Chinese University of Hong Kong, Hong Kong, China
Date of Web Publication
19-Jun-2017
Correspondence Address: Weixin Si Department of Computer Science and Engineering, Chinese University of Hong Kong, Rm903, SHB, CUHK, Shatin, N.T., Hong Kong China
Source of Support: None, Conflict of Interest: None
Check
DOI: 10.4103/digm.digm_7_17
Abstract
Background and Objectives: Realistically and efficiently simulating dynamic behavior of human organs under interactions is crucial for the immersive user experience of the surgical simulator. Conventional methods are time-consuming to simulate this phenomenon due to topological modifications. Materials and Methods: This paper proposes a robust and efficient point-based framework for surgical simulation, allowing realistically simulating mechanical response of human organs under interactions with visual and tactile feedback. Considering the inevitable topological modifications occurred in surgical simulation, we adopt sparse point cloud to model the mechanics of deformable bodies while employ surface mesh to represent morphological details of human organ, which can not only disconnect mechanical complexity from geometrical details, but also enable precise boundary conditions to be solved with surface mesh. Results: We validate our method on a variety of challenging surgical scenarios, and the results demonstrate that our method can realistically and efficiently provide the visuo-haptic feedback for surgical simulation. Conclusions: Our method can well tackle the inefficiency limitation of mesh-based methods related to topological modifications issue, and has great potential to be adopted in practical surgical simulators.
How to cite this article: Si W, Heng PA. Point-based visuo-haptic simulation of multi-organ for virtual surgery. Digit Med 2017;3:18-24
How to cite this URL: Si W, Heng PA. Point-based visuo-haptic simulation of multi-organ for virtual surgery. Digit Med [serial online] 2017 [cited 2023 Mar 24];3:18-24. Available from: http://www.digitmedicine.com/text.asp?2017/3/1/18/208450
Introduction
Virtual reality-based surgical simulators can give the surgeon visual and haptic cues promising to be powerful aids for training medical personnel and monitoring their performance, which greatly improves patient care and reduces risks. The major concern of such simulators is to reproduce the biomechanical response of organs in the interactive environment.[1] Biomechanical simulation with user interaction involves many challenges such as real-time computation of soft tissues deformation, collision detection, contact modeling, topological modification and haptic feedback (Payan [2] for a broad survey). To well handle the topological modifications, such as large deformation and cutting, this paper presents a meshless method based computational framework to achieve the fast and robust surgical simulation. Point-based method, one of the most promising meshless methods, has been well developed in computer graphics.[3],[4],[5] It uses the total Lagrangian formulation to calculate elastic forces with reference to the initial configuration, which makes it competitive compared to finite element method (FEM) due to its low computational cost, reasonable precision, and stability. However, the above advantages are not enough for the point-based method being applied in surgical simulation; there are still some open topics for point based computational framework simulating the dynamic, complex, and unpredictable mechanical response of human organs in virtual surgery.
Considering the complex boundary conditions of deformable organ, the method should be able to handle frictional contact problem induced by tissue-tissue and tool-tissue interactions to obtain a realistic global behavior. Besides, haptic rendering is also required in such simulators since there is evidence that haptics interface can better aid the user in the learning process over simulation without force feedback.[6],[7] We suggest readers knowing more haptic technology through the book,[8] and recent reviews of haptics in surgery.[9],[10] A common solution to deal with contact is to use the penalty method, however, the selection of penalty coefficient is problem-dependent, and depends strongly on the stiffness ratio between the contacting objects. Consequently, the penalty method is limited for some applications, such as the interaction between deformable bodies. Moreover, Duriez et al.[11] introduced Signorini model for contact handling in the field of haptic rendering. However, this method is currently only applied in mesh-based methods. To solve above issues, a hybrid geometric model, including a high-resolution triangular surface and internal points cloud, is proposed to adopt the high-resolution triangular surface to resolve frictional contact problem precisely while not increase the computational complexity in solving mechanical response, allowing real-time visuo-haptic feedback when human organ undergoes deformations.
Materials and Methods
In this paper, we work on investigating a step-by-step approach to apply point-based method in biomechanical simulation with complex boundary conditions, aiming at providing the robust, real-time, and realistic visuo-haptic feedback.
Hybrid geometric model
Our framework makes use of a hybrid geometric model comprising both surface meshes and volumetric points cloud. The high-resolution triangular surfaces are employed to represent the exterior structures of human organs in virtual surgery. Meanwhile, the interior structures of organs are constructed by arbitrary points clouds, which are also employed as physical model participating in dynamic simulation. The hybrid geometric model organization can be seen in [Figure 1]. This hybrid geometric model can make the mechanical resolution become independent from the rendering model, allowing to achieve real-time performance with certain mechanical resolution while preserve the detailed surface features of the object. The functional diagram of hybrid geometric model is illustrated in [Figure 2]. However, this leads to a dilemma; on the one hand, we resolve the frictional contact problem for mesh surface, rather than for mechanical model. On the other hand, the dynamic behavior of deformable bodies is solved on the internal point cloud, rather than the rendering model. Thus, we have to dynamically bridge these two geometries, allowing the displacement and force to be transferred in a light manner.{Figure 1}{Figure 2}
Here, we adopt the moving least square shape function [12] to connect the surface mesh and internal points cloud. In more details, each surface vertex can be represented by the internal points in the support domain. Suppose a particular surface vertex xi, the support domain S of xi contains n internal points, Us is an n-vector containing the value of the field variables at each node in the support domain Us= (u1, u2,…, un) T, we have
Where Φ (xi) is the shape function of xi in the support domain.
Assuming the contact point λ is inside the triangle (Ψ1, Ψ2, Ψ3), Ψ can be expressed as
Where is the barycentric coordinates for Ψ is the barycentric coordinates for Ψ.
Thus, we can easily transfer the contact forces on surface vertices to internal points
Where fs= (f1, f2,…, fn) T, and fc is contact force, contact normal N is equal to the surface normal of triangle where contact point locates in. Besides, the displacement of internal points can also be reflected on surface vertices according to Equation 1.
Point-based deformation
Following the work of Ller et al.,[3] we approximate the deformation gradient Fi for each particle i by minimizing
Where the sum is taken over the neighbors j, and wij is a weighting kernel evaluated in rest space, uij is the distance in the reference state, while xij is the distance in the present state. We compute the minimizer by solving the following linear system,
Where we refer to the quantity as the particle i's basis matrix, Ai, which we invert to solve the system.
We describe the biomechanical property of human organ with Neo-Hookean hyperelastic model; here, stress tensor can be formulated as
Where μ, λ are the material parameters, Ji = det (Fi), I is the identity matrix, and Ci the right Cauchy-Green deformation tensor .
After that, we can compute the symmetrized elastic forces
Where vi is the volume of i. For more details, we suggest readers refer to.[5]
In this paper, we model the dynamics of human organ using the following mathematical formulation:
Where t is the time step, M is a (constant) mass matrix mass, X are the current nodes' displacements. is the damping force, represented by the popular Rayleigh assumption:
, here ϑ and ξ are mass-proportional damping coefficient and stiffness-proportional damping coefficient, respectively. In addition, K(X) X represents elastic forces, and R is the current applied force field equivalently distributed to each node of the object's mesh.
Finally, it is worth noting that explicit time integration is used in this paper due to its efficiency. However, explicit time integration is only conditionally stable and requires an estimation of the maximum stable time step. Recently, Joldes et al.[13] developed an effectively method of estimating the stable time step for element free Galerkin (EFG) methods and we find it is also suitable for our work.
Where, in which n is the number of influence points for integration, c is the dilatational wave speed.
Where ρ is the density of human organ.
Frictional contact mechanics
Tool-tissue and tissue-tissue interactions in the surgical simulation are frictional contact problems, which are rather complicated from both the theoretical and numerical points of view. They are characterized by a geometric and material discontinuity at the interface instead of the usual continuity property holding in solid mechanics. As a consequence, frictional contact problems are inherently nonlinear (even nonsmooth), involving variational inequalities and constrained minimizations. In this paper, we model frictional contact problems through Coulomb's friction law with Signorini's condition,
Where δN and δT are the normal and tangential gap in the contact space, and fN and fT are their corresponding forces. ς is the coefficient of kinetic friction.
Thanks to the hybrid geometric model, we can solve the frictional contact mechanics with surface mesh, and further transfer the boundary conditions for internal point cloud. We first solve the tissue-tissue interaction with method proposed in.[11] In addition, we resolve the tool-tissue interaction with penalty method, where penalty force fc in tool-tissue interaction can be defined as,
Where δ is the interpenetration, and M, D, K, h are the mass, damping, stiffness matrix, and time step, respectively, which can be precomputed before simulation.
In addition, the refresh rate of visual feedback in surgical simulation range typically between 25 and 60 Hz (vision rendering rates), while haptic feedback should run at a faster rate (1000 Hz). However, the dynamic behavior of human organ with complex boundary conditions can be hardly simulated at haptic refresh rate, so that their simulation loop for visual and haptic rendering should be separated. Here, we introduce the multirate technique proposed in [14] to penalty method. In more detail, the dynamics of the human organ is computed in the simulation, while the feedback forces induced by tool-tissue interaction are solved using penalty method, which is recomputed at a higher rate in the haptic loop based on an intermediate representation shared between two loops. The overall process of visuo-haptic simulation is illustrated in [Figure 3].{Figure 3}
Results
In this section, we present results of experiments demonstrating the performance of our method. A companion video is available illustrating the detailed experiment setup and results. The experimental platform is Intel (R) Xeon (R) CPU, 3.40 GHz, 4 GB memory, NVIDIA Quadro K600.
We produced a simulation with a liver (8597 mesh vertices and 10,450 internal points) and applied the Neo-Hookean material on the biomedical models. We recorded the force data of an instrument controlled through the Geomagic Touch X haptic interface. [Figure 4] illustrates the biomechanical deformation simulation of the liver model with force feedback, and the experimental results demonstrate the effects of our method in deformation simulation. We also simulate the tissue-tissue interaction [Figure 5], which is a complex multiple frictional contact problem. The liver is contact with the stomach, and the interaction between will significantly affects the deformation of each other. The experimental results show our method can achieve continuous tissue-tissue interactions. [Figure 6] illustrates the force feedback by our method, which can achieve a stable force feedback for the visuo-haptic simulation. The dynamic procedure of visuo-haptic simulation can be seen in the accompanied video.{Figure 4}{Figure 5}{Figure 6}
In addition, we demonstrate that our method is generic enough to address several kinds of simulations in a medical context. We use it to simulate the mechanical response of human organs in laparoscopy and thoracoscopy respectively involving haptic feedback with multi-contacts and frictions. We show our method can fulfill the real-time requirement of such simulations and handle complex boundary conditions. The visual effects of visuo-haptic medical simulation can be more obviously observed in the supplementary video.
Laparoscopic surgery
Laparoscopic surgery is a kind of minimally invasive surgery, which allows doctors using fine instruments to cut or trim tissues, perform biopsies, grasp organs, etc., inside the abdomen. Except for the specific operations, biomechanical response of target organ induced by complex tissue-tissue interaction is important for the global behavior. We performed a simulation involving seven deformable organs (liver, spleen, pancreas, gallbladder, kidney, colon, and stomach) in contact with each other. Experiment results in [Figure 7] show that our method can simulate the visuo-haptic feedback with physical fidelity.{Figure 7}
Video-assisted thoracoscopic surgery
Video-assisted thoracoscopic surgery is also a minimally invasive surgical technique used to diagnose and treat problems in your chest. Deformation, induced by tool-tissue interaction, is vital for this virtual surgery. This is because doctors have to frequently operate the lung to find the legions, which is accompanied with the deformation. Here, we model two major organs in thoracoscopy, lung, and heart, through modeling the mechanical response under the interaction with medical device to demonstrate the effectiveness of our method [Figure 8].{Figure 8}
Discussion
Note that FEM is the most commonly used in biomechanical simulation and the complex nonlinear behavior of human organ can be directly accounted for through constitutive relations and provide high biomechanical realism. However, the accuracy and robustness of the FEM rely heavily on the mesh that discretizes the geometry that severe limit its applications in surgical simulation. This is because complicated and irregular meshes are required to represent the morphological details of organs in complex surgical scenarios. Furthermore, topological modifications are often accompanied with tool-tissue interactions, such as large deformation and cutting, the quality of elements could be deteriorated, leading to severely distorted elements.[15] Further progress of simulation is meaningless or even impossible without taking care of those distorted or degenerated elements. After years of constant progress, meshing and remeshing remain difficult issues.[16],[17]
With the latest developments, meshless methods [18],[19],[20] have been suggested as a possible alternative to solve the above difficulties. One advantage of meshless methods over FEM is that the domain of interest is totally discretized only by nodes, and hence, they are completely free from mesh dependence.[21] They only utilize an unstructured cloud of nodes to discretize the geometry instead of elements, and the arrangement of these nodes is almost arbitrary. A further advantage of using a meshless method over FEM is their greater flexibility in constructing shape functions. In meshless methods, connectivity between nodes is not defined or loosely defined. Meshless methods are thus more attractive in dealing with geometric discontinuities, such as large deformation.[22] Horton et al. compared the EFG methods using the total Lagrangian formulation with conventional FEM, and it was concluded that their methods were superior in the case of large deformations.[23],[24] Dehghan et al.[25] compared EFG method with conventional FEM imposing certain boundary conditions. The results show that their method can run faster than FEM since it does not require remeshing for large deformation. Besides, they also proved that for small deformations FEM and EFG can result in the same responses, while for large deformations EFG leads to a much smaller error compared with FEM.
In this regard, this paper employed a meshless-based computational framework to achieve the visuo-haptic interaction with human organ. Different from the FEM method, our method adopts sparse point cloud to model the mechanics of deformable bodies while employ surface mesh to represent morphological details of human organ, which can not only disconnect mechanical complexity from geometrical details, but also enable precise boundary conditions to be solved with surface mesh. In this way, our method can effectively release the limitation of the mesh dependency of FEM and take the advantages of the meshless method and simultaneously achieve the accurate continuous collision detection and real-time visuo-haptic simulation.
Conclusions
This paper has explored visuo-haptic simulation of multi-organ interactions, which plays a core role in realistic surgical simulation. We propose a robust and efficient point-based framework based on a hybrid geometric model, which is to adopt the point-based model to simulate the mechanical response of deformable body, represent morphological details with a surface mesh and and achieve the bijective mapping between internal point cloud and exterior surface mesh with MLS shape function. The frictional contact mechanics is solved with the Coulomb's friction law with Signorini's condition and the visuo-haptic feedback are controled by the multirate technique. The effectiveness of the proposed framework is demonstrated through a variety of challenging surgical scenarios. Our method can achieve efficient and realistic visuo-haptic simulation with both tool-tissue and tissue-tissue interactions.
However, our present method cannot model the mechanical behavior of heterogeneous human organ, such as tumorized organ so that our immediate plan next is to model the heterogeneity of human organ. We are also interested in integrating the GPU-based preconditioner proposed in [26] for contact problems to our present implementation, to further optimize the time performance of our method. In the long run, we will adopt the continuous penalty force [27] to improve the robustness and stability of our method and extend this computational framework to handle more complex topological modifications, such as cutting.
Financial support and sponsorship
The work is supported by Hong Kong Research Grants Council under General Research Fund (Project No. CUHK 14202514, CUHK 14203115).
Basdogan C, Sedef M, Harders M, Wesarg S. VR-based simulators for training in minimally invasive surgery. IEEE Comput Graph Appl 2007;27:54-66. [PUBMED]
Payan Y. Soft tissue biomechanical modeling for computer assisted surgery. Studies in Mechanobiology Tissue Engineering and Biomaterials. Berlin Heidelberg: Springer; 2012.
Ller M, Keiser R, Nealen A, Gross M. Point Based Animation of Elastic, Plastic and Melting Objects. ACM Siggraph/Eurographics Symposium on Computer Animation. Eurographics Association; 2004. p. 141-51.
Tholey G, Desai JP, Castellanos AE. Force feedback plays a significant role in minimally invasive surgery: Results and analysis. Ann Surg 2005;241:102-9. [PUBMED]
Panait L, Akkary E, Bell RL, Roberts KE, Dudrick SJ, Duffy AJ. The role of haptic feedback in laparoscopic simulation training. J Surg Res 2009;156:312-6. [PUBMED]
Westebring-van der Putten EP, Goossens RH, Jakimowicz JJ, Dankelman J. Haptics in minimally invasive surgery – A review. Minim Invasive Ther Allied Technol 2008;17:3-16. [PUBMED]
Puangmali P, Althoefer K, Seneviratne LD, Murphy D. State-of-the-art in force and tactile sensing for minimally invasive surgery. IEEE Sens J 2008;8:371-81.
Duriez C, Dubois F, Kheddar A, Andriot C. Realistic haptic rendering of interacting deformable objects in virtual environments. IEEE Trans Vis Comput Graph 2006;12:36-47.
Peterlik I, Nouicer M, Duriez C, Cotin S, Kheddar A. Constraint-based haptic rendering of multirate compliant mechanisms. IEEE Trans Haptics 2011;4:175-87.
Luo Y. Dealing with extremely large deformation by nearest-nodes FEM with algorithm for updating element connectivity. Int J Solids Struct 2008;45:5074-87.
Aras R. Meshless elasticity model and contact mechanics-based verification technique. MICCAI Computational Biomechanics for Medicine. Berlin Heidelberg: Springer; 2014.
Belytschko T, Krongauz Y, Organ D, Fleming M, Krysl P. Meshless methods: An overview and recent developments. Comput Methods Appl Mech Eng 1996;139:3-47.
Horton A, Wittek A, Joldes G R, Miller K. A meshless total Lagrangian explicit dynamics algorithm for surgical simulation. Int J Numer Method Biomed Eng 2010;26:977-98.
Zhang GY, Wittek A, Joldes GR, Jin X, Miller K. A three-dimensional nonlinear meshfree algorithm for simulating mechanical responses of soft tissue. Eng Anal Bound Elem 2014;42:60-6.
Dehghan MR, Rahimi A, Talebi HA, Zareinejad M. A three-dimensional large deformation model for soft tissue using meshless method. Int J Med Robot 2016;12:241-53.
Courtecuisse H, Allard J, Kerfriden P, Bordas SP, Cotin S, Duriez C. Real-time simulation of contact and cutting of heterogeneous soft-tissues. Med Image Anal 2014;18:394-410.
is the barycentric coordinates for Ψ is the barycentric coordinates for Ψ.
as the particle i's basis matrix, Ai, which we invert to solve the system.
.
is the damping force, represented by the popular Rayleigh assumption:
, here ϑ and ξ are mass-proportional damping coefficient and stiffness-proportional damping coefficient, respectively. In addition, K (X) X represents elastic forces, and R is the current applied force field equivalently distributed to each node of the object's mesh.
and M, D, K, h are the mass, damping, stiffness matrix, and time step, respectively, which can be precomputed before simulation.
Search Pubmed for