Method and apparatus for efficient three-dimensional contouring of medical images

36 Pages·2014·2.53 MB·English

USOO8731258B2
United States Patent
(12) (10) Patent N0.: US 8,731,258 B2
Hibbard et al. (45) Date of Patent: May 20, 2014
(54) METHOD AND APPARATUS FOR EFFICIENT 6,259,943 B1 7/2001 Cosman et a1.
THREE-DIMENSIONAL CONTOURING OF g} ylggal er atl l
, , au man e a .
MEDICAL IMAGES 6,606,091 B2 8/2003 Liang et al.
_ _ 6,683,933 B2 1/2004 Saito et al.
(75) Inventors: Lyndon S. Hibbard, St. Lou1s, MO 6,947,584 B1 9/2005 Avila er al,
(US); Xiao Han, Chester?eld, MO (US) 7,010,164 B2 3/2006 Weese et a1.
7,110,583 B2 * 9/2006 Yamauchi ................... .. 382/128
(73) Assignee: IMPAC Medical Systems, Inc., 7,167,172 B2 1/2007 Kaus et al'
Sunn vale CA (Us) 7,333,644 B2 2/2008 Jerebko et a1.
y ’ 7,428,334 B2 9/2008 Sch01sswohl et al.
_ _ _ _ _ 7,620,224 B2 * 11/2009 Matsumoto ................. .. 382/128
( * ) Notlce: Subject to any d1scla1mer, the term ofth1s 8,098,909 B2 1/2012 Hibbard et 31,
patent is extended or adjusted under 35 2005/0168461 A1 55/2005 Acosta et al.
U.S.C. 154(b) by 0 days. (Continued)
(21) Appl. N0.: 13/295,494 OTHER PUBLICATIONS
- _ Adelson et al., “Pyramid Methods in Image Processing”, RCA Engi
(22) Flled' NOV“ 14’ 2011 neer, Nov/Dec. 1984, pp. 33-41, vol. 29-6.
(65) Prior Publication Data (Continued)
Us 2012/0057768 A1 Man 8’ 2012 Primary Examiner * Anand Bhatnagar
_ _ (74) Attorney, Agent, or Firm * Finnegan, Henderson,
Related US. Application Data FarabOW’ Garrett & Dunner, LLp
(63) Continuation of application No. 11/848,624, ?led on
Aug. 31, 2007, noW Pat. No. 8,098,909. (57) ABSTRACT
A technique is disclosed for generating a neW contour and/or
(51) Int- Cl- a 3D surface such as a variational implicit surface from con
G06K 9/00 (2006-01) tour data. In one embodiment, B-spline interpolation is used
(52) US. C1. to e?iciently generate a neW contour (preferably a transverse
USPC ......................................... .. 382/128; 128/922 contour), from a plurality of input contours (preferably, sag
(58) Field of Classi?cation Search ittal and/ or coronal contours). In another embodiment, a point
USPC ............... .. 382/ 100, 128, 129, 130, 131, 132; reduction operation is performed on data sets corresponding
128/ 922; 378/4e27 to any combination of transverse, sagittal, or coronal contour
See application ?le for complete search history. data prior to processing those data sets to generate a 3D
surface such as a variational implicit surface. A neW contour
(56) References Cited can also be generated by the intersection of this surface With
an appropriately placed and oriented plane. In this manner,
US. PATENT DOCUMENTS the computation of the variational implicit surface becomes
suf?ciently ei?cient to make its use for neW contour genera
5,859,891 A * 1/1999 Hibbard ........................ .. 378/62 tion practical.
6,075,538 A * 6/2000 Shu et a1. . .. 345/419
6,112,109 A * 8/2000 D’Urso ....................... .. 600/407
6,142,019 A 11/2000 Venchiaruttiet al. 17 Claims, 23 Drawing Sheets
602 Receive a plurality of input points, the input points comprising a plurality of
different initial sets of input points, wherein each initial point set is representative
of a an S contour or a C contour
604: For each initial point set, reduce the number of points therein to generate a
plurality of different reduced sets of input points
i
606: Re-order the points within each reduced set with respect to a frame of
reference and a direction of rotation
l
608: Find the points of intersection in a T plane for the re-ordered points Within
each reduced set
i
610: Generate a contour in the T plane by interpolating through the points of
intersection using B—spiine interpolation
i
l 612: Does the generated T contour match the image? l
Yes No
614 Archive the generated contour
US 8,731,258 B2
Page 2
(56) References Cited Han et al., “A Morphing Active Surface Model for Automatic Re
Contouring in 4D Radiotherapy”, Proc. of SPIE, 2007, vol. 6512, 9
U.S. PATENT DOCUMENTS pages.
Ho et al., “SNAP: A Software Package for User-Guided Geodesic
2005/0231530 A1 10/2005 Liang et al. Snake Segmentation”, Technical Report, UNC Chapel Hill, Apr.
2005/0276455 A1 12/2005 Fidrich et al.
2003.
2006/0147114 A1 7/2006 Kaus et al.
Huang et al., “Semi-Automated CT Segmentation Using Optic Flow
2006/0149511 A1 7/2006 Kaus et al.
2006/0159322 A1 7/ 2006 Rinck et al. and Fourier Interpolation Techniques”, Computer Methods and Pro
2006/0159341 A1 7/ 2006 Pekar et al. grams in Biomedicine, 84, 124-134, 2006.
2006/0177133 A1 8/2006 Kee Igarashi et al., “Smooth Meshes for Sketch-based Freeform Model
2006/0204040 A1 9/2006 Freeman et al. ing” In ACM Symposium on Interactive 3D Graphics, (ACM
2006/0256114 A1 11/2006 Nielsen et al.
I3D’03), pp. 139-142, 2003.
2007/0014462 A1 1/2007 Rousson et al.
Igarashi et al., “Teddy: A Sketching Interface for 3D Freeform
2007/0041639 A1 2/2007 Von Berg et al.
Design”, Proceedings of SIGGRAPH 1999, 409-416.
2007/0092115 A1 4/2007 Usher et al.
2007/0167699 A1 7/ 2007 Lathuiliere et a1. Ijiri et al., “Seamless Integration of Initial Sketching and Subsequent
2009/0016612 A1 1/2009 Lobregt et a1. Detail Editing in Flower Modeling”, Eurographics 2006, 25, 617
2009/0190809 A1 7/2009 Han et al. 624, 2006.
2012/0057769 A1 3/2012 Hibbard et al. Jain et al., “Deformable Template Models: A Review”, Signal Proc.,
OTHER PUBLICATIONS 1998, pp. 109-129, vol. 71.
Jain, “Fundamentals of Digital Image Processing”, Prentice-Hall,
Anderson et al., “LAPACK User’s Guide”, Third Edition, SIAMi New Jersey, 1989.
Society for Industrial and Applied Mathematics, 1999, Philadelphia. Jehan-Besson et al., “Shape Gradients for Histogram Segmentation
Using Active Contours”, 2003, presented at the 9th IEEE Int. Conf.
Barrett et al., “Interactive Live-Wire Boundary Extraction”, Medical
Image Analysis, 1, 331-341, 1997. Comput. Vision, Nice, France, 8 pages.
Kalet et al., “The Use of Medical Images in Planning and Delivery of
Bertalmio et al., “Morphing Active Countours”, IEEE Trans. Patt.
Radiation Therapy”, J. Am. Med. Inf. Assoc., Sep./Oct. 1997, pp.
Anal. Machine Intell., 2000, pp. 733-737, vol. 22.
Bloomenthal, “An Implicit Surface Polygonizer”, Graphics Gems IV, 327-339, vol. 4 (5).
Karpenko et al., “Free-Form Sketching with Variational Implicit
P Heckbert, Ed., Academic Press, New York, 1994.
Surfaces”, Computer Graphics Forum, 21, 585-594, 2002.
Burnett et al., “A Deformable-Model Approach to Semi-Automatic
Segmentation of CT Images Demonstrated by Application to the Karpenko et al., “SmoothSketch: 3D Free-Form Shapes From Com
plex Sketches”, Proceedings of SIGGRAPH 06, pp. 589-598.
Spinal Canal”, Med. Phys., Feb. 2004, pp. 251-263, vol. 31 (2).
Leyrnarie et al., “Tracking Deformable Objects in the Plane Using an
Carr et al., “Reconstruction and Representation of 3D Objects with
Active Contour Model”, IEEE Trans. Patt. Anal. Machine Intell., Jun.
Radial Basis Functions”, Proceedings of SIGGRAPH 01, pp. 67-76,
1993, pp. 617-634, vol. 15 (6).
2001 . Lipson et al., “Conceptual Design and Analysis by Sketching”, Jour
Carr et al., “Surface Interpolation with Radial Basis Functions for
nal ofAI in Design and Manufacturing, 14, 391-401, 2000.
Medical Imaging”, IEEE Transactions on Medical Imaging, 16,
Lorenson et al., “Marching Cubes: A High Resolution 3D Surface
96-107, 1997.
Construction Algorithm”, Computer Graphics, Jul. 1987, pp. 163
Cover et al., “Elements of Information Theory”, Chapter 2, 1991,
169, vol. 21 (4).
Wiley, NewYork, 33 pages.
Lu et al., “Automatic Re-Contouring in 4D Radiotherapy”, Phys.
Cover et al., “Elements of Information Theory”, Chapter 8, 1991,
Med. Biol., 2006, pp. 1077-1099, vol. 51.
Wiley, New York, 17 pages.
Lu et al., “Fast Free-Form Deformable Registration Via Calculus of
Davis et a1 ., “Automatic Segmentation of Intra-Treatment CT Images
Variations”, Phys. Med. Biol., 2004, pp. 3067-3087, vol. 49.
for Adaptive Radiation Therapy of the Pro state”, presented at 8th Int.
Marker et al., “Contour-Based Surface Reconstruction Using
Conf. MICCAI 2005, Palm Springs, CA, pp. 442-450. Implicit Curve Fitting, and Distance Field Filtering and Interpola
DeBoor, “A Practical Guide to Splines”, Springer, NewYork, 2001. tion”, The Eurographics Association, 2006, 9 pages.
Digital Imaging and Communications in Medicine (DICOM), http:// Paragios et al., “Geodesic Active Contours and Level Sets for the
medical.nema.org/. Detection and Tracking of Moving Objects”, IEEE Trans. Patt. Anal.
Dinh et al., “Reconstructing Surfaces by Volumetric Regularization Machine Intell., Mar. 2000, pp. 266-280, vol. 22 (3).
Using Radial Basis Functions”, IEEE Trans. Patt. Anal. Mach. Intell., Pekar et al., “Automated Model-Based Organ Delineation for Radio
24, 1358-1371, 2002. therapy Planning in Prostate Region”, Int. J. Radiation Oncology
Dinh et al., “Texture Transfer During Shape Transformation”, ACM Biol. Phys., 2004, pp. 973-980, vol. 60 (3 ).
Transactions on Graphics, 24, 289-310, 2005. Pentland et al., “Closed-Form Solutions for Physically Based Shape
DoCarmo, “Differential Geometry of Curves and Surfaces”, Prentice Modeling and Recognition”, IEEE Trans. Patt. Anal. Machine Intell.,
Hall, New Jersey, 1976. Jul. 1991, pp. 715-729, vol. 13 (7).
Falcao et al., “An Ultra-Fast User-Steered Image Segmentation Para Piegl et al., “The NURBS Book”, Second Edition, Springer, New
York, 1997.
digm: Live Wire on the Fly”, IEEE Transactions on Medical Imaging,
Press et al., “Numerical Recipes in C”, Second Edition, Cambridge
19, 55-62, 2000.
University Press, 1992.
Freedman et al., “Active Contours for Tracking Distributions”, IEEE
Rogelj et al., “Symmetric Image Registration”, Med. Imag. Anal.,
Trans. Imag. Proc., Apr. 2004, pp. 518-526, vol. 13 (4).
2006, pp. 484-493, vol. 10.
Gao et al., “A Deformable Image Registration Method to Handle
Sarrut et al., “Simulation of Four-Dimensional CT Images from
Distended Rectums in Prostate Cancer Radiotherapy”, Med. Phys.,
Deformable Registration Between Inhale and Exhale Breath-Hold
Sep. 2006, pp. 3304-3312, vol. 33 (9). CT Scans”, Med. Phys., Mar. 2006, pp. 605-617, vol. 33 (3).
Gering et al., “An Integrated Visualization System for Surgical Plan Schmidt et al., ShapeShop: Sketch-Based Solid Modeling with Blob
ning and Guidance Using Image Fusion and an Open MR”, Journal of Trees, EUROGRAPHICS Workshop on Sketch-Based Interfaces and
Magnetic Resonance Imaging, 13, 967-975, 2001. Modeling, 2005.
Gering, “A System for Surgical Planning and Guidance Using Image Schroeder et al., “The Visualization Toolkit”, 2nd Edition, Kitware,
Fusion and Interventional MR”, MS Thesis, MIT, 1999. 2006, Ch. 5 & 13,65 pp.
Golub et al., “Matrix Computations”, Third Edition, The Johns Sethian, “Level Set Methods and Fast Marching Methods”, 2nd ed.,
Hopkins University Press, Baltimore, 1996. 1999, Cambridge University Press, Chapters 1, 2 & 6, 39 pages.
US 8,731,258 B2
Page 3
(56) References Cited De Berg et al., “Computational Geometry: Algorithms and Applica
tions”, 1997, Chapter 5, Springer-Verlag, New York.
OTHER PUBLICATIONS Dice’s coef?eient, Wikipedia, 1945.
Dinh et al., “Reconstructing Surfaces by Volumetric Regularization
Stefanescu, “Parallel Nonlinear Registration of Medical Images With Using Radial Basis Functions”, IEEE Transactions on Pattern Analy
a Priori Information on Anatomy and Pathology”, PhD Thesis. sis and Machine Intelligence, Oct. 2002, pp. 1358-1371, vol. 24, No.
Sophia-Antipolis: University of Nice, 2005, 140 pages. 10.
Duchon, “Splines Minimizing Rotation-Invariant SBMI-NORMS in
Strang, “Introduction to Applied Mathematics”, 1986, Wellesley,
Soboley Spaces”, 1977, Universite Scienti?que et Medicale
MA: Wellesley-Cambridge Press, pp. 242-262.
Laboratoire de Mathematiques Appliques, Grenoble France.
Thirion, “Image Matching as a Diffusion Process: An Analog with Gelas et al., “Variatonal Implicit Surface Meshing”, Computers and
Maxwell’s Demons”, Med. Imag. Anal., 1998, pp. 243-260, vol. 2 Graphics, 2009, pp. 312-320, vol. 33.
(3) Gering et al., “An Integrated Visualization System for Surgical Plan
Thomas, “Numerical Partial Differential Equations: Finite Differ ning and Guidance using Image Fusion and Interventional Imaging”,
ence Methods”, Springer, NewYork, 1995. Int Conf Med Image Comput Assist Interv, 1999, pp. 809-819, vol. 2.
Turk et al., “Shape Transformation Using Variational Implicit Func Girosi et al., “Priors, Stabilizers and Basis Functions: from regular
tions”, Proceedings of SIGGRAPH 99, Annual Conference Series, ization to radial, tensor and additive splines”, Massachusetts Institute
(Los Angeles, California), pp. 335-342, Aug. 1999. of Technology Arti?cial Intelligence Laboratory, Jun. 1993, 28
Vemuri et al., “Joint Image Registration and Segmentation”, Geo pages.
metric Level Set Methods in Imaging, Vision, and Graphics, S. Osher Ibanez et al., “The ITK Software Guide” Second Edition, 2005.
and N. Paragios, Editors, 2003, Springer-Verlag, NewYork, pp. 251 J ackowski et al., “A Computer-Aided Design System for Re?nement
269. of Segmentation Errors”, MICCAI 2005, LNCS 3750, pp. 717-724.
Wahba, “Spline Models for Observational Data”, SIAM (Society for Kalbe et al., “High-Quality Rendering of Varying Isosurfaces with
Industrial and Applied Mathematics), Philadelphia, PA, 1990. Cubic Trivariate Cl-continuous Splines”, ISVC 1, LNCS 5875,
Wang et al., “Validation of an Accelerated ‘Demons’ Algorithm for 2009, pp. 596-607.
Deformable Image Registration in Radiation Therapy”, Phys. Med. Kaus et al., “Automated 3-D PDM Construction From Segmented
Biol, 2005, pp. 2887-2905, vol. 50. Images Using Deformable Models”, IEEE Transactions on Medical
Wolf et al., “ROPES: a Semiautomated Segmentation Method for Imaging, Aug. 2003, pp. 1005-1013, vol. 22, No. 8.
Accelerated Analysis of Three-Dimensional Echocardiographic Kho et al., “Sketching Mesh Deformations”, ACM Symposium on
Data”, IEEE Transactions on Medical Imaging, 21, 1091-1 104, 2002. Interactive 3D Graphics and Games, 2005, 8 pages.
Xing et al., “Overview of Image-Guided Radiation Therapy”, Med. Knoll et a1 ., “Fast and Robust Ray Tracing of General Implicits on the
Dosimetry, 2006, pp. 91-'112, vol. 31 (2). GPU”, Scienti?c Computing and Imaging Institute, University of
Xu et al., “Image Segmentation Using Deformable Models”, Hand Utah, Technical Report No. UUSCI-2007-014, 2007, 8 pages.
book of Medical Imaging, vol. 2, M. Sonka and J.M. Fitzpatrick, Leventon et al., “Statistical Shape In?uence in Geodesic Active Con
Editors, 2000, SPIE Press, Chapter 3. tours”, IEEE Conference on Computer Vision and Pattern Recogni
Yezzi et al., “A Variational Framework for Integrating Segmentation tion, 2000, pp. 1316-1323.
and Registration Through Active Contours”, Med. Imag. Anal., 2003, Nealen et al., “A Sketch-Based Interface for Detail-Preserving Mesh
pp. 171-185, vol. 7. Editing”, Proceedings of ACM SIGGRAPH 2005, 6 pages, vol. 24,
Yoo, “Anatomic Modeling from Unstructured Samples Using Varia No. 3.
tional Implicit Surfaces”, Proceedings of Medicine Meets Virtual Osher et al., “Level Set Methods and Dynamic Implicit Surfaces”,
Reality 2001, 594-600. Chapters 11-13, 2003, Springer-Verlag, New York, NY
Young et al., “Registration-Based Morphing of Active Contours for Pieper et al., “The NA-MIC Kit: ITK, VTK, Pipelines, Grids and 3D
Segmentation of CT Scans”, Mathematical Biosciences and Engi Slicer as An Open Platform for the Medical Image Computing Com
neering, Jan. 2005, pp. 79-96, vol. 2 (1). munity”, Proceedings of the 3rd IEEE International Symposium on
Yushkevich et al., “User-Guided 3D Active Contour Segmentation of Biomedical Imaging: From Nano to Macro 2006, pp. 698-701, vol. 1.
Anatomical Structures: Signi?cantly Improved Ef?ciency and Reli Pohl et al., “A Bayesian model for joint segmentation and registra
ability”, NeuroImage 31, 1116-1128, 2006. tion”, NeuroImage, 2006, pp. 228-239, vol. 31.
Zagrodsky et al., “Registration-Assisted Segmentation of Real-Time Sapiro, “Geometric Partial Differential Equations and Image Analy
3-D Echocardiographic Data Using Deformable Models”, IEEE sis”, Chapter 8, 2001, Cambridge University Press.
Trans. Med. Imag., Sep. 2005, pp. 1089-1099, vol. 24 (9). Schroeder et al., “The Visualization Toolkit”, 2nd edition, Chapter 5,
Zeleznik et al., “SKETCH: An Interface for Sketching 3D Scenes”, 1998, Prentice-Hall, Inc.
Proceedings of SIGGRAPH 96, 163-170, 1996. Singh et al., “Real-Time Ray-Tracing of Implicit Surfaces on the
Zhong et al., “Object Tracking Using Deformable Templates”, IEEE GP ”, IEEE Transactions on Visualization and Computer Graphics,
Trans. Patt. Anal. Machine Intell., May 2000, pp. 544-549, vol. 22 2009, pp. 261-272, vol. 99.
(5). Tikhonov et al., “Solutions of III-Posed Problems”, Introductioni
Notice of Allowance for US. Appl. No. 12/022,929 dated May 8, Chapter 2, 1977, John Wiley & Sons.
2012. Tsai et al, “A Shape-BasedApproach to the Segmentation of Medical
Bookstein, “Principal Warps: Thin-Plate Splines and the Decompo Imagery Using Level Sets”, IEEE Transactions on Medical Imaging,
sition of Deformations”, IEEE Transactions on Pattern Analysis and Feb. 2003, pp. 137-154, vol. 22, No. 2.
Machine Intelligence, Jun. 1989, pp. 567-585, vol. 11, No. 6. Tsai et al., “An EM algorithm for shape classi?cation based on level
Botsch et al., “On Linear Variational Surface Deformation Methods”, sets”, Medical Image Analysis, 2005, pp. 491-502, vol. 9.
IEEE Transactions on Visualization and Computer Graphics, 2008, International Search Report and Written Opinion for PCT/US2012/
pp. 213-230, vol. 14, No. 1. 048938 dated Oct. 16, 2012.
Cruz et al., “A sketch on Sketch-Based Interfaces and Modeling”, Of?ce Action for US. Appl. No. 13/295,525 dated Nov. 28, 2012.
Graphics, Patterns and Images Tutorials, 23rd SIBGRAPI Confer
ence, 2010, pp. 22-33. * cited by examiner
U S. Patent May 20, 2014 Sheet 1 0f 23 US 8,731,258 B2
102 100 {W
US. Patent May 20, 2014 Sheet 2 0f 23 US 8,731,258 B2
Figure 2(a)
US. Patent May 20, 2014 Sheet 3 0f 23 US 8,731,258 B2
00/
z ’ OG
lsocenter C"
\ I
AYW Yw \ YC
View Plane Window ZW XVI \(kl
)Xw
ZW
The Window Coordinate System
Transverse Coronal
Figure 2(b)
US. Patent May 20, 2014 Sheet 4 0f 23 US 8,731,258 B2
Figure 3
US. Patent May 20, 2014 Sheet 5 0f 23 US 8,731,258 B2
Processor
(502)
M‘ M
Input Points for Output T
Contours Contour(s)
(500) (504)
Figure 5(a)
Processor
(502)
MM MM
Input Points for Output 3D
Contours Surface
(500) (506)
Figure 5(b)
US. Patent May 20, 2014 Sheet 6 0f 23 US 8,731,258 B2
602: Receive a plurality of input points, the input points comprising a plurality of
different initial sets of input points, wherein each initial point set is representative 4
of a an S contour or a C contour
604: For each initial point set, reduce the number of points therein to generate a
plurality of different reduced sets of input points
V
606: Re-order the points within each reduced set with respect to a frame of
reference and a direction of rotation
V
608: Find the points of intersection in a T plane for the re-ordered points within
each reduced set
V
610: Generate a contour in the T plane by interpolating through the points of
intersection using B~spline interpolation
‘
612: Does the generated T contour match the image?
Yes No
v
614: Archive the generated contour
Figure 6
US. Patent May 20, 2014 Sheet 7 0f 23 US 8,731,258 B2
r
Curv1at-ur:e
3] @111 5'3!) {BUD ZEUS
Arc Length
Figure 7