Sketch-based Image-independent Editing of 3D Tumor Segmentations using Variational Interpolation Frank Heckel1, Stefan Braunewell1, Grzegorz Soza2, Christian Tietjen2, Horst K. Hahn1 1 Fraunhofer MEVIS, Germany, 2 Siemens AG, Healthcare Sector, Imaging & Therapy Division, Computed Tomography, Germany

Motivation Why do we need efficient segmentation editing tools? Solution Results Outlook Conclusion Why do we need efficient segmentation editing tools? Segmentation is one of the essential tasks in medical image analysis Many sophisticated automatic segmentation algorithms exist … … which might fail in some cases (low contrast, noise, biological variability) What to do? Manual segmentation?  Takes too long Different algorithm?  Might fail as well Locally correct the error! An appropriate segmentation is often mandatory

Motivation What makes segmentation editing a difficult problem? Solution Results Outlook Conclusion What makes segmentation editing a difficult problem? Requirements: Intuitive interaction in 2D – estimate the user’s intention in 3D Local modifications Real-time feedback Provide a general tool (for different objects and modalities) Be independent of the preceding automatic algorithm The user expects the tool to allow him or her to correct all errors With only a few steps! The segmentation problems are typically hard (noise, low contrast, …) Do not use the image! Image-based manual correction methods suffer from the same problems as the automatic algorithms

Solution Segmentation Formulated as an Object Reconstruction Problem Motivation Results Outlook Conclusion Segmentation Formulated as an Object Reconstruction Problem Use methods known from object reconstruction Contour-based representation  Can be treated as a point cloud Reconstruct a smooth surface using variational interpolation Variational interpolation is energy-minimizing  generates a smooth surface

without hole-handling Solution Motivation Results Outlook Conclusion Segmentation Formulated as an Object Reconstruction Problem Hole-handling: Recursively check the level of embedding Holes have an odd level  Invert the sign of the normals without hole-handling with hole-handling

Part containing the center of gravity Solution Motivation Results Outlook Conclusion Sketch-based Editing in 2D User input Correction result Edited region Part containing the center of gravity

Solution Sketch-based Editing in 2D We have to deal with imperfection: Motivation Results Outlook Conclusion Sketch-based Editing in 2D We have to deal with imperfection: add remove add + remove replcae

Solution Sketch-based Editing in 2D Motivation Results Outlook Conclusion Sketch-based Editing in 2D A correction might generate new “holes”: Remove all contours whose level of embedding has changed

Solution 3D Extrapolation using Variational Interpolation 𝒔 𝒔 𝒆𝒏𝒅 Motivation Results Outlook Conclusion 3D Extrapolation using Variational Interpolation 𝒔 𝒔 𝒆𝒏𝒅 𝑪 𝒔 𝒖 𝒔 𝒔 𝐬𝐭𝐚𝐫𝐭 𝑪 𝒔 𝒆 𝑑 𝐶 𝑠 𝑢 = max 𝑖 min 𝑗 𝐶 𝑠 𝑒 𝑖 − 𝐶 𝑠 𝑢 [𝑗] 𝑑 𝑠 Compute a correction depth Reconstruct the new surface between start and end d approximates the thickness of the error Result continuously fits into initial segmentation 𝑠 start = max 𝑠−𝑑 𝐶 𝑠 𝑢 , 𝑠 min 𝑠 end = min 𝑠+𝑑 𝐶 𝑠 𝑢 , 𝑠 max

Solution 3D Extrapolation using Variational Interpolation Motivation Results Outlook Conclusion 3D Extrapolation using Variational Interpolation Making the correction local: Dilate the edited region Duplicate it to all slices of the reconstruction Use new segmentation in this region only 𝑘=2 1 4 3 3𝑛 4𝜋 +1 Sphere volume: 𝑉= 4 3 𝜋 𝑟 3 → Assumption: the object is roughly spherical 𝒌

Solution Manual Correction Workflow Motivation Results Outlook Conclusion Manual Correction Workflow Corrections can be performed in any view User can arbitrarily switch between views Previously performed corrections should be part of the new surface Keep all user-inputs and use them for reconstruction user-input (1st step, axial view) user-input (2nd step, sagittal view)

Solution Motivation Results Outlook Conclusion

Results Solution Outlook Conclusion Data: 89 tumors in CT (lung nodules, liver metastases, lymph nodes) Participants: 2 technical experts with 6+ years experience in tumor segmentation and assessment Qualitative rating of the correction tool Median: 4 steps (53s), Avg. time per step: 0.4s1 Rating Meaning # Cases Percentage ++ Perfect 14 15.7% + Good 43 47.3% Acceptable 25 28.1% - Bad 6 6.7% -- Unacceptable 1 1.1% 92.1% Time includes rating 1 Intel Xeon X5550 (2.66GHz), 12GB RAM, Windows 7 64-Bit, 4 cores used

Outlook Main problem currently: Contradictory user inputs Results Conclusion Main problem currently: Contradictory user inputs Reconstruction is currently too slow for large objects (like the liver) Comparison the other methods Strongly depends on the specific segmentation task and the experience and requirements of the users Proposal: Segmentation editing challenge Existing manual correction algorithms are not well evaluated

Conclusion Outlook Segmentation editing is an indispensable step in the segmentation process Efficient editing in 3D is challenging Sketching provides an intuitive interface for segmentation editing in 2D We have proposed a general, efficient editing tool 2D corrections are extrapolated to 3D using object reconstruction Can be used for any 3D modality and any compact object Slice-wise fallback possible if extrapolation fails

Thank you! frank.heckel@mevis.fraunhofer.de

Grafische Elemente Kästen, Pfeile, Verbindungen und Linien (Auswahl) ! DIESE FOLIE AUS FINALER PRÄSENTATION LÖSCHEN ! folgende Elemente können hier per Rechtsklick kopiert und an gewünschter Stelle in der neuen Präsentation per Rechtsklick wieder eingesetzt werden: Kasten gefüllt Grau Pfeile und Verbindungen Fraunhofer-Grün Kasten mit Outline Fraunhofer-Grün Linienstärke je nach Größe 2, 4, 6, 8 pt Je größer der Kasten, desto stärker die Linie Linien Fraunhofer-Grün, bis 1 pt auch Schwarz Linienstärke gestrichelt: 1 pt Linienstärke durchgezogen: 1, 2, 3, 4 und 5 pt

! DIESE FOLIE AUS FINALER PRÄSENTATION LÖSCHEN ! Farben ! DIESE FOLIE AUS FINALER PRÄSENTATION LÖSCHEN ! folgende Farben können über die Powerpoint-Farbauswahl hier aufgenommen und damit in der neuen Präsentation angewendet werden: Überschriften / Fließtext / Quellenangaben / Bildunterschriften / Grafikauszeichnungen Grafikauszeichnungen Aufzählungen / Nummerierungen erster Ebene / grafische Elemente Grafiken Fonds hinter Grafiken

Fhe: Weitere grafische Elemente ! DIESE FOLIE AUS FINALER PRÄSENTATION LÖSCHEN ! Diese Box kann z.B. für Definitionen verwendet werden. Leider wird die Stärke der Abrundung der Ecken mit der Größe der Box verändert. Die Rundung muss dann per Hand korrigiert werden. Die Gesamtanzahl der Folien („xx“ in der Fußzeile) wird aktuell leider nicht automatisch eingefügt und muss per Hand im Folienmaster angepasst werden. Begriff Definition …