Reconstruction of 3D Histology Volumes from Breast Lumpectomy Specimens
by
Rushin Shojaii
A thesis submitted in conformity with the requirements for the degree of Doctor of Philosophy
Department of Medical Biophysics University of Toronto
©Copyright by Rushin Shojaii (2017)
ii
Reconstruction of 3D Histology Volumes from Breast
Lumpectomy Specimens
Rushin Shojaii
Doctor of Philosophy
Department of Medical Biophysics University of Toronto
2017
Abstract
Three dimensional reconstruction and examination of microscopic images of tissue sections can
enhance the study of normal and disease processes, particularly in studies concerned with
structural changes. 3D histology volumes can also be used as ground truth in validating new
volumetric medical imaging techniques. The accuracy of the histology volume reconstruction is
hindered by sparse sampling of the clinical specimens, the deformation of the tissue during the
histology processes, and variations in staining. The existing methods for histology volume
construction of human breast specimens are either manual or applicable only to small cropped
densely sampled regions.
In this thesis, we hypothesize that by using image registration algorithms, 3D histology volumes
can be automatically reconstructed from stacked 2D images of histology sections. We present an
image registration framework for histology volume reconstruction, which is also extendable for
registration of the histology to ex vivo medical images. The framework is first validated using a
pre-clinical model and then further improved by optimizing the deformable registration for
histology images obtained from highly deformable human breast specimens. Finally, the
histology volumes of five lumpectomy specimens are reconstructed using the proposed
framework to study tumour features in 3D, specifically tumour volume and focality.
The proposed approach is shown to be suitable for 3D histology volume reconstruction from 2D
histology images of breast lumpectomy specimens. The average target registration error for the
iii
breast specimens was measured as 0.47 ± 0.31 mm based on manually selected landmarks and
the optimized deformable registration was shown to significantly improve the registration
comparing to rigid registration. It is also shown that this method has the potential to provide
more accurate information about 3D tumour measurements, such as volume, and presence of
multi-focality.
iv
Acknowledgments
I would like to thank the following people for their help and support throughout my Ph.D. work:
My supervisor Dr. Anne Martel and members of my supervisory committee Dr. Martin Yaffe
and Dr. John Sled for their insightful comments and guidance;
Dr. Sharon Nofech-Mozes and Dr. Naomi Matsuura for lending their expertise to my research;
The past and present members of Dr. Yaffe’s group, Melissa Hill, Gordon Mawdsley, Tigran
Karavardanyan, Sam Shen, Yulia Yerofeyeva, Alison Cheung, Taha Rashed, Kela Liu, Rachel
Peters, Adebayo Adeeko, Cindy Wang and James Mainprize for their great assistance in my
experimental work and scientific discussions;
All past and present members of Dr. Martel’s group for their support and encouragement;
All my friends and fellow graduate students, Nazanin Hosseinkhah, Golnaz Farhat, Hatef
Mehrabian, Janet Arani, Melissa Hill, Rachel Chan, Wendy Oakden, Mahta Karimpoor, Zahra
Faraji-Dana, Laleh Golestanirad, Farzad Khalvati, Cristina Gallego, Mohammad Peikari,
Sharmila balasingham, and Taraneh Azizi for their help, support and useful discussions;
My research collaborators, Dr. Arun Seth, Stephanie Bacopulos, Wenyi Yang, for their valuable
help in my research;
My husband, Ali Tizghadam, for his constant love, support and encouragement;
My son, Pedram Tizghadam, for his patience over many years of my graduate studies;
My sister, Nooshin Shojaei for her endless support and love;
This thesis is dedicated to the memory of my beloved parents, Farkhondeh Mohaghegh and
Kamal Shojaei, whom I miss every day of my life. The completion of this work would not have
been possible without their unconditional support.
v
Table of Contents
Acknowledgments.......................................................................................................................... iv
Table of Contents .............................................................................................................................v
List of Tables ...................................................................................................................................x
List of Figures ................................................................................................................................ xi
List of Abbreviations .................................................................................................................. xvii
Academic Curriculum Vitae ........................................................................................................ xix
Statement of Contributions .......................................................................................................... xxi
1 Introduction and Background ......................................................................................................1
1.1 Background ..........................................................................................................................1
1.1.1 Breast Cancer ...........................................................................................................1
1.1.2 Breast Anatomy, Physiology ...................................................................................1
1.1.3 Cellular Biology .......................................................................................................3
1.1.4 DCIS and its progression to invasive and metastatic disease ..................................3
1.1.5 Histology and Histopathology .................................................................................3
1.1.5.1 H&E Stain .................................................................................................5
1.1.5.2 Other commonly used staining processes ..................................................6
1.1.6 Surgical Pathology ...................................................................................................6
1.1.7 Whole-mount vs Conventional Histopathology .......................................................8
1.1.8 Digital Histology ......................................................................................................9
1.1.9 3D Histology and its Significance .........................................................................10
1.2 Histology Volume Reconstruction .....................................................................................10
1.3 Image Registration .............................................................................................................11
1.3.1 Image Transformations ..........................................................................................12
vi
1.3.1.1 Rigid Registration ....................................................................................12
1.3.1.2 Affine Registration ..................................................................................12
1.3.1.3 Deformable Registration .........................................................................12
1.3.2 Image Registration Metrics ....................................................................................13
1.3.2.1 Point-Based ..............................................................................................13
1.3.2.2 Intensity-Based ........................................................................................13
1.3.2.3 Features-Based ........................................................................................14
1.3.1 Research Context ...................................................................................................14
1.3.2 Challenges ..............................................................................................................18
1.3.2.1 Sparse Sampling of Clinical Specimens ..................................................18
1.3.2.2 Artifacts in Histology Sections ................................................................19
1.3.2.3 Deformation of Breast Specimen ............................................................19
1.3.2.4 Validation of Histology Image Registration ............................................19
1.4 Hypothesis and Outline of this Thesis ...............................................................................20
2 A Pipeline for 3D Histology Volume Reconstruction ..............................................................21
2.1 Introduction ........................................................................................................................21
2.2 Material and Methods ........................................................................................................24
2.2.1 Specimen and Histology and .................................................................................24
2.2.2 Reference Images ...................................................................................................25
2.2.3 Alignment of Blockface Images ............................................................................26
2.2.4 Image Segmentation and Point Selection ..............................................................27
2.2.5 Fourier Descriptors (FDs) ......................................................................................28
2.2.6 Iterative Closest Points (ICP).................................................................................30
2.3 Results ................................................................................................................................30
2.4 Summary and Discussion ...................................................................................................33
3 Registration of Histology with Volumetric Images ..................................................................36
vii
3.1 Introduction ........................................................................................................................36
3.2 Requirements of the Image Registration Pipeline .............................................................38
3.2.1 Deformable Registration ........................................................................................38
3.2.2 Extendable for registration of histology to medical images ..................................39
3.2.3 Validation ...............................................................................................................39
3.3 Material and Methods ........................................................................................................39
3.3.1 Specimen ...............................................................................................................39
3.3.2 Reference Images ...................................................................................................40
3.3.3 A Multi-Modality Fiducial Marker ........................................................................40
3.3.4 Imaging ..................................................................................................................41
3.3.5 Histology ................................................................................................................42
3.3.6 Image Registration Pipeline ...................................................................................43
3.3.6.1 Segmentation, Point Selection, and Rigid Registration ...........................44
3.3.6.2 Deformable Registration .........................................................................47
3.3.6.3 Validation ................................................................................................48
3.3.6.4 Blockface image spacing .........................................................................49
3.4 Results ................................................................................................................................50
3.4.1 Fiducial Markers ....................................................................................................50
3.4.2 Blockface Volume .................................................................................................56
3.4.3 Reconstruction of a 3D histology volume from serial sections .............................56
3.4.4 Correlation of ex vivo MR to Histology ................................................................57
3.5 Summary and Discussion ...................................................................................................60
4 Optimization of Deformable Registration for Breast Histology Images ..................................63
4.1 Introduction ........................................................................................................................63
4.2 Materials and Methods .......................................................................................................65
4.2.1 Specimen ................................................................................................................66
viii
4.2.2 Dense SIFT Features ..............................................................................................67
4.2.3 Affine Registration.................................................................................................68
4.2.4 Deformable Registration ........................................................................................68
4.2.5 SIFTFlow Parameter Optimization ........................................................................69
4.2.6 Registration using Edge Points ..............................................................................71
4.2.7 Validation ...............................................................................................................72
4.3 Results ................................................................................................................................72
4.3.1 Affine Registration.................................................................................................72
4.3.2 Deformable Registration ........................................................................................74
4.3.2.1 Effect of regularization parameters on SIFTFlow registration ................74
4.3.2.2 Identifying paired landmarks for the assessment of deformable registration ...............................................................................................75
4.3.2.3 Comparing edge-based deformable registration with SIFTFlow. ...........77
4.3.3 Accuracy of optimized SIFTFlow on an independent test dataset. .......................80
4.4 Discussion ..........................................................................................................................80
4.5 Summary and Conclusion ..................................................................................................82
5 Breast Histology Volume Reconstruction and 3D Tumour Measurement ...............................83
5.1 Introduction ........................................................................................................................83
5.2 Materials and Methods .......................................................................................................85
5.2.1 Overview ................................................................................................................85
5.2.2 One Specimen and Multiple Tissue Blocks ...........................................................86
5.2.3 Specimen ................................................................................................................87
5.2.4 Alignment of the blockface images .......................................................................89
5.2.5 Outlining the Tumour Regions ..............................................................................90
5.2.6 Histology Image Registration ................................................................................90
5.2.7 Validation ...............................................................................................................90
ix
5.2.8 Tumour 3D Measurements ....................................................................................91
5.2.8.1 Volume ....................................................................................................91
5.2.8.2 Tumour Focality ......................................................................................91
5.3 Results ................................................................................................................................92
5.3.1 Alignment of the blockface images .......................................................................92
5.3.2 Histology Image Registration and Validation........................................................94
5.3.3 Tumour Volume and Multi-focality.......................................................................95
5.4 Summary and Discussion .................................................................................................100
6 Summary and Future Directions .............................................................................................102
6.1 Summary ..........................................................................................................................102
6.2 Future Improvements to the Image Registration Framework ..........................................104
6.2.1 Correlation at Higher Resolutions .......................................................................104
6.2.2 A Generic Modular Image Registration Software ...............................................104
6.3 Future Directions of Applications ....................................................................................105
6.3.1 Correlation of Different Disease Bio-Markers ....................................................105
6.3.2 Correlation of ex vivo Medical Images with Histology .......................................109
6.3.3 Correlation of in vivo Medical Images with Histology ........................................110
6.4 Conclusion .......................................................................................................................112
Bibliography ................................................................................................................................115
x
List of Tables
Table 3-1. Summary of the image registration techniques used at each step of the workflow H =
Histology image, B = Blockface image, MR = MR image ……………………………………..48
Table 3-2.Fiducial markers and their delectability in different imaging modalities ……………52
Table 3-3.Mean Absolute Target Registration Error and the Standard Deviation.rM = rigid
registration of MRI, rH = rigid registration of histology, B = blockface image, dM = deformable
registration of MRI, dH = deformable registration of histology ……………………………… 58
Table 4-1. Target Registration Errors for rigid and affine registration. Values are the mean TREs
(± stdev) based on implanted fiducial markers …………………………………………………72
Table 4-2 Average TREs for LOOCV experiment using edge-based, SIFTFlow with default
parameters, and SIFTFlow with optimized parameters measured using both fiducial markers and
manually selected landmarks. Values are given as mean (stdev) in mm. ………………………77
Table 4-3 TRE based on the internal structures of five histology sections from five different
patients with the default parameters (η = 0.005 α = 2.00) and the optimized parameters
(η=4.9x10-3, α = 1.169)...............................................................................................................80
Table 5-1. TRE based on the internal structures of five specimens from five different patients for
rigid and deformable registration ……………………………………………………..………..95
Table 5-2 Largest tumour volume,
calculated before and after alignment and after multi-modality process ……………..…..……100
xi
List of Figures
Figure 1-1: “Breast Anatomy Female”: For the National Cancer Institute © 2011Terese Winslow
LLC, U.S. Govt. has certain rights. (Image used by permission from Terese Winslow LLC). ...... 2
Figure 1-2: Duct and its transformation from normal to IDC. (Image used with permission from
Aegis Creative Communications, Inc.) ........................................................................................... 3
Figure 1-3: A part of 4μm thick histology section of breast stained with H&E showing different
structures of the breast. ................................................................................................................... 5
Figure 1-4. General block diagram of image registration workflow ............................................ 11
Figure 1-5.Transformations in Image registration: a) Source image. b) Rigid. c) Affine. d)
Deformable. .................................................................................................................................. 13
Figure 1-6 The coronal view of all ducts and their branches in an autopsy breast, reconstructed
by Going et al. (Going & Moffat, 2004). Each Roman numeral refers to a different independent
duct system. Some colours are used for more than one duct system (Image used by permission
from John Wiley & Sons, Ltd). ..................................................................................................... 15
Figure 1-7 Three-dimensional reconstruction of a nipple by Rusby et al. (Rusby et al., 2007).
Skin in tan, cut edge in yellow and ducts in purple. (A) the ducts are arranged in a central bundle
(B) the bundle narrows to a waist just beneath the skin (C) some ducts originate on the areola or
part way up the nipple (D) most ducts narrow as they approach the tip of the nipple (E) many of
the ducts have a common orifice (Image used by permission from Springer Science + Business
Media, LLC). ................................................................................................................................ 15
Figure 1-8 The selected 3D reconstruction of the subvolume in case B containing DCIS (yellow)
and surrounding invasion (orange). (A) and (B) show the DCIS and the surrounding invasion and
C, the DCIS alone. No continuity between in situ and invasive disease was identified. D shows
the lumina (green) within the DCIS. The lumina were mostly continuous throughout the ductal
xii
structure, with additional multiple small, unconnected lumina (Image used by permission from
John Wiley & Sons, Ltd). ............................................................................................................. 16
Figure 1-9 The scanned whole-mount images aligned manually using Sedeen viewer for volume
measurement. The reconstructed 3D tumour volume in following dimensions: (a) superior to
inferior, (b) medial to lateral (Image distributed under the Creative Commons Attribution
License). ........................................................................................................................................ 18
Figure 2-1 Histology volume reconstruction; histology images are registered to their optical
blockface images, which represent the shape of the specimen before it is cut into histology
sections. ......................................................................................................................................... 24
Figure 2-2. a) Blockface imaging setup with telecentric lens. b) a blockface image of a mouse
mammary gland with three 1mm holes drilled in three corners of the block. .............................. 26
Figure 2-3 Alignment of the blockface images using the registration holes. ............................... 27
Figure 2-4 Block diagram of histology volume reconstruction using boundary points ............... 29
Figure 2-5: a) blockface volume of the wild-type gland. b) histology volume of the wild-type
gland. c) blockface volume of Igfbp7-null gland. d) histology volume of Igfbp7-null gland ...... 31
Figure 2-6. 5x magnification of a cropped region from two aligned consecutive histology images
of the Igfbp7-null mammary gland. Due to large separation between slices structures are not
continuous but that the alignment process has worked as it is clear that the two images are clearly
from the same region. ................................................................................................................... 32
Figure 2-7. A region cut from four aligned consecutive sections at high resolution shows the
continuation of the 3D alveolar structures. Top) Wild-type. Bottom) Igfbp7-null gland. ............ 33
Figure 3-1. Overall pipeline for the histology volume reconstruction and registration of medical
images with histology. .................................................................................................................. 38
Figure 3-2 A mouse kidney with fiducial markers embedded in agarose..................................... 39
Figure 3-3. Images of a mouse tumour with validation fiducial markers: a) ex vivo MRI, b)
blockface image, c) Histology. ..................................................................................................... 42
xiii
Figure 3-4.a) Edge points of a histology image, b) Surface points of the MR volume, c) Surface
points of the Blockface volume. ................................................................................................... 44
Figure 3-5.The block diagram of the 3D histology volume reconstruction. ................................. 46
Figure 3-6.Eliminating pairs of blockface and histology images to increase the spacing. ........... 50
Figure 3-7. Mouse Kidney with Catheter filled with Cuttlefish ink & flour (left) and Liquid iron
& flour (right) ............................................................................................................................... 53
Figure 3-8. a) Optical image of a mouse tumour with three different fiducial markers: Black
Spaghetti, Catheter filled with bronze metallic paint and glue, Catheter filled with ground black
spaghetti and water. b) MR image of the same specimen scanned at 7T scanner. c) x-ray of a
mouse kidney with black spaghetti. .............................................................................................. 54
Figure 3-9. Top: mixture of bronze metallic paint and glue in catheter, a) blockface image b)
histology image. Bottom: black spaghetti in thigh muscle of a mouse, c) blockface image d) and
the histology image. ...................................................................................................................... 55
Figure 3-10. a) Blockface image, b) Blockface volume, arrows show three holes in the corners of
the paraffin tissue block ................................................................................................................ 56
Figure 3-11.a) cross-section of the blockface volume. b) same cross-section on histology volume
reconstructed using blockface images as a reference. c) same cross-section of the histology
volume reconstructed using pairwise registration without the blockface images. ........................ 57
Figure 3-12.a) Registered MR volume, b) blockface volume, c) histology volume. ................... 58
Figure 3-13.a) Registered MR image. b) Blockface image. c) Correspondent deformed Histology
image. ............................................................................................................................................ 59
Figure 3-14.Fitted lines to the centroid of the fiducial markers in 3D; a) rigid MRI in blue and
correspondent histology in red, b) deformed MRI in green and correspondent histology in red . 60
Figure 4-1 (a) 4mm slice of the fresh tissue. (b) Blockface image (90 angle) (c) Blockface
image (oblique angle) (d) H&E stained whole-mount image. ...................................................... 67
xiv
Figure 4-2 Flow chart outlining the joint parameter optimization method ................................... 70
Figure 4-3 Rigid registration of a histology image to the corresponding blockface image; a)
blockface image with boundary in blue (figure best viewed in colour). b) histology image with
manually defined boundary shown in red and blockface boundary in blue. Aligned histology
images: c) using edge-based method and d) using dense SIFT features. The boundary of the
blockface image is overlaid on both aligned histology images for comparison. .......................... 73
Figure 4-4 The deformation fields calculated with different settings for η and α are applied to a
grid image in order to visualize the effects of changing the regularization parameters. The top
row shows that increasing α results in smoother displacement: a) η =0.005, α =1. b) η =0.005, α
=2. The bottom row shows the effect of increasing η: c) η =0.0005,α =2 d) η=0.1, α =2. d was
fixed at the default value of 40 in all cases. .................................................................................. 74
Figure 4-5 Sensitivity of TRE (measured using the implanted fiducial markers) to the SIFTFlow
parameters: a) η, b) α, and c) d. .................................................................................................... 75
Figure 4-6 Two pairs of blockface and histology images with different sizes and composition.
Top row shows the a) blockface and b) histology images of a small section, which mostly
contains fat. Bottom row shows the c) blockface and d) histology images of a larger section with
more fibroglandular tissue and higher number of selected landmarks. The arrows on the
blockface images show the selected landmarks. ........................................................................... 76
Figure 4-7 Images from case # 04; a) Blockface image. b) histology registered using edge-based
method, the circle shows a missing part of the tissue section in histology image, which can
degrade the result of the registration. c) histology registered using default SIFTFlow parameters.
d) histology registered using optimized SIFTFlow. The arrows point to some of the areas that are
more accurately registered when the optimized parameters are used. .......................................... 79
Figure 5-1 Illustration of a) the misalignment of tissue slices, when they are embedded in
paraffin separately, b) the use of fiducial markers inserted in gel around the tissue for correcting
the alignment of the blockface images obtained from multiple tissue slices, d) the guiding device
for accurate insertion of the fiducial markers. .............................................................................. 87
xv
Figure 5-2. (a) The jig to guide the fiducial markers. (b) Tissue block fixed on a platform for CT
scan. (c) 4mm slices of the tissue block. ....................................................................................... 88
Figure 5-3. (a) CT image. (b) 4mm slice of the fresh tissue. (c) Straight blockface image. (d)
Angled blockface image. (e) H&E stained histology image. ........................................................ 89
Figure 5-4.Alignment of the blockface images using the fiducial markers in gel. Centre points of
the fiducial markers are shown in blue and the intersection points of the fitted lines in red. a)
Fiducial markers and the fitted lines before alignment, b) fiducial markers after alignment, c) top
view of the fiducial markers before registration, d) top view of the fiducial markers after
registration. ................................................................................................................................... 93
Figure 5-5. Comparison of the alignment of the fiducial markers in (a) blockface images with (b)
CT images. c) Segmented fiducial markers from CT images ....................................................... 94
Figure 5-6. Screen shot of Sedeen viewer software showing two outlined tumour regions. ........ 96
Figure 5-7 3D stack of the tumour regions. a) before registration. b) Overlay of the tumour
regions on the histology sections after alignment ......................................................................... 96
Figure 5-8. Verification of tumour focality using X-ray images of the tissue blocks. a) Histology
image with one non-overlapping tumour region shown with large arrow. b) X-ray image of the
correspondent block. c) The region with ambiguous connectivity in X-ray image, larger arrow, is
zoomed in and shows no connectivity, small arrow, between the regions. d) The same region in
the X-ray image of the previous block shows no connectivity either. e) A histology image from a
different block with one non-overlapping tumour regions shown with large arrow, f) the
correspondent X-ray image. g) The region with a non-overlapping region is zoomed in and
shows the connection, specified with a small arrow, of the isolated region to the main tumour
region. ........................................................................................................................................... 98
Figure 5-9. 3D tumour volume, a) original tumour regions. b) main tumour after removing the
non-overlapping tumour regions, c) after registration, d) after X-ray assessment. ...................... 99
Figure 6-1 Consecutive histology images with different stains. a) H&E Histology image. b) ER
Histology image .......................................................................................................................... 106
xvi
Figure 6-2 Fusion of the registered images, a) with Liu et al.’s parameters, b) with optimized
parameters. Arrow show the areas, which have been improved with optimized parameters. .... 107
Figure 6-3 Selected regions at high resolution (the images are downsized to fit in this space). a)
H&E region, b) ER region, c) deformed ER using Liu et al.’s parameters, d) deformed ER using
optimized parameters. e) fusion of H&E and the deformed ER using Liu et al.’s parameters, f)
fusion of H&E and the deformed ER using optimized parameters. Arrows mark some of the
places that the deformable registration has been improved by using optimized parameters. ..... 108
Figure 6-4. Correspondent histology (a) and CT (b) images of a human breast specimen. ....... 109
Figure 6-5 Study of the Impact of Tumour Focality on Surgical Management by Tot et al. (Tot &
Gere, 2016). (Image obtained from the open access article distributed under the Creative
Commons Attribution License)................................................................................................... 111
Figure 6-6. Registration of in vivo medical images with histology using the ex vivo images as
intermediate reference images. ................................................................................................... 112
xvii
List of Abbreviations
2D Two-dimensional
3D Three-dimensional
ADH Atypical Ductal Hyperplasia
BIRL Biomarker Imaging Research Laboratory
CT Computed Tomography
DCIS Ductal Carcinoma in Situ
DWI Diffusion Weighted Imaging
ER Estrogen Receptor
FD Fourier Descriptors
H&E Haematoxylin and eosin
ICP Iterative Closest Points
IDC Invasive Ductal Carcinoma
Igfbp7 Insulin-like growth factor-binding protein 7
IHC Immunohistochemistry
ILC Invasive Lobular Carcinoma
ITK Insight ToolKit
kV kilo Voltage (applied to X-ray tube)
LOOCV Leave One Out Cross Validation
xviii
MDA Metastatic Ductal Adenocarcinoma
MI Mutual Information
MRI Magnetic Resonance Imaging
mTRE Mean Target Registration Error
NBF Neutral Buffered Formalin
OCT Optical Coherence Tomography
PR Progesterone Receptor
SCID Severe Combined Immunodeficiency
Sedeen Selective Decoding and Encoding Engine
SIFT Scale Invariant Feature Transform
SRI Sunnybrook research Institute
stdev Standard Deviation
TDLU Terminal Ductal Lobular Unit
TPS Thin Plate Spline
TRE Target Registration Error
US Ultrasound
xix
Academic Curriculum Vitae
Peer Reviewed Publications
The work presented in this thesis is based on the following manuscripts:
Chapter 2:
Rushin Shojaii, Stephanie Bacopulos, Wenyi Yang, Tigran Karavardanyan, Demetri
Spyropoulos, Afshin Raouf, Anne Martel, and Arun Seth, “Reconstruction of 3-Dimensional
Histology Volume and its Application to Study Mouse Mammary Glands”, Journal of Visualized
Experiments (JoVE), (89), e51325, doi: http://dx.doi.org/10.3791/51325, (2014)
Chapter 4:
Rushin Shojaii, and Anne L. Martel, “Optimized SIFTFlow for Registration of Whole-Mount
Histology to Reference Optical Images”, SPIE Journal of medical Imaging (JMI),
doi:http://dx.doi.org/10.1117/1.JMI.3.4.047501, (2016)
Chapter 5:
Rushin Shojaii, Martin J. Yaffe, Anne L. Martel, “Breast Histology Volume Reconstruction from
whole-mount Slides and its Application in Characterization of 3D Features of Tumour” In
submission to Histopathology, (2016)
xx
Conference Presentations and Proceedings
The work presented in this thesis has been shared at the following conferences:
Chapter 3:
Rushin Shojaii, Tigran Karavardanyan, Martin Yaffe, and Anne L. Martel, “Validation of Histology Image Registration”, SPIE Medical Imaging, 2011.
Rushin Shojaii, and Anne L. Martel, “Multi-Modality fiducial marker for validation of registration of medical images with histology”, SPIE, Medical Imaging, 2010.
Rushin Shojaii, and Anne L. Martel, “3D Histology Volume Reconstruction”, Pathology Informatics, September 2010.
Rushin Shojaii, and Anne L. Martel “A Novel Edge Point Selection Method for Registration of Histology Images”, MICCAI (OPTIMHisE) Workshop, 2009.
Additional Collaborative Work:
Melissa L. Hill, Kela Liu, James G. Mainprize, Ronald B. Levitin, Rushin Shojaii, Martin J.
Yaffe, “Pre-clinical Evaluation of Tumour Angiogenesis with Contrast-Enhanced Breast
Tomosynthesis”, IWDM, LNCS 7361, pp. 1-8, 2012
xxi
Statement of Contributions
In chapter 2, I developed the 3D histology volume reconstruction methodology with the
supervision and guidance of my supervisor, Dr. Anne Martel. I also directed the preparation of
the image data in compliance with the developed methodology. Dr. Spyropoulos, Dr. Raouf, and
Dr. Seth were responsible for providing the preclinical animal models, and the biological and
histological findings were analyzed by Stephanie Bacopulos and Wenyi Yang under the
supervision of Dr. Arun Seth.
In chapters 3, 4 and 5, I developed the methodology, and conducted the experiments. The animal
specimen was obtained from Dr. William Tran in Dr. Czarnota’s lab. The human breast
specimens were obtained through a study conducted by Dr. Gina Clarke at Biomarker Imaging
Research Laboratory (BIRL) under the supervision of Dr. Martin Yaffe. All human tissue
processing was carried out by the BIRL, a clinically accredited facility. Image data (digitized
slides, X-ray and CT images) from human specimens was acquired by BIRL and Sam Shan in
Dr.Yaffe’s lab according to the specifications provided by me. For the work in these chapters, I
also developed the multimodality fiducial markers, designed a device to guide and implant the
markers and oversaw the construction of the device.
In Chapter 3 the Ultrasound images were acquired in Dr. Czarnota’s lab, MR images by Dr.
Stanisz’s lab and X-ray images by Dr. Melissa Hill in Dr. Yaffe’s lab.
The blockface imaging used in all chapters was done in collaboration with Mr. Tigran
Karavardanyan under the supervision of Dr. Martin Yaffe and Gord Mawdsley.
1
1 Introduction and Background
This Chapter will outline the motivation for the development of an automatic image registration
workflow for 3D histology volume reconstruction. An overview is given of breast cancer and the
role of pathology in diagnosis and prognosis. A general outline of 3D histology volume
reconstruction and its applications in pathology is also explained. An introduction is given to
different image registration techniques. It is proposed that image registration methods can be
employed to develop an automatic pipeline to accurately reconstruct histology volume from 2D
images of the whole-mount histology sections obtained from clinical specimens.
1.1 Background
1.1.1 Breast Cancer
Breast cancer is the most common cancer among Canadian women after non-melanoma skin
cancer (“Breast cancer statistics,” 2015). It is the second cause of cancer death for women after
lung cancer (“Breast cancer statistics,” 2015). Routine breast screening program with X-ray
mammography has significantly reduced mortality (“National Cancer Institute,” 2015). Breast
cancer is diagnosed through X-ray mammography along with ultrasound and magnetic resonance
imaging for cases with discordant imaging findings (Michell, 2010). There is ongoing research
into new techniques to detect and treat cancer in its earlier stages in order to increase the survival
rate. Cancer diagnosis is always confirmed by surgical pathology as the gold standard. Surgical
pathology is concerned with the diagnosis based on the gross (macroscopic) and histologic
(microscopic) findings and may involve immunohistochemistry for molecular examination of the
excised tissue or other laboratory tests (Weider, 2009).
1.1.2 Breast Anatomy, Physiology
Breast parenchyma (Figure 1-1) consists of 15 to 20 lobes, which drain secretions (milk) into the
ductal system that converges and opens into the nipple (Navaz, 2011). The functional unit of the
breast is the terminal duct lobular unit (TDLU), which is composed of the terminal (intralobular)
2
duct, and its ductules/acini (also referred to as lobules). The ducts, ductules, and lobular acinar
units, which compose the glandular tissue, are embedded within stroma that is composed of
fibrous (ligaments) and adipose (fat) tissue to support the glandular tissue and build the shape of
the breast (Collins & Schnitt, 2007), (Navaz, 2011). The relative proportions of fibrous and
adipose tissue vary with age and among individuals (Collins & Schnitt, 2007). The ductal-lobular
system of the breast is arranged in the form of segments, or lobes. Each ductal-lobular unit
consists of a branching structure that is similar to a flowering tree (Collins & Schnitt, 2007). The
lobules represent the flowers (Figure 1-1), draining milk into ductules and ducts (twigs and
branches), which, in turn, drain into the collecting ducts (trunk) that open onto the surface of the
nipple (Collins & Schnitt, 2007). Figure 1-1also shows a part of the breast lymphatic system,
which drains the lymph fluid from breast.
Figure 1-1: “Breast Anatomy Female”: For the National Cancer Institute © 2011Terese
Winslow LLC, U.S. Govt. has certain rights. (Image used by permission from Terese
Winslow LLC).
3
1.1.3 Cellular Biology
The entire ductal system, extralobular large and intermediate ducts, as well as the intralobular
(terminal ducts and ductules/lobules), is lined by two cell layers: inner epithelial cells and an
outer layer of myoepithelial cells surrounded by basement membrane (Navaz, 2011) as shown in
Figure 1-2-a for a normal duct.
1.1.4 DCIS and its progression to invasive and metastatic disease
Cancer arises primarily in the TDLUs as an abnormality in the epithelial cells. This abnormality
results in proliferation of epithelial cells leading to hyperplasia (Jansen, 2012). The intraductal
proliferations are classified as usual or atypical ductal hyperplasia (ADH) (Figure 1-2-b and c).
ADH is differentiated from usual hyperplasia by observing some atypical cells (Hoda, Brogi,
Koerner, & Rosen, 2014). An early stage tumour that is still confined within a mammary duct is
known as ductal carcinoma in situ (DCIS) (Figure 1-2-d) and eventually, some DCIS becomes
invasive ductal carcinoma (IDC) (Figure 1-2-e) by invading the basement membrane and
infiltrating into the stroma (Jansen, 2012), (Hoda et al., 2014). IDC can subsequently create
distant metastases by cancer cells leaving the breast via the lymphatic or vasculature system and
invading other organs.
1.1.5 Histology and Histopathology
Histology is the microscopic study of the anatomy of tissue at cellular level, while
histopathology is the microscopic study of diseased tissue. Histopathological examination is
routinely performed on biopsies and surgical tissue to provide diagnostic information. In some
cases they also provide information for treatment planning and prognosis.
a) b) c) d)
Figure 1-2: Duct and its transformation from normal to IDC. (Image used with permission from Aegis Creative Communications, Inc.)
4
After gross examination, the excised tissue undergoes a series of histological processes to be
observed with light or electron microscopes. The specimen is first bread loafed into 4mm slices
to make it more exposed to the chemical solutions and to increase the diffusion of the chemical
solutions to the tissue (Sun, Wang, Zubovits, Yaffe, & Clarke, 2009). The tissue slices then
undergo different steps of histological processes, which are briefly explained below (Pakurar &
Bigbee, 2004):
Chemical Fixation is the first step of the histological processes, which preserves the
cellular structure and maintains the distribution of the organelles. One of the most
commonly used chemical fixatives is Formaldehyde, which is also referred to as
Formalin in neutral buffered solution. It stabilizes protein by forming cross-link
between primary amino groups.
Dehydration of the tissue is the next step. Tissue water is not miscible with the
embedding solution. The purpose of dehydration of tissue is to replace tissue water
with alcohol at a series of increasingly higher concentrations. The alcohol is then
replaced by an intermediate solvent, which is miscible with both alcohol and
embedding solution.
Infiltration and embedding of the tissue are the two steps which replace the
intermediate solvent with the embedding compound (e.g. paraffin wax or epoxy
plastic) and when the embedding medium is solidified in a mold a tissue block is
built. Infiltration and embedding provide rigidity to the tissue and prepare it for the
next step, which is sectioning the tissue block.
Sectioning of the tissue involves cutting thin sections of the tissue block. The sections
should be thin enough to allow light or electrons to pass through. The thickness of the
sections is between 1μm to 20 μm for light microscopy and 60 μm to 100μm for
electron microscopy.
Staining is one of the important steps of histological processing, which provides contrast to the
tissue section for visualization. The choice of staining dyes depends on the structures to be
visualized (Pakurar & Bigbee, 2004). Some common staining mechanisms are explained in the
next section.
5
1.1.5.1 H&E Stain
Figure 1-3 demonstrates the microscopic appearance of different structures of breast in a 4μm
thick histology section stained with hematoxylin and eosin (H&E). Hematoxylin and eosin are
the most commonly used stains in histology and histopathology for light microscopy.
Hematoxylin binds to the negatively charged basophilic tissue components, mostly nucleic acids
in cell nuclei, and colours them purple-blue. On the other hand Eosin binds to the positively
charged eosinophilic components such as cytoplasm, stroma, and muscle fibers and colours them
pink. Red blood cells are stained red by eosin (Pakurar & Bigbee, 2004).
Collagen
Normal Duct
Blood vessel
Fat
100μm
Ductal Hyperplasia
Normal Ducts
Figure 1-3: A part of 4μm thick histology section of breast stained with H&E showing different
structures of the breast.
6
1.1.5.2 Other commonly used staining processes
Histochemistry localizes chemical groups by using chemical reactions between laboratory
chemicals and different tissue components (Pakurar & Bigbee, 2004).
Immunohistochemistry (IHC) localizes specific antigens by using labeled antibodies. It uses the
principal of the antibody-antigen interaction to visualize the location of the antigens.
In situ hybridization uses labeled nucleotide probes for detection of messenger RNA or genomic
DNA.
Electron microscopy requires staining the tissue sections with salts of heavy metals (i.e. lead
citrate and uranyl acetate). These metals bind to the negatively charged areas and block the
passage of the electrons through the tissue and create a dark area in the electron micrograph. The
structures which bind the metals are electron dense and the structures which do not bind to the
metals are referred to electron lucent.
1.1.6 Surgical Pathology
Biopsies and surgical resections are the main types of specimens which are submitted for
surgical pathology. Surgical pathologists examine the stained section to provide an accurate,
specific, and sufficiently comprehensive diagnosis to enable the clinician to come up with an
optimal treatment plan and, to some extent, estimate prognosis (Connolly, Schnitt, Wang, & Al.,
2003). They complete a gross examination of the fresh excised tissue followed by a thorough
examination of the stained tissue sections in a microscope. At the time of excision the specimen
is marked with clips or sutures to provide anatomic orientation.
The gross examination of the specimen includes the description of its appearance and
characteristics. The specimen is measured in three dimensions. If the lesion is visible, it is also
measured in three dimensions along with the distances between the lesion edges and the excision
margins. Excision margins are marked with ink prior to any dissection, permitting accurate
measurement of these distances later in microscopic examination of the sections. All lymph
nodes associated with a cancer specimen are dissected out, described along with their location,
and processed for histology. In addition, breast specimens with calcification often require
specimen radiography of each breadloaf of tissue. The tissue slices are then coded and correlated
7
with radiographs. After grossing, the tissue slices undergo histological processes for microscopic
examination.
Histology sections are examined by the surgical pathologist and, based on the presence or
absence of cellular abnormalities or invasion they decide whether a neoplasm exists. The
pathologist then classifies the neoplasm based on its origin accompanied by other descriptors
such as the gross and microscopic appearance, the degree of cellular differentiation, and benign
or malignant behavior. The tumours with epithelial cell origin are called adenomas or papillomas
when benign and carcinoma when malignant. The carcinomas originating from glandular
epithelium (e.g. in breast) are termed as adenocarcinoma, and squamous cell carcinoma when
from squamous type of epithelium.
Several different types of tumors may arise within an organ, each with its own special
characteristics, prognosis, and therapy response. Infiltrating ductal carcinoma, which accounts
for ~80% of infiltrating breast cancers, and infiltrating lobular carcinoma, which accounts for
~9% of breast cancers, are the two most common types of malignant breast cancer. These two
types of breast cancer have poorer prognosis than other less common types of breast carcinoma
(i.e. tubular, mucinous or colloid, medullary, papillary, and adenoid cystic carcinomas)
(Connolly et al., 2003).
Malignant epithelial tumors that have not extended through the underlying basement membrane
are reported as in situ carcinomas and are potentially cured by complete excision. Malignant
tumor cells that invade lymphatics and veins and travel by lymph or blood flow to distant sites
can cause metastasis and are reported as metastatic carcinomas.
One of the other important tasks of surgical pathologists is to examine the resection margins to
determine whether resection margins are adequate following the surgery. For some tumours the
margins are evaluated based on the frozen section, which can be prepared rapidly while the
patient is still in surgery, so that additional tissue can be removed immediately. As they arise in
soft tissue, the surgical margins for breast tumours are best evaluated in paraffin embedded
sections. It is difficult to obtain good results when cutting sections from frozen tissues containing
fat, such as breast. Sections cut from paraffin embedded tissue, on the other hand, are thinner
with better quality and provide greater certainty of interpretation. If resection margins are
inadequate then the patient is sent for re-excision. In the case of a discrete mass the margins are
8
easily evaluated, but in the case of diffusely infiltrating tumours (e.g. infiltrating lobular
carcinoma) the margins are hardly defined and more difficult to assess. Usually factors such as
the extent of intraductal growth must also be considered when evaluating resection margins
(Connolly et al., 2003).
Another important role of surgical pathologists is in staging, tumour grading, and receptor status,
which has a great value in treatment planning and prognosis. Staging represents the extent of
cancer based on tumour size, lymph node involvement, and the presence of metastasis. Tumour
grading defines the degree of differentiation of the tumour cells and their growth rate and is
based on the pathologist’s judgment while evaluating tumour-specific factors (e.g. mitotic count,
tubule formation). The pathologist also evaluates the expression of cell receptors and antigens
such as estrogen (ER) and progesterone (PR) receptors, and Her2/nue cell surface antigen. The
status of the receptors has a great impact in treatment planning and prognosis (Jansen, 2012),
(Connolly et al., 2003).There are on-going studies on identifying markers to predict the patient
response to different treatments.
1.1.7 Whole-mount vs Conventional Histopathology
Conventional histopathology is based on small rectangles of tissue, roughly 15 x 15 mm in size,
cut from the breadloafed resection specimen. The samples are taken from the tumour regions
during the gross pathologic assessment and the location of each sample is based on the
pathologist’s or pathology assistant’s judgment. One of the limitations of conventional
histopathology is that the specimen is under-sampled and positive margins or multiple foci of
tumour might not be detected (Gupta, Nath, & Layfield, 2003), (Gina M Clarke, Peressotti,
Mawdsley, & Yaffe, 2006). Furthermore, it is difficult to determine the orientation of sections
obtained from the separate samples with respect to each other and to the original 3D specimen.
In order to overcome these limitations, Clarke et al (Gina M Clarke et al., 2006) developed a
method of embedding entire breadloafed resection specimens in paraffin and then cutting whole-
mount sections form the resulting block in order to provide cross-sections through the whole
specimen. The orientation of tumour regions is intact and surgery margins, and multifocality can
be assessed more precisely in 2D whole-mount sections (Gina M Clarke et al., 2006), (Sun et al.,
2009). Microscopic measurement of tumour size, which is used in staging, can also be more
9
accurate in 2D (maximum extent of tumour) using whole-mount sections and also in 3D (tumour
volume) using reconstructed histology volume from whole-mount sections (G M Clarke et al.,
2012).
Tumour size is one of the strong predictive factors of breast cancer survival and is used for
staging of the disease. It is conventionally measured as the maximum linear extent of the tumour
based on the 2D histology sections from the sampled tissue, which might not represent the whole
tumour extent due to under-sampling. Although the use of whole-mount histology sections
improves the 2D measurement of the maximum tumour diameter, tumour volume may still be a
more accurate surrogate for tumour size (G M Clarke et al., 2012). Tumours with different
shapes can have the same 2D extension, but their volumes might be different. It has been shown
(G M Clarke et al., 2012) that tumour volume estimated based on the maximum tumour diameter
overestimates the volume compared to 3D volumetric calculations.
Multimodality is defined as two or more foci, which are separated by intervening normal breast
tissue (Coombs & Boyages, 2005). Multifocality is another important surrogate for prognosis
and is associated with local recurrence (Leopold et al., 1989) and distant metastasis (Pedersen,
Gunnarsdottir, Rasmussen, Moeller, & Lanng, 2004), (Weissenbacher et al., 2010).
Multifocality also affects the assessment of the tumour extent, which is currently based on the
size of the largest foci (Coombs & Boyages, 2005). The total tumour burden predicts the
tendency of breast tumour to metastasize and ignoring the additional tumour burden from small
foci may under-stage the disease (Coombs & Boyages, 2005). Whole-mount histopathology
reduces the problem of losing small tumour foci due to under-sampling the tumour in
conventional histopathology and therefore improves the detection of multiple tumour foci in 2D
sections. The problem of determining whether separate foci seen in one 2D section are actually
part of a single tumour is not resolved by the use of 2D whole-mount histology however.
1.1.8 Digital Histology
Digital pathology was created as a tool to educate students in histology courses (Pakurar &
Bigbee, 2004). It started as a series of digitized micrographs and evolved rapidly into its present
form, which includes the digitization and visualization of all types of stained tissue sections. The
10
images, which are referred to as virtual slides, can be zoomed in and out to simulate the use of a
microscope. Digital pathology is used for both as educational tool at the universities and as a
substitute for traditional microscope laboratories in clinic. It also facilitates the transfer of slides
between different institutes.
1.1.9 3D Histology and its Significance
Although whole-mount histology has made the microscopic assessment of the specimen easier in
two dimensions, the problems with the accurate orientation of the tissue sections with respect to
each other and to the whole intact specimen and the uncertainty in multifocality are still present.
3D histology volume reconstruction has the potential to vastly improve gross pathology
assessments in the clinic. Tumour size, including maximum tumour diameter and tumour volume
may be more accurately measured in 3D vs 2D. Visualization of 3D histology volume can also
help the pathologist when they report the orientation of the tumour or when investigating the
spatial relationship between the distributions of different disease biomarkers.
Another important application area for 3D histology is in radiology-pathology correlative
studies. There is ongoing research into the development of new volumetric medical imaging
techniques for early detection of cancer. These new techniques are often validated by correlating
the acquired medical images with histology images of tissue specimens obtained from the
corresponding location after imaging. 3D histology volume reconstruction is an important
intermediate step in co-registering histology findings with medical images acquired using 3D
modalities such as MRI or ultrasound.
1.2 Histology Volume Reconstruction
3D histology volumes are reconstructed from 2D images of histology sections. While the
histology sections are being mounted on microscope slides, they will be shifted and rotated with
respect to each other. Furthermore, the sections might expand and deform while they are being
de-wrinkled in hot water bath. In addition to all of these transformations, the tissue sections can
tear and fold, which introduces artefacts. All of these transformations mean that reconstructing a
3D histology volume is far more complicated than simply stacking the series of 2D images.
Manual correction and alignment of histology sections is very challenging and may not even be
possible. Image registration techniques can be employed to automatically align and correct the
11
2D images of the whole-mount histology sections so that they can be used to reconstruct a 3D
histology volume.
1.3 Image Registration
Image registration is the process of aligning two or more images, which contain similar objects.
In image registration a source image, in this case a histology image, usually undergoes a series of
geometric transformations to match a target or reference image, such as another histology or
medical image. Image registration, as shown in Figure 1-4, is basically an optimization problem
that finds a transformation by minimizing or maximizing a cost function or a distance measure
(e.g. distance between specific points or landmarks, difference of pixel intensities or image
features) between the two images. The distance measure is also referred to as image registration
metric. The type of the transformation and the distance measure specifies the type of the image
registration.
Source Image
Target Image Metric Optimizer
Transform Interpolator
Distance measure
Transform
Parameters
Transformed
Image
Output Parameters
Initial Transform
Figure 1-4. General block diagram of image registration workflow
12
1.3.1 Image Transformations
1.3.1.1 Rigid Registration
Rigid registration finds the global translation and rotation of the source image with respect to the
target image and can therefore correct for the shifts and rotations that occur when tissue sections
are positioned on the microscope slides. This transformation does not change the size and shape
of the objects in the source image (Figure 1-5-a,b). The rigid transformation can be written as a
matrix equation as shown below:
=
cos − sinsin cos0 0 1 1
=cos − sin +sin + cos +
1
Where T is the transformation matrix, θ is the rotation angel and tx and ty are the translation
values in x and y directions. With this transformation all pixels, p(x,y), are rotated and translated
with the same values.
1.3.1.2 Affine Registration
In affine image registration global scaling (i.e. shrinkage and expansion) and global skewing are
also allowed, but parallel lines remain parallel in the transformed image (Figure 1-5-c. Affine
registration of histology images is therefore suitable for correction of shrinkage and expansion of
the tissue sections. The matrix equation for affine transformation is similar to the rigid
transformation with the addition of scale (S) and skew (K) factors to the matrix.
cos − sinsin cos0 0 1
.1
=cos − sin +sin + cos +
1
1.3.1.3 Deformable Registration
Deformable or non-rigid registration corrects the local deformation of the source image.
Deformable registration finds a displacement field or a deformation map for the pixels of the
source image and can vastly change the shape of the objects in the source image (Figure 1-5).
Local deformation of tissue sections can be corrected in the histology images by using
13
deformable registration techniques. There are different approaches to model the local
deformation of tissue. Deformable registration is discussed more in chapter 3.
1.3.2 Image Registration Metrics
Different distance measures based on points, intensity, and features can be used to find the
parameters for rigid, affine, and deformable transformation (i.e. translation, rotation, scale, skew,
and pixel displacement) of the histology images.
1.3.2.1 Point-Based
In point-based registration techniques the Euclidean distance between correspondent points is
minimized to find the registration parameters. These points can be boundary points, corner points
or landmarks which are detectable in both source and target images based on the internal
structures of tissue.
1.3.2.2 Intensity-Based
Intensity differences of the overlapping pixels in the source and target images define another
category of distance measures for image registration. One of the metrics in this category is
Mutual Information (MI), which is commonly used in registration of images obtained from
different modalities with different intensity variations (Mattes, Haynor, Vesselle, Lewellen, &
Eubank, 2001). This metric is based on the similarity of the information measured for source and
target images individually.
x
y
x
y
x
y
x
y
a) b) c) d)
Figure 1-5.Transformations in Image registration: a) Source image. b) Rigid.
c) Affine. d) Deformable.
14
1.3.2.3 Features-Based
The difference between corresponding features extracted from the source and target images
drives another group of image registration algorithms categorized as feature-based registration
methods. Edges and blob-like structures are examples of image features, which can be used for
registration of histology images.
1.3.1 Research Context
In this work, we are particularly interested in automated volume reconstruction for breast whole-
mount histology which has not been previously reported. The highly deformable nature of breast
tissue and the lack of clear boundaries between different tissue types make this a particularly
difficult problem.
Going et al. studied the 3D anatomy of breast ducts and measured the volumes of all 20 complete
duct systems (lobes) by reconstructing the 3D volume of all TDLUs, Figure 1-6, from the
histology images of the 2mm serial sections obtained from an autopsy breast (Going & Moffat,
2004). They traced parenchymal elements from enlarged photographic prints onto transparent
acetate sheets. In order to reconstruct the histology volume of the ductal system they stacked the
printed sheets and manually identified the connected duct segments and trace every duct system
from nipple to periphery, slice by slice.
In order to study the number, diameter and structure of ducts in nipples, Rusby et al.
reconstructed the histology volume of a nipple from the H&E stained 5µm sections with 45µm
separation (Rusby, Brachtel, Michaelson, Koerner, & Smith, 2007). They aligned histology
images by eye in corroboration with microscope examination of the original section,
Figure 1-7.
15
Figure 1-6 The coronal view of all ducts and their branches in an autopsy breast, reconstructed by Going
et al. (Going & Moffat, 2004). Each Roman numeral refers to a different independent duct system. Some
colours are used for more than one duct system (Image used by permission from John Wiley & Sons,
Ltd).
Figure 1-7 Three-dimensional reconstruction of a nipple by Rusby et al. (Rusby et al., 2007). Skin in tan,
cut edge in yellow and ducts in purple. (A) the ducts are arranged in a central bundle (B) the bundle
narrows to a waist just beneath the skin (C) some ducts originate on the areola or part way up the nipple
(D) most ducts narrow as they approach the tip of the nipple (E) many of the ducts have a common orifice
(Image used by permission from Springer Science + Business Media, LLC).
16
Booth et al. (Booth et al., 2015) investigated the use of an automated 3D reconstruction of breast
tissue sections in the study of the biology of DCIS. The sections were cut from a small sample of
breast tissue around the DCIS. They reconstructed the histology volumes, shown in Figure 1-8,
by registering the neighbouring sections to the middle section as reference image (Roberts et al.,
2012) and this relies on having consecutive 4µm thick sections which means that only small
tissue volumes can be processed in this way. There is a much greater separation (i.e. 4-5mm)
between sections from clinical specimens, which highly reduces the similarity between the
sections.
Figure 1-8 The selected 3D reconstruction of the subvolume in case B containing DCIS (yellow)
and surrounding invasion (orange). (A) and (B) show the DCIS and the surrounding invasion and
C, the DCIS alone. No continuity between in situ and invasive disease was identified. D shows
the lumina (green) within the DCIS. The lumina were mostly continuous throughout the ductal
structure, with additional multiple small, unconnected lumina (Image used by permission from
John Wiley & Sons, Ltd).
17
Reis et al. (Reis et al., 2015) investigated the maximum spacing between the breast histology
sections required for 3D histology reconstruction to a specific tolerance. They focused on the
accuracy of reconstruction of individual ductal structures that were extracted from the whole
slide. They also selected the section at the centre of the block as the reference image and used
pair-wise rigid or affine registration approach to align the sections to the reference image. They
concluded that a spacing of 41µm between sections was sufficient to assess ductal structures,
however, in the clinical setting the spacing between the breast sections is about 4mm and
methods that rely on slice to slice registration will likely fail.
All these works show that 3D reconstruction is required for the study of the normal and disease
breast at the cellular level. These methods are either laborious or apply to tissue sections which
are cut from small samples of the excised breast tissue with a very fine separation between the
sections. Clarke et al developed a tissue preparation method for producing whole-mount
histology sections from whole lumpectomy and mastectomy specimens (Sun et al., 2009) and
then manually aligned the histology images to reconstruct the breast histology volume, shown in
Figure 1-9, in order to compare 3D with 2D tumour measurements (G M Clarke et al., 2012). A
3D reference image obtained using MRI or CT can be used as a reference volume for 3D
histology volume reconstruction if it is available (Grégoire Malandain, Bardinet, Nelissen, &
Vanduffel, 2004); in breast Clarke et al. (G. M. Clarke et al., 2008) demonstrated that manually
identified landmarks could be used to align a whole-mount lumpectomy histology image with a
specimen cone-beam CT slice. All these examples reveal that there is a need for an accurate
automatic image registration approach for reconstruction of the histology volumes from 2D
whole-mount histology images of breast specimens.
18
Figure 1-9 The scanned whole-mount images aligned manually using Sedeen viewer for volume
measurement. The reconstructed 3D tumour volume in following dimensions: (a) superior to
inferior, (b) medial to lateral (Image distributed under the Creative Commons Attribution
License).
1.3.2 Challenges
1.3.2.1 Sparse Sampling of Clinical Specimens
A few millimeters separation between the histology sections obtained from clinical specimens
reduces the similarities of the tissue structures on the consecutive sections. These tissue
structures include the cross-section of the blood vessels and fibroglandular tissue. Since image
registration techniques work based on the similarities between the images, this low similarity
between the sections makes the registration of the histology images more challenging.
19
1.3.2.2 Artifacts in Histology Sections
Histology artifacts can occur during the histology processing and these change the appearance of
the sections. One of these histology artifacts is shrinkage, which occurs during dehydration of
the tissue. Intensity variation is another artifact, which can happen in the staining step. Different
patches of slides are stained at different times and depending on the freshness of the chemical
used at this step the stain might be stronger or weaker. Tears and folds are the artifacts that occur
when the sections are cut and mounted on microscope slides. Usually sections with bad tears or
folds are not kept and the tissue block is re-sectioned, but small tears and folds are to some
extent acceptable. Digitizing artifacts, such as tiling and stripes, can occur during scanning and
the appearance of these artifacts will depend on the type of scanner used, although as scanner
technology improves, these artifacts are becoming less common. Having artifacts in histology
sections reduces the similarity between the sections and makes the manual or automatic
correlation of the consecutive sections more difficult.
1.3.2.3 Deformation of Breast Specimen
Human breast is mostly composed of fat, which is highly deformable. Therefore, during the
tissue preparation the changes in conformation of the breast specimen are much greater
compared to other more rigid organs (e.g. prostate). The highly deformable nature of breast
makes the alignment and deformable registration of the breast tissue sections more challenging.
1.3.2.4 Validation of Histology Image Registration
Image registration is usually assessed by overlaying the two histology images and visualizing or
calculating the distance between the similar structures. Due to the great separation (4-5mm)
between the consecutive histology sections cut from a clinical specimen, similar structures might
not be detectable in both images. Using manually selected landmarks based on the internal
structures in the serial histology images also suffers from the lack of similarity between the
sections. Furthermore, depending on the shape of the specimen, a reconstructed histology
volume with the perfectly aligned internal structures might not represent the original
conformation of the tissue. A reconstructed histology volume can also be validated by comparing
it with the ex-vivo medical images of the specimen obtained by volumetric imaging modalities
such as MRI or CT. The problem with this approach is that the ex-vivo imaging of the specimen
is not always feasible for clinical specimens with the limited cold ischemic time, which is
20
calculated from when the tissue is removed from the body to when it is placed into fixative. The
ischemic time should be less than one hour to prevent it from altering the staining intensity of
IHC. Another approach for validating histology image registration is to implant fiducial markers
in the excised tissue and use them as landmarks in histology images for calculating the
registration error. Implanting fiducial markers in the specimen might cause distortion in the
tissue, and is not usually permitted for clinical specimens. Validation of the histology volume
reconstruction is therefore a challenging step.
1.4 Hypothesis and Outline of this Thesis
This work is motivated by the hypothesis, which stated as follows:
Accurate 3D Histology volumes of breast specimens can be automatically reconstructed from 2D
digital images of whole-mount histology slides. These reconstructed volumes have the potential
to assist surgical pathologists with more precise assessment of tumour extent in 3D.
This hypothesis is investigated in four parts in this thesis. Chapter Two introduces the image
registration pipeline developed for 3D histology volume reconstruction. Chapter Three presents
the extension of the pipeline for co-registration of histology with volumetric MRI. This chapter
also includes the introduction of a multi-modality fiducial marker for validation of the image
registration pipeline. Chapter Four describes how a deformable image registration method can be
optimized to correct for breast tissue deformation when registering histology images with respect
to their correspondent blockface images. Chapter Five shows the efficacy of using these
optimum parameters for histology volume reconstruction of different breast specimens and
applies the method to carry out 3D tumour measurement. Chapter Six summarizes the results and
contributions of the work presented in this thesis and highlights some future research
applications in the field of breast histopathology.
21
2 A Pipeline for 3D Histology Volume Reconstruction
Chapter two introduces a novel image registration workflow for 3D histology volume
reconstruction, which overcomes some of the challenges described in chapter 1. This chapter
also demonstrates the utility of the developed methodology in pre-clinical studies. The image
registration workflow is used in a pilot study to characterize the deficiency of Igfbp7 in mouse
mammary glands. 3D-reconstructions of mammary glands were generated for mutant and wild
type mice in order to compare the size and structures of the two.
2.1 Introduction
Histology sections are shifted, rotated, and deformed relative to the original tissue block.
Therefore the 3D conformation of the specimen is lost if the 2D images of the histology sections
are simply stacked up to rebuild the histology volume. Assessment of tumour features in 3D
requires an accurate and realistic histology volume. Accurate reconstruction of the histology
volume is also important in finding and correlating the correspondent histology images to the
slices of volumetric medical imaging. In order to reconstruct the histology volume it is necessary
to find the transformations, which bring the sections back into alignment and correct the
deformation of the sections.
A common approach to histology volume reconstruction consists in the pair-wise registration of
consecutive sections, but this method is typically unable to recover the overall shape of the
reconstructed volume (Malandain et al. 2004) unless additional constraints are applied. This is
true when there is a large separation between the histology sections, which greatly reduces the
similarities between the consecutive sections. Another problem of only registering two slices at a
time is that registration errors may be propagated throughout the volume. If one section is mis-
registered, the next section, which is being registered to this one will be also mis-registered and
22
in the same way the error propagates to the rest of the sections. Methods that take into account
multiple sections simultaneously or add regularization terms have been proposed in order to
overcome these problems. For example, smoothness of the structures of the brain has been used
as a constraint by Cifor et al. (Cifor, Bai, & Pitiot, 2011) to guide the 3D histology volume
reconstruction of the brain. Cahill et al. (Cahill, Noble, & Hawkes, 2010) attempted to regularize
a nearly rigid descriptor feature transformation to be consistent across the neighbourhood
sections. Christensen et al. (Christensen &Johnson, 2001) also estimated the forward and reverse
transformations jointly, and produced a consistent transformation that has low pairwise
registration errors. All of these approaches require that consistent anatomical structures are
visible on neighbouring sections and that their location in the volume can be approximated by a
smoothly varying function. This generally means that the separation between sections must be
small enough for such structures to be tracked and in clinical applications this may require an
impractically large number of sections to be prepared. In order to overcome this problem several
groups (Bardinet et al., 2002; Chakravarty, Bertrand, Hodge, Sadikot, & Collins, 2006; Dauguet
et al., 2007) have employed 2D images of the tissue block as a reference to improve the accuracy
of 3D histology reconstruction. A blockface image is a photographic image of the surface of the
tissue block acquired immediately before the desired section is cut. Since the tissue block is
relatively rigid and is mounted into a microtome during the sectioning process, these blockface
images act as a reference which preserves the conformation of the anatomical structures of the
tissue.
To reconstruct the histology volume, a 2D to 2D image registration is required to register each
tissue section to a reference image regardless of whether a blockface image or another histology
section is used as the reference. Various manual, semi-automatic, and automatic image
registration techniques have been applied to 2D images.
Manual approaches, such as (Deverell et al., 1993), are laborious and time consuming and the
registration accuracy is operator-dependent. Semi-automatic and automatic approaches may
include intensity-based and feature-based methods. Implanted fiducial marker methods have also
been investigated but these may cause tissue distortion. Although a wide range of intensity-based
methods (Kim, Boes, Frey, & Meyer, 1997; Ourselin, Bardinet, Dormont, Malandain, & Roche,
2001) can be used to register two 2-D images, the differences in the appearance of the histology
images and the reference images makes intensity-based approaches challenging. Even when both
23
the section to be registered and the reference image have been stained with the same reagent,
intensity differences still have to be taken into account. These differences make the localization
of the intrinsic landmarks and their correspondences difficult (Zhan et al., 2007), but landmark-
based registration techniques may be more robust. Feature-based methods consist in
automatically extracting features such as points (Guest, Berry, Baldock, Fidrich, & Smith, 2001),
edges (Kim, Frey, Mukhopadhyay, Ross, & Meyer, 1995) or contours (Cohen, Yang, Huang, &
Nissanov, 1998;Weizhao, Young, & Ginsberg, 1993) from the input slices and using them to
guide the 2-D alignment. The accuracy of the registration is dependent upon the accuracy of the
feature extraction methods. These features include boundary points (Jacobs, Windham,
Soltanian-Zadeh, Peck, & Knight, 1999; Weizhao et al., 1993) and/or the features from the
internal anatomical structures. Biomechanical model based methods have been developed for
registration of histology to MRI of prostate (Samavati et al., 2011) by assigning specimen-
specific material properties obtained using an ex vivo quasi-static magnetic resonance
elastography (MRE) technique. This is a very complex technique and it has not been validated on
breast tissue. Although several works have been done to measure the hyperelastic parameters of
various types of breast tissues, further work and statistical analysis is required to obtain accurate
parameters that can be used in reliable correction of breast tissue deformation (Samani & Plewes,
2004) (Samani, Zubovits, & Plewes, 2007).
Because the boundaries of the pre-clinical specimens used in this work are well detectable in
histology, edge-based registration techniques were used in this work to reconstruct the histology
volumes of the specimens. One of the advantages of using boundary points for image registration
is that it makes the pipeline independent from tissue type and also independent from histology
stain as long as the tissue boundary is well detectable. Furthermore, edge-based methods are
more efficient than intensity-based approaches especially when working with large histology
images.
This chapter introduces an image registration workflow for histology volume reconstruction.
Figure 2-1 shows the overall pipeline. Histology images are registered to their correspondent
blockface images to reconstruct a histology volume. The stack of segmented tissue regions in
blockface images represent the shape of the specimen before it is cut into histology sections.
24
2.2 Material and Methods
2.2.1 Specimen and Histology and
The fourth mammary glands of a wild-type CDH1 and an Igfbp7-null mice were surgically
excised three days post onset of lactation. The two glands were spread onto glass slides to help
regain native mammary gland morphology.
The mammary glands were fixed in neutral buffered 4% PFA O/N at 4˚C and stored in 70%
ethanol prior to tissue processing. After the tissue processing and embedding the tissue in
paraffin the sections were cut from the paraffin block with 5 μm thickness and 5 µm separation
between sections. The sections were stained with H&E and then digitized with the magnification
of 20x and the resolution of 0.5μm using a Leica SCN400 slide scanner. The histology images
are then down-sampled to 18µm resolution, which is the resolution of the reference images used
in the pipeline and is explained in the next section.
2D registration
Figure 2-1 Histology volume reconstruction; histology images are registered to their optical
blockface images, which represent the shape of the specimen before it is cut into histology
sections.
25
2.2.2 Reference Images
As it is explained in 1.3.2.1 the sparse sampling of the clinical specimens reduces the similarity
between the sections and therefore makes the image registration more challenging. A common
approach to address this problem is to employ reference images (Bardinet et al., 2002),
(Chakravarty, Bertrand, Hodge, Sadikot, & Collins, 2006), (Dauguet et al., 2007). In the
proposed image registration pipeline optical images of the tissue blocks were used as reference
images. Blockface images are taken from the paraffin blocks of the tissue, mounted on the
microtome, before each section is cut. Since the blockface images are obtained from the tissue
fixed in its place in paraffin block, they represent the correct alignment of the images with
respect to the tissue in block. These optical images are obtained by a telecentric lens to eliminate
the barrel distortion, which usually occurs when using regular lenses. A telecentric lens is a
compound lens which has an entrance pupil placed effectively at infinity (Opto Engineering,
2015). This property makes the chief rays parallel to the optical axis and the image magnification
independent from the object's distance or position in the field of view(Opto Engineering, 2015).
The diameter of the telecentric lens used to image the mammary gland is 4mm and the resolution
of the optical blockface images is 18 μm.
Figure 2-2-a shows the blockface imaging setup used to capture optical images of the tissue
blocks. The images are taken at an oblique angle (≈15°) to use the reflection of the surface of the
block for contrast enhancement between the tissue and paraffin surface and to eliminate the
shadow of the tissue in depth, under the paraffin surface. A photographic filter is also used to
polarize the light coming from the block surface and the tissue to balance the contrast. To correct
for the displacement of the block on the rotary microtome, three vertical 1mm holes are drilled in
three corners of the block using a vertical milling machine. The holes are easily detectable in the
blockface images as shown in Figure 2-2-b. The centroids of these holes are used along with
landmark-based rigid registration to align the blockface images.
26
2.2.3 Alignment of Blockface Images
Due to the movement of the tissue blocks on the rotary microtome, the blockface images might
be vertically shifted with respect to each other. The centroids of the registration holes were used
along with landmark-based rigid registration to correct the displacement of the blockface images.
Figure 2-3 shows the block diagram of the blockface image alignment. The registration holes
were first segmented in blockface images using the average pixel value of the regions
representing the holes as a fixed threshold. Since some additional parts might also be segmented
by using the fixed threshold, the circularity and the area features of the segmented objects were
used to find the holes and discard the extra objects. Circularity of an object can be specified by
the ratio of (4π x area)/(perimeter)2. This ratio for round objects is 1. One blockface image is
selected as reference and the rest of the blockface images were aligned to the reference by using
the centre of the registration holes and landmark-based registration technique developed using
Insight ToolKit (ITK) image registration libraries.
Figure 2-2. a) Blockface imaging setup with telecentric lens. b) a blockface image of a mouse
mammary gland with three 1mm holes drilled in three corners of the block.
a)
b)
5mm
27
2.2.4 Image Segmentation and Point Selection
In order to use an edge-based registration method it is required to segment the tissue sections and
find their boundary points both in aligned blockface and histology images. Figure 2-4 shows the
block diagram of the image segmentation and registration of the histology images to their
correspondent blockface images.
The tissue sections were manually segmented from the background in the aligned blockface
images. The histology images were first down-sampled to the resolution of the blockface images
Figure 2-3 Alignment of the blockface images using the registration holes.
28
and then segmented using Otsu thresholding technique (Ridler, T.W. Calvard, 1978) to create
binary masks of the histology images.
The largest segmented object in both sets of images was identified by using the histogram of the
labeled objects in the mask images and one pixel wide boundary points were extracted from both
histology and blockface masks. Chain code algorithm (Freeman, 1974) was used to represent the
boundary points by a sequence of piecewise linear fits, and this representation of boundary
points were used for the rest of the steps in the protocol.
2.2.5 Fourier Descriptors (FDs)
Given that some expansion and shrinkage occurs during histology processes, the size of the
histology images is different from the blockface images and an initial registration is required to
achieve an estimate of the scale factor along with the initial translation and rotation. To obtain
this initial transformation we use Fourier Descriptors which are invariant to rotation and
translation of the boundary points and also independent of the starting point (Giardina & Kuhl
1977). We employed normalized Fourier descriptors to find the rotation angle and translation
values as well as the scale factor for initial alignment of the boundaries of the blockface and the
histology images (Duan, Kuester, Gaudiot, & Hammami, 2008).
30
2.2.6 Iterative Closest Points (ICP)
To refine the alignment we use the Iterative Closest Points (ICP) algorithm (Besl & McKay,
1992). This technique uses point sets to find the transformation parameters (Besl & McKay,
1992). ICP iteratively finds the transformation, which minimizes the average distance between
the moving edge points and their closest points on the fixed point set and transforms the moving
points.
The high curvature edge sections were first removed from the boundary contour using a rolling
ball filter (with a 50 pixel ball structuring element) with the assumption that high-curvature
edges correspond to tears in the section. Five hundred random edge points are then selected from
the remained smooth boundary sections using uniform sampling. These random edge points are
transformed by the transformation parameters obtained from the initial alignment using Fourier
Descriptors.
The transformed histology random points are used as a moving point set and the whole set of
blockface edge points are used as fixed point set. ICP was then employed to refine the rigid
registration with Euclidean distance as similarity measure and Levenberg Marquardt as optimizer
with maximum iteration of 200.
2.3 Results
Figure 2-5 shows the blockface volume and histology volume of the mammary glands. In order
to investigate the tissue structures at 5x magnification, the original histology images were down
sampled to 5x magnification. The region of interest was then cropped from one of the histology
images and using the rigid registration parameters the location of that region in other 5x
histology images were calculated and the regions were cropped. Finally the alignment of this
regions were refined manually. A program was developed in MATLAB to overlay two images
and select the values for rotation and translation of one of the images over the other one. Figure
2-6 shows the 5x magnification of two consecutive histology for Igfbp7-null glands, which
agrees that the image registration pipeline is capable of aligning the images and the cropped
regions are from the same region.
31
By looking at the overall structures we can see the difference in size between mutant and wild
type glands. However, using the approach described herein, it becomes apparent that this size
difference is in length and width but interestingly not depth. For the glands used in this pilot
experiment, the wild-type gland was 1.06 mm deep, while the Igfbp7-null gland was 1.02 mm
deep. The other phenotype immediately noticeable is the difference in stromal components of the
two glands, as marked by eosin staining (pink areas). The wild-type glands have little stromal
tissue, while the null-glands appear to be predominantly stromal tissue Figure 2-6. To further
investigate this, images were aligned near the lymph node in higher resolution, this allows us to
see how the glands are changing in serial sections, as shown in Figure 2-7. In the wild-type gland
we can see large alveolar structures, which would have been filled with milk, and can be
followed in several consecutive sections. In contrast the Igfbp7-null gland has few well-
developed structures and it is difficult to follow them in even a couple of consecutive sections .
Moreover, these structures were crowded with fibroblast-like cells. The alveolar volume is
greatly reduced within the null-glands indicating inadequate milk volume for feeding large
litters. The total volume of the wild-type gland was 82.88 mm3 while the null gland measured
only 19.62 mm3.
a) b)
c) d)
Figure 2-5: a) blockface volume of the wild-type gland. b) histology volume of the wild-type
gland. c) blockface volume of Igfbp7-null gland. d) histology volume of Igfbp7-null gland
32
0.2m
Figure 2-6. 5x magnification of a cropped region from two aligned consecutive histology images
of the Igfbp7-null mammary gland. Due to large separation between slices structures are not
continuous but that the alignment process has worked as it is clear that the two images are clearly
from the same region.
33
2.4 Summary and Discussion
In this study, we have developed an image registration workflow to reconstruct a 3D histology
volume from serial 2D histology images, which does not require internal manually selected
landmarks or implanted fiducial markers within the tissue, which might distort the tissue. By the
method described, optical blockface images themselves are used as the reference images prior to
sectioning. We use external holes drilled in the paraffin block to aid in aligning the blockface
images and to correct for the 2D transversal movement of the paraffin block in front of the
camera. The 2D histology images are aligned to the corresponding 2D blockface images to
prevent the propagation of the registration error and reconstruct an accurate histology volume,
even when some sections are missing due to problems with tissue section preparations. In order
to make the workflow independent of the type of the tissue and the histology stain used,
boundary points are used to carry out the registration. This point-based approach has the
advantage (over intensity-based approaches) that it is less computationally demanding and
therefore better able to cope with very large digital pathology images. Another advantage of
using blockface images to align the histology images is that the spacing between the histology
images does not affect the quality of their alignment to create the histology volume. This is
Figure 2-7. A region cut from four aligned consecutive sections at high resolution shows the
continuation of the 3D alveolar structures. Top) Wild-type. Bottom) Igfbp7-null gland.
34
important in the clinical setting where the spacing between sections can vary widely, often as
great as half a centimeter.
The important capability of this approach is that it is independent from the tissue type and
intensity variations and thus it can be used to reconstruct the histology volume of different pre-
clinical and clinical specimens. One of the other advantages of this approach is that it is not
dependent on a specific stain. This contour based approach is compatible with any stain, so long
as it provides a clear contour of the whole section or a clear contour of a structure, which is
detectable in both histology and blockface images.
The limitation of the edge-based image registration approach is that it is requires the boundary
points of the tissue sections on both histology and blockface images and therefore an image
segmentation step has to be added to the pipeline. Automatic segmentation of tissue boundaries
might not always be feasible especially for specimens with no clear boundaries in the images,
such as breast. Segmentation error can also contribute to the registration error.
In this work the accuracy of the histology volume reconstruction pipeline was assessed by visual
comparison of the histology volumes with blockface volumes. In order to be able to evaluate the
results of the image registration approach, it is required to have a suitable validation strategy. A
quantitative validation technique is discussed in the next chapter.
Throughout the work of this chapter we have shown that the approach can be used in a research
context as used for the study of the effect of the effect of down-regulation of Igfbp7 protein in
the development of the mammary glands.
A pitfall of traditional microscopy techniques is that the understanding of an organ at the
microscopic level is limited to one field-of-view at a time. Even “total disclosure” slides, which
provide entire slide sections, fail to provide three-dimensional information. To better
characterize the deficiency of the Igfbp7-null mouse, 3D-reconstruction of mammary glands
were performed on glands excised 3 days post commencement of lactation. The reconstructed
histology volumes were used to compare the shape, volume, and structures of the mammary
glands.
35
This chapter presents a point-based rigid registration approach for reconstruction of the histology
volume. The tissue sections also deform during the preparation of the tissue slides. Next chapter
discusses a point-based deformable registration used to correct these deformations. The
capability of the histology volume reconstruction approach for another pre-clinical model, a
mouse tumour, is also further discussed in the next chapter along with the extension of the
pipeline for registration of histology to volumetric medical images.
36
3 Registration of Histology with Volumetric Images
Chapter three gives an overview to the existing approaches for 3D co-registration of histology
medical images, followed by introducing the expansion of the image registration pipeline,
presented in chapter two, for registration of the histology images with volumetric medical
images. This chapter also gives an overview of the required elements and how they are addressed
in the proposed workflow. A multimodality fiducial marker is also introduced, which can be used
to investigate the accuracy of the image registration pipeline. The overall accuracy of the image
registration pipeline is assessed by using a pre-clinical tumour model.
3.1 Introduction
In order to transfer new developments in medical imaging techniques to clinic, they need to be
validated in pre-clinical studies, and this validation is usually done by correlating the acquired
images to the underlying molecular structures in histology (Sztrókay et al., 2012) (Tot & Gere,
2016) (Magnitsky et al., 2005) (Weissleder et al., 2000). For example, correlation of histology
with medical images gives us a better understanding of new contrast mechanisms (Cao, Liang,
Shen, Miller, & Stantz, 2009; Knopp et al., 1999), and the relationship between parameters
derived from medical images and any underlying pathology or physiology (Chopra et al. 2009;
Pauly et al. 2006; Aref et al. 2008). In order to accurately correlate the findings in medical
images with the underlying structures in histology, it is required to compare the correspondent
histology and medical images. Selecting the correspondent 2D histology and medical images
from volumetric images, e.g. MRI and CT, is challenging, and is usually accomplished by first
37
registering the medical image volume with histology volume and then selecting the
correspondent images.
Different methods have been proposed to register ex-vivo and in-vivo volumetric images with
reconstructed 3D histological images. Some techniques involve intensity variations (Alic et al.,
2011; D’Aische et al., 2005; Wachowiak, Smolíková, Zheng, Zurada, & Elmaghraby, 2004).
Alic et al. (Alic et al., 2011) have shown that the contrast of MR images might not completely
correspond with the contrast in histology images and that the combination of different contrast
mechanisms in MRI, such as Diffusion Weighted Imaging (DWI), is required to improve the
correlation. However, the acquisition of specialized MR sequences may not be possible in the
clinic. Other techniques are based on the shape or boundaries of the volumetric images and
histology volume (M. A. Jacobs, Windham, Soltanian-Zadeh, Peck, & Knight, 1999; Taylor et
al., 2004; Zhan et al., 2007). For some specimens the anatomical structures can be used, such as
the bloblike structures of prostate (Zhan et al., 2007) along with landmark-based registration
methods. However these internal structures might not be detectable through the whole volume
and in many cases, e.g. xenografts extracted from animal models, no reliable anatomical
structures can be identified. Extrinsic landmarks, such as needle paths (Breen et al. 2005;
Lazebnik et al., 2003), can cause more distortion and deformation, and may interfere with the
tissue harvesting and processing protocol.
This chapter introduces the expansion of the proposed image registration workflow for histology
volume reconstruction for correlation of histology to volumetric medical images along with the
validation approach. Figure 3-1 shows the overall pipeline. As explained in chapter two, the
histology images are registered to their correspondent blockface images to reconstruct histology
volume. The volumetric ex-vivo MR images are also registered to the blockface volume. Since
both histology and ex-vivo images are registered to the same set of reference images, they will
be registered as well. After resampling the ex-vivo images, the correspondent histology image
can be selected based on the depth of each image.
38
3.2 Requirements of the Image Registration Pipeline
3.2.1 Deformable Registration
Histology volume reconstruction of highly deformable human breast specimens requires a
suitable deformable registration algorithm to be able to correct the deformations that occur at
different steps of tissue handling and histology slide preparation.
The deformable registration used in this pipeline is discussed in detail in a later section in this
chapter.
Registered ex vivo Volumee)
Figure 3-1. Overall pipeline for the histology volume reconstruction and registration of
medical images with histology.
39
3.2.2 Extendable for registration of histology to medical images
One of the applications of histology volume reconstruction is in co-registration of histology
images with the volumetric medical images. This application indicates that it is desirable that the
histology volume reconstruction pipeline possesses the capability of being extendable for
registration of histology to medical images. As it is explained in detail in section 2.2.2, having
optical references images in the pipeline facilitates this requirement.
3.2.3 Validation
In order to be able to assess the performance of the image registration pipeline we need to have a
reliable method for validation of the different steps of the pipeline. For some specimens the
anatomical structures can be used as landmarks to calculate the registration error. But these
internal structures might not be detectable through the whole volume and for some specimens no
reliable anatomical structures can be identified, especially for human breast tissue where the
spacing between the sections is a few millimeters and the structures are not continuous in
different histology sections. Section 3.3.3 explains the experiments pursued to develop a multi-
modality fiducial marker, which is detectable in volumetric medical images as well as in
histology and optical images of the paraffin blockface. A marker with these characteristics can
be reliably used to validate the registration techniques.
3.3 Material and Methods
3.3.1 Specimen
In order to investigate the characteristics of different fiducial
markers in different tissue types, a mouse thigh muscle, two
mouse kidneys, and a mouse tumour model were used. The
mouse tumour was a xenograft of human prostate cancer cell
line PC3. The fiducial markers were implanted in the fresh
tissue and then the tissue was embedded in 3% agarose to keep
the sample fixed in a petri dish for imaging as shown in Figure
3-2.
In the second part of the work in this chapter a xenograft of human breast cancer cell line (MDA)
in the hip of a SCID mouse was excised and fixed in 10% Neutral Buffered Formalin (NBF) and
Figure 3-2 A mouse kidney with fiducial markers embedded in agarose
40
used for validation of the image registration workflow. The MRI, histology and blockface
images of the specimen with implanted fiducial markers were prepared as explained in the
following sections.
3.3.2 Reference Images
As explained in chapter two, optical blockface images are used in this pipeline. The blockface
images were first aligned using the centres of three drilled holes in the three corners of the tissue
block.
3.3.3 A Multi-Modality Fiducial Marker
A multi-modality fiducial marker is introduced in this section, which can be used for validating
the histology volume reconstruction as well as the correlation of histology images with medical
images.
An MR detectable fiducial marker containing metallic elements can create high contrast by
perturbing the main magnetic field and consequently nullifying the signal driven from the
hydrogen nuclei. In conventional radiography and computed tomography a detectable fiducial
marker should contain a material with high atomic number (such as iodine) to produce greater X-
ray attenuation. In Ultrasound the fiducial marker should have different acoustic impedance
which causes the acoustic wave reflection from the interface and as a result the ultrasound signal
from the interface is enhanced. On the other hand, in histology a fiducial marker should be easily
cut and remain stable on positively charged microscope slides. An organic material is suitable for
this purpose. Considering all mentioned requirements we tried different fiducial markers
combining metallic and organic materials. The present work reports on some initial results of the
experiments that we have accomplished on these fiducial markers.
The requirements for a multi-modality fiducial marker to be used for validation as follows. It
should be rigid enough to be implanted in the tissue and also it should be easily cut during
sectioning the tissue. Finally the fiducial marker needs to remain on the positively charged
microscope slides so that it can be located in histology images.
Different fiducial markers were tested for these requirements in this work. First we assessed a
26G catheter filled with a mixture of bronze metallic paint and marine apoxy glue (Rouvière et
41
al., 2006), which is conventionally used as a marker in MRI. The results of the experiment with
this marker showed that it falls off the microscope slide and also causes scratches on the tissue
section while it is being cut. We then tried various markers based on flour, a biological material
that does remain on the final microscope slide, i.e. regular spaghetti, black spaghetti, a 26G
catheter, a mixture of ground black spaghetti and water, a mixture of squid ink and flour, and a
mixture of liquid iron supplement and flour. The two types of spaghetti were implanted directly
in the tissue. For other fiducials a catheter is first placed in the tissue and the mixture is then
injected into the catheter tube to fill it completely. The tube was then cut from the catheter end
and stayed in the tissue. Since the size of the catheter is small, it is suitable for small specimens
such as mouse tumours.
Based on the results shown in section 3.4.1, three 26 Gauge catheters perfused with a mixture of
cuttlefish ink, flour, and water were implanted in the tumour. Since the specimen was fixed
before, the fiducial markers were implanted in the fixed tissue. However, the localization of the
fiducial markers will be more accurate when they are implanted in the fresh tissue. In order to be
able to correct for the displacement of the blockface images due to the movement of the
microtome block holder in front of the lens of camera, three holes were drilled in the three
corners of the paraffin block, in which the tissue is embedded. The centroids of the holes were
then used as landmarks to align the blockface images.
3.3.4 Imaging
After implanting the fiducial markers, the ex-vivo images of the specimens were obtained using
different imaging modalities. In order to characterize the fiducial markers, the samples were
scanned at 7 Tesla from Bruker Biospin or 3 Tesla MR Scanner from GE and using a head coil.
T1 weighted images at 3T were obtained using Fast Spin Gradient Echo with TE= 6.3 and TR =
13.2 and the matrix size of 256 x 256. The visibility of the fiducial markers was investigated
along with the susceptibility artifact. Following MRI the sample was imaged by high frequency
ultrasound (25 kHz) using a VisualSonics Vivo 2100 system. The X-ray images were then
obtained from a GE 2000D mammography system with both low and high energy.
42
For validation of the image registration pipeline, magnetic resonance imaging of the specimen
was carried out using a 3 Tesla MR scanner from GE with the head coil. T1 weighted images
were obtained using Fast Spin Gradient Echo with TE = 6.3 and TR = 13.2 and the matrix size of
256 x 256 Figure 3-3-a. The voxel size was calculated as 100x100x400µm.
3.3.5 Histology
After scanning the tissue, the agarose was peeled off and the tissue was fixed in 10% formalin
for 48 hours. The specimen was processed with an automatic tissue processor and embedded in
paraffin. Sections with 5μm thickness and 20μm separation were cut from the paraffin block and
stained with Hematoxylin and Eosin (H&E). Before each section is cut, an optical blockface
image was also obtained from the surface of the block, Figure 3-3-b. The sections were digitized
by an Aperio ScanScope scanner with a resolution of 0.2µm and then downsampled to 18µm
resolution resulting in an image size of 8400 x 5700 pixels Figure 3-3-c. As shown in Figure 3-3
the fiducial markers in the tumour are detectable on MR, blockface and histology images;
therefore, it is feasible to use them for validation.
1mm 1mm 5mm
a) b) c)
Figure 3-3. Images of a mouse tumour with validation fiducial markers: a) ex vivo MRI, b)
blockface image, c) Histology.
43
3.3.6 Image Registration Pipeline
This section gives an overview of the development and validation of the registration workflow
that can be used to reconstruct 3D histology images and to co-register ex vivo MR images with
2D histology sections.
Similar to the work presented in chapter two, point-based registration techniques have been used
in this pipeline in order to avoid registration problems caused by intensity dissimilarity of the
images and also to make the workflow independent from the tissue type. In addition, using
boundary points rather than all the pixels of the images is more efficient when working with
large histology images.
As it is shown in the block diagram in Figure 3-5, the MR images, the blockface images, and the
histology images are first segmented into tissue and background. The surface points of the
segmented MR and blockface volumes are then selected for registration. The MR volume is
registered to the blockface volume using a 3D registration approach consisting of an initial rigid
registration followed by a more accurate rigid registration and finally a deformable registration.
The registered MR volume is then re-sliced. For each MR slice the depth of the tissue is
calculated based on the MR spacing and the correspondent blockface and histology image is
selected from the same location as shown in equation 3-1.
H
MRMRH SP
SPNN
(3-1)
NH is the index of the correspondent histology section to the MR slice with the index NMR. SPMR
and SPH are the spacing of the MR slices and histology sections respectively.
The selected histology image is then registered with its correspondent blockface image using the
edge points. Similar to the 3D registration part, the 2D registration is also achieved using an
initial alignment followed by a more accurate rigid and deformable registration. The mean
absolute Target Registration Error (TRE) is then calculated based on the implanted fiducial
markers in the specimen.
44
In this workflow, the MR volume and the histology images are registered to the same reference
(blockface images). Additionally, we registered each histology image to its correspondent
blockface image to reconstruct the histology volume.
Each step is explained in the following sections in detail.
3.3.6.1 Segmentation, Point Selection, and Rigid Registration
Segmentation and point selection were pursued using the same approach explained in chapter
two. Histology segmentation was carried out on the blue channel, where the contrast between
different tissue types was higher. To detect the edges of H&E sections, the sections were
segmented from the background using the Otsu thresholding technique (Gonzalez & Woods,
2002).This worked well with H&E histology images, but the contrast between the sample and the
background is lower in blockface images and Otsu thresholding did not perform as well. In this
work, the boundaries of the optical blockface images are segmented manually to minimize the
registration error caused by the segmentation error.
Figure 3-4.a) Edge points of a histology image, b) Surface points of the MR volume, c) Surface
points of the Blockface volume.
The one-pixel wide edges of the binary images were extracted by using the Sobel filter followed
by a thinning operation and then the longest closed boundary in each image was selected (Figure
a) b) c)
45
3-4-a). The closed boundaries were then represented by a sequence of piecewise linear fits using
chain code which is a method to encode boundary points on a closed curve (Freeman, 1974).
The resulting edge points were used for initial alignment by Fourier Descriptors (Giardina &
Kuhl 1977).
The transformed random points of the histology and the complete set of edge points of the
blockface are used in the rest of the 2D image registration steps. MR images are segmented using
seeded region growing technique (Gonzalez & Woods, 2002). The surface points are selected
from the segmented MR images and blockface images in a spiral way as shown in Figure 3-4-b
and Figure 3-4-c. Similar to 2D registration, after initial rigid registration the random MR surface
points are used in the subsequent image registration steps.
As explained in chapter two, 2D Fourier Descriptors (Duan et al., 2008) and Iterative Closest
Points (ICP) algorithm (Besl & McKay, 1992) were employed for initial and refinement of the
rigid alignment of the histology images with their correspondent blockface images. Similarly, 3D
Fourier Descriptors and ICP were used for 3D initial and refinement of the rigid registration of
MR volume with blockface volume.
The point-based deformable registration method is explained in the next chapter.
47
3.3.6.2 Deformable Registration
A deformable registration is necessary to compensate for the deformation of the sections that
might occur during histology processes. For non-rigid registration we have used Thin-Plate
Splines (TPS), which estimates the displacement by minimizing the bending energy over a fixed
set of control points (Bookstein, 1989). Following the rigid registration, the closest points on
blockface edges to the rigidly aligned histology edge points are extracted as control points and
used to deform the histology images (Shojaii &Martel 2009; Shojaii et al. 2011). We have used a
variation of thin-plate splines method, which is regularized by the stiffness (λ) of the object
(Equation 3-2).
)(|)(|)(1
2 uJpuquJn
i
dmii
(3-2)
As it is described in (Sprengel, Rohr, & Stiehl, 1997) the first term of (2) measures the sum of
the quadratic Euclidean distances between the transformed source landmarks u(pi) and the target
landmarks qi. u is the desired transformation which approximates the displacement of the source
landmarks to the target landmarks by minimizing the functional Jλ. The second term measures
the smoothness of the resulting transformation. Jdm is the functional defined in (Sprengel et al.,
1997), for d-dimenional images and for an arbitrary order m of derivatives. The relative weight
between the approximation behavior and the smoothness of the transformation is determined by
the parameter λ>0 (Sprengel et al., 1997). In this work we compared the error of deformable
registration when using the values of 0 and 1 for the smoothness parameter λ. With λ=0 there is
no regularization and we have an interpolating transformation, which overlaps the control points
of the two images and finds the displacement of the other pixels by interpolating the
displacements of the control points. Therefore, λ=0 causes higher or less smooth deformation
and is suitable for less stiff or more deformable tissue types, such as muscle. In contrast, with
higher values for λ, for example 1, the variation of the displacement of the pixels is regularized,
and as a result the deformation is less, which makes it suitable for stiffer tissue types such as
tumour.
48
The ex-vivo MR volume represents the shape of the specimen before sectioning and the
blockface volume represent the same specimen after sectioning. Therefore, the 3D surface points
of the two volumes can be used for image registration. Similar to 2D histology to 2D blockface
image registration, for the deformable registration of MR volume with the blockface volume, the
closest surface points from the rigid registration are selected as control points and then 3D TPS is
used to compensate the 3D deformation.
The image registration techniques used at each step of the workflow are summarized in Table
3-1.
Table 3-1. Summary of the image registration techniques used at each step of the workflow H =
Histology image, B = Blockface image, MR = MR image
Source to Target
Type of Registration
Algorithm Cost function Optimizer Interpolator
2D H to B initial rigid FD - - Linear
2D H to B rigid ICP L2 norm Levenberg Marquardt
Nearest Neighbor
2D H to B non-rigid TPS bending energy Gradient Descent
Linear
3D MR to B initial rigid FD - - Linear 3D MR to B
rigid ICP L2 norm Levenberg
Marquardt Nearest Neighbor
3D MR to B non-rigid
TPS Bending energy Gradient Descent
Linear
3.3.6.3 Validation
The mean absolute TRE was calculated at each step of the workflow based on the centres of the
fiducial markers and showed the statistical dispersion by calculating the standard deviation (sd).
In order to evaluate the histology volume reconstruction, the mean absolute TRE was calculated
based on the centres of the fiducial markers manually localized on histology and the
correspondent blockface images. The registration error was measured after the 2D rigid
registration, using FDs and ICP, and also after 2D deformable registration using TPS with two
different stiffness values.
49
To confirm that the blockface images improve the histology volume reconstruction, the histology
images were also registered using the pair-wise approach without using blockface images and
compared the reconstructed volume with the volume made by using the proposed approach.
The MR volume is then registered to blockface volume and re-sliced. Ten slices were selected
from the middle of the MR volume, where the tumour and fiducial markers were detectable. The
mean absolute TRE was then measured using fiducial markers on the 2D MR slices and their
correspondent blockface and histology images. If the correspondent histology image was torn or
had artefacts then the next or previous section was selected as closest section for correlation.
To assess the registration error in 3D, the fitted lines to the centres of the fiducial markers in MR
and histology volumes were also visualized.
3.3.6.4 Blockface image spacing
Since the blockface images are used as reference and play an important role in the workflow, we
have investigated the effect of the spacing of the blockface imaging on the target registration
error of MR slices and their correspondent histology images. This provides us with an estimate
of the registration error with fewer histology sections and enables us to select the number of
histology sections that we need to meet the desired registration accuracy. As it is mentioned in
3.3.5, the original spacing of the blockface images and the histology images is 0.02 mm. To
increase the blockface spacing we have eliminated some of the blockface images from the
registration process. First we increased the blockface spacing to 0.095 mm by including one
image in every four images (Figure 3-6) in the registration process and then we repeated all the
steps to measure the mean absolute TRE of the registration of MR images to the correspondent
histology images. The spacing was further increased to 0.195 mm, 0.395 mm, and 0.795 mm by
including one image in every 8, 16, and 32 images respectively and repeating the whole process.
The measured mean absolute TREs are shown and compared in section 3.4.
50
Figure 3-6.Eliminating pairs of blockface and histology images to increase the spacing.
3.4 Results
3.4.1 Fiducial Markers
This section reports on the preliminary results of a set of experiments for developing a multi-
modality fiducial marker, which can also be identified in histology and optical blockface images.
The purpose of this marker is to validate the registration histology images to their correspondent
optical blockface image. This fiducial marker can also be used in validation of the registration of
medical images with histology images. This fiducial marker can also be used to provide
landmarks for image registration. Among seven fiducial markers, black spaghetti has shown to
be the most suitable for big specimens. For small specimens, such as a mouse tumour, a
promising marker is shown to be a 26G catheter filled with a mixture of the squid ink and the
flour.
The susceptibility artifact in MRI can be eliminated by adjusting the concentration of the squid
ink based on the iron concentration in black spaghetti. The detectability, localization and stability
of black spaghetti in different imaging modalities and blockface images is very good and in
100% of the histology sections it stays on microscope slides after staining. Catheter filled with
the mixture of cuttlefish ink and flour and water is very well visualized, localized and stable in
medical images and blockface images and in around 40% of the sections it remains on
microscope slides after staining. Even when the catheter does not remain on the slide, the hole
caused by the catheter's insertion is visible in the sections. The reason that the catheter falls off is
that paraffin supports the tissue and the fiducial marker during sectioning, but with regular
51
catheter tubes and also with inorganic mixtures of paint and glue, the paraffin infiltration is poor,
therefore the fiducial marker drops off during the sectioning. The localization of this fiducial
marker on histology sections can be improved by choosing a sturdy permeable catheter to
increase the paraffin infiltration during the tissue processing.
Table 3-2 shows a summary of the results of the preliminary experiments toward a multi-
modality fiducial marker which is also visible in histology and blockface images. The markers
which are detectable in medical images are stable in terms of contrast enhancement. As the table
shows, the regular spaghetti cannot be identified in MRI and therefore it has not been further
investigated with US and X-ray. Black spaghetti is made the same way as regular spaghetti
except that the cuttlefish ink has been added to it for colour. Since cuttlefish ink contains iron
(the composite of Melanin-Fe (Lei et al., 2008)), it meets all the criteria for a fiducial marker
mentioned in the introduction and is therefore detectable in all imaging modalities. For histology
the positively charged microscope slides were used for these experiments. These slides allow
specimens to adhere to the slide easily. The organic nature of the spaghetti makes it more similar
to the specimens and suitable to stay on the microscope slides. However its large diameter
restricts the number of fiducials in small specimens and makes it more suitable for big
specimens, as it is usually required to have several markers for precise validation.
The 26G catheter is well detectable in medical images (air inside the catheter and agar at both
ends) and the blockface image, but since it is not organic or porous, it cannot be infiltrated with
paraffin so it does not stay on histology sections and remains a hole. This makes it difficult to
accurately localize the marker in histology images. Bronze metallic paint (Rouvière et al., 2006)
contains iron-oxide particles, which acts as a negative contrast agent Since the mixture of the
bronze metallic paint and the marine epoxy glue(Rouvière et al., 2006) does not stay on the
histology sections either, it has also a poor localization in histology images similar to the catheter
alone.
52
Table 3-2.Fiducial markers and their delectability in different imaging modalities
no: not detectable, yes: detectable, NT: not tested
No. Fiducial Markers MRI Ultrasound X-ray Blockface
image Histology
Size (diameter)
1 Spaghetti no NT NT yes yes (100%) ~0.95mm
2 Black spaghetti yes
Size(mm):2.0 ± 0.4 NT yes yes yes (100%) ~1.75mm
3 26G Catheter yes
Size(mm):0.6 ± 0.02 NT yes yes
no (poor localization)
Outer:0.6mm inner:0.4mm
4
Catheter filled with bronze metallic paint and marine epoxy glue (1:2)
yes
Size(mm):0.6 ± 0.05 NT NT yes no (poor localization)
Outer:0.6mm inner:0.4mm
5 Catheter filled with ground black spaghetti and water
yes
Size(mm):0.6 ± 0.05 NT NT yes
yes (83%) (better localization)
Outer:0.6mm inner:0.4mm
6 Catheter filled with liquid iron supplement and flour (1:1)
yes (high susceptibility
artifact)
Size(mm): 1.2 ± 0.5
yes yes yes
yes (44%) (better localization)
size(mm):0.71 ± 0.07
(inner) 0.49 ± 0.06
Outer:0.6mm inner:0.4mm
7 Catheter filled with cuttlefish ink and flour (1.5:1)
yes (susceptibility artifact)
Size(mm): 0.6 ± 0.1
yes yes yes
yes (46%) (better localization)
size(mm):0.69 ± 0.03
(inner) 0.41 ± 0.06
Outer:0.6mm inner:0.4mm
53
2mm 2mm
2cm
X-ray
MRI
US
Mouse kidneys
Histology
Figure 3-7. Mouse Kidney with
Catheter filled with Cuttlefish ink &
flour (left) and Liquid iron & flour
5mm
m
5mm
Liquid iron&flour
5mm
Cuttlefish ink&flour
54
The mixture of the ground black spaghetti with water shows up very well in medical images and
histology and blockface images, but the problem with this marker is that the ground black
spaghetti is not very soluble in water and it is difficult to make a uniform mixture therefore water
gradually separates from the mixture. The localization of this fiducial marker on histology is
good where the mixture is denser (as shown inTable 3-2), but poor where it is less dense. The
mixture of the liquid iron supplement (20 mg elemental iron in 1 ml) with flour was also tested,
but because of the high iron concentration the susceptibility artifact in MRI is high. Since the
black spaghetti presents good characteristics, a mixture of the cuttlefish ink and flour (1.5:1)
wasalso tested as another fiducial marker. The mixtures of cuttlefish ink showed less
susceptibility artifact compared to the liquid iron but higher susceptibility artifact comparing to
the black spaghetti. In order to decrease the susceptibility artifact, the concentration of iron in
black spaghetti (12 ppm) and the condensed squid ink (45 ppm) was measured to determine the
amount of cuttlefish ink required for fiducial marker. Figure 3-7 show a complete set of images
with the 6th and 7th fiducial markers in a mouse kidney.
a)
Black Spaghetti
c)
Figure 3-8. a) Optical image of a mouse tumour with three different fiducial markers: Black
Spaghetti, Catheter filled with bronze metallic paint and glue, Catheter filled with ground black
spaghetti and water. b) MR image of the same specimen scanned at 7T scanner. c) x-ray of a
mouse kidney with black spaghetti.
1mm
Black
Spaghetti
Catheter + Paint
& Glue
Catheter +
Paint & Glue
Catheter +
Ground BS & W
Catheter +
Ground BS & W
b)
55
Figure 3-8-a shows an optical image of three different types of fiducial markers implanted in a
xenograft of human prostate cancer cell line PC3. Figure 3-8-b displays the T1 weighted image
of the tumour, where the fiducial markers can be easily localized. The size of the black spaghetti
in the middle limits the number of implanted fiducial markers especially in small specimens. On
the other hand the size of the 26G catheters is appropriate for small specimens. The mixture of
the ground black spaghetti and water and also bronze metallic paint and marine epoxy glue
facilitate the localization of the catheter. Figure 3-8-c also shows that two black spaghettis
implanted in a mouse kidney are well detectable in the x-ray image of the specimen.
b) a)
c) d)
Figure 3-9. Top: mixture of bronze metallic paint and glue in catheter, a) blockface image b) histology image. Bottom: black spaghetti in thigh muscle of a mouse, c) blockface image d) and the histology image.
56
Figure 3-9-a and Figure 3-9-b show the appearance of a catheter filled with mixture of bronze
metallic paint and glue in blockface and histology images respectively. Although the catheter and
the mixture get washed off during histology processes the location of the fiducial marker remains
detectable in the histology image. The same result was obtained with the mixtures of cuttlefish
ink and flour and water, and the mixture of ground black spaghetti and water, but with more
attachment of the mixture to the microscope slide. Two implanted black spaghettis implanted in
a mouse thigh muscle are shown in Figure 3-9-c and Figure 3-9-d. The localization of the black
spaghetti is feasible in blockface and histology images.
3.4.2 Blockface Volume
Figure 3-10-a shows a blockface image with the three holes in the three corners of the paraffin
block. The stack of the blockface images after the alignment of the drilled holes is shown in
Figure 3-10-b.
3.4.3 Reconstruction of a 3D histology volume from serial sections
As it is shown in Figure 3-11a-c, we can visually detect the registration error propagation or
banana effect, which occurs when a curved 3D object is reconstructed from serial sections. The
Figure 3-10. a) Blockface image, b) Blockface volume, arrows show three holes in the corners
of the paraffin tissue block
57
shape of the tissue has changed and it is also detectable that the implanted fiducial markers have
shifted to the left compared to the blockface volume. In Figure 3-11-b (reconstruction using the
blockface as a reference) we can see that the fiducial markers are in the correct alignment.
The mean absolute TRE after an initial rigid alignment using Fourier Descriptors was calculated
as 0.4± 0.18mm. The calculated mean absolute TRE after the refinement of the rigid alignment
using ICP is 0.3± 0.12 mm (figure. 8-b). The deformable registration has been done with two
values of smoothness parameter in Thin-Plate Spline approximation method. With the
smoothness parameter of zero the mean absolute TRE after deformable registration is calculated
as 0.33 ± 0.17mm and with the smoothness parameter of 1.00 the mean TRE is 0.38 ± 0.2mm.
3.4.4 Correlation of ex vivo MR to Histology
The 3D registration of MRI to the blockface volume can be carried out using rigid registration
(rM-B) or rigid registration followed by deformable registration (dM-B) (Figure 3-12) and TREs
for each step are shown in Table 3-3. The registration of the histology images to the block face
images (Figure 3-13) can also be carried out using rigid registration (rH) or deformable
registration (dH) giving us 4 possible ways of registering MRI to histology; these results are also
summarized in Table 3-3.
Figure 3-11.a) cross-section of the blockface volume. b) same cross-section on histology
volume reconstructed using blockface images as a reference. c) same cross-section of the
histology volume reconstructed using pairwise registration without the blockface images.
58
The results of a paired t-test indicate that there is a significant improvement in overall
registration accuracy when deformable registration is used to align the MRI volume with the
blockface images compared to the use of simple rigid registration ((dM-rH vs rM-rH, p=0.01);
(dM-dH vs rM-dH, p<0.0001)).
Table 3-3.Mean Absolute Target Registration Error and the Standard Deviation.rM = rigid
registration of MRI, rH = rigid registration of histology, B = blockface image, dM = deformable
registration of MRI, dH = deformable registration of histology
Step Mean ± SD (mm)
rM-B 0.41 ± 0.12 dM-B 0.37 ± 0.11 rM-rH 0.68 ± 0.39 dM-rH 0.54 ± 0.37 rM-dH 1.07 ± 0.4 dM-dH 0.86 ± 0.45
It was shown in section 3.4.3 that the addition of a deformable registration step to align the 2D
histology and blockface images increases the TREs and this is reflected in the higher errors for
(rM-dH) and (dM-dH). The best results were obtained using deformable registration to align the
3D blockface and MR images and a rigid registration for the histology images.
a) b) d)
Figure 3-12.a) Registered MR volume, b) blockface volume, c) histology volume.
59
We also investigated the effect of the spacing of the blockface imaging on the target registration
error of 10 MR slices and their correspondent histology images. A one-way ANOVA test
showed that there is no statistically significant difference between the five groups (p=0.63).
Therefore increasing the spacing of the blockface and histology images up to about 0.8mm has
no significant effect on the accuracy of the registration of MR images to histology images;
however, this may be due to our small sample size and it is required to further investigate it using
more samples.
Figure 3-13.a) Registered MR image. b) Blockface image. c) Correspondent deformed Histology
image.
In order to investigate the through plane error in finding the correspondent histology images to
the re-sliced MR images we plotted the fitted lines to the centroid of the fiducial markers
detected in the re-sliced MR images after rigid (Figure 3-14-a) and deformable registration
(Figure 3-14 -b) and compared them with the fitted lines to the centroid of the fiducial markers
detected in the aligned histology images shown in solid red visually. As it is shown in Figure
3-14, the distances between the fitted lines have been decreased after deformable registration of
MR volume.
60
Figure 3-14.Fitted lines to the centroid of the fiducial markers in 3D; a) rigid MRI in blue and
correspondent histology in red, b) deformed MRI in green and correspondent histology in red
3.5 Summary and Discussion
This chapter reports on the results of a set of experiments for developing a multi-modality
fiducial marker, which can also be identified in histology and optical blockface images. The
purpose of this marker is to validate the registration histology images to their correspondent
optical blockface image. This fiducial marker can also be used in validation of the registration of
medical images with histology images. If required, this fiducial marker can also be used to
provide landmarks for image registration. Among seven fiducial markers, black spaghetti has
shown to be the most suitable for big specimens. For small specimens, such as a mouse tumour, a
promising marker is shown to be a 26G catheter filled with a mixture of the squid ink and the
flour.
The susceptibility artifact in MRI can be eliminated by adjusting the concentration of the squid
ink based on the iron concentration in black spaghetti. The localization and stability of black
spaghetti in different imaging modalities and blockface images is very good and in 100% of the
histology sections it stays on microscope slides after staining. Catheter filled with the mixture of
cuttlefish ink and flour and water is very well visualized, localized and stable in medical images
and blockface images and in around 40% of the sections it remains on microscope slides after
staining. Even when the catheter does not remain on the slide, the hole caused by the catheter's
insertion is visible in the sections. The reason that the catheter falls off is that paraffin supports
a) b)
61
the tissue and the fiducial marker during sectioning, but with regular catheter tubes and also with
inorganic mixtures of paint and glue, the paraffin infiltration is poor, therefore the fiducial
marker drops off during the sectioning. The localization of this fiducial marker on histology
sections improves by choosing a sturdy permeable catheter to increase the paraffin infiltration
during the tissue processing.
This chapter also introduces an image registration workflow to register ex vivo MRI to histology
images of a preclinical model. Optical blockface images are used as reference images. The 2D
histology images are aligned to the correspondent 2D blockface images and the 3D MR image is
registered to the blockface volume. The registered MR volume is then re-sliced and the
correspondent histology image to each MR slice is found. In order to make the workflow
independent of the type of the tissue and the histology stain used, boundary points are used to
carry out the registration. Catheters filled with the mixture of cuttlefish ink and flour and water
were implanted in the specimen as multi-modality validation landmarks and the mean absolute
target registration error was measured based on the centres of these markers.
Another advantage of using a point based approach rather than an intensity based approach is
that it is less computationally demanding and therefore better able to cope with very large digital
pathology images.
In order to evaluate the effect of the spacing on the registration of MR image with histology
images we eliminated some pairs of the histology and blockface images, or in other words
increased the spacing between the histology and blockface images, and repeated all the steps to
measure the registration error of the registration of MR images to the correspondent histology
images. We have shown that increasing the spacing of the blockface and histology images up to
about 0.8mm has no significant effect on the accuracy of the registration of MR images to
histology images of a mouse tumour model. Further investigation into the effect of increasing the
spacing between the blockface images and histology images should be carried out with more
specimens for confirmation.
Deformable registration of MR volume has improved the overall accuracy of the registration of
MRI to histology (0.68 mm vs 0.54 mm). The tissue is highly deformed during fixation,
processing, and embedding. Therefore the 3D deformation of the whole specimen, including the
tumour, between the MR volume (fresh tissue) and the blockface volume (embedded tissue) is
62
high relative to the 2D deformation between the blockface and histology images. The results
show that the 3D deformable registration using the thin-plate spline method has corrected this
deformation to some extent and has improved the overall registration of MRI to histology. The
2D distortions between blockface and histology sections are much smaller than the 3D
distortions between MR and blockface volumes, and as the results show, the 2D deformable
registration using the edge points has not improved the accuracy of the global registration. A
possible reason may be that the tumour is more rigid than the surrounding muscle and the
connective tissue, therefore the tumour deformation is more subtle than the other regions of the
section. The Thin-Plate Spline method assumes that the tissue is homogeneous and assigns one
stiffness value to the whole object and as a result, the deformation field obtained from TPS for
2D images is dominated by the effect of deformation in the muscle.
We have demonstrated that this registration workflow applied to a pre-clinical xenograft model
produces a target registration error of 0.54mm. Whether this registration accuracy is "good
enough" depends on the application and on the smallest structure that is of interest. In (Eli
Gibson, Fenster, & Ward, 2012), Gibson et al. utilized the fractional overlap between the
reference and target volumes to derive a statistical power calculation relating registration error,
sample size, and the minimum detectable difference between the region of interest and
background on imaging, which can be used to assess whether an image registration algorithm has
sufficient accuracy for a particular application. For example, one of the applications is the
validation of the utility of medical imaging for localization of a specific anatomy. For this
application and by using the proposed image registration pipeline, with the registration error of
0.54mm, structures such as breast ducts (~2 mm), prostate glands (3-4cm), and brain cerebral
cortex (1.5-4.5mm) detected in MR images can be reliably registered to histology.
It is difficult to generalize the accuracy of the pipeline to other applications. Different tissue
types have different biomechanical properties, such as highly deformable fat and less deformable
fibroglandular tissue, therefore the registration error is expected to change for different
specimens. Therefore, in chapter four a deformable image registration approach is introduced to
improve the 2D deformable image registration of the pipeline and an optimization strategy is also
presented to enhance the registration of breast lumpectomy specimens.
63
4 Optimization of Deformable Registration for Breast Histology Images
Chapter four explains the work that is done to improve the 2D to 2D deformable registration of
histology to blockface images. An overview is given of the deformable registration of the 2D
image registration of multi-modal images. This chapter shows the feasibility of using densely
sampled scale-invariant feature transform (SIFT) features and a SIFTFlow deformable
registration algorithm for co-registering whole-mount histology images with blockface optical
images. An optimization strategy is also introduced to estimate the optimum parameters for
SIFTFlow objective function and used to determine the most appropriate values for the
registration of breast lumpectomy specimens. The work in this chapter also demonstrates that
tuning the regularization parameters results in significant improvements in accuracy and also
shows that SIFTFlow outperforms the previously described edge-based registration method.
Further evaluation of the optimum parameters is also presented using an independent test set of
images from five different lumpectomy specimens.
4.1 Introduction
An image registration pipeline is presented in chapter two for histology volume reconstruction
and registration of histology with volumetric medical images. The study with edge-based
registration methods in the previous chapter suggests that the 2D deformable registration method
requires improvement. Due to the highly deformable nature of breast tissue, it is important to
employ an effective method to correct the deformation of the tissue sections with respect to the
original form of the tissue. In the proposed pipeline the histology images are registered to their
correspondent optical blockface images with different intensity variation. Therefore, this task
demands for a deformable images registration, which is capable of registering images obtained
from multiple modalities. In the previous chapter the boundary point of the sections were used
64
for registration of images obtained from animal models. Lacking a clear boundary between the
breast tissue and the surrounding gel in histology images and blockface images, rules out using
the edge-based registration methods for human breast tissue.
Several approaches for correcting for the 2D misalignment and deformations have been
described. Some groups have used extrinsic fiducial markers to align the slices (Breen et al.,
2005), (E Gibson et al., 2013), however this is an invasive procedure which has the potential to
affect the pathologists’ ability to interpret the histological sections, and is frequently not
permitted on clinical specimens.
Both intensity-based and landmark-based registration methods have the advantage that they do
not interfere with the appearance of the tissue sections. Intensity based methods have been
shown to be very effective in medical image registration but they may fail in histology, where
the intensities across histology images are not constant (Ju et al., 2006), (Grégoire Malandain et
al., 2004), (Ourselin, Roche, Pennec, & Sattonnet, 1998). Bağci et al. have demonstrated that
intensity variations across images can lead to registration errors and this problem cannot be fixed
through a simple intensity normalization approach (Bağci & Bai, 2008), (Bagci, Udupa, & Bai,
2010). Feature based image registration techniques (Guest, Berry, Baldock, Fidrich, & Smith,
2001), (Cohen, Yang, Huang, & Nissanov, 1998) rely on identifying corresponding features such
as points and lines in both images and the choice of a feature space plays a significant role in
registration quality.
The existing methods have been developed and validated for fairly rigid organs such as prostate
and brain, which also have clear boundaries in the images. Breast tissue is highly deformable and
does not have well defined boundaries. There is no automatic deformable registration technique,
which has been validated for the histology images obtained from large clinical breast specimens.
The aim of the work presented in this chapter is to improve the accuracy of the deformable
registration used to co-register the 2D histology sections to the 2D optical blockface images. In
our previous work, points at the boundary between the tissue specimen and the surrounding
paraffin block were automatically identified in both histology and blockface images and used to
generate a deformation field. Scale Invariant Feature Transform (SIFT) features (Lowe, 2004)
take into account the spatial information obtained from the gradient of the pixel intensities and
are invariant to brightness, rotation, and scale making them a suitable choice for use in histology.
65
Here we propose the use of DenseSIFT, which does not require the detection of key points. Our
proposed approach overcomes two of the problems mentioned above. First, using the reference
optical images acquired from the surface of the tissue blocks, before each section is cut, makes
the alignment of the histology images independent from the problem of dissimilarity of the
sections because of the spacing between them. Second, a noninvasive automatic feature based
image registration method has been used and optimized for breast specimens to find the
parameters suitable for deformation of the whole-mount breast histology sections. Using SIFT
features, which are invariant to brightness, rotation, and scale, eliminates the shortcoming of the
intensity based methods. To evaluate the accuracy of the proposed deformable registration
method, fiducial markers have been implanted in one breast specimen, which allow us to
calculate the target registration error (TRE) at each step of the pipeline accurately. For five
additional specimens, manually selected landmarks are used to validate the approach. The
average Target Registration Error using the optimized parameters are compared with the TRE
using the parameters used by Liu et al. (Liu, Yuen, & Torralba, 2011a) and visual assessments
are shown. Finally we compare the performance of the proposed deformable registration method
with the edge based method described previously.
4.2 Materials and Methods
This section describes the pipeline for co-registration of the histology images and their
corresponding blockface images. Rigid registration parameters are calculated by minimizing the
distance between the densely sampled Scale Invariant Feature Transform (SIFT) features (Lowe,
2004), (Liu, Yuen, Torralba, Sivic, & Freeman, 2008) of the two sets of images. The same
features are used to correct for the deformation of the histology images using a feature-based
image registration technique, SIFTFlow (Liu et al., 2011a). An iterative optimization approach is
used to estimate the optimum parameters for deformable registration of the breast histology and
blockface images. Both rigid and deformable registration are validated by visual assessment and
calculation of average TRE using multi-modality fiducial markers, which are detectable in
histology and reference images (R Shojaii & Martel, 2010). Moreover, manually selected
structural landmarks are also used for validation.
66
4.2.1 Specimen
Two sets of image data were acquired which we will refer to as the training set and the test set.
The training set of 10 whole-mount/blockface image pairs came from a single specimen and was
used to determine an optimum set of parameters for the SIFTFlow algorithm and to carry out the
comparison with the edge based approach. The test dataset, comprising 5 image pairs from 5
different patients, was used to test the registration accuracy obtained using the optimized
SIFTFlow method.
For the training data, we were able to obtain institutional research ethics board approval
to implant multimodality fiducial markers into part of a lumpectomy specimen from a patient
with invasive ductal carcinoma (the rest of the specimen was retained by the anatomic pathology
lab for routine clinical assessment). Four multimodality fiducial markers (black spaghetti), were
implanted in the fresh tissue using a biopsy needle (R Shojaii & Martel, 2010). The specimen
was suspended in 3.5% agar gel and four additional fiducial markers were placed in the agar gel
close to the specimen; these markers were not used for the histology to blockface registration as
the gel tended to become detached from the tissue during sectioning. The fresh tissue was sliced
at 4 mm. Figure 4-1-a shows one slice of the fresh tissue with the fiducial markers both in agar
gel and the specimen. The tissue slices were then formalin fixed, processed and embedded in
paraffin (Sun et al., 2009) to produce 10 blocks. One 4µm thick whole mount section was
obtained from each paraffin block.
Immediately prior to cutting each section, a blockface image was captured using a
telecentric lens. Imaging the blocks at a 90° angle provides an image with low contrast between
the tissue at the surface of the block and the surrounding paraffin and also between different
tissue types (Figure 4-1-b). Positioning both the lens and light source at an oblique angle (≈15
from 90 angle) produces a high contrast image (Table 4-1-c) but introduces a horizontal scaling
artefact. We correct for this by calculating a scaling factor based on the horizontal distances
between the fiducial markers in the gel. After applying the scaling correction, the oblique
blockface images are used as reference images. The pixel size of the blockface images was 186
µm and the corrected blockface images were cropped to 340x410 pixels to remove excessive
background.
67
The tissue sections were stained with Haematoxylin and Eosin (H&E) which provides
good contrast between fibroglandular tissue (purple), stroma (pink) and fatty tissue (mostly
white) and is the most commonly used histological stain. They were then digitized at 2µm
resolution, down-sampled to the resolution of the blockface images and padded with the average
background pixel values to the size of blockface images (Table 4-1-d).
Figure 4-1 (a) 4mm slice of the fresh tissue. (b) Blockface image (90 angle) (c) Blockface
image (oblique angle) (d) H&E stained whole-mount image.
For validation, a testing set of five pairs of blockface and histology images were also acquired
from lumpectomy specimens from five different patients. It was not possible to get approval to
implant fiducial markers in these specimens and only a single section was used from each
specimen. A smaller lens was used to obtain the blockface images for these testing images
therefore the resolution of blockface images and the down-sampled whole-mount images was 55
µm.
4.2.2 Dense SIFT Features
The Dense SIFT features (Lowe, 2004), (Liu et al., 2011a) are used in this work to find the
transformation of the histology images with respect to their correspondent blockface images.
SIFT features have been developed for automatic detection of landmarks in images and to find
objects in different scenes (Lowe, 2004). Based on the basic Dense SIFT algorithm, for each
pixel an eight bin histogram of gradient orientations is calculated for sixteen 4 x 4 neighboring
68
blocks, which results in 8 x 16 = 128 features. Therefore, for each w x h image, we have a 3D
SIFT image of dimension [w x h x 128] (Liu et al., 2008).
4.2.3 Affine Registration
The image pairs are first aligned using an affine registration. To find the rotation angle, shift
parameters and scaling factors we minimized the Euclidean distance between the SIFT features
of the histology and blockface images as formulated in (1).
Where S1 and S2 are the SIFT features of the blockface and the histology images respectively.
Variable θ is the rotation angle and tx and ty are the shift values in the x and y directions. Sx and Sy
are the scale factors, which correct for any shrinkage and expansion of the tissue sections during
processing. In the interest of speed, affine registration using fewer Dense SIFT features was also
investigated. Reducing the number of features to ten, i.e. w x h x 10, demonstrates results similar
to those for w x h x 128. Therefore the first ten features only were used for rigid registration.
Given that minimizing this objective function is an unconstrained multivariable optimization
problem, the Nelder-Mead algorithm (Nelder & Mead, 1965) is used to find the registration
parameter. This algorithm is a simplex-based direct search method that approximates a local
minimum of an optimization problem with multiple variables.
4.2.4 Deformable Registration
After the images have been roughly aligned using a similarity transformation, we use a
deformable image registration to further improve the correspondence between the 2 images.
Finding the pixel-wise displacement field is typically achieved by solving an optimization
problem. Liu et al. showed that the SIFT Flow objective function works well for deformable
registration of different scenes containing similar objects (Liu et al., 2008), (Liu et al., 2011a).
69
They adopted a discrete, discontinuity preserving, flow estimation algorithm to match the dense
SIFT features, instead of raw pixels (Liu et al., 2008).
The SIFTFlow objective function (2) comprises three terms; data term, small displacement term,
and smoothness term:
S1 and S2 are the dense SIFT features of the source and target images. Variable
( ) [ ( ), ( )]w p u p v p is the displacement of pixel p=(x,y). Parameter η controls the displacement
of the pixels and α controls the smoothness of this displacement with respect to the neighbouring
pixels. Small values for these parameters allow for high deformations while higher values allow
for milder deformations. The smoothness term, which encourages pairwise smoothness in the
pixel displacements, is controlled by α and is bounded by a threshold parameter d. The use of a
truncated L1 norm for the smoothness term accounts for outliers in the deformation field such as
the presence of large tears where neighbouring pixels may be subject to very different
deformations.
4.2.5 SIFTFlow Parameter Optimization
Liu et al do not discuss their choice of regularization parameter values in their papers (Liu et al.,
2008) (Liu et al., 2011a) suggesting that that the default parameter settings of η=0.005, α=2 and
d=40 used in their software implementation (Liu, Yuen, & Torralba, 2011b) were suitable for the
many image types used in (Liu et al., 2011a) to illustrate the SIFTFlow method. Our task of
registering breast histology images to blockface images, however, is very different to that of
registering natural images due to the presence of tears, folds and other artefacts in the histology
slices and it is necessary to determine what effect the parameter settings have on registration
accuracy and also to identify the most appropriate settings to use.
70
We developed the iterative optimization method summarized in Figure 4-2 to determine the most
appropriate parameter settings for lumpectomy specimens. In the first step, the objective function
(2) is minimized to find the displacement field using the initial parameter values from (Liu,
Yuen, & Torralba, 2011b). In this step the efficient belief propagation method (Felzenszwalb &
Huttenlocher, 2006; Liu et al., 2011a) is used to find the deformation field. In the second step,
the objective function is optimized using the Nelder-Mead optimization algorithm to find
updated parameter estimates for the displacement field obtained from the first step. The new
estimated parameters are then used as inputs to the first step to find a new displacement field.
These two steps iterate until either the change of the function value is lower than an upper limit
or it reaches a maximum number of iterations.
Figure 4-2 Flow chart outlining the joint parameter optimization method
71
Since implantation of the validation fiducial markers was permitted for only one clinical
specimen, which resulted in ten pairs of images, Leave-One-Out Cross Validation (LOOCV) is
adopted to investigate the efficacy of parameter estimation approach. The formulated objective
function with LOOCV is shown in equation (3).
The summation on nImage is added for LOOCV.
In order to make sure that the optimization was not influenced by the presence of the fiducial
markers, the regions of the images with fiducial markers were marked manually for each pair of
histology and blockface images after the rigid alignment of the images and then excluded from
the calculation of the objective function.
4.2.6 Registration using Edge Points
The edge-based registration techniques introduced in chapters two and three are also applied to
the whole-mount lumpectomy images in order to make a comparison with the SIFTFlow method.
For the initial affine registration step, the segmented boundary points are aligned using Fourier
descriptors (Kuhl & Giardina, 1982), (Duan et al., 2008) and then the registration is refined using
the Iterative Closest Points (ICP) (Besl & McKay, 1992) algorithm. The aligned boundary
points are then used as control points to find the non-rigid deformation and a Thin-Plate Spline
(TPS) (Bookstein, 1989) is used to model the deformation field.
The edge-based approach is simple to apply when there is a good contrast between the tissue and
background in both histology and blockface images. In breast histology images, however, there
is very low contrast between the fatty tissue and the background, which makes the automatic
segmentation of the boundary points very challenging. Therefore in this work we manually
segmented the boundaries of the tissue in both sets of images.
72
4.2.7 Validation
The rigid alignment of the image pairs in the training set was assessed by measuring the mean
target registration error (TRE) based on the centre of the four fiducial markers implanted in the
specimen. In order to measure the deformable error we also selected some internal landmarks in
each pair of images and calculated the mean TRE based on these internal points. We attempted
to identify an equal number of evenly distributed landmarks in each image pair for both the
training and the testing datasets.
4.3 Results
4.3.1 Affine Registration
Figure 4-3 shows an example of one pair of blockface and histology images together with the
aligned histology images obtained by using the edge-based and the feature-based methods
respectively. The images are overlaid by grids to facilitate the visual comparison of the image
alignment. The large amount of fatty tissue leads to very poorly defined tissue boundaries,
particularly in the histology images, and it was necessary to manually define boundaries for the
edge based approach.
The TRE calculated using the centre of the implanted fiducial markers in the tissue is reported in
Table 4-1.The improvement in TRE when using dense SIFT feature-based method is small with
respect to the edge-based method, but the feature-based method has the advantage that it is not
dependent on image segmentation. The results in Table 4-1 also show that using a similarity
transformation, which allows for scaling, improves the alignment compared with a simple rigid
transformation.
Table 4-1. Target Registration Errors for rigid and affine registration. Values are the
mean TREs (± stdev) based on implanted fiducial markers
Rigid registration Affine registration
Edge-based Feature-based Edge-based Feature-based
3.88 ± 0.87 mm 3.55 ± 0.5 mm 2.93 ± 1.82 mm 2.88 ± 0.24 mm
73
The histology images were aligned with their correspondent blockface images using dense SIFT
features. The TRE calculated using the centre of the implanted fiducial markers in the tissue is
reported inError! Reference source not found..
Figure 4-3 Rigid registration of a histology image to the corresponding blockface image; a) blockface
image with boundary in blue (figure best viewed in colour). b) histology image with manually defined
boundary shown in red and blockface boundary in blue. Aligned histology images: c) using edge-based
method and d) using dense SIFT features. The boundary of the blockface image is overlaid on both
aligned histology images for comparison.
74
4.3.2 Deformable Registration
4.3.2.1 Effect of regularization parameters on SIFTFlow registration
Figure 4-4 illustrates the effect of changing the parameters controlling the magnitude of the
displacement of the pixels η and the smoothness of this displacement, α. White areas on the
images represent regions where compression has occurred.
Figure 4-4 The deformation fields calculated with different settings for η and α are applied to a
grid image in order to visualize the effects of changing the regularization parameters. The top
row shows that increasing α results in smoother displacement: a) η =0.005, α =1. b) η =0.005, α
=2. The bottom row shows the effect of increasing η: c) η =0.0005,α =2 d) η=0.1, α =2. d was
fixed at the default value of 40 in all cases.
75
The effect of changing each of the regularization parameters was assessed by holding 2 of the
parameters fixed at the default values and then varying the third parameter. The plots in Figure
4-5 show the mean TRE calculated using the fiducial markers in all 10 training image pairs as
each parameter is varied over a range of values. We see that the largest variation is caused by
changes in α, the smoothness parameter, and Figure 4-5-c) suggests that changing d has very
little effect once d>20. We therefore kept d fixed at the default value of 40 when carrying out the
multivariate optimization.
4.3.2.2 Identifying paired landmarks for the assessment of deformable registration
Most of the landmarks were selected at the corners of the fibroglandular tissue and a few at the
cross-section of the blood vessels, where the correspondence of the points in histology and
blockface images could be determined. It was not possible to identify landmarks in the fatty
tissue with confidence. Figure 6 shows two pairs of blockface and histology images. The smaller
section contains mostly fat, while the larger section has more fibroglandular tissue making it
possible for more landmarks to be identified. The number of manually selected landmarks varied
for different sections in the training dataset as there was a wide variation in section size and
composition and, although we attempted to evenly distribute landmarks throughout the images,
Figure 4-5 Sensitivity of TRE (measured using the implanted fiducial markers) to the SIFTFlow
parameters: a) η, b) α, and c) d.
76
this was not always possible as can be seen in figure 6. The number of landmarks in the 10
training sections varied from 4 to 10.
The 5 sections taken from 5 different patients were all selected through the centre of the
specimens and as well as being larger, they also contained more fibroglandular tissue, allowing
us to identify 10 landmarks in each test image pair.
Figure 4-6 Two pairs of blockface and histology images with different sizes and composition.
Top row shows the a) blockface and b) histology images of a small section, which mostly
contains fat. Bottom row shows the c) blockface and d) histology images of a larger section with
more fibroglandular tissue and higher number of selected landmarks. The arrows on the
blockface images show the selected landmarks.
77
4.3.2.3 Comparing edge-based deformable registration with SIFTFlow.
The procedure outlined in Figure 4-2 was used to estimate the optimum parameters η and α in the
SIFTFlow objective function (d was fixed at 40 throughout). The optimized parameters and the
associated mean TREs (calculated using both the implanted fiducials and the manually defined
landmarks) from the LOOCV experiment using ten pairs of images are reported in Table 4-2.
Table 4-2 Average TREs for LOOCV experiment using edge-based, SIFTFlow with default
parameters, and SIFTFlow with optimized parameters measured using both fiducial markers and
manually selected landmarks. Values are given as mean (stdev) in mm.
Mean TRE (mm) Fiducial Markers
Mean TRE (mm) Manual landmarks
Case
# η (e-3) α
# of
land-
marks
Edge-based Default
SIFTFlow
Optimized
SIFTFlow Edge-based
Default
SIFTFlow
Optimized
SIFTFlow
02 5.185 1.149 5 7.46 (0.00) 8.09 (0.00) 7.56 (0.00) 6.82 (0.00) 7.83 (1.80) 6.24 (1.50)
03 5.397 1.129 5 3.1 (0.00) 3.03 (0.00) 0.41 (0.00) 2.97 (0.00) 3.86 (0.95) 0.43 (0.07)
04 5.283 1.137 8 5.39 (0.57) 1.56 (1.78) 0.27 (0.05) 5.13 (1.10) 1.50 (1.53) 0.71 (0.93)
05 5.279 1.137 10 1.22 (0.63) 0.44 (0.49) 0.36 (0.54) 1.09 (0.74) 0.80 (0.88) 0.79 (0.90)
06 5.080 0.913 10 0.54 (0.28) 0.46 (0.25) 0.17 (0.16) 0.98 (0.31) 1.17 (1.60) 0.79 (1.12)
07 5.125 1.160 10 1.38 (0.67) 0.43 (0.22) 0.22 (0.16) 1.14 (0.53) 0.44 (0.63) 0.30 (0.19)
08 5.215 1.149 8 1.56 (0.32) 1.07 (1.17) 0.24 (0.06) 1.52 (0.40) 0.79 (0.70) 0.37 (0.15)
09 5.442 0.862 5 1.89 (0.15) 0.81 (0.64) 0.25 (0.23) 1.78 (0.20) 1.10 (1.25) 0.56 (0.18)
10 5.125 1.160 4 0.81 (0.00) 0.19 (0.00) 0.19 (0.00) 0.76 (0.00) 0.21 (0.15) 0.22 (0.15)
11 5.282 1.137 4 077. (0.00) 0.45 (0.00) 0.18 (0.00) 0.68 (0.00) 0.52 (0.53) 0.51 (0.51)
78
In image 2 the error is very high; this is because a part of the tissue section is missing on the
histology slide leading to a large discordance between the stained section and the corresponding
blockface image.
For the TREs measured using fiducial markers, the results confirm that optimizing the
regularization parameters significantly improves registration accuracy (p = 0.0088 using a paired
two tailed t-test, normal distribution of TREs was confirmed using a Kolmogorov-Smirnov test).
The average difference between the TRE obtained from the optimized and default parameters is
0.67mm which corresponds to approximately 4 pixels at 186 µm resolution. The results of the
deformable registration using the edge-based method are also shown in Table 4-2. Errors were
significantly higher than those obtained with the SIFTFlow method.
Figure 4-7 shows one of the blockface images and the corresponding histology image after
registration with each of the methods tested. The circle on Figure 4-7-a) and Figure 4-7-b)
highlights an area of tissue that is missing on the histology slides. In Figure 4-7-b) it appears, as
expected, that structures close to the edge of the tissue are well aligned but structures further
away from the boundary points are not as well registered with those in the blockface image. In
Figure 4-7-c) and Figure 4-7-d) arrows point out areas of difference in the default and optimized
SIFTFlow results.
In order to obtain a single set of SIFTFlow parameters to use in the test dataset we
repeated the optimization procedure using all 10 image pairs and obtained values of η=4.9x10-3,
α = 1.169.
79
Figure 4-7 Images from case # 04; a) Blockface image. b) histology registered using edge-based
method, the circle shows a missing part of the tissue section in histology image, which
can degrade the result of the registration. c) histology registered using default SIFTFlow
parameters. d) histology registered using optimized SIFTFlow. The arrows point to some
of the areas that are more accurately registered when the optimized parameters are used.
80
4.3.3 Accuracy of optimized SIFTFlow on an independent test dataset.
In order to confirm that parameters optimized using the training dataset are suitable for different
breast specimens, we applied both these parameters and the default parameters to five pairs of
blockface and histology images from 5 different patients and the results are presented in Table
4-3. The edge based method was not applied to this data as the results from the training data
clearly indicated that it is inferior to the SIFTFlow method.
TREs were comparable with those obtained in the training set and the optimized parameters
resulted in an improvement over the default parameters. The mean registration error was
calculated as 0.32±0.22mm for this independent test set.
Table 4-3 TRE based on the internal structures of five histology sections from five different
patients with the default parameters (η = 0.005 α = 2.00) and the optimized parameters
(η=4.9x10-3, α = 1.169).
Image Default parameters TRE ± stdev (mm)
Optimized parameters TRE ± stdev (mm)
Pt_1 0.40± 0.25 0.35 ± 0.21
Pt_2 0.57± 0.34 0.49 ± 0.38
Pt_3 0.47± 0.34 0.35 ± 0.21
Pt_4 0.10± 0.07 0.10 ± 0.06
Pt_5 1.58± 2.44 0.33 ± 0.26
4.4 Discussion
In this work we have demonstrated that SIFTFlow is successful in aligning blockface and
histology images and we have developed a method of tuning the regularization parameters used
by this approach.
SIFTFlow removes the need for image segmentation and is more accurate than our previously
described edge-based approach. Another advantage is that it uses SIFT features, which are scale,
rotation, and brightness invariant and relatively robust to variations in illumination and intensity
in different images. This means the same technique could be used for registration of histology
with other imaging modalities such as ex vivo MRI or OCT or for the alignment of histology
images with different complementary stains.
81
For the rigid registration step we found that it was useful to use a similarity transformation model
rather than a rigid transformation suggesting that the tissue sections may expand when they are
placed in a water bath during the mounting procedure. We also found that a lower value of α than
that suggested by Liu et al., significantly improved the results of registration. The mixture of
highly deformable fatty tissue and less deformable fibro-glandular tissue in breast tissue
specimens, together with the presence of folds and tears which inevitably occur when mounting a
very flexible 4µm tissue section onto a glass slide, leads to the presence of more complex
deformation fields in histology images than would have been present in the natural images used
to demonstrate the SIFTFlow method in (Liu et al., 2011a). Lower values for parameter α have
the effect of relaxing the smoothness constraint in the objective function and this allows for
higher variability in the deformation field. Our method for carrying out a joint optimization of
the regularization parameters allows SIFTFlow to be tuned for specific tissue types which could
have very different biomechanical properties leading to differences in the range of deformations.
One of the limiting factors of the proposed parameter estimation approach for SIFTFlow is that it
is sensitive to the initial values. In this work we used the parameters published by Liu et al. as
initial values for optimization. We are currently working on a pre-optimization fast search
algorithm to find suitable ranges for initial values to overcome this problem.
In this work detection of misregistered images due to the convergence of the optimizer to a local
minimum has been done manually. In future, an automatic method for registration error detection
can be employed to automate this process. For example, in order to evaluate the non-rigid
alignment of pre-operative brain MRI with intraoperative images, Fedorov et al. (Fedorov et al.,
2008) measured a modified Hausdorff distance between the edges segmented from both set of
images images. This method can be applied on breast images by measuring the distance between
extracted features from the histology and blockface images. The applicability of this method for
breast images will be investigated in future.
The motivation of this work was to correct for deformations in the histology images prior to
reconstructing a 3D volume. We make the assumption that the blockface images are free from
distortion and can therefore be used as a reference image. In practice significant distortions occur
as a result of the tissue processing steps carried out in order to embed the tissue block in paraffin
and these will also have to be accounted for if the 3D volume is to be subsequently aligned with
3D ultrasound or MRI images.
82
Although our results are encouraging, in applications where a region of interest in an aligned
histology image needs to be assessed at a higher resolution even a sub-millimeter misalignment
could become very significant. For example, when a region of interest in a medical image is
correlated with the correspondent region in a histology image, the assessment of the region at
higher resolution is required to reveal the underlying structures in the histology image. An
average difference of 0.67mm corresponds to an error of four pixels at 186 µm resolution and
335 pixels at 2 µm resolution, which means that zooming in a region of interest selected at low
resolution could lead to the selection of a totally misaligned patch at a higher resolution. In
future the accuracy of the image registration will be further improved by the employment of
multi-scale image registration methods.
4.5 Summary and Conclusion
This chapter presents an automatic approach for correcting for deformations in whole-mount
histology images from human breast lumpectomy specimens. The contribution of the work is
two-fold: 1) it is demonstrated that a noninvasive, automatic, feature-based image registration
method can be applied to the task of aligning histology images with optical blockface images; 2)
a method is proposed for optimizing this method and shown that this significantly reduced target
registration error.
Quantitative and qualitative evaluations of the image registration results for different breast
specimens demonstrate lower registration error when optimized parameters are used. The
optimized feature-based method also outperforms a previously described edge-based method.
This method will be used in the next chapter as a pre-processing step in the generation of 3D
volume reconstructions of whole-mount histology images.
83
5 Breast Histology Volume Reconstruction and 3D Tumour Measurement
Chapter five describes how the image registration approach described in previous chapters, can
be extended to reconstruct 3D histology volumes from 2D whole-mount slides of breast
specimens. In order to achieve this, two main challenges have had to be overcome. In the clinical
setting, the breast specimen is cut into thick breadloaf sections before paraffin embedding which
means that an additional alignment step is required. Furthermore, only a single wholemount
section is cut from each block therefore the sampling is very sparse. This chapter presents an
approach for automatic aligning reference blockface images followed by histology volume
reconstruction from breast wholemount images. The feasibility of using the reconstructed
volumes to assess two 3D tumour features, i.e.tumour volume and tumour focality, is
demonstrated.
5.1 Introduction
In the era of digital pathology, the study of the 3D microscopic features of organs and tissue
structures has been the research topic of different groups. While some groups study the biology
and structure of breast (Going & Moffat 2004), (Rusby et al. 2007), others have focused on the
3D features of breast diseases, e.g. Ductal Carcinoma In Situ (DCIS) (Booth et al. 2015). Clarke
et al. have also shown that tumour volume can be a better estimate of the extent of the disease
when calculated based on the 3D histology volume reconstructed from whole-mount histology
slides compared to the 2D measurements (Clarke et al. 2012).
Reconstruction of 3D histology volumes is also an important intermediate step in co-registration
of pathologic findings in volumetric medical images, such as MRI, CT, and US, with the
histology images as a ground truth (Sztrókay et al., 2012), (Tot & Gere, 2016).
84
An accurate histology volume reconstruction is hindered by the spacing between the sections as
well as the local distortions resulting from histological preparation procedures, including
shrinkage, expansion, tears, and folds. Both the spacing and the local distortions reduce the
similarities between the consecutive sections and consequently make the image registration more
challenging.
While several approaches have been reported for histology volume reconstruction (Gefen,
Tretiak, & Nissanov, 2003), (Ju et al., 2006), (G Malandain, Bardinet, Nelissen, & Vanduffel,
2004), (Nikou, Heitz, Nehlig, Namer, & Armspach, 2003), (Wirtz, Fischer, Modersitzki, &
Schmitt, 2004), there are only a few works in the field of breast histology volume reconstruction.
Clarke et al. have manually aligned breast whole-mount slides to reconstruct the breast histology
volume to correlate it with CT images (G. M. Clarke et al., 2008) or to compare the 3D with 2D
tumour measurements (G M Clarke et al., 2012). Booth et al. (Booth et al., 2015) investigated the
use of 3D reconstruction of DCIS in the study of the biology of DCIS. They reconstructed the
histology volume by selecting the middle section as reference image and registering the
neighbouring sections to this reference image (Roberts et al., 2012). This registration proceeds
out from the centre, with subsequent sections aligned to their neighbours. Hawkes et al. (Reis et
al., 2015) investigated the maximum spacing between the breast histology sections required for
3D histology reconstruction to a specific tolerance. Rather than the volume reconstruction from
the whole-mount sections, Hawkes et al and Booth et al extracted small regions around structures
of interest from the specimen which were then finely sectioned. They selected the section at the
centre of the structure as the reference image and used a pair-wise rigid or affine registration
approach to align the sections from the same structure to the reference image. Since the spacing
between the breast sections in the clinic is about 4mm, the similarity of the sections is
diminished and choosing one section as the reference image for a breast specimen is not
advantageous.
The existing methods for 3D histology volume reconstruction of breast specimens are either
manual or applicable to small samples of the breast tissue, which are finely sectioned to preserve
the continuation of the tissue structures, which are used for image registration.
In chapters two and three an image registration framework was presented for histology volume
reconstruction. The workflow was validated for preclinical applications. As it is explained in
85
section 3.3.6.4, this framework makes the reconstruction of 3D histology volumes independent
from the problem of dissimilarity of the sections due to the spacing between them by using
blockface images. In the work of this chapter the deformable registration approach of the
pipeline is replaced with the feature based image registration method introduced in chapter four
and optimized for breast images. The reference blockface images obtained from small pre-
clinical specimens embedded in one block are well aligned with each other. Whereas for large
clinical specimens, the blockface images obtained from multiple tissue blocks are misaligned.
This problem is also addressed in this chapter. It is also demonstrated that it is feasible to use the
reconstructed histology volume for tumour volume measurement. A strategy is also introduced to
address the effect of tumour multifocality in volume measurement.
5.2 Materials and Methods
5.2.1 Overview
In the proposed pipeline, the histology images are registered to their corresponding blockface
images. Using blockface images eliminates the propagation of registration error, which
commonly happens when registering the histology images to one another without reference
images. The blockface images obtained from multiple tissue blocks of a large clinical specimen,
such as breast lumpectomy specimens, are misaligned and need to be registered. This chapter
explains a strategy for alignment of the blockface images. The histology images are then
registered to the aligned blockface images to reconstruct the histology volume. In our pipeline
for deformable registration of histology to blockface images we use a feature-based image
registration technique, SIFTFlow, which we have previously optimized for correction of the
deformation of the breast lumpectomy specimen. The pipeline is validated by visual assessment
and calculation of Target Registration Error (TRE) using internal landmarks, which are
detectable both in histology and reference images.
86
5.2.2 One Specimen and Multiple Tissue Blocks
As it is explained in chapter one, excised breast specimens are cut into multiple 4mm slices in
order to increase the penetration of the fixatives and other required chemicals for histology
processes. While these processed tissue slices are being embedded in paraffin they are rotated
and shifted with respect to each other, Figure 5-1-a. Therefore, unlike the reference blockface
images obtained from a small pre-clinical specimens embedded in one block, the blockface
images obtained from multiple tissue blocks of a larger clinical specimen are misaligned. In
order to be able to retrieve the correct position of the blockface images three fiducial markers are
inserted in the gel around the tissue before the tissue is sliced, Figure 5-1-b. A guiding device
(Figure 5-1-c) was designed and built at the in house machine shop, and used to accurately insert
the fiducial markers. Based on the results shown in chapter three, black spaghetti was used as
fiducial markers for alignment of the blockface images. Black spaghetti gradually absorbs water
from the surrounding gel and softens. Therefore it has no interference with slicing the specimen.
Due to the short ischemic time for human breast specimen in clinic, the softening process of the
fiducial markers was boosted up by immersing them in 45˚C water for 8 minutes. A set of
experiments with different water temperatures and immersion periods showed that 45˚C and 8
minutes are suitable for the softening process to start while the core of the spaghetti is still rigid.
After the insertion of the fiducial markers, the softening process still continues due to the water
absorption and by the time the gel tissue block is cooled down, the rigid cores are also softened
and ready for slicing. The centres of these fiducial markers can later be used to align the
blockface images as it is explained later in section 5.2.4.
87
Figure 5-1 Illustration of a) the misalignment of tissue slices, when they are embedded in
paraffin separately, b) the use of fiducial markers inserted in gel around the tissue for correcting
the alignment of the blockface images obtained from multiple tissue slices, d) the guiding device
for accurate insertion of the fiducial markers.
5.2.3 Specimen
The image pairs of the histology sections from all tissue slices of five lumpectomy specimens
and a part of a lumpectomy from the work explained in chapter four were used for the work in
this chapter. While the specimens were suspended in 3.5% agar gel four fiducial markers were
placed in the agar gel close to the specimen as it is shown in Figure 5-2(a). Three fiducial
markers were perpendicular to the cutting plane and one is at a 30˚ angle to be able to track the
alignment of the fiducial markers in the z-direction. These fiducial markers in the agar gel are
used to align the blockface images from different breadloafed slices of the tissue. After the agar
gel is set in the freezer for 30 minutes the tissue block is trimmed and mounted on a platform for
CT scanning, Figure 5-2(b), at the isotropic resolution of 0.1mm (500 projections, 45 KV, 100
mA, 200 ms). The volumetric CT images are used for validation of the alignment of the
blockface images.
2cm
a) b) a)
88
The fresh tissue block is then breadloafed into 4 mm slices (Figure 5-2-c). Figure 5-3-a and
Figure 5-3-b show a CT image and a slice of the fresh tissue respectively. The tissue slices were
then formalin fixed, processed and embedded in paraffin (Sun et al., 2009). One 5µm thick
whole mount section was obtained from each paraffin block along with its blockface image with
the resolution of 186 µm for the partial lumpectomy and 55 µm for the five whole lumpectomy
specimens. As explained in chapter four, a telecentric lens was used to capture two sets of
blockface images were captured; one by placing the lens straight in front of the object (Figure
5-3-c) and the other one with both the lens and light source at slight angle (≈15◦) (Figure 5-3-d).
As explained in section 4.2.1, the angled images provide higher contrast between the tissue at the
surface of the block and the surrounding paraffin, but since the lens is at angle, foreshortening
artefact in horizontal direction occurs. The straight images, on the other hand, have lower
contrast between the tissue at the surface of the block and paraffin, but with no foreshortening
artefact and as a result can be used to correct the angled images. The sections were then stained
with H&E (Figure 5-3-e) and digitized at 2µm resolution for the partial lumpectomy and 0.5µm
for the rest of the specimens. The histology images were then down sampled to the resolution of
the blockface images.
Figure 5-2. (a) The jig to guide the fiducial markers. (b) Tissue block fixed on a platform
for CT scan. (c) 4mm slices of the tissue block.
89
5.2.4 Alignment of the blockface images
This section explains an approach for blockface image alignment. This approach is tested and
validated using a part of a lumpectomy specimen, which has validation fiducial markers in the
specimen. The blockface images of the five specimens were then aligned using the same
approach.
The blockface images were first cropped to 680x820 to remove the excessive background. The
angled images are first corrected for the horizontal foreshortening artefact, section 4.2.1, and
then registered through the alignment of the fiducial markers implanted in gel block. A 3D line is
first fit to the set of points at the centre of the fiducial markers. The best fitting line has been
used, which passes through the mean of the data points, p( , , ̅), in the direction of the
eigenvector of the largest eigenvalue v(a, b, c). The intersection of the fitted lines and the
different z-planes, or different blockface images, can be found by using the vector representation
of the line:
⟨ , , ⟩ = ⟨ + , + , + ⟩ (5-1)
z = Z level of each section
t = (Z- )/c
X = + , Y = +
a) b) c) d) e)
Figure 5-3. (a) CT image. (b) 4mm slice of the fresh tissue. (c) Straight blockface image.
(d) Angled blockface image. (e) H&E stained histology image.
90
The rigid transformation parameters, shift value and the angle of rotation, are then obtained by
minimizing the Euclidean distance between the centre points of the fiducial markers and their
correspondent intersection points of the fitted lines at each z-plane. In the next iteration, a line is
fit to the transformed centre points and then the intersect points and the transformation
parameters are recalculated. This iterative process aligns the centres of the fiducial markers along
3D straight lines after approximately three iterations.
The alignment of the blockface images are compared to the 3D orientation of the fiducial
markers segmented from volumetric CT images of the whole specimen in gel block, which
obtained before slicing the tissue.
5.2.5 Outlining the Tumour Regions
The tumour regions were outlined on histology images by a pathology assistant (PA) using
Sedeen Viewing Program (Selective Decoding and Encoding Engine (PathCore, 2016);
developed by Dr. Anne Martel and Danoush Hosseinzadeh at Sunnybrook Research Institute).
5.2.6 Histology Image Registration
To reconstruct the histology volumes, the histology images are registered to their correspondent
blockface images. As explained in chapter four, dense SIFT features were used for both affine
and deformable registration of the histology images. For a few pairs of images with a very few
low contrast structures, rigid registration using dense SIFT failed. Therefore, Mutual Information
(MI) technique (Mattes et al., 2001) was used for the affine alignment of these images. Dense
SIFT technique outperformed MI for the rest of the image pairs.
SIFTFlow deformable registration method, which was optimized for breast images (explained in
chapter four), was used to correct the deformation of the breast histology images with respect to
their correspondent blockface images.
5.2.7 Validation
The alignment of the blockface images are visually assessed using the volumetric CT images.
91
The performance of the image registration is evaluated by calculating the mean TRE based on
ten manually selected anatomical landmarks on both histology and blockface images.
5.2.8 Tumour 3D Measurements
It has been shown that tumour volume may be a more accurate surrogate for size (G M Clarke et
al., 2012). Tumour volume calculated from the 2D measurements, i.e. using the 2D maximum
tumour diameter, might overestimate volume compared to the 3D volumetric calculation (G M
Clarke et al., 2012). We calculated the tumour volume of one of the cases with large Invasive
Lobular Carcinoma (ILC) to show the feasibility of using the 3D histology volumes for 3D
tumour measurements. We also considered the effect of tumour multifocality in volume
calculation and visualization.
5.2.8.1 Volume
The boundary points of the tumour regions, outlined by a pathologist assistant, were used as
surface points of the tumour for 3D shape visualization and measurement of the tumour. The
boundary points were read from an XML files created by Sedeen Viewer and turned into binary
masks. To calculate the tumour volume for the largest foci, the area of the largest connected 3D
object was multiplied by the voxel size. The overlapping tumour regions in 3D were considered
to be connected. On the other hand, non-overlapping tumour regions can either represent
different foci or be branches of a single main foci which are connected deeper in the tissue.
In the latter case, all connected regions should be identified and included in the tumour volume
calculation. The problem of multifocality in volume measurement has been addressed in the next
section.
5.2.8.2 Tumour Focality
Tumour focality has a great impact on the estimation of tumour extent. Isolated tumour regions
in the histology sections are considered as separate foci, but the extension of these regions to the
main tumour deeper in the tissue slice is not clear from the 2D histology sections cut from the
92
surface of the block. To address this problem, X-ray images of the tissue blocks were obtained
and observed to verify the connectivity of the tumour regions detected in the histology section
cut from each block, shown inFigure 5-8. A Faxitron X-ray machine with 30 kV was used to
image the tissue blocks. If the isolated tumour region was not connected in the X-ray image of
the correspondent block, the X-ray image of the previous block was also observed for
connectivity between the regions. The isolated tumour regions, which are connected to the
largest foci are labeled the same as the largest object in the label image created from the binary
masks of the tumour regions and therefore included in the volume measurement of the largest
foci.
5.3 Results
5.3.1 Alignment of the blockface images
The blockface images were aligned by using the implanted fiducial markers in gel. Figure 5-4
shows the fiducial markers and the fitted lines before and after the alignment (images on the left
and on the right respectively).
Figure 5-5-a shows all fiducial markers both in gel and in tissue in a 3D diagram. The bend in the
middle of two fiducial markers corresponds well with the fiducial markers in volumetric CT
images, Figure 5-5-b. In order to have a better 3D view of the fiducal markers in CT images,
Figure 5-5-c, they are segmented using active contour segmentation method in ITK-Snap
(Yushkevich et al., 2006).
93
Top View (before registration) Top View (after registration)
Figure 5-4.Alignment of the blockface images using the fiducial markers in gel.
Centre points of the fiducial markers are shown in blue and the intersection points of
the fitted lines in red. a) Fiducial markers and the fitted lines before alignment, b)
fiducial markers after alignment, c) top view of the fiducial markers before
registration, d) top view of the fiducial markers after registration.
94
Figure 5-5. Comparison of the alignment of the fiducial markers in (a) blockface
images with (b) CT images. c) Segmented fiducial markers from CT images
5.3.2 Histology Image Registration and Validation
We selected ten internal landmarks in each pair of histology and blockface images and calculated
the average TRE based on these internal points. As it is reported in Table 5-1 the average TRE
after deformable registration is lower for each patient.
A paired two sample t-test was also conducted to compare the TRE following rigid and
deformable registration. The average TRE based on the fiducial markers, Table 5-1, were used to
perform t-test. The result of the t-test (two-tail p-value=0.006 < 0.05) also shows that the
deformable registration has significantly improved the registration, with the average TRE of 2.12
± 1.03 mm for rigid registration and 0.47 ± 0.31 mm for deformable registration.
c) b) a)
95
Table 5-1. TRE based on the internal structures of five specimens
from five different patients for rigid and deformable registration
Patient Rigid
# of Pixels Deformable # of Pixels
Rigid TRE ± Std
(mm)
Deformable TRE ±Std
(mm)
Pt_1 30.72± 7.72 4.2 ±1.06 1.69±0.41 0.23 ± 0.06
Pt_2 10.72 ± 4.09 3.34±1.84 0.59 ± 0.22 0.18 ± 0.1
Pt_3 52.38 ± 33.54 16.07 ± 11.78 2.9 ± 1.85 0.88 ± 0.64
Pt_4 41.03 ± 18.45 13.25 ± 9.97 2.25 ± 1.02 0.73 ± 0.55
Pt_5 57.88 ± 24.98 6.13 ± 4.74 3.18 ± 1.37 0.33 ± 0.26
5.3.3 Tumour Volume and Multi-focality
Figure 5-6 shows a screen shot of sedeen viewer with two outlined tumour regions.Tumour
regions for patient Pt_4 were detectable on eight histology sections, Figure 5-7-a, out of thirteen
sections. Figure 5-7-b shows the stack of the tumour regions after the alignment of the images.
96
a)
b)
Figure 5-7 3D stack of the tumour regions. a)
before registration. b) Overlay of the tumour
regions on the histology sections after
alignment
Figure 5-6. Screen shot of Sedeen viewer software showing two outlined tumour regions.
97
Figure 5-8 shows the X-ray images of two 4mm tissue blocks. Figure 5-8-a to d show the
separation of different tumour regions in the X-ray images of the same tissue block and one
block before. The X-ray image of another block show that a separate tumour region in Figure-e
is connected to the main focus deeper in the tissue, Figure 5-8-f and g. Therefore, it is labeled as
main tumour and included in the volume measurement of the largest focus. The connection of the
tumour regions based on X-ray images is not certain, because they might be overlapping but not
connected. However, the assumption is that the overlapping regions are connected.
The 3D view of the tumour regions is shown in Figure 5-9-a before registration and the largest
focus of the tumour is shown in Figure 5-9-b after removing the non-overlapping tumour
regions.
As it is shown in Figure 9-c more separate regions overlapped the main tumour after image
registration and included in the measurement. Figure 9-d shows the inclusion of a separate region
to the main tumour after X-ray images showed its connection to the main tumour.
98
f)
b)
c)
d)
e)
a)
g)
Figure 5-8. Verification of tumour focality using X-ray images of the tissue blocks. a)
Histology image with one non-overlapping tumour region shown with large arrow. b) X-ray
image of the correspondent block. c) The region with ambiguous connectivity in X-ray
image, larger arrow, is zoomed in and shows no connectivity, small arrow, between the
regions. d) The same region in the X-ray image of the previous block shows no connectivity
either. e) A histology image from a different block with one non-overlapping tumour
regions shown with large arrow, f) the correspondent X-ray image. g) The region with a
non-overlapping region is zoomed in and shows the connection, specified with a small
arrow, of the isolated region to the main tumour region.
99
Figure 5-9. 3D tumour volume, a) original
tumour regions. b) main tumour after
removing the non-overlapping tumour
regions, c) after registration, d) after X-ray
assessment.
(µm)
(µm) (µm)
(µm)
(µm)
(µm)
(µm)
(µm)
(µm)
(µm)
(µm)
(µm)
a)
(b
(c
(d
100
The tumour volume was also calculated based on the number of the voxels multiplied by the
voxel size in four different steps, before the image alignment, after the alignment and after the
isolated tumour regions were included in the tumour volume measurements if the x-ray images
revealed their connection to the main tumour. The results are shown in Table 5-2.
Table 5-2 Largest tumour volume,
calculated before and after alignment and after multi-modality process
no registration rigid deformable after X-ray
assessment
Volume of largest foci (cm3) 1.38 1.59 1.57 1.65
5.4 Summary and Discussion
In the work presented in this chapter, the image registration framework developed in the
previous chapters [(R. Shojaii et al., 2011), (Rushin Shojaii et al., 2014)] was employed to
reconstruct a 3D histology volume from the images obtained from sparsely sampled human
breast specimen. A practical approach was also presented to align the blockface images, which is
required for this image registration pipeline. The feasibility of evaluating of 3D tumour features
using the reconstructed histology volume was also assessed in this chapter. The X-ray images of
the tissue blocks were used to address the problem of tumour multi-focality in 3D tumour
reconstruction and tumour volume measurement.
An advantage of using blockface images to align the histology images is that the spacing
between the histology images does not affect the quality of their alignment. This is important in
the clinical setting where the spacing between sections can vary widely, and the gap between
sections may be as great as half a centimetre.
Although this image registration pipeline requires fiducial markers in the gel near the tissue to
align the blockface image, it does not require selection of internal landmarks or implanted
fiducial markers within the tissue, which might distort the tissue.
101
The alignment of the blockface images were evaluated by visual comparison of the alignment of
the fiducial markers in blockface images with the 3D orientation of the fiducial markers
segmented from CT images.
Tumour volume has been calculated based on the number of the voxels from the connected
overlapping tumour regions in 3D with no interpolation. The calculated volumes show that
deformable registration has a small impact in tumour volume calculation comparing to the rigid
registration, and this is consistent with the rigidity of the tumour.
In this chapter we have shown that our proposed approach is capable of reconstructing 3D
histology volumes from 2D whole-mount histology images of breast specimens. This approach
can help the surgical pathologists and researchers in comparison, visualization and analysis of
clinical specimens. Our work here shows that it is feasible to use this approach for investigation
of the tumour shape, volume, and multifocality.
102
6 Summary and Future Directions
6.1 Summary
Histology volume reconstruction may improve the study of anatomy and biology of normal and
diseased tissue. Studies concerned with structural changes and the spatial relationship of disease
biomarkers may particularly benefit from 3D analysis of tissue. Furthermore, cancer diagnosis
and prognosis can be assisted by assessment of 3D tumour features, such as volume and multi-
focality. In addition to the field of pathology, 3D histology volume can be employed as a ground
truth in validating new volumetric medical imaging techniques. Sparse sampling of clinical
specimens and the deformation of specimens during histology processes encumber an accurate
reconstruction of histology volume. Validation of histology image reconstruction is also a
challenging process when working with clinical specimens. The existing methods for histology
volume construction of human breast specimens are either manual or applicable only to small
densely sampled regions, cropped from the specimen.
Throughout the work of this dissertation, an automatic image registration pipeline has been
developed and validated for histology volume reconstruction from 2D images of the whole-
mount histology sections of pre-clinical and clinical specimens, a deformable registration
technique has been optimized to correct the deformation of human breast histology images, and
the application of this framework for assessment of 3D tumour features has been evaluated.
The work in chapter 2 includes the development of the image registration framework for
histology volume reconstruction using a set of reference optical images. The use of the blockface
images prevents registration errors from propagating throughout the reconstructed volume and
therefore maintains the conformation of the tissue. Chapter 2 also provides experimental
103
evidence that this pipeline can be successfully employed in 3D comparison of normal and
diseased tissue of pre-clinical mouse models.
Chapter 3 shows the expansion of the image registration framework for registration of the
histology to ex vivo medical images. In order to make the pipeline independent from different
tissue types and different histological stains, edge-based image registration methods were used in
the work presented in chapter 3. A multimodality fiducial marker was also developed for
validation of the histology volume reconstruction. The framework was validated based on the
Euclidean distance between the implanted fiducial markers in a pre-clinical model. The average
target registration error demonstrated that the improvement of the deformable registration
method of the framework is required.
Chapter 4 describes a feature-based deformable registration method for registration of the
histology images with their correspondent blockface images. This method obtains the pixel
displacement field by minimizing the distance between the densely sampled SIFT features,
which are scale, rotation and illumination invariant. The results confirms that the feature–based
deformable registration performs effectively and can replace the edge-based method used in
chapter 3.The optimum registration parameters for human breast tissue were also estimated by an
iterative optimization approach and it was demonstrated that this improves registration accuracy
significantly.
Furthermore, in chapter 5, the histology volume of five lumpectomy specimens are reconstructed
using the proposed framework, to assess the feasibility of using histology volumes to study the
tumour features in 3D, i.e. tumour volume and focality. The proposed approach is shown to be
suitable for 3D histology volume reconstruction from 2D histology images of breast specimen.
The average target registration error for the breast specimens was measured as 0.47 ± 0.31 mm
based on manually selected landmarks and the optimized deformable registration was shown to
significantly (p-value=0.006 < 0.05) improve the registration comparing to rigid registration. It is
also shown that this method has the potential to provide more accurate information about 3D
tumour measurements, such as tumour volume, and multi-focality.
104
6.2 Future Improvements to the Image Registration Framework
6.2.1 Correlation at Higher Resolutions
Although the image registration accuracy of the presented framework is reasonable at the
resolution of blockface images, at finer resolution the precision of the images registration might
not be acceptable, for example when consecutive histology images are correlated at cellular
level. The accuracy of the image registration at a desired higher resolution can be improved by
the employment of multi-scale image registration methods and the proposed framework can still
provide an initial alignment for these methods, which uses a coarse-to-fine strategy. An example
of this would be registration of consecutive histology images at cellular level to reconstruct the
3D structures such as DCIS.
6.2.2 A Generic Modular Image Registration Software
Different organs of the human body have different structures. For example histology images of
human liver have fewer detectable structures at the resolution of blockface images than a human
brain or breast. The feature-based image registration method, which is proposed in this thesis for
breast histology images, might not perform well for liver images, whereas the edge-based
method might be a better choice for liver images. One of the advantages of the proposed
framework is that it has no limitation on the choice of different types of rigid and non-rigid
image registration methods. The proposed approach can be expanded toward a generic modular
image registration software that can register images from any tissue type and/or imaging
modality. Image registration methods can be developed as separate modules, which can be linked
together to build a proper pipeline based on the application in hand. Segmentation and
visualization modules can also be included in the software to facilitate the image registration and
3D visualization.
105
6.3 Future Directions of Applications
6.3.1 Correlation of Different Disease Bio-Markers
In the era of personalized medicine, target specific drugs are increasingly introduced and
developed for different cancer types. To ensure that cancer patients receive optimal treatment,
tumour response to these targeted drugs is investigated in pathology by using different
established predictive biomarkers, e.g. ER, PR, and Her2/neu for breast. More disease
biomarkers are continuously being studied and developed to clarify the mechanisms of tumour
response and to identify the optimal strategy for drug administration. Additionally, the spatial
correlation of these tumour biomarkers in may also help with the identification and classification
of different cancer subtypes. Currently, surgical pathologists correlate differently stained
histology slides by eye under a microscope by switching between the slides. Automatic image
registration of the histology images with different stains facilitates the correlation of different
disease biomarkers for surgical pathologists.
To investigate the feasibility of automatic registration of the histology images with different
stains, a preliminary experiment was pursued using the image registration methods presented in
chapter 4 of this thesis. Two consecutive differently stained sections from a lumpectomy
specimen were obtained from Biomarker Imaging Research Laboratory (BIRL) at Sunnybrook
Research Institute. The sections were 4μm thick and one was stained with H&E and the other
one was stained for Estrogen Receptors (ER). The sections were scanned at 2μm resolution,
Figure 6-1.
106
Figure 6-1 Consecutive histology images with different stains. a) H&E Histology image. b) ER
Histology image
The images were down-sampled to 40 μm resolution. Since the two sections were well aligned
even before image registration, the rigid registration step was skipped for this pair of images.
Similar structures, which can be detected in the consecutive sections, were used for validation of
the image registration. Figure 6-2 shows the fusion of the registered images both with the
optimized parameters, obtained in chapter 3, and the default parameters. The smooth
continuation of the structures in Figure 6-2-b confirms the superiority of deformable registration
using optimized parameters.
The sections expand differently when they are placed in a hot water bath to remove wrinkles.
Furthermore, different tissue types also shrink or expand differently and this can deform the
sections. Therefore, it is required to register the sections with a suitable deformable registration
method to be able to accurately correlate the distribution of different stains.
1cm
107
Furthermore, a region was selected from both images at the original resolution, 2 μm, and
registered using two sets of registration parameters. The quality of registration was explored at
high resolution, where more structures can be visualized. Figure 6-3 confirms that the optimized
parameters still outperform the default parameters in registration of H&E and ER images at high
resolution.
The result of this preliminary study confirms that the proposed methods can be employed for
automatic correlation of the images scanned from the consecutive breast histology sections
stained for different disease biomarkers. Additional validation experiments in future are required
to confirm the suitability of the proposed methods for this application.
a) b)
a) b)
1cm
Figure 6-2 Fusion of the registered images, a) with Liu et al.’s parameters, b) with
optimized parameters. Arrow show the areas, which have been improved with
optimized parameters.
108
a b
c d
e f)
a) b)
c) d)
e) f)
200µm
Figure 6-3 Selected regions at high resolution (the images are downsized to fit in this space).
a) H&E region, b) ER region, c) deformed ER using Liu et al.’s parameters, d) deformed ER
using optimized parameters. e) fusion of H&E and the deformed ER using Liu et al.’s
parameters, f) fusion of H&E and the deformed ER using optimized parameters. Arrows mark
some of the places that the deformable registration has been improved by using optimized
parameters.
109
6.3.2 Correlation of ex vivo Medical Images with Histology
Correlation of ex vivo medical images with histology has been used by scientists for validation
of new improvements in medical imaging field. In order to accurately compare the findings in
medical images with the underlying structures in histology, it is necessary to compare the
correspondent images with each other. Finding the correspondent images manually is
challenging especially when the two volumes are misaligned. 3D image registration of a
reconstructed histology volume with a volumetric medical image of an excised specimen brings
the two volumes in alignment and assists with the selection of the closest correspondent images.
Figure 6-4 shows an example of the correspondent histology and Computed Tomography (CT)
images of a human breast specimen, which are, in this case, manually selected throughout the
two volumes. 3D Registration of ex vivo volumetric images with histology volume can also be
used as the first step of registration of in vivo volumetric images with histology, which is
explained in the next section.
A preliminary experiment in Chapter 3 of this thesis shows that the proposed image registration
framework is capable of accurately finding the correspondent ex vivo MR and histology images
a) b)
Figure 6-4. Correspondent histology (a) and CT (b) images of a human breast specimen.
110
of a small pre-clinical model. In future, additional experiments are required to assess the use of
this pipeline for registration of ex vivo and histology images of clinical specimens.
6.3.3 Correlation of in vivo Medical Images with Histology
Moving further, correlation of histology images with in vivo medical images is desirable for
many different applications. In order to transfer the new improvements in medical imaging
techniques to clinic, it is required for the scientists to first validate their new approaches. The
validation is usually pursued by comparison of the medical images with histology, which is still
the gold standard. A valid comparison requires the correct selection of the correspondent
histology and medical images. For example Sztrókay et al. (Sztrókay et al., 2012) co-registered
high resolution CT images of breast with histology for validation of the use of Phase Contrast
Imaging (PCI) in Computed Tomography for breast cancer imaging. On the reconstructed CT
volumes they investigated several slice orientations to find the best fitting slices to the histology
image that they prepared from the tumour site.
Another example of radiological-pathological correlation is shown in Figure 6-5, which is a work
done by Tot et al. (Tot & Gere, 2016) to investigate the proportion of radiologically occult
extensive disease and its impact on surgical management. They focused on tumors judged as
unifocal based on pre-operative multimodality breast imaging and found out the frequency of
additional radiologically occult malignant lesions in whole slide images of histology tissue
sections.
In these studies, the correspondent histology and radiological slices were found manually based
on the visual similarity assessment, which is a laborious task. These examples reveal the need for
an automatic image registration approach to accurately find the correspondent histology and
medical images.
111
Registration of histology with in vivo images may also assist the surgical pathologists to report
the location of the tumour with respect to the anatomical planes. The orientation of the excised
tissue is marked by the surgeon using sutures. To maintain the orientation of the specimen, the
pathologist assistant uses different color inks to mark the surface of the specimen at different
reference planes, before removing the sutures. In rare cases the orientation might get lost during
this process.
Figure 6-6 shows a pipeline, which can be for registration of in vivo medical images with
histology images. Because of the remarkable difference between histology and in vivo images,
finding correspondent images for comparison is a challenging task. On the other hand, ex vivo
images, which are scanned by the same imaging modality, share more similarities with in vivo
images. Therefore, ex vivo volumetric images can serve as an intermediator in the pipeline for
registration of histology with in vivo images.
Figure 6-5 Study of the Impact of Tumour Focality on Surgical Management by Tot et al. (Tot &
Gere, 2016). (Image obtained from the open access article distributed under the Creative
Commons Attribution License).
112
6.4 Conclusion
The work in this thesis presents an image registration framework for histology volume
reconstruction using optical blockface images as reference images. The validation experiments
using multimodality fiducial markers demonstrated that this framework can be used for histology
volume reconstruction of pre-clinical and clinical specimens with no dependency on the spacing
between the histology sections. The framework can be conveniently extended for registration of
histology images with ex vivo medical images. It is also shown that the introduced optimized
deformable registration has significantly improved the registration of histology images to their
correspondent optical blockface images. In a preliminary study it is shown that the optimized
deformable registration also performs well when used for registration of the histology images
with different stains. The reconstructed histology volume facilitates the study of tumour features
in 3D as it is shown for tumour volume and multi-focality.
Figure 6-6. Registration of in vivo medical images with histology using the ex vivo images as
intermediate reference images.
113
The contributions of this work can be summarized as follows:
An accurate image registration pipeline for histology volume reconstruction and co-
registration of histology with images ex vivo has been introduced in this thesis. The use
of optical blockface images makes the pipeline independent from the spacing between
the histology sections and avoids the propagation of registration error from one slice to
the next.
It has been shown that increasing the spacing of the blockface and histology images up
to about 0.8mm has no significant effect on the accuracy of the registration of MR
images to histology images of a preclinical tumour model.
A multi-modality fiducial marker has been introduced, which is suitable for validation of
image registration.
The use of dense SIFT features has been introduced for the automatic registration of
histology images.
The use of SIFTFlow deformable registration algorithm has been introduced for
deformable registration of histology images.
An optimization approach has been developed for estimating the optimum parameters for
SIFTFlow. The developed optimization software can be used to optimize SIFTFlow
parameters for deformable registration of histology images obtained from other tissue
types and other organs.
It has been demonstrated that the reconstructed volumes have the potential to assist
surgical pathologists with more precise assessment of tumour extent in 3D by using
tumour volume calculated from 3D histology.
The problem of tumour multi-focality in the assessment of tumour extent has been
partially addressed.
114
A feasibility test has been performed to show that the work in this thesis can be used in
correlation of the histology images of the sections stained for different disease
biomarkers.
The work in this thesis has demonstrated that accurate 3D histology volumes of breast specimens
can be automatically reconstructed from 2D digital images of whole-mount histology slides.
These reconstructed volumes can also assist surgical pathologists with more precise assessment
of tumour extent in 3D.
115
Bibliography
Alic, L., Haeck, J. C., Bol, K., Klein, S., van Tiel, S. T., Wielepolski, P. A., … Veenland, J. F.
(2011). Facilitating tumor functional assessment by spatially relating 3D tumor histology
and in vivo MRI: image registration approach. PloS One, 6(8), e22835.
http://doi.org/10.1371/journal.pone.0022835
Aref, M., Chaudhari, A. R., Bailey, K. L., Aref, S., & Wiener, E. C. (2008). Comparison of
tumor histology to dynamic contrast enhanced magnetic resonance imaging-based
physiological estimates. Magnetic Resonance Imaging, 26(9), 1279–93.
http://doi.org/10.1016/j.mri.2008.02.015
Bağci, U., & Bai, L. (2008). Registration of standardized histological images in feature space. In
J. M. Reinhardt & J. P. W. Pluim (Eds.), SPIE Medical Imaging (p. 69142V–69142V–9).
http://doi.org/10.1117/12.770219
Bagci, U., Udupa, J. K., & Bai, L. (2010). The role of intensity standardization in medical image
registration. Pattern Recognition Letters. http://doi.org/10.1016/j.patrec.2009.09.010
Bardinet, E., Ourselin, S., Dormont, D., Malandain, G., Tand, D., Parain, K., … Yelnik, J.
(2002). Co-registration of Histological , Optical and MR Data of the Human Brain.
MICCAI, 2488, 548–555.
Besl, P., & McKay, N. (1992). A Method for Registration of 3-D Shapes. IEEE Transactions on
Pattern Analysis and Machine Intelligence, 14(2), 239–256.
Bookstein, F. L. (1989). Principal Warps: Thin-Plate Splines and the Decomposition of
Deformations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 11(6),
567–585.
Booth, M. E., Treanor, D., Roberts, N., Magee, D. R., Speirs, V., & Hanby, A. M. (2015). Three-
dimensional reconstruction of ductal carcinoma in situ with virtual slides. Histopathology,
66(7), 966–973. http://doi.org/10.1111/his.12561
116
Breast cancer statistics. (2015). Retrieved September 13, 2016, from
http://www.cancer.ca/en/cancer-information/cancer-type/breast/statistics/?region=on
Breen, M. S., Lazebnik, R. S., & Wilson, D. L. (2005). Three-dimensional registration of
magnetic resonance image data to histological sections with model-based evaluation.
Annals of Biomedical Engineering, 33(8), 1100–12. http://doi.org/10.1007/s10439-005-
5778-8
Cahill, N. D., Noble, J. A., & Hawkes, D. J. (2010). Extending the quadratic taxonomy of
regularizers for nonparametric registration. Image (Rochester, N.Y.), 7623, 76230B–
76230B–12. http://doi.org/10.1117/12.844464
Cao, M., Liang, Y., Shen, C., Miller, K. D., & Stantz, K. M. (2009). Developing DCE-CT to
quantify intra-tumor heterogeneity in breast tumors with differing angiogenic phenotype.
IEEE Transactions on Medical Imaging, 28(6), 861–71.
http://doi.org/10.1109/TMI.2008.2012035
Chakravarty, M. M., Bertrand, G., Hodge, C. P., Sadikot, A. F., & Collins, D. L. (2006). The
creation of a brain atlas for image guided neurosurgery using serial histological data.
NeuroImage, 30(2), 359–76. http://doi.org/10.1016/j.neuroimage.2005.09.041
Chopra, R., Tang, K., Burtnyk, M., Boyes, A., Sugar, L., Appu, S., … Bronskill, M. (2009).
Analysis of the spatial and temporal accuracy of heating in the prostate gland using
transurethral ultrasound therapy and active MR temperature feedback. Physics in Medicine
and Biology, 54(9), 2615–33. http://doi.org/10.1088/0031-9155/54/9/002
Clarke, G. M., Mainprize, J. G., Peressotti, C., Sun, L., Wang, D., Zubovits, J. T., … Yaffe, M. J.
(2008). Validation of cone-beam CT measurements of tumour burden using three-
dimensional histopathology: Initial results for a lumpectomy. Lecture Notes in Computer
Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in
Bioinformatics), 5116 LNCS, 440–446. http://doi.org/10.1007/978-3-540-70538-3_61
Clarke, G. M., Murray, M., Holloway, C. M. B., Liu, K., Zubovits, J. T., & Yaffe, M. J. (2012).
3D Pathology Volumetric Technique: A Method for Calculating Breast Tumour Volume
from Whole-Mount Serial Section Images. International Journal of Breast Cancer, 2012,
117
1–9. http://doi.org/10.1155/2012/691205
Clarke, G. M., Peressotti, C., Mawdsley, G. E., & Yaffe, M. J. (2006). Design and
characterization of a digital image acquisition system for whole-specimen breast
histopathology. Physics in Medicine and Biology, 51(20), 5089–103.
http://doi.org/10.1088/0031-9155/51/20/002
Cohen, F. S., Yang, Z., Huang, Z., & Nissanov, J. (1998). Automatic matching of homologous
histological sections. IEEE Transactions on Bio-Medical Engineering.
http://doi.org/10.1109/10.668755
Collins, L. C., & Schnitt, S. J. (2007). Histology for Pathologists. (S. E. Mills, Ed.).
Philadelphia.
Connolly, J., Schnitt, S., Wang, H., & Al., E. (2003). Holland-Frei Cancer Medicine. (D. Kufe,
R. Pollock, R. Weichselbaum, & E. Al., Eds.) (6th ed.). Hamilton (ON). Retrieved from
http://www-ncbi-nlm-nih-gov.myaccess.library.utoronto.ca/books/NBK13237/
Coombs, N. J., & Boyages, J. (2005). Multifocal and multicentric breast cancer: Does each focus
matter? Journal of Clinical Oncology, 23(30), 7497–7502.
http://doi.org/10.1200/JCO.2005.02.1147
D’Aische, A. du B., Craene, M. De, Geets, X., Gregoire, V., Macq, B., & Warfield, S. K. (2005).
Efficient multi-modal dense field non-rigid registration: alignment of histological and
section images. Medical Image Analysis, 9(6), 538–46.
http://doi.org/10.1016/j.media.2005.04.003
Dauguet, J., Delzescaux, T., Condé, F., Mangin, J.-F., Ayache, N., Hantraye, P., & Frouin, V.
(2007). Three-dimensional reconstruction of stained histological slices and 3D non-linear
registration with in-vivo MRI for whole baboon brain. Journal of Neuroscience Methods,
164(1), 191–204. http://doi.org/10.1016/j.jneumeth.2007.04.017
Duan, W., Kuester, F., Gaudiot, J., & Hammami, O. (2008). Automatic object and image
alignment using Fourier Descriptors. Image (Rochester, N.Y.), 26, 1196–1206.
http://doi.org/10.1016/j.imavis.2008.01.009
118
Fedorov, A., Billet, E., Prastawa, M., Gerig, G., Radmanesh, A., Warfield, I. K., …
Chrisochoides, N. (2008). The Use of Robust Local Hausdorff Distances in Accuracy
Assessment for Image Alignment of Brain MRI. In International Symposium on Visual
Computing (ISVC) (pp. 594–603).
Felzenszwalb, P. F., & Huttenlocher, D. P. (2006). Efficient belief propagation for early vision.
International Journal of Computer Vision. http://doi.org/10.1007/s11263-006-7899-4
Freeman, H. (1974). Computer Processing of Line-Drawing Images. ACM Computing Surveys
(CSUR), 6(1), 57–97. http://doi.org/10.1145/356625.356627
Gefen, S., Tretiak, O., & Nissanov, J. (2003). Elastic 3-D alignment of rat brain histological
images. IEEE Transactions on Medical Imaging, 22(11), 1480–9.
http://doi.org/10.1109/TMI.2003.819280
Giardina, C., & Kuhl, F. (1977). Accuracy of Curve Approximation by Harmonically Related
Vectors with Elliptical Loci. Computer Graphics and Image Processing, 6, 277–285.
Gibson, E., Fenster, A., & Ward, A. D. (2012). Registration Accuracy : How Good Is Good
Enough ? A Statistical Power Calculation Incorporating Image Registration Uncertainty.
MICCAI, 7511, 643–650.
Gibson, E., Gaed, M., Gómez, J. a, Moussa, M., Romagnoli, C., Pautler, S., … Ward, A. D.
(2013). 3D prostate histology reconstruction: an evaluation of image-based and fiducial-
based algorithms. Medical Physics, 40(9), 93501. http://doi.org/10.1118/1.4816946
Going, J. J., & Moffat, D. F. (2004). Escaping from Flatland: Clinical and biological aspects of
human mammary duct anatomy in three dimensions. Journal of Pathology, 203(1), 538–
544. http://doi.org/10.1002/path.1556
Gonzalez, R., & Woods, R. (2002). Digital Image Processing. Prentice Hall.
Guest, E., Berry, E., Baldock, R. A., Fidrich, M., & Smith, M. A. (2001). Robust Point
Correspondence Applied to Two- and Three-Dimensional Image Registration. IEEE
Transactions on Pattern Analysis and Machine Intelligence, 23(2), 165–179.
119
Gupta, D., Nath, M., & Layfield, L. J. (2003). Utility of four-quadrant random sections in
mastectomy specimens. The Breast Journal, 9(4), 307–311.
Hoda, S. A., Brogi, E., Koerner, F. C., & Rosen, P. P. (2014). Rosen’s breast pathology.
Jacobs, M. A., Windham, J. P., Soltanian-Zadeh, H., Peck, D. J., & Knight, R. A. (1999).
Registration and warping of magnetic resonance images to histological sections. Medical
Physics, 26(8), 1568–78. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/10501057
Jansen, S. A. (2012). Breast Cancer Biology for the Radiation Oncologist. (J. Strauss, W. Small,
& G. E. Woloschak, Eds.). Berlin, Heidelberg: Springer Berlin Heidelberg.
Ju, T., Warren, J., Carson, J., Bello, M., Kakadiaris, I., Chiu, W., … Eichele, G. (2006). 3D
volume reconstruction of a mouse brain from histological sections using warp filtering.
Journal of Neuroscience Methods, 156(1–2), 84–100.
http://doi.org/10.1016/j.jneumeth.2006.02.020
Knopp, M. V, Weiss, E., Sinn, H. P., Mattern, J., Junkermann, H., Radeleff, J., … van Kaick, G.
(1999). Pathophysiologic basis of contrast enhancement in breast tumors. Journal of
Magnetic Resonance Imaging : JMRI, 10(3), 260–6. Retrieved from
http://www.ncbi.nlm.nih.gov/pubmed/10508285
Kuhl, F., & Giardina, C. (1982). Elliptic Fourier Features of a Closed Contour. Computer
Graphics and Image Processing, 18, 236–258.
Lazebnik, R. S., Lancaster, T. L., Breen, M. S., Lewin, J. S., & Wilson, D. L. (2003). Volume
registration using needle paths and point landmarks for evaluation of interventional MRI
treatments. IEEE Transactions on Medical Imaging, 22(5), 653–60.
http://doi.org/10.1109/TMI.2003.812246
Lei, M., Xue, C. H., Wang, Y. M., Li, Z. J., Xue, Y., & Wang, J. F. (2008). Effect of squid ink
melanin-Fe on iron deficiency anemia remission. Journal of Food Science, 73(8), H207-11.
http://doi.org/10.1111/j.1750-3841.2008.00930.x
Leopold, K. A., Recht, A., Schnitt, S. J., Connolly, J. L., Rose, M. A., Silver, B., & Harris, J. R.
(1989). Results of conservative surgery and radiation therapy for multiple synchronous
120
cancers of one breast. International Journal of Radiation Oncology, Biology, Physics, 16(1),
11–6.
Liu, C., Yuen, J., & Torralba, A. (2011a). SIFT flow: dense correspondence across scenes and its
applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(5), 978–
94. http://doi.org/10.1109/TPAMI.2010.147
Liu, C., Yuen, J., & Torralba, A. (2011b). SIFT flow: Matlab/C++ code. Retrieved September 8,
2016, from http://people.csail.mit.edu/celiu/SIFTflow/SIFTflow.zip
Liu, C., Yuen, J., & Torralba, A. (2011c). SIFT flow: Matlab/C++ code.
Liu, C., Yuen, J., Torralba, A., Sivic, J., & Freeman, W. T. (2008). SIFT Flow: Dense
Correspondence across Different Scenes. European Conference on Computer Vision
(ECCV), 1(1), 28–42.
Lowe, D. G. (2004). Distinctive Image Features from Scale-Invariant Keypoints. International
Journal of Computer Vision, 60(2), 91–110.
http://doi.org/10.1023/B:VISI.0000029664.99615.94
Magnitsky, S., Watson, D. J., Walton, R. M., Pickup, S., Bulte, J. W. M., Wolfe, J. H., &
Poptani, H. (2005). In vivo and ex vivo MRI detection of localized and disseminated neural
stem cell grafts in the mouse brain. NeuroImage, 26(3), 744–54.
http://doi.org/10.1016/j.neuroimage.2005.02.029
Malandain, G., Bardinet, E., Nelissen, K., & Vanduffel, W. (2004). Fusion of autoradiographs
with an MR volume using 2-D and 3-D linear transformations. NeuroImage, 23(1), 111–
127. http://doi.org/10.1016/j.neuroimage.2004.04.038
Malandain, G., Bardinet, E., Nelissen, K., & Vanduffel, W. (2004). Fusion of autoradiographs
with an MR volume using 2-D and 3-D linear transformations. NeuroImage, 23(1), 111–27.
http://doi.org/10.1016/j.neuroimage.2004.04.038
Mattes, D., Haynor, D. R., Vesselle, H., Lewellen, T. K., & Eubank, W. (2001). Nonrigid
multimodality image registration. In SPIE Medical Imaging (Vol. 4322, pp. 1609–1620).
121
Michell, M. J. (2010). Breast Cancer. (M. J. Michell, Ed.), Asco (1st ed.). Cambridge:
Cambridge University Press. http://doi.org/10.1093/med/9780199363315.003.0020
National Cancer Institute. (2015). Retrieved September 13, 2016, from
http://www.cancer.gov/types/breast/hp/breast-screening-pdq#link/_10_toc
Navaz, S. (2011). Breast Cancer. (L. Jacobs & C. A. Finlayson, Eds.).
Nelder, B. J. a, & Mead, R. (1965). A simplex method for function minimization. Computer
Journal, 7, 308–313.
Nikou, C., Heitz, F., Nehlig, A., Namer, I. J., & Armspach, J.-P. (2003). A robust statistics-based
global energy function for the alignment of serially acquired autoradiographic sections.
Journal of Neuroscience Methods, 124(1), 93–102. http://doi.org/10.1016/S0165-
0270(02)00369-2
Opto Engineering. (2015). Telecentric lenses tutorial. Retrieved September 13, 2016, from
http://www.opto-engineering.com/resources/telecentric-lenses-tutorial
Ourselin, S., Roche, A., Pennec, X., & Sattonnet, C. (1998). Automatic Alignment of Histological
Sections for 3D Reconstruction and Analysis. Analytical Cellular Pathology (Vol. RR-3595,
I).
Pakurar, A. S., & Bigbee, J. W. (2004). Digital Histology. Hoboken, NJ, USA: John Wiley &
Sons, Inc. http://doi.org/10.1002/0471728551
PathCore. (2016). Sedeen Viewer. Retrieved from
http://www.pathcore.ca/products/visualizations
Pauly, K. B., Diederich, C. J., Rieke, V., Bouley, D., Chen, J., Nau, W. H., … Sommer, G.
(2006). Magnetic resonance-guided high-intensity ultrasound ablation of the prostate.
Topics in Magnetic Resonance Imaging : TMRI, 17(3), 195–207.
http://doi.org/10.1097/RMR.0b013e31803774dd
Pedersen, L., Gunnarsdottir, K. a, Rasmussen, B. B., Moeller, S., & Lanng, C. (2004). The
prognostic influence of multifocality in breast cancer patients. Breast (Edinburgh,
122
Scotland), 13(3), 188–93. http://doi.org/10.1016/j.breast.2003.11.004
Reis, S., Eiben, B., Mertzanidou, T., Hipwell, J., Hermsen, M., van der Laak, J., … Hawkes, D.
(2015). Minimum slice spacing required to reconstruct 3D shape for serial sections of breast
tissue for comparison with medical imaging. SPIE Medical Imaging; Digital Pathology,
9420, 94200K. http://doi.org/10.1117/12.2081909
Ridler, T.W. Calvard, S. (1978). Picture Thresholding Using an Iterative Slection Method. IEEE
Transactions on Systems, Man and Cybernetics, 8(8), 630–632.
http://doi.org/10.1109/TSMC.1978.4310039
Roberts, N., Magee, D., Song, Y., Brabazon, K., Shires, M., Crellin, D., … Treanor, D. (2012).
Toward Routine Use of 3D Histopathology as a Research Tool. The American Journal of
Pathology, 180(5), 1835–1842. http://doi.org/10.1016/j.ajpath.2012.01.033
Rouvière, O., Reynolds, C., Le, Y., Lai, J., Roberts, L. R., Felmlee, J. P., & Ehman, R. L. (2006).
Fiducial markers for MR histological correlation in ex vivo or short-term in vivo animal
experiments: a screening study. Journal of Magnetic Resonance Imaging : JMRI, 23(1), 50–
9. http://doi.org/10.1002/jmri.20466
Rusby, J. E., Brachtel, E. F., Michaelson, J. S., Koerner, F. C., & Smith, B. L. (2007). Breast
duct anatomy in the human nipple: Three-dimensional patterns and clinical implications.
Breast Cancer Research and Treatment, 106(2), 171–179. http://doi.org/10.1007/s10549-
006-9487-2
Samani, A., & Plewes, D. (2004). A method to measure the hyperelastic parameters of ex vivo
breast tissue samples. Physics in Medicine and Biology, 49(18), 4395–4405.
http://doi.org/10.1088/0031-9155/49/18/014
Samani, A., Zubovits, J., & Plewes, D. (2007). Elastic moduli of normal and pathological human
breast tissues: an inversion-technique-based investigation of 169 samples. Phys. Med. Biol,
52, 1565–1576. http://doi.org/10.1088/0031-9155/52/6/002
Samavati, N., McGrath, D., Lee, J., van der Kwast, T., Jewett, M., Ménard, C., & Brock, K. K.
(2011). Biomechanical model-based deformable registration of MRI and histopathology for
123
clinical prostatectomy. Journal of Pathology Informatics, 2(2), S10.
http://doi.org/10.4103/2153-3539.92035
Shojaii, R., Bacopulos, S., Yang, W., Karavardanyan, Tigran Spyropoulos, Demetri, Raouf, A.,
Martel, A., & Seth, A. (2014). Reconstruction of 3-Dimensional Histology Volume and its
Application to Study Mouse Mammary Glands. JoVE (Journal of Visualize Experiments).
Shojaii, R., Karavardanyan, T., Yaffe, M., & Martel, A. L. (2011). Validation of histology image
registration. SPIE Medical Imaging, 7962, 79621E–79621E–7.
http://doi.org/10.1117/12.878762
Shojaii, R., & Martel, A. (2009). A Novel Edge Point Selection Method for Registration of
Histology Images. Optical Tissue Image Analysis in Microscopy, Histopathology and
Endoscopy (OPTIMHisE) Workshop, MICCAI.
Shojaii, R., & Martel, A. (2010). Multi-Modality fiducial marker for validation of registration of
medical images with histology. SPIE Medical Imaging, 7623, 762331-762331–8.
http://doi.org/10.1117/12.845551
Sprengel, R., Rohr, K., & Stiehl, H. S. (1997). Thin-Plate Spline Approximation for Image
Registration. IEEE Engineering in Medicine and Biology, 18, 1190–1191.
Sun, L., Wang, D., Zubovits, J. T., Yaffe, M. J., & Clarke, G. M. (2009). An improved
Processing Method for Breast Whole-Mount Serial Sections for Three-Dimensional
Histopathology Imaging. American Journal of Clinical Pathology, 131(3), 383–392.
http://doi.org/10.1309/AJCPVBZZ4IKJHY3U
Sztrókay, A., Diemoz, P. C., Schlossbauer, T., Brun, E., Bamberg, F., Mayr, D., … Coan, P.
(2012). High-resolution breast tomography at high energy: a feasibility study of phase
contrast imaging on a whole breast. Physics in Medicine and Biology, 57(10), 2931–42.
http://doi.org/10.1088/0031-9155/57/10/2931
Taylor, L. S., Porter, B. C., Nadasdy, G., di Sant’Agnese, P. A., Pasternack, D., Wu, Z., …
Parker, K. J. (2004). Three-dimensional registration of prostate images from histology and
ultrasound. Ultrasound in Medicine & Biology, 30(2), 161–8.
124
http://doi.org/10.1016/j.ultrasmedbio.2003.10.005
Tot, T., & Gere, M. (2016). Radiologically Unifocal Invasive Breast Carcinomas: Large-Section
Histopathology Correlate and Impact on Surgical Management. Journal of Cancer Science
& Therapy, 8(3), 50–54. http://doi.org/10.4172/1948-5956.1000389
Wachowiak, M. P., Smolíková, R., Zheng, Y., Zurada, J. M., & Elmaghraby, A. S. (2004). An
Approach to Multimodal Biomedical Image Registration Utilizing Particle Swarm
Optimization. IEEE Transactions on Evolutionary Computation, 8(3), 289–301.
Weider, N. (Ed.). (2009). Modern Surgical Pathology. Elsevier B.V.
Weissenbacher, T. M., Zschage, M., Janni, W., Jeschke, U., Dimpfl, T., Mayr, D., … Dian, D.
(2010). Multicentric and multifocal versus unifocal breast cancer: Is the tumor-node-
metastasis classification justified? Breast Cancer Research and Treatment, 122(1), 27–34.
http://doi.org/10.1007/s10549-010-0917-9
Weissleder, R., Moore, a, Mahmood, U., Bhorade, R., Benveniste, H., Chiocca, E. a, & Basilion,
J. P. (2000). In vivo magnetic resonance imaging of transgene expression. Nature Medicine,
6(3), 351–5. http://doi.org/10.1038/73219
Wirtz, S., Fischer, B., Modersitzki, J., & Schmitt, O. (2004). Super-fast elastic registration of
histologic images of a whole rat brain for three-dimensional reconstruction. Medical
Imaging 2004: Image Processing, Pts 1-3, 5370, 328–334\r2158.
http://doi.org/10.1117/12.534110
Yushkevich, P. A., Piven, J., Hazlett, Heather Cody Smith, R. G., Ho, S., Gee, J. C., & Gerig, G.
(2006). User-guided 3D active contour segmentation of anatomical structures: Significantly
improved efficiency and reliability. Neuroimage, 31(3), 1116–28.
Zhan, Y., Ou, Y., Feldman, M., Tomaszeweski, J., Davatzikos, C., & Shen, D. (2007).
Registering histologic and MR images of prostate for image-based cancer detection.
Academic Radiology, 14(11), 1367–81. http://doi.org/10.1016/j.acra.2007.07.018
Top Related