logo

SCIENCE CHINA Information Sciences, Volume 63, Issue 1: 111101(2020) https://doi.org/10.1007/s11432-019-2708-1

Application of machine learning method in optical molecular imaging: a review

More info
  • ReceivedJul 5, 2019
  • AcceptedOct 22, 2019
  • PublishedDec 25, 2019

Abstract

Optical molecular imaging (OMI) is an imaging technology that uses an optical signal, such as near-infrared light, to detect biological tissue in organisms. Because of its specific and sensitive imaging performance, it is applied in both preclinical research and clinical surgery. However, it requires heavy data analysis and a complex mathematical model of tomographic imaging. In recent years, machine learning (ML)-based artificial intelligence has been used in different fields because of its ability to perform powerful data processing. Its analytical capability for processing complex and large data provides a feasible scheme for the requirement of OMI. In this paper, we review ML-based methods applied in different OMI modalities.


Acknowledgment

This work was supported by Ministry of Science and Technology of China (Grant Nos. 2018YFC091- 0602, 2017YFA0205200, 2017YFA0700401, 2016YFA0100902, 2016YFC0103702), National Natural Science Foundation of China (Grant Nos. 61901472, 61671449, 81227901, 81527805), the Strategic Priority Research Program of Chinese Academy of Sciences (Grant Nos. XDB32030200, XDB01030200), Chinese Academy of Sciences (Grant Nos. GJJSTD20170004, YJKYYQ20180048, KFJ-STS-ZDTP-059, QYZDJ-SSW-JSC005), Beijing Municipal Science Technology Commission (Grant Nos. Z161100002616022, Z171100000117023), and General Financial Grant from the China Postdoctoral Science Foundation (Grant No. 2017M620952). The authors would like to acknowledge the instrumental and technical support of Multi-modal biomedical imaging experimental platform, Institute of Automation, Chinese Academy of Sciences.


References

[1] Conway J R W, Carragher N O, Timpson P. Developments in preclinical cancer imaging: innovating the discovery of therapeutics.. Nat Rev Cancer, 2014, 14: 314-328 CrossRef PubMed Google Scholar

[2] Maldiney T, Bessière A, Seguin J. The in vivo activation of persistent nanophosphors for optical imaging of vascularization, tumours and grafted cells.. Nat Mater, 2014, 13: 418-426 CrossRef PubMed Google Scholar

[3] Ellenbroek S I J, van Rheenen J. Imaging hallmarks of cancer in living mice.. Nat Rev Cancer, 2014, 14: 406-418 CrossRef PubMed Google Scholar

[4] Weissleder R, Pittet M J. Imaging in the era of molecular oncology.. Nature, 2008, 452: 580-589 CrossRef PubMed Google Scholar

[5] Massoud T F, Gambhir S S. Molecular imaging in living subjects: seeing fundamental biological processes in a new light.. Genes Dev, 2003, 17: 545-580 CrossRef PubMed Google Scholar

[6] Fan-Minogue H, Cao Z W, Paulmurugan R. Noninvasive molecular imaging of c-Myc activation in living mice.. Proc Natl Acad Sci USA, 2010, 107: 15892-15897 CrossRef PubMed Google Scholar

[7] Nguyen Q T, Tsien R Y. Fluorescence-guided surgery with live molecular navigation--a new cutting edge.. Nat Rev Cancer, 2013, 13: 653-662 CrossRef PubMed Google Scholar

[8] Weissleder R. Molecular imaging in cancer.. Science, 2006, 312: 1168-1171 CrossRef PubMed Google Scholar

[9] Jobsis F F. Noninvasive, infrared monitoring of cerebral and myocardial oxygen sufficiency and circulatory parameters.. Science, 1977, 198: 1264-1267 CrossRef PubMed Google Scholar

[10] Gao Y, Wang K, An Y. Nonmodel-based bioluminescence tomography using a machine-learning reconstruction strategy. Optica, 2018, 5: 1451-1454 CrossRef Google Scholar

[11] Jiang S X, Liu J, Zhang G L. Reconstruction of Fluorescence Molecular Tomography via a Fused LASSO Method Based on Group Sparsity Prior.. IEEE Trans Biomed Eng, 2019, 66: 1361-1371 CrossRef PubMed Google Scholar

[12] Li Y C, Charalampaki P, Liu Y. Context aware decision support in neurosurgical oncology based on an efficient classification of endomicroscopic data.. Int J CARS, 2018, 13: 1187-1199 CrossRef PubMed Google Scholar

[13] LeCun Y, Bengio Y, Hinton G. Deep learning.. Nature, 2015, 521: 436-444 CrossRef PubMed Google Scholar

[14] Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks. Commun ACM, 2017, 60: 84-90 CrossRef Google Scholar

[15] Farabet C, Couprie C, Najman L. Learning hierarchical features for scene labeling.. IEEE Trans Pattern Anal Mach Intell, 2013, 35: 1915-1929 CrossRef PubMed Google Scholar

[16] Tompson J J, Jain A, LeCun Y, et al. Joint training of a convolutional network and a graphical model for human pose estimation. In: Proceedings of Advances in Neural Information Processing Systems 27. 2014. Google Scholar

[17] Szegedy C, Liu W, Jia Y Q, et al. Going deeper with convolutions. In: Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, 2015. 1--9. Google Scholar

[18] Mikolov T, Deoras A, Povey D, et al. Strategies for training large scale neural network language models. In: Proceedings of 2011 IEEE Workshop on Automatic Speech Recognition & Understanding, Waikoloa, 2011. 196--201. Google Scholar

[19] Hinton G, Deng L, Yu D, et al. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Process Magaz, 2012, 29: 82--97. Google Scholar

[20] Sainath T N, Kingsbury B, Saon G. Deep Convolutional Neural Networks for large-scale speech tasks.. Neural Networks, 2015, 64: 39-48 CrossRef PubMed Google Scholar

[21] Bengio Y, Ducharme R, Vincent P. A neural probabilistic language model. J Mach Learn Res, 2003, 3: 1137--1155. Google Scholar

[22] Sutskever I, Vinyals O, Le Q V. Sequence to sequence learning with neural networks. 2014,. arXiv Google Scholar

[23] Quan W Z, Wang K, Yan D M. Distinguishing Between Natural and Computer-Generated Images Using Convolutional Neural Networks. IEEE TransInformForensic Secur, 2018, 13: 2772-2787 CrossRef Google Scholar

[24] Bayar B, Stamm M C. Constrained Convolutional Neural Networks: A New Approach Towards General Purpose Image Manipulation Detection. IEEE TransInformForensic Secur, 2018, 13: 2691-2706 CrossRef Google Scholar

[25] Yang Y, Zhang W S, He Z W. Locator slope calculation via deep representations based on monocular vision. Neural Comput Applic, 2019, 31: 2781-2794 CrossRef Google Scholar

[26] Ma J S, Sheridan R P, Liaw A. Deep neural nets as a method for quantitative structure-activity relationships.. J Chem Inf Model, 2015, 55: 263-274 CrossRef PubMed Google Scholar

[27] Lema?tre G, Rastgoo M, Massich J. Classification of SD-OCT Volumes Using Local Binary Patterns: Experimental Validation for DME Detection.. J Ophthalmology, 2016, 2016(6): 1-14 CrossRef PubMed Google Scholar

[28] Srinivasan P P, Kim L A, Mettu P S. Fully automated detection of diabetic macular edema and dry age-related macular degeneration from optical coherence tomography images.. Biomed Opt Express, 2014, 5: 3568-3577 CrossRef PubMed Google Scholar

[29] Lee C S, Baughman D M, Lee A Y. Deep learning is effective for the classification of OCT images of normal versus Age-related Macular Degeneration.. Ophthalmology Retina, 2017, 1: 322-327 CrossRef PubMed Google Scholar

[30] Roy A G, Conjeti S, Karri S P K. ReLayNet: retinal layer and fluid segmentation of macular optical coherence tomography using fully convolutional networks.. Biomed Opt Express, 2017, 8: 3627 CrossRef PubMed Google Scholar

[31] Roy A G, Conjeti S, Carlier S G. Lumen Segmentation in Intravascular Optical Coherence Tomography Using Backscattering Tracked and Initialized Random Walks.. IEEE J Biomed Health Inform, 2016, 20: 606-614 CrossRef PubMed Google Scholar

[32] Wang Z, Jenkins M W, Linderman G C. 3-D Stent Detection in Intravascular OCT Using a Bayesian Network and Graph Search.. IEEE Trans Med Imag, 2015, 34: 1549-1561 CrossRef PubMed Google Scholar

[33] Schwab J, Antholzer S, Nuster R, et al. Real-time photoacoustic projection imaging using deep learning. 2018,. arXiv Google Scholar

[34] Hauptmann A, Lucka F, Betcke M. Model-Based Learning for Accelerated, Limited-View 3-D Photoacoustic Tomography.. IEEE Trans Med Imag, 2018, 37: 1382-1393 CrossRef PubMed Google Scholar

[35] Antholzer S, Schwab J, Bauer-Marschallinger J, et al. Nett regularization for compressed sensing photoacoustic tomography. In: Proceedings of SPIE, 2019. 10878. Google Scholar

[36] Huang C, Meng H, Gao Y, et al. Fast and robust reconstruction method for fluorescence molecular tomography based on deep neural network. In: Proceedings of SPIE, 2019. 10881. Google Scholar

[37] André B, Vercauteren T, Buchner A M. A smart atlas for endomicroscopy using automated video retrieval.. Med Image Anal, 2011, 15: 460-476 CrossRef PubMed Google Scholar

[38] Kamen A, Sun S H, Wan S H. Automatic Tissue Differentiation Based on Confocal Endomicroscopic Images for Intraoperative Guidance in Neurosurgery.. Biomed Res Int, 2016, 2016: 1-8 CrossRef PubMed Google Scholar

[39] Ravì D, Szczotka A B, Shakir D I. Effective deep learning training for single-image super-resolution in endomicroscopy exploiting video-registration-based reconstruction.. Int J CARS, 2018, 13: 917-924 CrossRef PubMed Google Scholar

[40] Zhang C, Wang K, An Y. Improved generative adversarial networks using the total gradient loss for the resolution enhancement of fluorescence images.. Biomed Opt Express, 2019, 10: 4742-4756 CrossRef PubMed Google Scholar

[41] Wang L V, Wu H I, Masters B R. Biomedical Optics, Principles and Imaging. J Biomed Opt, 2008, 13: 049902 CrossRef Google Scholar

[42] Gessert N, Lutz M, Heyder M. Automatic Plaque Detection in IVOCT Pullbacks Using Convolutional Neural Networks.. IEEE Trans Med Imag, 2019, 38: 426-434 CrossRef PubMed Google Scholar

[43] Foot B, MacEwen C. Surveillance of sight loss due to delay in ophthalmic treatment or review: frequency, cause and outcome.. Eye, 2017, 31: 771-775 CrossRef PubMed Google Scholar

[44] Ting D S W, Pasquale L R, Peng L. Artificial intelligence and deep learning in ophthalmology.. Br J Ophthalmol, 2019, 103: 167-175 CrossRef PubMed Google Scholar

[45] Liu Y Y, Chen M, Ishikawa H. Automated macular pathology diagnosis in retinal OCT images using multi-scale spatial pyramid and local binary patterns in texture and shape encoding.. Med Image Anal, 2011, 15: 748-759 CrossRef PubMed Google Scholar

[46] Karen Simonyan, Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. 2014,. arXiv Google Scholar

[47] Venhuizen F G, van Ginneken B, Liefers B. Deep learning approach for the detection and quantification of intraretinal cystoid fluid in multivendor optical coherence tomography.. Biomed Opt Express, 2018, 9: 1545 CrossRef PubMed Google Scholar

[48] de Fauw J, Ledsam J R, Romera-Paredes B. Clinically applicable deep learning for diagnosis and referral in retinal disease.. Nat Med, 2018, 24: 1342-1350 CrossRef PubMed Google Scholar

[49] Tsantis S, Kagadis G C, Katsanos K, et al. Automatic vessel lumen segmentation and stent strut detection in intravascular optical coherence tomography. Med Phys, 2012, 39: 503--513. Google Scholar

[50] Lu H, Gargesha M, Wang Z. Automatic stent detection in intravascular OCT images using bagged decision trees.. Biomed Opt Express, 2012, 3: 2809-2824 CrossRef PubMed Google Scholar

[51] Yabushita H, Bouma B E, Houser S L. Characterization of human atherosclerosis by optical coherence tomography.. Circulation, 2002, 106: 1640-1645 CrossRef PubMed Google Scholar

[52] Wang Z, Chamie D, Bezerra H G. Volumetric quantification of fibrous caps using intravascular optical coherence tomography.. Biomed Opt Express, 2012, 3: 1413-1426 CrossRef PubMed Google Scholar

[53] Zahnd G, Karanasos A, van Soest G. Quantification of fibrous cap thickness in intracoronary optical coherence tomography with a contour segmentation method based on dynamic programming.. Int J CARS, 2015, 10: 1383-1394 CrossRef PubMed Google Scholar

[54] Wang L V. Multiscale photoacoustic microscopy and computed tomography.. Nat Photon, 2009, 3: 503-509 CrossRef PubMed Google Scholar

[55] Kruger R A, Liu P Y, Fang Y R. Photoacoustic ultrasound (PAUS)--reconstruction tomography.. Med Phys, 1995, 22: 1605-1609 CrossRef PubMed Google Scholar

[56] Karabutov A A, Podymova N B, Letokhov V S. Time-resolved laser optoacoustic tomography of inhomogeneous media. Appl Phys B-Lasers Opt, 1996, 63: 545-563 CrossRef Google Scholar

[57] Ntziachristos V, Razansky D. Molecular imaging by means of multispectral optoacoustic tomography (MSOT).. Chem Rev, 2010, 110: 2783-2794 CrossRef PubMed Google Scholar

[58] Antholzer S, Haltmeier M, Schwab J. Deep learning for photoacoustic tomography from sparse data.. Inverse Problems Sci Eng, 2019, 27: 987-1005 CrossRef PubMed Google Scholar

[59] Xu M H, Wang L V. Universal back-projection algorithm for photoacoustic computed tomography.. Phys Rev E, 2005, 71: 016706 CrossRef PubMed Google Scholar

[60] Burgholzer P, Bauer-Marschallinger J, Grün H. Temporal back-projection algorithms for photoacoustic tomography with integrating line detectors. Inverse Problems, 2007, 23: S65-S80 CrossRef Google Scholar

[61] Zeng L, Xing D, Gu H M. High antinoise photoacoustic tomography based on a modified filtered backprojection algorithm with combination wavelet.. Med Phys, 2007, 34: 556-563 CrossRef PubMed Google Scholar

[62] Hoelen C G A, de Mul F F M. Image reconstruction for photoacoustic scanning of tissue structures.. Appl Opt, 2000, 39: 5872-5883 CrossRef PubMed Google Scholar

[63] Rosenthal A, Razansky D, Ntziachristos V. Fast semi-analytical model-based acoustic inversion for quantitative optoacoustic tomography.. IEEE Trans Med Imag, 2010, 29: 1275-1285 CrossRef PubMed Google Scholar

[64] Paltauf G, Viator J A, Prahl S A. Iterative reconstruction algorithm for optoacoustic imaging.. J Acoust Soc Am, 2002, 112: 1536-1544 CrossRef PubMed Google Scholar

[65] Jetzfellner T, Rosenthal A, Englmeier K H. Interpolated model-matrix optoacoustic tomography of the mouse brain. Appl Phys Lett, 2011, 98: 163701 CrossRef Google Scholar

[66] Treeby B E, Cox B T. k-Wave: MATLAB toolbox for the simulation and reconstruction of photoacoustic wave fields.. J Biomed Opt, 2010, 15: 021314 CrossRef PubMed Google Scholar

[67] Xu Y, Wang L V. Time reversal and its application to tomography with diffracting sources.. Phys Rev Lett, 2004, 92: 033902 CrossRef PubMed Google Scholar

[68] Hristova Y, Kuchment P, Nguyen L. Reconstruction and time reversal in thermoacoustic tomography in acoustically homogeneous and inhomogeneous media. Inverse Problems, 2008, 24: 055006 CrossRef Google Scholar

[69] Dean-Ben X L, Ntziachristos V, Razansky D. Acceleration of optoacoustic model-based reconstruction using angular image discretization.. IEEE Trans Med Imag, 2012, 31: 1154-1162 CrossRef PubMed Google Scholar

[70] Dean-Ben X L, Buehler A, Ntziachristos V. Accurate model-based reconstruction algorithm for three-dimensional optoacoustic tomography.. IEEE Trans Med Imag, 2012, 31: 1922-1928 CrossRef PubMed Google Scholar

[71] Huang C, Wang K, Nie L. Full-wave iterative image reconstruction in photoacoustic tomography with acoustically inhomogeneous media.. IEEE Trans Med Imag, 2013, 32: 1097-1110 CrossRef PubMed Google Scholar

[72] Arridge S R, Betcke M M, Cox B T. On the adjoint operator in photoacoustic tomography. Inverse Problems, 2016, 32: 115012 CrossRef Google Scholar

[73] Arridge S R, Beard P, Betcke M. Accelerated high-resolution photoacoustic tomography via compressed sensing.. Phys Med Biol, 2016, 61: 8908-8940 CrossRef PubMed Google Scholar

[74] Hauptmann A, Cox B, Lucka F, et al. Approximate k-space models and deep learning for fast photoacoustic reconstruction. In: Machine Learning for Medical Image Reconstruction. Belrin: Springer, 2018. 103--111. Google Scholar

[75] Ntziachristos V, Ripoll J, Wang L V. Looking and listening to light: the evolution of whole-body photonic imaging.. Nat Biotechnol, 2005, 23: 313-320 CrossRef PubMed Google Scholar

[76] Ntziachristos V, Bremer C, Weissleder R. Fluorescence imaging with near-infrared light: New technological advances that enable in vivo molecular imaging. European Radiology, 2003, 13(1): 195-208 https://xs.scihub.ltd/https://doi.org/10.1007/s00330-002-1524-x. Google Scholar

[77] Wang G, Li Y, Jiang M. Uniqueness theorems in bioluminescence tomography.. Med Phys, 2004, 31: 2289-2299 CrossRef PubMed Google Scholar

[78] Gao Y, Wang K, Jiang S. Bioluminescence Tomography Based on Gaussian Weighted Laplace Prior Regularization for In Vivo Morphological Imaging of Glioma.. IEEE Trans Med Imag, 2017, 36: 2343-2354 CrossRef PubMed Google Scholar

[79] Qin C H, Zhu S P, Feng J C. Comparison of permissible source region and multispectral data using efficient bioluminescence tomography method.. J Biophoton, 2011, 4: 824-839 CrossRef PubMed Google Scholar

[80] Arridge S R, Schweiger M, Hiraoka M. A finite element approach for modeling photon transport in tissue.. Med Phys, 1993, 20: 299-309 CrossRef PubMed Google Scholar

[81] Arridge S R. Optical tomography in medical imaging. Inverse Problems, 1999, 15: R41-R93 CrossRef Google Scholar

[82] Lu Y J, Zhang X Q, Douraghy A. Source reconstruction for spectrally-resolved bioluminescence tomography with sparse a priori information.. Opt Express, 2009, 17: 8062-8080 CrossRef PubMed Google Scholar

[83] Liu K, Tian J, Qin C H. Tomographic bioluminescence imaging reconstruction via a dynamically sparse regularized global method in mouse models.. J Biomed Opt, 2011, 16: 046016 CrossRef PubMed Google Scholar

[84] Chehade M, Srivastava A K, Bulte J W M. Co-registration of bioluminescence tomography, computed tomography,and magnetic resonance imaging for multimodal in vivo stem cell tracking. Tomography, 2016, 2: 158-165 CrossRef PubMed Google Scholar

[85] Zhang X Q, Lu Y J, Chan T. A Novel Sparsity Reconstruction Method from Poisson Data for 3D Bioluminescence Tomography. J Sci Comput, 2012, 50: 519-535 CrossRef Google Scholar

[86] Dutta J, Ahn S, Li C Q, et al. Joint l1 and total variation regularization for fluorescence molecular tomography. Phys Med Biol, 2015, 57: 1459--1476. Google Scholar

[87] Davis S C, Samkoe K S, O'Hara J A. Comparing implementations of magnetic-resonance-guided fluorescence molecular tomography for diagnostic classification of brain tumors.. J Biomed Opt, 2010, 15: 051602 CrossRef PubMed Google Scholar

[88] Davis S C, Samkoe K S, Tichauer K M. Dynamic dual-tracer MRI-guided fluorescence tomography to quantify receptor density in vivo.. Proc Natl Acad Sci USA, 2013, 110: 9025-9030 CrossRef PubMed Google Scholar

[89] Holt R W, Demers J L H, Sexton K J. Tomography of epidermal growth factor receptor binding to fluorescent Affibody in vivo studied with magnetic resonance guided fluorescence recovery in varying orthotopic glioma sizes.. J Biomed Opt, 2015, 20: 026001 CrossRef PubMed Google Scholar

[90] Schulz R B, Ale A, Sarantopoulos A. Hybrid system for simultaneous fluorescence and x-ray computed tomography.. IEEE Trans Med Imag, 2010, 29: 465-473 CrossRef PubMed Google Scholar

[91] Baikejiang R, Zhao Y, Fite B Z. Anatomical image-guided fluorescence molecular tomography reconstruction using kernel method.. J Biomed Opt, 2017, 22: 055001 CrossRef PubMed Google Scholar

[92] Cho K, van Merrienboer B, Gulcehre C, et al. Learning phrase representations using rnn encoder-decoder for statistical machine translation. 2014,. arXiv Google Scholar

[93] Machida H, Sano Y, Hamamoto Y. Narrow-band imaging in the diagnosis of colorectal mucosal lesions: a pilot study.. Endoscopy, 2004, 36: 1094-1098 CrossRef PubMed Google Scholar

[94] Gerger A, Koller S, Weger W. Sensitivity and specificity of confocal laser-scanning microscopy for in vivo diagnosis of malignant skin tumors.. Cancer, 2006, 107: 193-200 CrossRef PubMed Google Scholar

[95] Gotoh K, Kobayashi S, Marubashi S, et al. Intraoperative detection of hepatocellular carcinoma using indocyanine green fluorescence imaging. In: ICG Fluorescence Imaging and Navigation Surgery. Tokyo: Springer, 2016. 325--334. Google Scholar

[96] Glatz J, Garcia-Allende P B, Becker V. Near-infrared fluorescence cholangiopancreatoscopy: initial clinical feasibility results.. Gastrointestinal Endoscopy, 2014, 79: 664-668 CrossRef PubMed Google Scholar

[97] Adler A, Pohl H, Papanikolaou I S. A prospective randomised study on narrow-band imaging versus conventional colonoscopy for adenoma detection: does narrow-band imaging induce a learning effect?. Gut, 2007, 57: 59-64 CrossRef PubMed Google Scholar

[98] Vahrmeijer A L, Hutteman M, van der Vorst J R. Image-guided cancer surgery using near-infrared fluorescence.. Nat Rev Clin Oncol, 2013, 10: 507-518 CrossRef PubMed Google Scholar

[99] Schaafsma B E, Mieog J S D, Hutteman M. The clinical use of indocyanine green as a near-infrared fluorescent contrast agent for image-guided oncologic surgery.. J Surg Oncol, 2011, 104: 323-332 CrossRef PubMed Google Scholar

[100] Kitai T, Inomoto T, Miwa M. Fluorescence navigation with indocyanine green for detecting sentinel lymph nodes in breast cancer.. Breast Cancer, 2005, 12: 211-215 CrossRef PubMed Google Scholar

[101] Tummers Q R J G, Verbeek F P R, Schaafsma B E. Real-time intraoperative detection of breast cancer using near-infrared fluorescence imaging and Methylene Blue.. Eur J Surgical Oncology (EJSO), 2014, 40: 850-858 CrossRef PubMed Google Scholar

[102] Keereweer S, van Driel P B A A, Snoeks T J A. Optical image-guided cancer surgery: challenges and limitations.. Clin Cancer Res, 2013, 19: 3745-3754 CrossRef PubMed Google Scholar

[103] Andre B, Vercauteren T, Buchner A M. Learning semantic and visual similarity for endomicroscopy video retrieval.. IEEE Trans Med Imag, 2012, 31: 1276-1288 CrossRef PubMed Google Scholar

[104] Mountney P, Yang G Z. Context specific descriptors for tracking deforming tissue.. Med Image Anal, 2012, 16: 550-561 CrossRef PubMed Google Scholar

[105] Hu J, Shen L, Sun G, et al. Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 7132--7141. Google Scholar

[106] Xu T, Zhang P C, Huang Q Y, et al. Attngan: fine-grained text to image generation with attentional generative adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 1316--1324. Google Scholar

[107] Woo S, Park J, Lee J Y, et al. Cbam: Convolutional block attention module. In: Proceedings of the European Conference on Computer Vision (ECCV), 2018. 3--19. Google Scholar

  • Figure 1

    (Color online) Main flow chart of machine learning (ML) applied to optical molecular imaging (OMI).

  • Figure 2

    (Color online) Optical coherence tomography. (a) Imaging mechanism of OCT and (b) OCT images. These images are reproduced from [42,43].

  • Figure 3

    (Color online) Overview of ML-based algorithms for OCT segmentation. (a) U-Net-based intraretinal cystoid fluid segmentation reproduced from [48]and (b) artificial intelligence (AI) framework used in ocular disease and corresponding segmentation results reproduced from [31].

  • Figure 4

    (Color online) Imaging mechanism of photoacoustic tomography (PAT) reproduced from [58].

  • Figure 5

    (Color online) Different methods of ML-based PAT reconstruction. (a) U-Net-based reconstruction method reproduced from [58]; (b) neural network-based method to simulate conventional iteration reproduced from [35]; and (c) convolutional neural network (CNN)-based regularization method reproduced from [36].

  • Figure 6

    (Color online) Optical scattering tomography. (a) Anatomical information of X-CT; (b) planar optical imaging; and (c) images of OST. These images are reproduced from [78].

  • Figure 7

    (Color online) Structure of the networks used in OST reconstruction and the corresponding reconstruction results. (a) MLP-based BLT reconstruction network reproduced from [10]and (b) CNN-RNN-based FMT reconstruction framework reproduced from [37].

  • Figure 8

    (Color online) Different imaging modalities used in surgery. (a) Narrowband cavity mirror imaging; (b) laser confocal microscopy imaging; and (c) near-infrared fluorescence imaging. These images are reproduced from [93-95].

  • Figure 9

    (Color online) ML-based methods in the application of minimally invasive surgery (MIS) imaging classification. (a) WoB-based method in [38]; (b) SVM-based classification method in [39]; and (c) CNN-RNN framework in [12].

  • Figure 10

    (Color online) Overview of methods applied in MIS imaging enhancement. (a) Ground-truth building strategy and super-resolution results of different DL-based methods and (b) registration method based on a decision tree. These images are reproduced from [40,104].

  • Table 1   Summary of the different ML methods applied in OMI
    OMI ML systemsYearModelAUC SensitivitySpecificity
    modality$^{\rm~a)}$(%)$^{\rm~b)}$(%)(%)
    7[0]* OCT Guillaume et al.[27] 2016 Random Forest NA81.20 93.70
    Pratul et al.[28] 2014 Support Vector Machine 86.67 NA NA
    Cecilia et al.[29] 2017 VGG-16 93.45 92.64 93.69
    Roy et al.[30] 2017 ReLayNet 99.00 NA NA
    Jeffrey et al.[31] 2018 3D U-Net 99.21 NA NA
    Abhijit et al.[32] 2016 Random Walks 97.86 NA NA
    Zhao et al.[33] 2015 Bayesian Network 91 NA NA
    3[0]* PAT Johannes Schwab et al. [34] 2018 DALnet 93.30 NA NA
    Andreas et al. [35] 2018 Model-based CNN 94.50 NA NA
    Stephan et al. [36] 2019 NETT 89 NA NA
    2[0]* OST Chao H et al. [37] 2019 CNN and RNN NA NA NA
    Yuan et al. [10] 2019 Multilayer Perceptron NA NA NA
    5[0]* OIS Andre et al. [38] 2011 Bag-of-Visual-Words 94.20 97.70 86.10
    Kamen et al. [39] 2016 Support Vector Machine 84.32 87.34 80.61
    Li et al. [12] 2018 CNN 99.49 NA NA
    Daniele et al. [40] 2018 Context Specific Descriptor NA 89.20 NA
    Chong et al. [41] 2019 Generative Adversarial Networks 91.64 NA NA

    a

Copyright 2020 Science China Press Co., Ltd. 《中国科学》杂志社有限责任公司 版权所有

京ICP备18024590号-1