logo

SCIENCE CHINA Information Sciences, Volume 61, Issue 6: 060421(2018) https://doi.org/10.1007/s11432-017-9303-0

Neuromorphic vision chips

Nanjian WU1,2,3,*
More info
  • ReceivedOct 7, 2017
  • AcceptedNov 13, 2017
  • PublishedFeb 2, 2018

Abstract

The paper reviews the progress of neuromorphic vision chip research in decades. It focuses on two kinds of the neuromorphic vision chips: frame-driven (FD) and event-driven (ED) vision chips. The FD and ED vision chips are very different from each other in system architecture, image sensing, image information coding, image processing algorithm, design methodology. The vision chips can overcome serial data transmission and processing bottlenecks in traditional image processing systems. They can perform the high speed image capture and real-time image processing operations. This paper selects two typical chips from the two kinds of vision chips, respectively, and introduces their architectures, image sensing schemes, image processing processors and system operation. The FD neuromorphic reconfigurable vision chip comprises a high speed image sensor, a processing element array and self-organizing map neural network. The FD vision chip has the advantages in image resolution, static object detection, time-multiplex image processing, and chip area. The ED neuromorphic vision chip system is based on address-event-representation image sensor and event-driven multi-kernel convolution network. The ED vision chip has the advantages in fast sensing, low communication bandwidth, brain-like processing, and high energy efficiency. Finally, this paper discusses the architecture and the challenges of the future neuromorphic vision chip and indicates that the reconfigurable vision chip with left- and right-brain functions integrated in the three dimensional (3D) large-scale integrated circuit (LSI) technology becomes a trend of the research on the vision chip.


Acknowledgment

This work was supported by National Natural Science Foundation of China (Grant Nos. 61234003, 61434004, 61504141), Brain Project of Beijing (Grant No. Z161100000216129), and CAS Interdisciplinary Project (Grant No. KJZD-EW-L11-04). The author would like to thank all members in the research group for their collaborations.


References

[1] Mead C. Neuromorphic electronic systems. Proc IEEE, 1990, 78: 1629-1636 CrossRef Google Scholar

[2] Aizawa K. Computational sensors — vision VLSI. IEICE Trans Inf Syst, 1999, 82: 580--588. Google Scholar

[3] Boahen K A. Communicating neuronal ensembles between neuromorphic chips. In: Neuromorphic Systems Engineering. Berlin: Springer, 1998. 229--259. Google Scholar

[4] Chung-Yu Wu , Chin-Fong Chiu . A new structure of the 2-D silicon retina. IEEE J Solid-State Circuits, 1995, 30: 890-897 CrossRef Google Scholar

[5] Funatsu E, Nitta Y, Miyake Y. An artificial retina chip with current-mode focal plane image processing functions. IEEE Trans Electron Devices, 1997, 44: 1777-1782 CrossRef ADS Google Scholar

[6] Dudek P, Hicks P J. A general-purpose processor-per-pixel analog SIMD vision chip. IEEE Trans Circuits Syst I, 2005, 52: 13-20 CrossRef Google Scholar

[7] Kim D, Cho J, Lim S, et al. A 5000S/s single-chip smart eye-tracking sensor. In: Proceedings of IEEE International Solid-State Circuits Conference — Digest of Technical Papers, San Francisco, 2008. Google Scholar

[8] Moini A, Bouzerdoum A, Eshraghian K. An insect vision-based motion detection chip. IEEE J Solid-State Circuits, 1997, 32: 279-284 CrossRef Google Scholar

[9] Oike Y, Ikeda M, Asada K. A 375 /spl times/ 365 high-speed 3-D range-finding image sensor using row-parallel search architecture and multisampling technique. IEEE J Solid-State Circuits, 2005, 40: 444-453 CrossRef Google Scholar

[10] Leon-Salas W D, Balkir S, Sayood K. A CMOS Imager With Focal Plane Compression Using Predictive Coding. IEEE J Solid-State Circuits, 2007, 42: 2555-2572 CrossRef Google Scholar

[11] Miao W, Lin Q, Wu N. A Novel Vision Chip for High-Speed Target Tracking. Jpn J Appl Phys, 2007, 46: 2220-2225 CrossRef ADS Google Scholar

[12] Komuro T, Kagami S, Ishikawa M. A Dynamically Reconfigurable SIMD Processor for a Vision Chip. IEEE J Solid-State Circuits, 2004, 39: 265-268 CrossRef Google Scholar

[13] Yamaguchi K, Watanabe Y, Komuro T, et al. Design of a massively parallel vision processor based on multi-SIMD architecture. In: Proceedings of IEEE International Symposium on Circuits and Systems, New Orleans, 2007.łinebreak 3498--3501. Google Scholar

[14] Miao W, Lin Q, Zhang W. A Programmable SIMD Vision Chip for Real-Time Vision Applications. IEEE J Solid-State Circuits, 2008, 43: 1470-1479 CrossRef Google Scholar

[15] Lin Q Y, Miao W, Zhang W C, et al. A 1000 frames/s programmable vision chip with variable resolution and row-pixel-mixed parallel image processors. Sensors, 2009, 9: 5933--5951. Google Scholar

[16] Zhang W, Fu Q, Wu N J. A Programmable Vision Chip Based on Multiple Levels of Parallel Processors. IEEE J Solid-State Circuits, 2011, 46: 2132-2147 CrossRef Google Scholar

[17] Shi C, Yang J, Han Y, et al. A 1000 fps vision chip based on a dynamically reconfigurable hybrid architecture comprising a PE array processor and self-organizing map neural network. IEEE J Solid-State Circ, 2014, 49: 2067--2082. Google Scholar

[18] Yang Y, Yang J, Liu L. High-Speed Target Tracking System Based on a Hierarchical Parallel Vision Processor and Gray-Level LBP Algorithm. IEEE Trans Syst Man Cybern Syst, 2017, 47: 950-964 CrossRef Google Scholar

[19] Yang J, Yang Y, Chen Z. A Heterogeneous Parallel Processor for High-Speed Vision Chip. IEEE Trans Circuits Syst Video Technol, 2018, 28: 746-758 CrossRef Google Scholar

[20] Li H L, Zhang Z X, Yang J, et al. A novel vision chip architecture for image recognition based on convolutional neural network. In: Proceedings of the 11th International Conference on ASIC, Chengdu, 2015. Google Scholar

[21] Schmitz J A, Gharzai M K, Balkir S, et al. A 1000 frames/s vision chip using scalable pixel-neighborhood-level parallel processing. IEEE J Solid-State Circ, 2017, 52: 556--568. Google Scholar

[22] Yamazaki T, Katayama H, Uehara S, et al. 4.9 A 1ms high-speed vision chip with 3D-stacked 140GOPS column-parallel PEs for spatio-temporal image processing. In: Proceedings of IEEE International Solid-State Circuits Conference, San Francisco, 2017. 82--83. Google Scholar

[23] Culurciello E, Etienne-Cummings R, Boahen K A. A biomorphic digital image sensor. IEEE J Solid-State Circuits, 2003, 38: 281-294 CrossRef Google Scholar

[24] Shoushun C, Bermak A. Arbitrated Time-to-First Spike CMOS Image Sensor With On-Chip Histogram Equalization. IEEE Trans VLSI Syst, 2007, 15: 346-357 CrossRef Google Scholar

[25] Lichtsteiner P, Posch C, Delbruck T. A 128$\times$128 120 dB 15 $\mu$s Latency Asynchronous Temporal Contrast Vision Sensor. IEEE J Solid-State Circuits, 2008, 43: 566-576 CrossRef Google Scholar

[26] Xu J, Zhang M, Yan S. A Method to Solve the Side Effects of Dual-Line Timed Address Event Vision System. J CIRCUIT SYST COMP, 2015, 24: 1550028 CrossRef Google Scholar

[27] Xu J, Zou J, Yan S. Effective target binarization method for linear timed address-event vision system. Opt Eng, 2016, 55: 063103 CrossRef ADS Google Scholar

[28] Chan V, Jin C, van Schaik A. An address-event vision sensor for multiple transient object detection.. IEEE Trans Biomed Circuits Syst, 2007, 1: 278-288 CrossRef PubMed Google Scholar

[29] Venier P, Mortara A, Arreguit X. An integrated cortical layer for orientation enhancement. IEEE J Solid-State Circuits, 1997, 32: 177-186 CrossRef Google Scholar

[30] Serrano-Gotarredona T, Andreou A G, Linares-Barranco B. AER image filtering architecture for vision-processing systems. IEEE Trans Circuits Syst I, 1999, 46: 1064-1071 CrossRef Google Scholar

[31] Serrano-Gotarredona R, Serrano-Gotarredona T, Acosta-Jimenez A. A neuromorphic cortical-layer microchip for spike-based event processing vision systems. IEEE Trans Circuits Syst I, 2006, 53: 2548-2566 CrossRef Google Scholar

[32] Serrano-Gotarredona R, Serrano-Gotarredona T, Acosta-Jimenez A. On Real-Time AER 2-D Convolutions Hardware for Neuromorphic Spike-Based Cortical Processing. IEEE Trans Neural Netw, 2008, 19: 1196-1219 CrossRef Google Scholar

[33] Choi T Y W, Merolla P A, Arthur J V. Neuromorphic implementation of orientation hypercolumns. IEEE Trans Circuits Syst I, 2005, 52: 1049-1060 CrossRef Google Scholar

[34] Camunas-Mesa L, Acosta-Jimenez A, Zamarreno-Ramos C. A 32$\,\times\,$32 Pixel Convolution Processor Chip for Address Event Vision Sensors With 155 ns Event Latency and 20 Meps Throughput. IEEE Trans Circuits Syst I, 2011, 58: 777-790 CrossRef Google Scholar

[35] Camunas-Mesa L, Zamarreno-Ramos C, Linares-Barranco A. An Event-Driven Multi-Kernel Convolution Processor Module for Event-Driven Vision Sensors. IEEE J Solid-State Circuits, 2012, 47: 504-517 CrossRef Google Scholar

[36] Serrano-Gotarredona R, Oster M, Lichtsteiner P. CAVIAR: a 45k neuron, 5M synapse, 12G connects/s AER hardware sensory-processing- learning-actuating system for high-speed visual object recognition and tracking.. IEEE Trans Neural Netw, 2009, 20: 1417-1438 CrossRef PubMed Google Scholar

[37] Zhao B, Ding R, Chen S. Feedforward Categorization on AER Motion Events Using Cortex-Like Features in a Spiking Neural Network.. IEEE Trans Neural Netw Learning Syst, 2015, 26: 1963-1978 CrossRef PubMed Google Scholar

[38] Perez-Carrasco J A, Bo Zhao J A, Serrano C. Mapping from frame-driven to frame-free event-driven vision systems by low-rate rate coding and coincidence processing--application to feedforward ConvNets.. IEEE Trans Pattern Anal Mach Intell, 2013, 35: 2706-2719 CrossRef PubMed Google Scholar

[39] Stromatias E, Soto M, Serrano-Gotarredona T. An Event-Driven Classifier for Spiking Neural Networks Fed with Synthetic or Dynamic Vision Sensor Data. Front Neurosci, 2017, 11: 350 CrossRef Google Scholar

[40] Wang H Y, Xu J T, Gao Z Y, et al. An event-based neurobiological recognition system with orientation detector for objects in multiple orientations. Front Neuros, 2016, 10: 498. Google Scholar

[41] Son B, Suh Y, Kim S, et al. 4.1 A 640$\times$480 dynamic vision sensor with a 9 $\mu$m pixel and 300 Meps address-event representation. In: Proceedings of IEEE International Solid-State Circuits Conference, San Francisco, 2017. 66--67. Google Scholar

[42] Shi C, Yang J, Han Y, et al. 7.3 A 1000fps vision chip based on a dynamically reconfigurable hybrid architecture comprising a PE array and self-organizing map neural network. In: Proceedings of IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), San Francisco, 2014. 128--129. Google Scholar

[43] Cao Z X, Zhou Y F, Li Q L, et al. Design of pixel for high speed CMOS image sensors. In: Proceedings International Image Sensor Workshop, Snowbird, 2013, 229--232. Google Scholar

[44] Kohonen T. Self-organizing Maps. Berlin: Springer, 2001. Google Scholar

[45] Chen Z, Yang J, Shi C. High speed vision processor with reconfigurable processing element array based on full-custom distributed memory. Jpn J Appl Phys, 2016, 55: 04EF08 CrossRef ADS Google Scholar

[46] Lenero-Bardallo J A, Serrano-Gotarredona T, Linares-Barranco B. A 3.6 $\mu$s Latency Asynchronous Frame-Free Event-Driven Dynamic-Vision-Sensor. IEEE J Solid-State Circuits, 2011, 46: 1443-1455 CrossRef Google Scholar

[47] Kim S J, Kang B, Kim J D K, et al. A 1920$\times$1080 3.65 $\mu$m-pixel 2D/3D image sensor with split and binning pixel structure in 0.11 pm standard CMOS. In: Proceedings of IEEE International Solid-State Circuits Conference, San Francisco, 2012. 396--398. Google Scholar

[48] Chen Z, Di S, Cao Z. A 256256 time-of-flight image sensor based on center-tap demodulation pixel structure. Sci China Inf Sci, 2016, 59: 042409 CrossRef Google Scholar

[49] Chen Y H, Krishna T, Emer J S. Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks. IEEE J Solid-State Circuits, 2017, 52: 127-138 CrossRef Google Scholar

[50] Shin D, Lee J, Lee J, et al. 14.2 DNPU: an 8.1 TOPS/W reconfigurable CNN-RNN processor for general-purpose deep neural networks. In: Proceedings of IEEE International Solid-State Circuits Conference, San Francisco, 2017. 240--241. Google Scholar

[51] Cao Y, Chen Y, Khosla D. Spiking Deep Convolutional Neural Networks for Energy-Efficient Object Recognition. Int J Comput Vis, 2015, 113: 54-66 CrossRef Google Scholar

[52] Merolla P A, Arthur J V, Alvarez-Icaza R. A million spiking-neuron integrated circuit with a scalable communication network and interface. Science, 2014, 345: 668-673 CrossRef PubMed ADS Google Scholar

[53] Wu H, Wang X H, Gao B. Resistive Random Access Memory for Future Information Processing System. Proc IEEE, 2017, 105: 1770-1789 CrossRef Google Scholar

[54] Zheng Z J, Weng J Y. Mobile device based outdoor navigation with on-line learning neural network: a comparison with convolutional neural network. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshops, Las Vegas, 2016. 11--18. Google Scholar

[55] Fan D, Shim Y, Raghunathan A. STT-SNN: A Spin-Transfer-Torque Based Soft-Limiting Non-Linear Neuron for Low-Power Artificial Neural Networks. IEEE Trans Nanotechnol, 2015, 14: 1013-1023 CrossRef ADS arXiv Google Scholar

[56] Koyanagi M, Nakagawa Y, Lee K W, et al. Neuromorphic vision chip fabricated using three-dimensional integration technology. In: Proceedings of IEEE International Solid-State Circuits Conference, San Francisco, 2001. 270--271. Google Scholar

Copyright 2020 Science China Press Co., Ltd. 《中国科学》杂志社有限责任公司 版权所有

京ICP备18024590号-1