Research on neuromorphic vision sensor and its applications

Yongsheng SANG; Renhao LI; Yaoqian LI; Qiangwei WANG; Yao MAO

doi:10.11959/j.issn.2096-3750.2019.00133

您当前的位置：

首页 >

文章列表页 >

Research on neuromorphic vision sensor and its applications

Theory and Technology | 更新时间：2024-06-05

- Research on neuromorphic vision sensor and its applications
- Chinese Journal on Internet of Things Vol. 3, Issue 4, Pages: 63-71(2019)
- 作者机构：
  
  1. 四川大学计算机学院，四川成都 610065
  2. 四川旅游学院，四川成都 610100
  3. 中国科学院光电技术研究所，四川成都 610209
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.2096-3750.2019.00133
  CLC： TP212
- Published：30 December 2019，
  
  Published Online：2019-09，
- 稿件说明：
移动端阅览
YONGSHENG SANG, RENHAO LI, YAOQIAN LI, et al. Research on neuromorphic vision sensor and its applications. [J]. Chinese journal on internet of things, 2019, 3(4): 63-71.
DOI：

YONGSHENG SANG, RENHAO LI, YAOQIAN LI, et al. Research on neuromorphic vision sensor and its applications. [J]. Chinese journal on internet of things, 2019, 3(4): 63-71. DOI： 10.11959/j.issn.2096-3750.2019.00133.

摘要

神经形态视觉传感器是一种模拟生物视觉系统工作机理的传感器，具有高时间分辨率、低时延、低功耗以及高动态范围等特点。首先，介绍了神经形态工程的研究背景、神经形态芯片以及神经形态视觉传感器的工作机理和主要优点。然后，详细介绍了神经形态视觉的主要计算方法，包括概率统计方法、脉冲神经网络方法以及深度神经网络方法等。最后，综述了神经形态视觉传感器在同步定位与地图构建、图像重建以及特征检测与跟踪等方面的应用研究。对神经形态视觉传感器从硬件、计算方法和应用等方面作了系统概述，为相关研究者提供了全面的参考。

Abstract

Neuromorphic vision sensor is a biologically inspired artificial neural system that mimics algorithmic behavior of biological vision systems

which has numerous advantages over standard vision sensors

such as high temporal resolution

low latency

low power

high dynamic range

etc.At first

a brief introduction to neuromorphic engineering

neuromorphic chips

and vision sensors was given.Then the main computing methods for neuromorphic vision were reviewed

including probability and statistics

spiking neural network and deep neural network.Finally

several kinds of applications based on neuromorphic vision sensors were given

such as simultaneous localization and mapping (SLAM)

image reconstruction (IR)

etc.A review of hardware

computing methods and applications for neuromorphic vision sensors was given

which provided a comprehensive reference for researchers.

关键词

神经形态视觉传感器事件相机帧相机神经网络即时定位与地图构建图像重构

Keywords

neuromorphic vision sensorevent-based cameraframe-based cameraneural networksimultaneous localization and mappingimage reconstruction

references

陶建华, 陈云霁 . 类脑计算芯片与类脑智能机器人发展现状与思考[J]. 中国科学院院刊, 2016,31(7): 803-811.

TAO J H, CHEN Y J . Current status and consideration on brain-like computing chip and brain-like intelligent robot[J]. Bulletin of Chinese Academy of Sciences, 2016,31(7): 803-811.

MEAD C . Neuromorphic electronic systems[J]. Proceedings of the IEEE, 1990,78(10): 1629-1636.

NAWROCKI R A, VOYLES R M, SHAHEEN S E . A mini review of neuromorphic architectures and implementations[J]. IEEE Transactions on Electron Devices, 2016,63(10): 3819-3829.

BOAHEN K . Neuromorphic microchips[J]. Scientific American, 2005,292(5): 56-63.

HASLER J, MARR B . Finding a roadmap to achieve large neuromorphic hardware systems[J]. Frontiers in Neuroscience, 2013,7:118.

GAO P, BENJAMIN B V, BOAHEN K . Dynamical system guided mapping of quantitative neuronal models onto neuromorphic hardware[J]. IEEE Transactions on Circuits and Systems I:Regular Papers, 2012,59(10): 2383-2394.

MEROLLA P A, ARTHUR J V, ALVAREZ-ICAZA R ,et al. A million spiking-neuron integrated circuit with a scalable communication network and interface[J]. Science, 2014,345(6197): 668-673.

YU S, CHEN P Y, CAO Y ,et al. Scaling-up resistive synaptic arrays for neuro-inspired architecture:challenges and prospect[C]// Electron Devices Meeting. IEEE, 2016.

MA L, XU Q, YANG Y . Organic nonvolatile memory by controlling the dynamic copper—ion concentration within organic layer[J]. Applied Physics Letters, 2004,84(24): 4908-4910.

BLOM P W, SMIT C W, HAVERKORT J E ,et al. Carrier capture into a semiconductor quantum well[J]. Physical Review B,Condense Matter, 1993,47(4): 2072-2081.

SON D I, YOU C H, KIM W T ,et al. Electrical bistabilities and memory mechanisms of organic bistable devices based on colloidal ZnO quantum dot-polymethylmethacrylate polymer nanocomposites[J]. Applied Physics Letters, 2009,94(13):>132103.

HA H, KIM O . Electrode-material-dependent switching characteristics of organic nonvolatile memory devices based on Poly(3,4ethylenedioxythiophene):poly (styrenesulfonate) film[J]. IEEE Electron Device Letters, 2010,31(4): 368-370.

MÜLLER S, PERLOV C, JACKSON W ,et al. A polymer/ semiconductor write-once read-many-times memory[J]. Nature, 2003,426(6963): 166-169.

NAWROCKI R A, VOYLES R M, SHAHEEN S E . Neurons in polymer:hardware neural units based on polymer memristive devices and polymer transistors[J]. IEEE Transactions on Electron Devices, 2014,61(10): 3513-3519.

VANARSE A, OSSEIRAN A, RASSAU A . A review of current neuromorphic approaches for vision,auditory,and olfactory sensors[J]. Frontiers in Neuroscience, 2016,10(275):115.

SARPESHKAR R, LYON R F, MEAD C . A low-power wide-dynamicrange analog VLSI cochlea[J]. Analog Integrated Circuits and Signal Processing, 1998,16(3): 245-274.

WATTS L, KERNS D A, LYON R F ,et al. Improved implementation of the silicon cochlea[J]. IEEE Journal of Solid-State Circuits, 1992,27(5): 692-700.

SARPESHKAR R, SALTHOUSE C, SIT J J ,et al. An ultra-low-power programmable analog bionic ear processor[J]. IEEE Transactions on Biomedical Engineering, 2005,52(4): 711-727.

CHAN V, LIU S C, VAN SCHAIK A . AER EAR:a matched silicon cochlea pair with address event representation interface[J]. IEEE Transactions on Circuits and Systems I:Regular Papers, 2007,54(1): 48-59.

LIU S C, VAN SCHAIK A, MINCH B A ,et al. Asynchronous binaural spatial audition sensor with 2×64×4 channel output[J]. IEEE transactions on biomedical circuits and systems, 2013,8(4): 453-464.

POSCH C, SERRANO-GOTARREDONA T, LINARES- BARRANCO B . Retinomorphic event-based vision sensors:bioinspired cameras with spiking output[J]. Proceedings of the IEEE, 2014,102(10): 1470-1484.

LICHTSTEINER P, POSCH C, DELBRUCK T . A 128×128 120dB 15μs latency asynchronous temporal contrast vision sensor[J]. IEEE Journal of Solid-State Circuits, 2008,43(2): 566-576.

BERNER R, BRANDLI C, YANG M ,et al. A 240×180 10mW 12us latency sparse-output vision sensor for mobile applications[C]// VLSI Circuits (VLSIC),2013 Symposium on IEEE. IEEE, 2013.

TOBI D, MEAD C A . Time-derivative adaptive silicon photoreceptor array[J]. Proceedings of SPIE-the International Society for Optical Engineering, 1993,1541: 92-99.

GALLEGO G, DELBRUCK T, ORCHARD G ,et al. Event-based vision:a survey[J]. arXiv:Computer Vision and Pattern Recognition, 2019.

POSCH C, MATOLIN D, WOHLGENANNT R . An asynchronous time-based image sensor[C]// IEEE International Symposium on Circuits ＆ Systems. IEEE, 2008.

CHEN S, AKSELROD P, ZHAO B ,et al. Efficient feedforward categorization of objects and human postures with address-event image sensors[J]. IEEE Transactions on Pattern Analysis ＆ Machine Intelligence, 2012,34(2):302.

ZHAO B, DING R, CHEN S ,et al. Feedforward categorization on AER motion events using cortex-like features in a spiking neural network[J]. IEEE Transactions on Neural Networks, 2015,26(9): 1963-1978.

BICHLER O, QUERLIOZ D, THORPE S J ,et al. Extraction of temporally correlated features from dynamic vision sensors with spike-timing-dependent plasticity[J]. Neural Networks, 2012,32(2): 339-348.

LI H, LI G, SHI L . Classification of spatiotemporal events based on random forest[C]// International Conference on Brain Inspired Cognitive Systems. 2016: 138-148.

SHAISTA H, ARINDAM B . Multiclass classification by adaptive network of dendritic neurons with binary synapses using structural plasticity[J]. Frontiers in Neuroscience, 2016,10:113.

PENG X, ZHAO B, YAN R ,et al. Bag of events:an efficient probability-based feature extraction method for AER image sensors[J]. IEEE Transactions on Neural Networks and Learning Systems, 2016: 1-13.

ZHANG D, LU G . Review of shape representation and description techniques[J]. Pattern recognition, 2004,37(1): 1-19.

LI H, LI G, JI X ,et al. Deep representation via convolutional neural network for classification of spatiotemporal event streams[J]. Neurocomputing, 2018:S0925231218301486.

吴金建 . 基于人类视觉系统的图像信息感知和图像质量评价[D]. 西安:西安电子科技大学, 2014.

WU J J . Image information perception and quality assessment based on the human visual system[D]. Xi’an:Xidian University, 2014.

QU X, ZHAO F, ZHOU M ,et al. A novel fast and robust binary affine invariant descriptor for image matching[J]. Mathematical Problems in Engineering, 2014,2014(1): 1-7.

LECUN Y, BENGIO Y, HINTON G . Deep learning[J]. Nature, 2015,521(7553):436.

SCHMIDHUBE R JÜRGE N . Deep learning in neural networks:an overview[J]. Neural Networks, 2015,61: 85-117.

CIREŞAN D, MEIER U, SCHMIDHUBER J . Multi-column deep neural networks for image classification[J].,2012:1202.2745. arXiv:Computer Vision and Pattern Recognition, 2012:1202.2745.

HE K, ZHANG X, REN S ,et al. Delving deep into rectifiers:surpassing human-level performance on image net classification[C]// IEEE International Conference on Computer Vision. IEEE, 2015: 1026-1034.

SCHWARTZ E L . Spatial mapping in the primate sensory projection:analytic structure and relevance to perception[J]. Biological Cybernetics, 1977,25(4): 181-194.

LAGORCE X, ORCHARD G, GALLUPPI F ,et al. Hots:a hierarchy of event-based time-surfaces for pattern recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017,39(7): 1346-1359.

SAEED A, COHEN G K, WANG R M ,et al. The ripple pond:enabling spiking networks to see[J]. Frontiers in Neuroscience, 2013,7:212.

RAMESH B, YANG H, ORCHARD G M ,et al. DART:distribution aware retinal transform for event-based cameras[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019.

GERSTNER W, KISTLER W . Spiking neuron models:an introduction[M]. Oxford: Cambridge University PressPress, 2002.

IZHIKEVICH E M, EDELMAN G M . Large-scale model of mammalian thalamocortical systems[J]. Proceedings of the National Academy of Sciences, 2008,105(9): 3593-3598.

ANANTHANARAYANAN R, ESSER S K, SIMON H D ,et al. The cat is out of the bag:cortical simulations with 109neurons,1013synapses[C]// Proceedings of the Conference on High Performance Computing Networking,Storage and Analysis. IEEE, 2009.

ELIASMITH C, STEWART T C, CHOO X ,et al. A large-scale model of the functioning brain[J]. Science, 2012,338(6111): 1202-1205.

SERRANO-GOTARREDONA T, LINARES-BARRANCO B, GALLUPPI F ，et al. ConvNets experiments on SpiNNaker[C]// IEEE International Symposium on Circuits and Systems. IEEE, 2015.

FOLOWOSELE F, VOGELSTEIN R J, ETIENNE-CUMMINGS R . Towards a cortical prosthesis:implementing a spike-based HMAX model of visual object recognition in silico[J]. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 2011,1(4): 516-525.

AMIR A, TABA B, BERG D ,et al. A low power,fully event-based gesture recognition system[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2017.

EVANGELOS S, DANIEL N, MICHAEL P ,et al. Robustness of spiking deep belief networks to noise and reduced bit precision of neuro-inspired hardware platforms[J]. Frontiers in Neuroscience, 2015,9(00222):222.

FARABET C, RAFAEL P, PÉREZ-CARRASCO PJ ,et al. Comparison between frame-constrained fix-pixel-value and frame-free spiking-dynamic-pixel ConvNets for visual processing[J]. Frontiers in Neuroscience, 2012,6: 32-32.

NGUYEN A, DO T T, CALDWELL D G ,et al. Real-time 6DOF pose relocalization for event cameras with stacked spatial LSTM networks[J].,2017. arXiv:Computer Vision and Pattern Recognition, 2017.

PARK P K J, CHO B H, PARK J M ,et al. Performance improvement of deep learning based gesture recognition using spatiotemporal demosaicing technique[C]// IEEE International Conference on Image Processing. IEEE, 2016.

ZHU A Z, YUAN L, CHANEY K ,et al. EV-FlowNet:self-supervised optical flow estimation for event-based cameras[J]. arXiv:Computer Vision and Pattern Recognition, 2018.

VIDAL A R, REBECQ H, HORSTSCHAEFER T ,et al. Ultimate SLAM? Combining events,images,and IMU for robust visual SLAM in HDR and high-speed scenarios[J]. IEEE Robotics and Automation Letters, 2018,3(2): 994-1001.

REBECQ H, HORSTSCHÄFER T, GALLEGO G ,et al. EVO:a geometric approach to event-based 6-DOF parallel tracking and mapping in real time[J]. IEEE Robotics and Automation Letters, 2016,2(2): 593-600.

NI Z, PACORET C, BENOSMAN R ,et al. Asynchronous event-based high speed vision for microparticle tracking[J]. Journal of Microscopy, 2012,245(3): 236-244.

Views

1295

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

A data-driven approach to wireless channel available throughput estimation and prediction

Neural network wind speed prediction based on multiple prediction model and nonlinear combination

Human activity recognition algorithm based on the spatial feature for WBAN

Related Author

Yao XIAO

Junshuo LIU

Zhifu LONG

Caiming QIU

Jiajun WANG

Wei CAO

Guilong ZHANG

Huaizhi ZHANG

Related Institution

Xi'an University of Posts and Telecommunications

Yalong River Basin Hydropower Development Co., Ltd.

⁰