神经形态视觉传感器及其应用研究

桑永胜; 李仁昊; 李耀仟; 王蔷薇; 毛耀

doi:10.11959/j.issn.2096-3750.2019.00133

您当前的位置：

首页 >

文章列表页 >

神经形态视觉传感器及其应用研究

理论与技术 | 更新时间：2024-06-05

- 神经形态视觉传感器及其应用研究
- Research on neuromorphic vision sensor and its applications
- 物联网学报 2019年3卷第4期页码：63-71
- 作者机构：
  
  1. 四川大学计算机学院，四川成都 610065
  2. 四川旅游学院，四川成都 610100
  3. 中国科学院光电技术研究所，四川成都 610209
- 作者简介：
  
  [ "桑永胜（1974- ），男，四川绵阳人，四川大学计算机学院副教授，主要研究方向为类脑智能、机器视觉以及深度学习等" ]
  [ "李仁昊（1997- ），男，北京人，四川大学计算金融专业学生，主要研究方向为计算机视觉、数据挖掘等" ]
  [ "李耀仟（1997- ），男，广西崇左人，四川大学计算金融专业学生，主要研究方向计算机视觉、大数据分析。" ]
  [ "王蔷薇（1981- ），男，四川绵阳人，四川旅游学院讲师，主要研究方向为智能数据分析。" ]
  [ "毛耀（1978- ），男，四川眉山人，中国科学院光电技术研究所研究员，主要研究方向为光束控制及计算机管理技术，包括预测跟踪、惯性稳定、自主控制、机器视觉、强化学习等。" ]
- 基金信息：
  
  国家自然科学基金资助项目;The National Natural Science Foundation of China(U1831121);四川省重大科技专项;The Key S＆T Special Projects of Sichuan(2019ZDZX0006);四川省重大科技专项;The Key S＆T Special Projects of Sichuan(2018GZDZX0028)
- DOI：10.11959/j.issn.2096-3750.2019.00133
  中图分类号： TP212
- 纸质出版日期：2019-12-30，
  
  网络出版日期：2019-09，
- 稿件说明：
移动端阅览
桑永胜, 李仁昊, 李耀仟, 等. 神经形态视觉传感器及其应用研究[J]. 物联网学报, 2019,3(4):63-71.

YONGSHENG SANG, RENHAO LI, YAOQIAN LI, et al. Research on neuromorphic vision sensor and its applications. [J]. Chinese journal on internet of things, 2019, 3(4): 63-71.
桑永胜, 李仁昊, 李耀仟, 等. 神经形态视觉传感器及其应用研究[J]. 物联网学报, 2019,3(4):63-71. DOI： 10.11959/j.issn.2096-3750.2019.00133.

YONGSHENG SANG, RENHAO LI, YAOQIAN LI, et al. Research on neuromorphic vision sensor and its applications. [J]. Chinese journal on internet of things, 2019, 3(4): 63-71. DOI： 10.11959/j.issn.2096-3750.2019.00133.

摘要

神经形态视觉传感器是一种模拟生物视觉系统工作机理的传感器，具有高时间分辨率、低时延、低功耗以及高动态范围等特点。首先，介绍了神经形态工程的研究背景、神经形态芯片以及神经形态视觉传感器的工作机理和主要优点。然后，详细介绍了神经形态视觉的主要计算方法，包括概率统计方法、脉冲神经网络方法以及深度神经网络方法等。最后，综述了神经形态视觉传感器在同步定位与地图构建、图像重建以及特征检测与跟踪等方面的应用研究。对神经形态视觉传感器从硬件、计算方法和应用等方面作了系统概述，为相关研究者提供了全面的参考。

Abstract

Neuromorphic vision sensor is a biologically inspired artificial neural system that mimics algorithmic behavior of biological vision systems

which has numerous advantages over standard vision sensors

such as high temporal resolution

low latency

low power

high dynamic range

etc.At first

a brief introduction to neuromorphic engineering

neuromorphic chips

and vision sensors was given.Then the main computing methods for neuromorphic vision were reviewed

including probability and statistics

spiking neural network and deep neural network.Finally

several kinds of applications based on neuromorphic vision sensors were given

such as simultaneous localization and mapping (SLAM)

image reconstruction (IR)

etc.A review of hardware

computing methods and applications for neuromorphic vision sensors was given

which provided a comprehensive reference for researchers.

关键词

神经形态视觉传感器事件相机帧相机神经网络即时定位与地图构建图像重构

Keywords

neuromorphic vision sensorevent-based cameraframe-based cameraneural networksimultaneous localization and mappingimage reconstruction

references

陶建华, 陈云霁 . 类脑计算芯片与类脑智能机器人发展现状与思考[J]. 中国科学院院刊, 2016,31(7): 803-811.

TAO J H, CHEN Y J . Current status and consideration on brain-like computing chip and brain-like intelligent robot[J]. Bulletin of Chinese Academy of Sciences, 2016,31(7): 803-811.

MEAD C . Neuromorphic electronic systems[J]. Proceedings of the IEEE, 1990,78(10): 1629-1636.

NAWROCKI R A, VOYLES R M, SHAHEEN S E . A mini review of neuromorphic architectures and implementations[J]. IEEE Transactions on Electron Devices, 2016,63(10): 3819-3829.

BOAHEN K . Neuromorphic microchips[J]. Scientific American, 2005,292(5): 56-63.

HASLER J, MARR B . Finding a roadmap to achieve large neuromorphic hardware systems[J]. Frontiers in Neuroscience, 2013,7:118.

GAO P, BENJAMIN B V, BOAHEN K . Dynamical system guided mapping of quantitative neuronal models onto neuromorphic hardware[J]. IEEE Transactions on Circuits and Systems I:Regular Papers, 2012,59(10): 2383-2394.

MEROLLA P A, ARTHUR J V, ALVAREZ-ICAZA R ,et al. A million spiking-neuron integrated circuit with a scalable communication network and interface[J]. Science, 2014,345(6197): 668-673.

YU S, CHEN P Y, CAO Y ,et al. Scaling-up resistive synaptic arrays for neuro-inspired architecture:challenges and prospect[C]// Electron Devices Meeting. IEEE, 2016.

MA L, XU Q, YANG Y . Organic nonvolatile memory by controlling the dynamic copper—ion concentration within organic layer[J]. Applied Physics Letters, 2004,84(24): 4908-4910.

BLOM P W, SMIT C W, HAVERKORT J E ,et al. Carrier capture into a semiconductor quantum well[J]. Physical Review B,Condense Matter, 1993,47(4): 2072-2081.

SON D I, YOU C H, KIM W T ,et al. Electrical bistabilities and memory mechanisms of organic bistable devices based on colloidal ZnO quantum dot-polymethylmethacrylate polymer nanocomposites[J]. Applied Physics Letters, 2009,94(13):>132103.

HA H, KIM O . Electrode-material-dependent switching characteristics of organic nonvolatile memory devices based on Poly(3,4ethylenedioxythiophene):poly (styrenesulfonate) film[J]. IEEE Electron Device Letters, 2010,31(4): 368-370.

MÜLLER S, PERLOV C, JACKSON W ,et al. A polymer/ semiconductor write-once read-many-times memory[J]. Nature, 2003,426(6963): 166-169.

NAWROCKI R A, VOYLES R M, SHAHEEN S E . Neurons in polymer:hardware neural units based on polymer memristive devices and polymer transistors[J]. IEEE Transactions on Electron Devices, 2014,61(10): 3513-3519.

VANARSE A, OSSEIRAN A, RASSAU A . A review of current neuromorphic approaches for vision,auditory,and olfactory sensors[J]. Frontiers in Neuroscience, 2016,10(275):115.

SARPESHKAR R, LYON R F, MEAD C . A low-power wide-dynamicrange analog VLSI cochlea[J]. Analog Integrated Circuits and Signal Processing, 1998,16(3): 245-274.

WATTS L, KERNS D A, LYON R F ,et al. Improved implementation of the silicon cochlea[J]. IEEE Journal of Solid-State Circuits, 1992,27(5): 692-700.

SARPESHKAR R, SALTHOUSE C, SIT J J ,et al. An ultra-low-power programmable analog bionic ear processor[J]. IEEE Transactions on Biomedical Engineering, 2005,52(4): 711-727.

CHAN V, LIU S C, VAN SCHAIK A . AER EAR:a matched silicon cochlea pair with address event representation interface[J]. IEEE Transactions on Circuits and Systems I:Regular Papers, 2007,54(1): 48-59.

LIU S C, VAN SCHAIK A, MINCH B A ,et al. Asynchronous binaural spatial audition sensor with 2×64×4 channel output[J]. IEEE transactions on biomedical circuits and systems, 2013,8(4): 453-464.

POSCH C, SERRANO-GOTARREDONA T, LINARES- BARRANCO B . Retinomorphic event-based vision sensors:bioinspired cameras with spiking output[J]. Proceedings of the IEEE, 2014,102(10): 1470-1484.

LICHTSTEINER P, POSCH C, DELBRUCK T . A 128×128 120dB 15μs latency asynchronous temporal contrast vision sensor[J]. IEEE Journal of Solid-State Circuits, 2008,43(2): 566-576.

BERNER R, BRANDLI C, YANG M ,et al. A 240×180 10mW 12us latency sparse-output vision sensor for mobile applications[C]// VLSI Circuits (VLSIC),2013 Symposium on IEEE. IEEE, 2013.

TOBI D, MEAD C A . Time-derivative adaptive silicon photoreceptor array[J]. Proceedings of SPIE-the International Society for Optical Engineering, 1993,1541: 92-99.

GALLEGO G, DELBRUCK T, ORCHARD G ,et al. Event-based vision:a survey[J]. arXiv:Computer Vision and Pattern Recognition, 2019.

POSCH C, MATOLIN D, WOHLGENANNT R . An asynchronous time-based image sensor[C]// IEEE International Symposium on Circuits ＆ Systems. IEEE, 2008.

CHEN S, AKSELROD P, ZHAO B ,et al. Efficient feedforward categorization of objects and human postures with address-event image sensors[J]. IEEE Transactions on Pattern Analysis ＆ Machine Intelligence, 2012,34(2):302.

ZHAO B, DING R, CHEN S ,et al. Feedforward categorization on AER motion events using cortex-like features in a spiking neural network[J]. IEEE Transactions on Neural Networks, 2015,26(9): 1963-1978.

BICHLER O, QUERLIOZ D, THORPE S J ,et al. Extraction of temporally correlated features from dynamic vision sensors with spike-timing-dependent plasticity[J]. Neural Networks, 2012,32(2): 339-348.

LI H, LI G, SHI L . Classification of spatiotemporal events based on random forest[C]// International Conference on Brain Inspired Cognitive Systems. 2016: 138-148.

SHAISTA H, ARINDAM B . Multiclass classification by adaptive network of dendritic neurons with binary synapses using structural plasticity[J]. Frontiers in Neuroscience, 2016,10:113.

PENG X, ZHAO B, YAN R ,et al. Bag of events:an efficient probability-based feature extraction method for AER image sensors[J]. IEEE Transactions on Neural Networks and Learning Systems, 2016: 1-13.

ZHANG D, LU G . Review of shape representation and description techniques[J]. Pattern recognition, 2004,37(1): 1-19.

LI H, LI G, JI X ,et al. Deep representation via convolutional neural network for classification of spatiotemporal event streams[J]. Neurocomputing, 2018:S0925231218301486.

吴金建 . 基于人类视觉系统的图像信息感知和图像质量评价[D]. 西安:西安电子科技大学, 2014.

WU J J . Image information perception and quality assessment based on the human visual system[D]. Xi’an:Xidian University, 2014.

QU X, ZHAO F, ZHOU M ,et al. A novel fast and robust binary affine invariant descriptor for image matching[J]. Mathematical Problems in Engineering, 2014,2014(1): 1-7.

LECUN Y, BENGIO Y, HINTON G . Deep learning[J]. Nature, 2015,521(7553):436.

SCHMIDHUBE R JÜRGE N . Deep learning in neural networks:an overview[J]. Neural Networks, 2015,61: 85-117.

CIREŞAN D, MEIER U, SCHMIDHUBER J . Multi-column deep neural networks for image classification[J].,2012:1202.2745. arXiv:Computer Vision and Pattern Recognition, 2012:1202.2745.

HE K, ZHANG X, REN S ,et al. Delving deep into rectifiers:surpassing human-level performance on image net classification[C]// IEEE International Conference on Computer Vision. IEEE, 2015: 1026-1034.

SCHWARTZ E L . Spatial mapping in the primate sensory projection:analytic structure and relevance to perception[J]. Biological Cybernetics, 1977,25(4): 181-194.

LAGORCE X, ORCHARD G, GALLUPPI F ,et al. Hots:a hierarchy of event-based time-surfaces for pattern recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017,39(7): 1346-1359.

SAEED A, COHEN G K, WANG R M ,et al. The ripple pond:enabling spiking networks to see[J]. Frontiers in Neuroscience, 2013,7:212.

RAMESH B, YANG H, ORCHARD G M ,et al. DART:distribution aware retinal transform for event-based cameras[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019.

GERSTNER W, KISTLER W . Spiking neuron models:an introduction[M]. Oxford: Cambridge University PressPress, 2002.

IZHIKEVICH E M, EDELMAN G M . Large-scale model of mammalian thalamocortical systems[J]. Proceedings of the National Academy of Sciences, 2008,105(9): 3593-3598.

ANANTHANARAYANAN R, ESSER S K, SIMON H D ,et al. The cat is out of the bag:cortical simulations with 109neurons,1013synapses[C]// Proceedings of the Conference on High Performance Computing Networking,Storage and Analysis. IEEE, 2009.

ELIASMITH C, STEWART T C, CHOO X ,et al. A large-scale model of the functioning brain[J]. Science, 2012,338(6111): 1202-1205.

SERRANO-GOTARREDONA T, LINARES-BARRANCO B, GALLUPPI F ，et al. ConvNets experiments on SpiNNaker[C]// IEEE International Symposium on Circuits and Systems. IEEE, 2015.

FOLOWOSELE F, VOGELSTEIN R J, ETIENNE-CUMMINGS R . Towards a cortical prosthesis:implementing a spike-based HMAX model of visual object recognition in silico[J]. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 2011,1(4): 516-525.

AMIR A, TABA B, BERG D ,et al. A low power,fully event-based gesture recognition system[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2017.

EVANGELOS S, DANIEL N, MICHAEL P ,et al. Robustness of spiking deep belief networks to noise and reduced bit precision of neuro-inspired hardware platforms[J]. Frontiers in Neuroscience, 2015,9(00222):222.

FARABET C, RAFAEL P, PÉREZ-CARRASCO PJ ,et al. Comparison between frame-constrained fix-pixel-value and frame-free spiking-dynamic-pixel ConvNets for visual processing[J]. Frontiers in Neuroscience, 2012,6: 32-32.

NGUYEN A, DO T T, CALDWELL D G ,et al. Real-time 6DOF pose relocalization for event cameras with stacked spatial LSTM networks[J].,2017. arXiv:Computer Vision and Pattern Recognition, 2017.

PARK P K J, CHO B H, PARK J M ,et al. Performance improvement of deep learning based gesture recognition using spatiotemporal demosaicing technique[C]// IEEE International Conference on Image Processing. IEEE, 2016.

ZHU A Z, YUAN L, CHANEY K ,et al. EV-FlowNet:self-supervised optical flow estimation for event-based cameras[J]. arXiv:Computer Vision and Pattern Recognition, 2018.

VIDAL A R, REBECQ H, HORSTSCHAEFER T ,et al. Ultimate SLAM? Combining events,images,and IMU for robust visual SLAM in HDR and high-speed scenarios[J]. IEEE Robotics and Automation Letters, 2018,3(2): 994-1001.

REBECQ H, HORSTSCHÄFER T, GALLEGO G ,et al. EVO:a geometric approach to event-based 6-DOF parallel tracking and mapping in real time[J]. IEEE Robotics and Automation Letters, 2016,2(2): 593-600.

NI Z, PACORET C, BENOSMAN R ,et al. Asynchronous event-based high speed vision for microparticle tracking[J]. Journal of Microscopy, 2012,245(3): 236-244.

浏览量

1295

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

数据驱动的无线信道可用吞吐量估计与预测方法

基于多预测模型与非线性组合的神经网络风速预测

基于空间特征的无线体域网人体姿态识别算法