CAREER: Reinventing Computer Vision through Bio-inspired Retinomorphic Vision Sensors, Corticomorphic Compute-In-Memory Processors and Event-based Algorithms

职业：通过仿生视网膜形态视觉传感器、皮质形态内存计算处理器和基于事件的算法重塑计算机视觉

• 批准号: 2338171
• 负责人: Rajkumar Chinnakonda Kubendran
• 金额: $ 54.98万
• 依托单位: University of Pittsburgh
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2024
• 资助国家: 美国
• 起止时间: 2024-05-01 至 2029-04-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2338171&HistoricalAwards=false
• 关键词: CAREER; Reinventing; Computer; Vision; through

State-of-the-art computer vision (CV) pipelines are compute/memory intensive and power hungry making them unsuitable for high-speed applications such as hypersonic missile tracking or resource-deficit edge applications such as autonomous drone navigation due to size, weight and power (SWaP) constraints. Neuromorphic engineering is a promising frontier to usher in the next generation of CV systems taking advantage of sparsity in the input and network architecture, reducing the number of operations through event-based computation i.e., compute only when necessary. This project aims to develop a versatile energy-efficient bio-inspired sensing, computing, and learning framework by developing a closely-knit system, from devices and circuits with rich spatio-temporal dynamics to network architectures inspired by the visual cortex and adaptive learning algorithms for visual perception. This will be achieved primarily using compute-in-memory (CIM) architectures that process and extract a variety of critical visual features in close physical proximity to where the data is stored in memory. The proposed research will embark on a uniquely integrated approach that addresses challenges at all levels, from devices, circuits, architectures, and algorithms leading to novel CV applications, inspired by neuroscience, such as low latency dynamic object classification, tracking and adaptive visual attention. The breadth of skillsets that are required to effectively train a new cadre of workforce in neuromorphic engineering for computer vision makes curriculum design and integration with existing frameworks incredibly challenging. The proposed BioVision educational consortium will address this issue. The main objective of this consortium is to collaborate and implement a comprehensive workforce development plan that incorporates evidence-based best practices to help train a new generation of engineers and researchers, who are equipped to satisfy the growing needs of the computer vision industry.The grand vision of this proposal is to reimagine modern computer vision (CV) pipelines that exist today and replace the components with bio-inspired sensors, processors and algorithms that can drastically improve energy efficiency, data efficiency and lower latency. To reinvent the CV pipeline, three research thrusts will be addressed simultaneously. Thrust 1 will focus on creating and building a new class of retina-inspired vision sensors, that outperforms existing cameras, such as frame-based or neuromorphic Dynamic Vision Sensors (DVS), in terms of features, efficiency and latency. Thrust 2 will focus on modeling, design and implementation of scalable corticomorphic networks on hardware, exhibiting non-linear neuromodulatory dynamics at multiple timescales using mixed-feedback control. Thrust 3 will focus on implementation of network architectures and algorithms inspired by neuroscience, such as reinforcement learning with stochastic rewards, event-based temporal pattern recognition. The proposed research has the potential to lead a generational shift in the fields of computer vision, neuromorphic computing, and artificial intelligence. Developing an energy-efficient event-based camera capable of versatile spatiotemporal pattern recognition and novel features inspired by the retina, along with a general purpose, programmable, event-based computer vision pipeline can have a transformative impact on our society, by impacting critical areas like healthcare, Internet of Things (IoT), military defense, edge computing and industrial automation. Enabling the use of advanced CV on personal electronics can revolutionize our lifestyle through technologies such as self-driving vehicles, always-on smart surveillance, and virtual/augmented reality (VR/AR) applications. Bio-inspired vision sensors, such as the DVS camera sold by companies like Prophesee and iniVation, are primarily developed in Europe and Asia and have no industry or academic contribution from USA. This proposal will address this national challenge by training a new generation of world-class researchers and provide the USA with a leading advantage in the deployment of next-generation computer vision systems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

最先进的计算机视觉（CV）管道是计算/记忆密集型和饥饿的电源，使其不适合高速应用程序，例如高超声导弹跟踪或资源缺陷边缘应用，例如由于尺寸，重量和功率（交换）约束而导致的自动无人机导航。神经形态工程是一个有前途的前沿，可以利用输入和网络体系结构中的稀疏性来吸引下一代的简历系统，从而减少通过基于事件的计算的操作数量，即仅在必要时计算。该项目旨在通过开发一个紧密联系的系统，从具有丰富时空 - 周期性动力学的设备和电路到受视觉cortex和适应性学习算法启发的网络体系结构，从而开发一种多功能节能的生物启发的感测，计算和学习框架。这将主要使用内存（CIM）体系结构来实现，该体系结构可处理并提取与数据存储在存储器中的位置的物理近距离的各种关键视觉特征。拟议的研究将启动一种独特的综合方法，该方法从设备，电路，体系结构和算法上解决各个层次的挑战，导致了新的CV应用，灵感来自神经科学的启发，例如低延迟动态对象分类，跟踪和适应性视觉注意力。有效培训新的用于计算机视觉的神经形态工程的新员工所需的技能集，使课程设计并与现有框架的集成变得令人难以置信的挑战。拟议的生物设施教育财团将解决这个问题。该财团的主要目标是协作并实施一项全面的劳动力发展计划，该计划结合了基于证据的最佳实践，以帮助培训新一代的工程师和研究人员，他们有能力满足计算机视觉行业不断增长的需求。该提议的宏伟愿景是重新想象现代计算机视觉（CV），并将其替换为“ CV”，并可以替代Bio-serngers和Sensors，并可以替代Bio-inspired Sensors，并将其替换为Sensors，并将其替换为启发的Sensors，并替代了Senserspiend的人，并替代了SENSERS的过程。效率，数据效率和较低的延迟。为了重新发明简历管道，将同时解决三个研究推力。推力1将专注于创建和建立一类新的视网膜风格的视觉传感器，以优于现有的相机，例如基于框架或神经形态的动态视觉传感器（DVS），就特征，效率和延迟而言。推力2将着重于硬件上可扩展的皮质形态网络的建模，设计和实现，并使用混合反馈控制在多个时间尺度上显示非线性神经调节动力学。推力3将重点放在受神经科学启发的网络体系结构和算法的实现，例如使用随机奖励，基于事件的时间模式识别的增强学习。拟议的研究有可能在计算机视觉，神经形态计算和人工智能领域的世代转变。开发一种基于节能的基于事件的相机，能够通过视网膜启发的多功能时空模式识别和新型功能以及通用，可编程的，基于事件的计算机视觉管道可以通过影响关键领域，例如医疗保健，物联网（IoT），军事防御，边缘计算和工业自动化，对我们的社会产生变革性的影响。在个人电子设备上启用高级简历可以通过诸如自动驾驶汽车，始终在智能监视以及虚拟/增强现实（VR/AR）应用等技术中彻底改变我们的生活方式。生物启发的视力传感器，例如预言和invation等公司出售的DVS摄像头，主要在欧洲和亚洲开发，并且没有美国的行业或学术贡献。该提案将通过培训新一代世界一流的研究人员来应对这一国家挑战，并为美国在部署下一代计算机视觉系统的部署方面具有领先优势。该奖项反映了NSF的法定任务，并被认为值得通过基金会的知识分子优点和更广泛的审查标准通过评估来进行评估。