CAREER:Bionic Eye: Heterogeneous Integration of Hemispherical Image Sensor with Artificial Neural Network

职业：仿生眼：半球图像传感器与人工神经网络的异构集成

• 批准号: 1942868
• 负责人: Kyusang Lee
• 金额: $ 50万
• 依托单位: University of Virginia Main Campus
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-02-01 至 2025-01-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1942868&HistoricalAwards=false
• 关键词: CAREER; Bionic; Eye; Heterogeneous; Integration

Non-Technical:Vision is one of the most crucial senses for humans when evaluating a scene. Natural imaging systems such as the eye provide an aberration-free image with a wide field of view. Artificial imaging systems can be made lighter, smaller, and with minimal optical aberrations by curving the focal plane array to match the curvature of the lens. Such a conformal architecture is analogous to the shape of the human eye. Furthermore, integration of an artificial synapse in image sensors will enable to processing of the high volumes of data acquired from a scene at low power. This approach is inspired by the interaction between the eye and visual cortex. In this project, the PI will design integrated hemispherical image sensors with an embedded neuromorphic chip that allows edge computing at hardware level with low-power operation. The system will provide wide-angle, aberration free images for various applications such as vehicle navigation, threat detection, and object identification. This will greatly facilitate further development of broad smart sensor applications combined with simplified AI technology. The PI will create introductory lab modules that connect basic material science and device physics to system level integration, allowing students to connect basic science with real-life applications.Technical:In this program, the PI will demonstrate an artificial retina integrated with neuromorphic circuits that consumes extremely low power. The PI will focus on designing and fabricating a thin-film InGaAs based hemispherical focal plane array and memrisror based artificial synapses that provides a wide field of view, simultaneous localization and mapping (SLAM), data reduction, ranging capability adaptable to variable and near infrared light levels while providing object recognition capabilities. This will be enabled by a combination of various unique technologies; from materials growth and lift-off, flexible device fabrication to heterogeneous and system level integration. The fabricated thin-film compound semiconductor based hemispherical image sensor array using remote epitaxy technique will be integrated with novel computing architecture based on emerging non-volatile resistive switching devices, providing an ideal implementation of a synaptic weight in artificial neural networks. This integrated image sensor will enable edge extraction by applying hexagonal kernel to the honeycomb image sensor array, and object recognition via a single layer of fully connected convolution neural network (CNN) similar to the human retina and visual cortex. Eventually, this proposed project will provide imagers that would enhance situational awareness and recognition with minimal power consumption in a power-constrained environment that is of central importance to robotics, autonomous driving, and military applications etc.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

非技术：视觉是人类在评估场景时最重要的感官之一。自然成像系统（例如眼睛）可提供具有宽视野的无像差图像。通过弯曲焦平面阵列以匹配透镜的曲率，可以使人工成像系统变得更轻、更小，并且光学像差最小。这种共形架构类似于人眼的形状。此外，在图像传感器中集成人工突触将能够以低功耗处理从场景获取的大量数据。这种方法的灵感来自于眼睛和视觉皮层之间的相互作用。在该项目中，PI 将设计带有嵌入式神经形态芯片的集成半球形图像传感器，该芯片允许在硬件级别以低功耗运行进行边缘计算。该系统将为车辆导航、威胁检测和物体识别等各种应用提供广角、无像差图像。这将极大地促进广泛的智能传感器应用与简化的人工智能技术相结合的进一步发展。 PI 将创建介绍性实验室模块，将基础材料科学和设备物理与系统级集成连接起来，使学生能够将基础科学与现实生活应用联系起来。技术：在该计划中，PI 将演示与神经形态电路集成的人造视网膜，功耗极低。该 PI 将专注于设计和制造基于薄膜 InGaAs 的半球形焦平面阵列和基于忆阻器的人工突触，提供宽视场、同步定位和建图 (SLAM)、数据缩减、适应可变和近红外的测距能力光照水平，同时提供物体识别功能。这将通过各种独特技术的结合来实现；从材料生长和剥离、灵活的器件制造到异构和系统级集成。使用远程外延技术制造的基于薄膜化合物半导体的半球形图像传感器阵列将与基于新兴非易失性电阻开关器件的新型计算架构集成，为人工神经网络中的突触权重提供理想的实现。这种集成图像传感器将通过将六边形内核应用于蜂窝图像传感器阵列来实现边缘提取，并通过类似于人类视网膜和视觉皮层的单层全连接卷积神经网络（CNN）进行物体识别。最终，该拟议项目将提供成像仪，在功率受限的环境中以最小的功耗增强态势感知和识别，这对于机器人、自动驾驶和军事应用等至关重要。该奖项反映了 NSF 的法定使命，并已通过使用基金会的智力优点和更广泛的影响审查标准进行评估，认为值得支持。

项目成果

2D materials-assisted heterogeneous integration of semiconductor membranes toward functional devices

二维材料辅助半导体膜异质集成到功能器件

• DOI: 10.1063/5.0122768
• 发表时间: 2022-11-21
• 期刊: Journal of Applied Physics
• 影响因子: 3.2
• 作者: Minseong Park;B. Bae;Taegeon Kim;H. Kum;Kyusang Lee
• 通讯作者: Kyusang Lee

Hetero‐Integrated InGaAs Photodiode and Oxide Memristor‐Based Artificial Optical Nerve for In‐Sensor NIR Image Processing

用于In传感器近红外图像处理的异质集成InGaAs光电二极管和氧化物忆阻器人工光神经

• DOI: 10.1002/adom.202201905
• 发表时间: 2022-12-08
• 期刊: Advanced Optical Materials
• 影响因子: 9
• 作者: B. Bae;Minseong Park;Doeon Lee;Inbo Sim;Kyusang Lee
• 通讯作者: Kyusang Lee

Quantized Neural Network via Synaptic Segregation Based on Ternary Charge‐Trap Transistors

基于三元电荷陷阱晶体管的突触隔离量化神经网络

• DOI: 10.1002/aelm.202300303
• 发表时间: 2023-09-01
• 期刊: Advanced Electronic Materials
• 影响因子: 6.2
• 作者: Y. Baek;B. Bae;Jeongyong Yang;Doeon Lee;H. Lee;Minseong Park;Taegeon Kim;Sihwan Kim;Bo;Geonwook Yoo;Kyusang Lee
• 通讯作者: Kyusang Lee

An artificial neuromuscular junction for enhanced reflexes and oculomotor dynamics based on a ferroelectric CuInP 2 S 6 /GaN HEMT

基于铁电 CuInP 2 S 6 /GaN HEMT 的人工神经肌肉接头，用于增强反射和动眼动动力学

• DOI: 10.1126/sciadv.adh9889
• 发表时间: 2023-09-22
• 期刊: Science Advances
• 影响因子: 13.6
• 作者: Park, Minseong;Yang, Jeong Yong;Yeom, Min Jae;Bae, Byungjoon;Baek, Yongmin;Yoo, Geonwook;Lee, Kyusang
• 通讯作者: Lee, Kyusang

Non-Line-of-Sight Detection Based on Neuromorphic Time-of-Flight Sensing

基于神经形态飞行时间传感的非视距检测

• DOI: 10.1021/acsphotonics.3c00448
• 发表时间: 2023-06-13
• 期刊: ACS Photonics
• 影响因子: 7
• 作者: Minseong Park;Yuan Yuan;Y. Baek;B. Bae;Bo;Young Hoon Kim;Nicholas Lin;J. Heo;Kyusang Lee
• 通讯作者: Kyusang Lee