CDSE: Collaborative: Cyber Infrastructure to Enable Computer Vision Applications at the Edge Using Automated Contextual Analysis

CDSE：协作：使用自动上下文分析在边缘启用计算机视觉应用的网络基础设施

• 批准号: 2104377
• 负责人: Vipin Chaudhary
• 金额: $ 10万
• 依托单位: Case Western Reserve University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-09-01 至 2024-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2104377&HistoricalAwards=false
• 关键词: CDSE; Collaborative; Cyber; Infrastructure; Enable

Digital cameras are deployed as network edge devices, gathering visual data for such tasks as autonomous driving, traffic analysis, and wildlife observation. Analyzing the vast amount of visual data is a challenge. Existing computer vision methods require fast computers that are beyond the computational capabilities of many edge devices. This project aims to improve the efficiency of computer vision methods so that they can run on battery-powered edge devices. Based on the visual data and complementary metadata (e.g., geographical location, local time), the project first extracts contextual information (such as a city street is expected to be busy at rush hour). The contextual information can help assist determine whether analysis results are correct. For example, a wild animal is not expected on a city street. Moreover, contextual information can improve efficiency. Only certain pixels need to be analyzed (pixels on the road are useful for detecting cars, while pixels in the sky are not) and this can significantly reduce the amount of computation, thus enabling analysis on edge devices. This project constructs a cyberinfrastructure for three services: (1) understand contextual information to reduce the search space of analysis methods, (2) reduce computation by considering only necessary pixels, and (3) automate evaluation of analysis results based on the contextual information without human effort.Understanding contextual information is achieved by using background segmentation, GPS-location-dependent logic, and image depth maps. Background analysis leverages semantic segmentation and analysis over time to identify the background pixels and then generate inference rules via a background-implies-foreground relationship. If a pixel is consistently marked by the same semantic label across a long period of time, this pixel is classified as a background pixel. The background information can infer certain types of foreground objects. For example, if the background is city streets, the foreground objects can be vehicles or pedestrians; if a bison is detected, this is likely a mistake. This project processes only the foreground pixels by adding masks to the neural network layers. Masking convolution can substantially reduce the amount of computation with no loss of accuracy and no additional training is needed. Meanwhile, hierarchical neural networks can skip sections of a model based on context. For example, pixels in the sky only need to be processed by the hierarchy nodes that classify airplanes. The project provides an online service that can accept input data and analysis programs for automatic evaluation of the programs, without human created labels. The evaluation is based on the correlations of background and foreground objects.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

数码相机被部署为网络边缘设备，为自动驾驶、交通分析和野生动物观察等任务收集视觉数据。分析大量视觉数据是一项挑战。现有的计算机视觉方法需要快速的计算机，这超出了许多边缘设备的计算能力。该项目旨在提高计算机视觉方法的效率，以便它们可以在电池供电的边缘设备上运行。基于视觉数据和补充元数据（例如地理位置、当地时间），该项目首先提取上下文信息（例如预计城市街道在高峰时段会很繁忙）。上下文信息可以帮助确定分析结果是否正确。例如，城市街道上不应出现野生动物。此外，上下文信息可以提高效率。 仅需要分析某些像素（道路上的像素对于检测汽车有用，而天空中的像素则不然），这可以显着减少计算量，从而能够在边缘设备上进行分析。该项目构建了三个服务的网络基础设施：（1）理解上下文信息以减少分析方法的搜索空间，（2）通过仅考虑必要的像素来减少计算量，以及（3）根据上下文信息自动评估分析结果，而无需理解上下文信息是通过使用背景分割、GPS 位置相关逻辑和图像深度图来实现的。 背景分析利用语义分割和随时间的分析来识别背景像素，然后通过背景-暗示-前景关系生成推理规则。如果一个像素在很长一段时间内一致地被相同的语义标签标记，则该像素被分类为背景像素。背景信息可以推断某些类型的前景物体。例如，如果背景是城市街道，则前景物体可以是车辆或行人；如果检测到野牛，这可能是一个错误。该项目通过向神经网络层添加掩模来仅处理前景像素。掩蔽卷积可以大大减少计算量，而不会损失准确性，并且不需要额外的训练。同时，分层神经网络可以根据上下文跳过模型的某些部分。例如，天空中的像素只需要由对飞机进行分类的层次结构节点进行处理。该项目提供在线服务，可以接受输入数据和分析程序，以自动评估程序，无需人工创建标签。评估基于背景和前景物体的相关性。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Irrelevant Pixels are Everywhere: Find and Exclude Them for More Efficient Computer Vision

不相关的像素无处不在：查找并排除它们以提高计算机视觉效率

• DOI: 10.1109/aicas54282.2022.9870012
• 发表时间: 2022-06
• 期刊: 2022 IEEE 4th International Conference on Artificial Intelligence Circuits and Systems (AICAS
• 作者: Tung, Caleb;Goel, Abhinav;Hu, Xiao;Eliopoulos, Nick;Amobi, Emmanuel S.;Thiruvathukal, George K.;Chaudhary, Vipin;Lu, Yung
• 通讯作者: Lu, Yung