CHS: Small: Robust Eye Tracking and Facial Context Sensing for Mobile Augmented Reality and Virtual Reality

CHS：小型：用于移动增强现实和虚拟现实的强大眼动追踪和面部情境感知

• 批准号: 1815347
• 负责人: Deepak Ganesan
• 金额: $ 50万
• 依托单位: University of Massachusetts Amherst
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-09-01 至 2023-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1815347&HistoricalAwards=false
• 关键词: CHS; Small; Robust; Eye; Tracking

Virtual and Augmented Reality headsets may soon become a major mass-market mobile consumer electronics device. However, there are challenges in making these headsets more portable and less power hungry, and in enabling a more immersive user experience by increasing awareness of the user's attention and emotional state. This project's goals are two-fold. The first goal is to enhance systems' ability to track the eye in a low-power yet robust manner by exploring new eye tracker designs combined with machine learning approaches. The second goal is to study methods to infer facial expressions and user emotion from wearable headsets by combining multiple sensing modalities including facial muscle activity and cameras. The project will also have educational impact on middle-school students and under-represented students through workshops. The project involves activities and innovations in several areas. On the eye tracking side, the project will explore new hardware designs with stereo cameras and machine learning approaches to enhance robustness to face and eye shapes as well as eyeglass movements. On the facial expression sensing side, the team will explore multimodal sensing methods that combine electrooculography (EOG) and multiple camera views to infer expressions as well as to reduce power consumption. On the networked systems side, the project will look at leveraging eye tracking and facial expression sensing to enable wireless offload of rendering to a nearby compute node. The techniques to be researched can significantly lower the power consumed by AR and VR systems, enhance their ability to sense human expressions to enable more immersive experience, and reduce computational complexity by enabling predictive pre-fetch from a wirelessly connected edge cloud.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

虚拟和增强现实耳机可能很快成为主要的大众市场移动消费电子设备。但是，使这些耳机更便携，更少的饥饿力以及通过提高对用户的注意力和情感状态的认识来实现更身临其境的用户体验存在挑战。该项目的目标是两个方面。第一个目标是通过探索与机器学习方法相结合的新型眼动式设计设计，以低功率但可靠的方式来增强系统以低功率但可靠的方式追踪眼睛的能力。第二个目标是研究通过组合包括面部肌肉活动和相机在内的多种感应方式来推断面部表情和用户情绪的方法。该项目还将通过研讨会对中学生和代表性不足的学生产生教育影响。该项目涉及多个领域的活动和创新。在眼睛跟踪方面，该项目将采用立体相机和机器学习方法探索新的硬件设计，以增强面部和眼睛形状以及眼镜运动的鲁棒性。在面部表达传感方面，团队将探索结合电视学（EOG）和多个相机视图以推断表达以及减少功耗的多模式感应方法。在网络系统方面，该项目将考虑利用眼睛跟踪和面部表达传感，以使无线渲染能够卸载到附近的计算节点。要研究的技术可以显着降低AR和VR系统所消耗的功率，增强其感知人类表达式以实现更多沉浸式体验的能力，并通过从无线连接的边缘云中启用预测性预先提取来降低计算复杂性。这一奖项反映了这一奖项。 NSF的法定使命，并使用基金会的知识分子优点和更广泛的影响审查标准来评估值得支持。

项目成果

Low-latency speculative inference on distributed multi-modal data streams

• DOI: 10.1145/3458864.3467884
• 发表时间: 2021-06
• 期刊: Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services
• 作者: Tianxing Li;Jin Huang;Erik Risinger;Deepak Ganesan

PhyMask: Robust Sensing of Brain Activity and Physiological Signals During Sleep with an All-textile Eye Mask

• DOI: 10.1145/3513023
• 发表时间: 2021-06
• 期刊: ACM Transactions on Computing for Healthcare (HEALTH)
• 作者: Soha Rostaminia;S. Z. Homayounfar;A. Kiaghadi;Trisha L. Andrew;Deepak Ganesan

Re-thinking computation offload for efficient inference on IoT devices with duty-cycled radios

• DOI: 10.1145/3570361.3592514
• 发表时间: 2023-07
• 期刊: Proceedings of the 29th Annual International Conference on Mobile Computing and Networking
• 作者: Jin Huang;H. Guan;Deepak Ganesan

CurtainNet: Enabling precise beamforming with a deformable antenna array on a fabric substrate

• DOI: 10.1145/3625687.3625787
• 发表时间: 2023-11
• 期刊: Proceedings of the 21st ACM Conference on Embedded Networked Sensor Systems
• 作者: Xingda Chen;Ankur Aditya;Zhenyu Lei;Deepak Ganesan

Multimodal Smart Eyewear for Longitudinal Eye Movement Tracking

• DOI: 10.1016/j.matt.2020.07.030
• 发表时间: 2020-10-07
• 期刊: MATTER
• 影响因子: 18.9
• 作者: Homayounfar, S. Zohreh;Rostaminia, Soha;Andrew, Trisha L.
• 通讯作者: Andrew, Trisha L.