Collaborative Research: CPS: Medium: Closing the Teleoperation Gap: Integrating Scene and Network Understanding for Dexterous Control of Remote Robots

协作研究：CPS：中：缩小远程操作差距：集成场景和网络理解以实现远程机器人的灵巧控制

• 批准号: 2039070
• 负责人: Keith Winstein
• 金额: $ 40万
• 依托单位: Stanford University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-02-15 至 2025-01-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2039070&HistoricalAwards=false
• 关键词: Collaborative; Research; CPS; Medium; Closing

The aim of this proposal is to enable people to control robots remotely using virtual reality. Using cameras mounted on the robot and a virtual reality headset, a person can see the environment around the robot. However, controlling the robot using existing technologies is hard: there is a time delay because it’s slow to send high quality video over the Internet. In addition, the fidelity of the image is worse than looking through human eyes, with a fixed and narrow view. This proposal will address these limitations by creating a new system which understands the geometry and appearance of the robot’s environment. Instead of sending high-quality video over the Internet, this new system will only send a smaller amount of information about how the environment’s geometry and appearance has changed over time. Further, understanding the geometry and appearance will let us expand the view visible to the person. Overall, these will improve a human’s ability to remotely control the robot by increasing fidelity and responsiveness. We will demonstrate this technology on household tasks, on assembly tasks, and by manipulating small objects.The aim of this proposal is to test the hypothesis that integrating scene and networking understanding can enable efficient transmission and rendering for dexterous control of remote robots through virtual reality interfaces. This system will result in dexterous teleoperation that enables remote human operators to perform complex tasks with remote robot manipulators, such as cleaning a room or repairing a machine. Such tasks have not previously been demonstrated to be teleoperated for two reasons: 1) lack of an intuitive awareness and understanding of the scene around the remote robot, and 2) lack of an effective low-latency interface to control the robot. We will address these problems by creating new scene- and network-aware algorithms which tightly couple sensing, display, interaction and transmission, enabling the operator to quickly and intuitively understand the environment around the robot. This project will research new interfaces which allow the operator to use their hand to directly specify the robot’s end effector pose in six degrees of freedom, combined with spatial- and semantic-object-based models that allow safe high-level commands. This project will evaluate the proposed system by assessing the speed and accuracy of the remote operator’s ability to complete complex tasks, including assembly tasks; the aim will be to complete unstructured assembly tasks that have never been demonstrated to be remotely teleoperated before.This project is in response to the NSF Cyber-Physical Systems 20-563 solicitation.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该提案的目的是使人们能够使用虚拟现实远程控制机器人，使用安装在机器人上的摄像头和虚拟现实耳机，人们可以看到机器人周围的环境。但是，使用现有技术控制机器人很困难：存在时间延迟，因为通过互联网发送高质量视频的速度很慢，此外，图像的保真度比通过固定且狭窄的视角观看的图像要差。该提案将通过创建新的图像来解决这些限制。理解几何形状和外观的系统这个新系统不会通过互联网发送高质量的视频，而是只会发送有关环境的几何形状和外观如何随时间变化的少量信息。此外，了解几何形状和外观将使我们扩展视野。总体而言，这些将通过提高保真度和响应能力来提高人类远程控制机器人的能力，我们将在家庭任务、组装任务和操纵小物体上演示这项技术。检验整合场景和网络的假设理解可以通过虚拟现实接口实现远程机器人灵巧控制的有效传输和渲染，从而使远程人类操作员能够使用远程机器人操纵器执行复杂的任务，例如清洁房间或修理机器等。以前没有证明任务可以远程操作，原因有两个：1）缺乏对远程机器人周围场景的直观认识和理解，2）缺乏有效的低延迟界面来控制机器人。将通过创建新的场景和网络感知算法来解决这些问题，这些算法将传感、显示、交互和传输紧密结合在一起，使操作员能够快速直观地了解机器人周围的环境。该项目将研究新的界面，使操作员能够用手直接指定机器人的六个自由度的末端执行器姿势，并结合基于空间和语义对象的模型，允许安全的高级命令。该项目将通过评估机器人的速度和准确性来评估所提出的系统。远程操作员完成任务的能力复杂的任务，包括组装任务；目标是完成以前从未被证明可以远程操作的非结构化组装任务。该项目是为了响应 NSF 网络物理系统 20-563 招标。该奖项反映了 NSF 的法定使命通过使用基金会的智力价值和更广泛的影响审查标准进行评估，并被认为值得支持。

项目成果

Sidecar: in-network performance enhancements in the age of paranoid transport protocols

Sidecar：偏执传输协议时代的网络内性能增强

• DOI: 10.1145/3563766.3564113
• 发表时间: 2022
• 期刊: The Twenty-first ACM Workshop on Hot Topics in Networks (HotNets 2022
• 作者: Yuan, Gina;Zhang, David K.;Sotoudeh, Matthew;Welzl, Michael;Winstein, Keith
• 通讯作者: Winstein, Keith

Computation-centric networking

以计算为中心的网络

• DOI: 10.1145/3563766.3564106
• 发表时间: 2022
• 期刊: HotNets
• 作者: Deng, Yuhan;Montemayor, Angela;Levy, Amit;Winstein, Keith
• 通讯作者: Winstein, Keith

R2E2: low-latency path tracing of terabyte-scale scenes using thousands of cloud CPUs

R2E2：使用数千个云CPU对TB级场景进行低延迟路径追踪

• DOI: 10.1145/3528223.3530171
• 发表时间: 2022
• 期刊: ACM Transactions on Graphics
• 影响因子: 6.2
• 作者: Fouladi, Sadjad;Shacklett, Brennan;Poms, Fait;Arora, Arjun;Ozdemir, Alex;Raghavan, Deepti;Hanrahan, Pat;Fatahalian, Kayvon;Winstein, Keith
• 通讯作者: Winstein, Keith