CAREER: Interactive Program Synthesis for Web Automation

职业：Web 自动化的交互式程序综合

• 批准号: 2236233
• 负责人: Xinyu Wang
• 金额: $ 52.49万
• 依托单位: Regents of the University of Michigan - Ann Arbor
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-03-01 至 2028-02-29
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2236233&HistoricalAwards=false
• 关键词: CAREER; Interactive; Program; Synthesis; Web

Process Automation continues to be a main driver of digital transformation. Yet, it is still technically very demanding to create such automation programs. This project aims to significantly lower the technical barrier of creating web automation programs. The project’s novelties are a suite of new rewrite-based program synthesis algorithms that can automatically generate programs from user demonstrations. The project’s impacts are to enable non-experts in need of performing a tedious but programmatic web-related task to create a program to automate the work, even if they have little or no background in web programming. At the core, this project is developing algorithms that take as input a user demonstration – in the form of a trace A of user actions (e.g., clicking buttons, scraping data) – and synthesize a parameterized program P with control-flow structures by rewriting A to P. The approach is based on finite tree automata and involves both neural and symbolic elements in the underlying web automation language, with the goal of being able to effectively reason about the webpage contents while still leveraging the underlying webpage structure. The project is curating a new suite of web automation tasks that are independently useful for future research beyond this project. The investigator is collaborating with partners at Michigan (e.g., Women in Science and Engineering (WISE), Michigan Louis Stokes Alliance for Minority Participation (MI-LSAMP) and Engineering Center for Academic Success (ECAS)) to increase participation of students from underrepresented groups in his research.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

流程自动化仍然是数字化转型的主要驱动力，但创建此类自动化程序在技术上仍然要求很高。该项目的新颖之处在于进行了一系列新的重写。基于的程序合成算法可以从用户演示中自动生成程序，该项目的影响是使需要执行繁琐但程序化的网络相关任务的非专家能够创建一个程序来自动化工作，即使他们几乎没有或没有能力。没有背景该项目的核心是开发算法，该算法将用户演示作为输入——以用户操作的轨迹 A 的形式（例如，单击按钮、抓取数据）——并通过控制合成参数化程序 P。该方法基于有限树自动机，涉及底层网络自动化语言中的神经和符号元素，目标是能够有效地推理网页内容，同时仍然利用底层网页结构.该项目正在策划中一套新的网络自动化任务，对本项目之外的未来研究独立有用。研究人员正在与密歇根州的合作伙伴（例如，科学与工程女性 (WISE)、密歇根州路易斯斯托克斯少数族裔参与联盟 (MI-LSAMP) 合作）。和学术成功工程中心 (ECAS)），以增加来自代表性不足群体的学生对其研究的参与。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力优点和更广泛的影响审查进行评估，被认为值得支持标准。

项目成果

DiLogics: Creating Web Automation Programs with Diverse Logics

• DOI: 10.1145/3586183.3606822
• 发表时间: 2023-08
• 期刊: Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology
• 作者: Kevin Pu;Jim Yang;Angel Yuan;Minyi Ma;Rui Dong;Xinyu Wang;Yuanchun Chen;Tovi Grossman

Efficient Bottom-Up Synthesis for Programs with Local Variables

具有局部变量的程序的高效自下而上综合

• DOI: 10.1145/3632894
• 发表时间: 2024
• 期刊: Proceedings of the ACM on Programming Languages
• 作者: Li, Xiang;Zhou, Xiangyu;Dong, Rui;Zhang, Yihong;Wang, Xinyu
• 通讯作者: Wang, Xinyu

MIWA: Mixed-Initiative Web Automation for Better User Control and Confidence

MIWA：混合主动 Web 自动化可实现更好的用户控制和信心

• DOI: 10.1145/3586183.3606720
• 发表时间: 2023
• 期刊: ACM
• 作者: Chen, Weihao;Liu, Xiaoyu;Zhang, Jiacheng;Lam, Ian Iong;Huang, Zhicheng;Dong, Rui;Wang, Xinyu;Zhang, Tianyi
• 通讯作者: Zhang, Tianyi