CAREER: Learning Structured Models with Natural Language Supervision

职业：利用自然语言监督学习结构化模型

• 批准号: 2238240
• 负责人: Jacob Andreas
• 金额: $ 60万
• 依托单位: Massachusetts Institute of Technology
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-07-01 至 2028-06-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2238240&HistoricalAwards=false
• 关键词: CAREER; Learning; Structured; Models; Natural

Current machine learning models struggle to understand visual scenes, perform household chores, and complete other tasks that require integrating low-level perception and action with high-level common-sense and background knowledge. This CAREER project will use language to bridge this gap by developing techniques that use language-based dataset annotations and large text corpora to guide training of machine learning models for robotics, computer vision, and other problem domains. New approaches for learning with natural language supervision will reduce the amount of data needed to train machine learning models and enable end users to shape model behavior without complex formal specifications. The project will provide research training to undergraduate and graduate students, and will be integrated into a new workshop series that connects academic language processing researchers and researchers in other application areas (with a focus on providing learning and community-building opportunities for students from historically marginalized groups). The educational component of the project will develop new curriculum materials on natural language processing and human factors in artificial intelligence systems, targeting high school and undergraduate students as well as non-technical industry groups (like journalists and policy researchers) studying the effects of automated decision-making systems.The technical core of this project is a new family of probabilistic latent variable models in which latent representations of plans or percepts jointly generate task data and natural language annotations. When language annotations are available, they can directly supervise the content of these latent representations; on unannotated examples, information from text corpora may be used to constrain latent representations' distribution. Language thus plays two roles: as a source of information about the structure of individual training examples and a source of general, task-level background knowledge. Research will yield concrete instantiations of this modeling framework for policy learning, language modeling, and scene understanding, using language to produce structured, composable models that combine the flexibility of the deep learning toolkit with the sample efficiency and controllability of symbolic representations, while requiring neither massive labeled datasets nor precisely formalized symbolic domains.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

当前的机器学习模型难以理解视觉场景，执行家务和完成其他任务，这些任务需要将低级感知和动作与高级常识和背景知识相结合。该职业项目将使用语言来通过开发使用基于语言的数据集注释和大型文本语料库来指导机器人技术，计算机视觉和其他问题域的机器学习模型的培训来弥合这一差距。通过自然语言监督进行学习的新方法将减少训练机器学习模型所需的数据量，并使最终用户无需复杂的形式规格就可以塑造模型行为。该项目将为本科生和研究生提供研究培训，并将整合到一个新的研讨会系列中，该系列连接其他应用领域的学术语言处理研究人员和研究人员（重点是为历史上边缘化群体的学生提供学习和社区建设机会）。 The educational component of the project will develop new curriculum materials on natural language processing and human factors in artificial intelligence systems, targeting high school and undergraduate students as well as non-technical industry groups (like journalists and policy researchers) studying the effects of automated decision-making systems.The technical core of this project is a new family of probabilistic latent variable models in which latent representations of plans or percepts jointly generate task data and natural language annotations.当有语言注释可用时，他们可以直接监督这些潜在表示的内容；在未经通知的示例中，可以使用来自文本语料库的信息来限制潜在表示的分布。因此，语言扮演两个角色：作为有关单个培训示例结构的信息来源，以及一般任务级背景知识的来源。研究将产生该建模框架的具体实例，用于政策学习，语言建模和场景理解，使用语言产生结构化的，可合并的模型，将深度学习工具包的灵活性与示例效率和符号表示的柔韧性相结合，同时既不需要大规模的标签，也不需要通过对符号域进行宣传。基金会的智力优点和更广泛的影响审查标准。