Multi-agent Self-improving of Large Language Models (LLMs)

大型语言模型 (LLM) 的多智能体自我改进

• 批准号: 2903811
• 金额: --
• 依托单位: University of Edinburgh
• 依托单位国家: 英国
• 项目类别: Studentship
• 财政年份: 2024
• 资助国家: 英国
• 起止时间: 2024 至 无数据
• 项目状态: 未结题
• 来源: https://gtr.ukri.org/projects?ref=studentship-2903811
• 关键词: Multi; agent; Self; improving; Large

In the rapidly evolving field of artificial intelligence (AI), Large Language Models (LLMs) stand out as powerful tools capable of understanding human instructions and generating helpful answers. However, the development of these models faces significant challenges. In general, improving LLMs' generation ability and aligning their generation with human values rely heavily on vast amounts of human feedback annotations. This approach, while effective, is difficult to scale and may inherently limit the models' potential. As an alternative, some researchers turn to train LLMs using self-generated data, i.e., self-learning. Self-learning also presents a set of problems, including the risk of reinforcing existing biases or inaccuracies without external correction. This dilemma sets the stage for a novel approach to advancing LLM capabilities without substantial demand for human resources or the pitfalls of self-learning. This project tries to propose an innovative self-improving framework through a multi-agent system that enables these models to learn and enhance themselves by leveraging feedback from other peer models. By integrating the strengths and diversity of various LLMs, the system is expected to refine its ability to follow instructions, align with human values, and perform across a broad spectrum of downstream tasks with minimal human supervision. The vision is to establish a scalable and efficient method for continuous improvement through inter-model interactions, sidestepping the constraints of human feedback and the limitations of self-generated data training. At the heart of this self-improving system are two pivotal questions: 1. Can the diversity of LLMs enrich the quality of self-generated training data? 2. Can collaboration among different LLMs reduce the necessity for human annotations while ensuring ongoing enhancement? Addressing these two open queries could open the door to a new paradigm in AI training/alignment methodologies. This exploration aims at fostering more efficient AI systems development with reduced reliance on human oversight and intervention. This project, therefore, is also an open-ended exploration into future AI training strategies. It seeks to contribute to the AI community by moving away from heavily human-supervision-dependent models to more data-efficient and self-improving LLM systems.

在快速发展的人工智能 (AI) 领域，大型语言模型 (LLM) 作为能够理解人类指令并生成有用答案的强大工具而脱颖而出。然而，这些模型的开发面临着重大挑战。一般来说，提高法学硕士的生成能力并使他们的生成与人类价值观保持一致在很大程度上依赖于大量的人类反馈注释。这种方法虽然有效，但难以扩展，并且可能固有地限制模型的潜力。作为替代方案，一些研究人员转而使用自我生成的数据（即自学习）来培训法学硕士。自学还带来了一系列问题，包括在没有外部纠正的情况下强化现有偏见或不准确性的风险。这种困境为一种提高法学硕士能力的新方法奠定了基础，而无需大量的人力资源需求或自学的陷阱。该项目试图通过多代理系统提出一个创新的自我改进框架，使这些模型能够通过利用其他同行模型的反馈来学习和增强自身。通过整合各种法学硕士的优势和多样性，该系统预计将提高其遵循指令、符合人类价值观以及在最少的人工监督下执行广泛的下游任务的能力。我们的愿景是建立一种可扩展且高效的方法，通过模型间交互进行持续改进，避开人类反馈的约束和自我生成数据训练的局限性。这个自我改进系统的核心是两个关键问题： 1. 法学硕士的多样性能否丰富自我生成的培训数据的质量？ 2. 不同法学硕士之间的合作能否减少人工注释的必要性，同时确保持续增强？解决这两个开放性问题可以为人工智能训练/对齐方法的新范式打开大门。这一探索旨在促进更高效的人工智能系统开发，减少对人类监督和干预的依赖。因此，这个项目也是对未来人工智能训练策略的开放式探索。它寻求通过从严重依赖人类监督的模型转向数据效率更高和自我改进的法学硕士系统来为人工智能社区做出贡献。