A Lebesgue Integral based Approximation for Language Modelling

基于勒贝格积分的语言建模近似

基本信息

批准号：
EP/X019063/1
负责人：
Lin Gui
金额：
$ 25.77万
依托单位：
King's College London
依托单位国家：
英国
项目类别：
Research Grant
财政年份：
2023
资助国家：
英国
起止时间：
2023 至无数据
项目状态：
未结题

来源：
https://gtr.ukri.org/projects?ref=EP%2FX019063%2F1
关键词：
Lebesgue Integral based Approximation Language

项目摘要

Deep learning (DL) based Natural Language Processing (NLP) technologies have attracted significant interest in recent years. The current SOTA language models, a.k.a. transformer-based language models, typically assume that the representation of a given word can be captured by the interpolation of its related context in a convex hull. However, it has recently been shown that in high-dimensional spaces, the interpolation almost surely never occurs regardless of the underlying intrinsic dimension of the data manifold. The representations generated by such transformer-based language models will converge into a dense cone-like hyperspace which is often discontinuous with many nonadjacent clusters. To overcome the limitation of current methods in most DL-based NLP models, this project aims to deploy Lebesgue integral, which can be defined as an ensemble of integrals among partitions (i.e., discontinuous feature clusters), to approximate the posterior distributions of clusters given input word features in finite measurable sets by automatically identifying the boundary of such discontinuous set, which in turn could help to generate better interpretations and quantify the uncertainty. By our proposed Lebesgue integral based approximation, the input text will be characterised by two properties: an indicator vector encoding its membership in clusters (i.e., measurable sets), and another continuous feature representation for better capturing its semantic meaning for downstream tasks. This not only allows for a more faithful approximation of commonly observed countably discontinuities in distributions of input text in NLP, but also enables learning text representations that are better understood by humans.

近年来，基于深度学习（DL）的自然语言处理（NLP）技术引起了极大的兴趣。当前的SOTA语言模型，又称基于变压器的语言模型，通常假定给定单词的表示可以通过在凸船上的相关上下文插值来捕获。但是，最近已经显示，在高维空间中，无论数据歧管的潜在固有维度如何，插值几乎绝对不会发生。这种基于变压器的语言模型产生的表示形式将收敛成一个浓密的锥形超空间，而这通常与许多非贴心簇不连续。为了克服大多数基于DL的NLP模型中当前方法的局限性，该项目旨在部署Lebesgue积分，可以将其定义为分区之间的积分集合（即不连续的特征簇），以近似于在有限的测量中，可以自动地识别出closeation nourdeation nourdeation nourdeation nourcyemation nourdeation nourdeation nourdeation nourdeation nourdeation，以近似为差异，以识别clusters的后部分布。并量化不确定性。通过我们提出的基于Lebesgue积分的近似值，输入文本将以两种属性为特征：一个指标向量编码其在簇中的成员资格（即可测量的集合），以及另一种连续的特征表示，以更好地捕获其在下游任务的语义含义。这不仅允许在NLP中输入文本的分布中更加忠实地逼近通常观察到的不连续性，而且还可以使学习文本表示形式更好地理解了人类。