Realizing competitive learning by information maximization and its application to language acquisition

信息最大化实现竞争性学习及其在语言习得中的应用

基本信息

批准号：
13680463
负责人：
KAMIMURA Ryotaro
金额：
$ 2.3万
依托单位：
TOKAI UNIVERSITY
依托单位国家：
日本
项目类别：
Grant-in-Aid for Scientific Research (C)
财政年份：
2001
资助国家：
日本
起止时间：
2001 至 2003
项目状态：
已结题

项目摘要

We propose a new information-theoretic approach to neural computing. In principle, we postulate that living systems manage to control information in a strategic way to keep their existence in extremely uncertain outer conditions. For example, living systems must maximize information content on outer environment to maintain their existence from destructive forces from outer environment. Applying this principle to neural networks, we have proposed a new information-theoretic competitive learning method that is completely different from conventional methods in that competition processes are realized by maximizing mutual information. Based upon this information-theoretic competitive learning, we have developed the following five different approaches to neural computing.(1) Cost-sensitive information maximization : in maximizing information, the associated cost must be minimized. This method can produce internal representations more faithful to input patterns.(2) Supervised information-theoretic competitive learning : unsupervised information-theoretic competitive learning is extended to supervised learning. In this method, we use multi-layered networks in which the output layer is added to the first competitive layer.(3) Teacher-directed learning: Teacher-directed learning is a new supervised learning method using information maximization in which no error back-propagation is used. Thus, the method is very efficient and biological sound method. In the previous method, we used the sigmoid activation functions with some problems in increasing information. To remedy this, we use the Gaiussian activation functions.(5) Unification of information maximization and minimization : We try to unify information maximization and minimization, because the shortcomings of both methods can be resolved in this framework. In this special session, we offer the present state of research of strategic information control.

我们提出了一种新的信息理论方法来进行神经计算。原则上，我们假设生活系统设法以战略方式控制信息，以使其存在在极不确定的外部条件下。例如，生活系统必须最大程度地提高外部环境的信息内容，以维持外部环境的破坏力。将此原理应用于神经网络，我们提出了一种新的信息理论竞争学习方法，该方法与传统方法完全不同，因为竞争过程是通过最大化互信息来实现的。基于此信息理论学习，我们为神经计算开发了以下五种不同的方法。（1）成本敏感的信息最大化：为了最大程度地提高信息，必须最大程度地减少相关成本。这种方法可以使内部表示形式更加忠实于输入模式。（2）监督信息理论竞争学习：无监督信息理论的竞争学习将扩展到监督学习。在这种方法中，我们使用多层网络，其中将输出层添加到第一个竞争层。（3）教师指导的学习：以教师为导向的学习是一种使用信息最大化的新的监督学习方法，在该方法中，无错误的错误 - 使用传播。因此，该方法是非常有效且生物学的方法。在上一个方法中，我们在增加信息时使用了Sigmoid激活函数以及一些问题。为了解决这个问题，我们使用Gaiussian激活功能。（5）信息最大化和最小化的统一：我们尝试统一信息最大化和最小化，因为这两种方法的缺点都可以在此框架中解决。在此特别会议中，我们提供了战略信息控制的当前研究状态。