A Long-term VIS-enabled Infrastructure for Supporting ML-assisted Human Decision-making

支持 ML 辅助人类决策的长期 VIS 基础设施

• 批准号: EP/X029557/1
• 负责人: Min Chen
• 金额: $ 74.45万
• 依托单位: University of Oxford
• 依托单位国家: 英国
• 项目类别: Research Grant
• 财政年份: 2023
• 资助国家: 英国
• 起止时间: 2023 至 无数据
• 项目状态: 未结题
• 来源: https://gtr.ukri.org/projects?ref=EP%2FX029557%2F1
• 关键词: Long; term; VIS; enabled; Infrastructure

Many large organisations maintains a large pool of trained human resources. When a new task arrives, the management constructs a team by selecting appropriate team members with different skills and arranges an effective operational structure for the team. In machine learning (ML), the model developers typically train many models for each individual task, then select the best model to perform the task, while discarding the unselected models. Considering that keeping a trained ML model costs much less than employing a person, there is a huge waste of model resources. The main reasons behind this wasteful practice include (i) the lack of effective means for apprehending the "skill profiles" a large number of ML models; (ii) the lack of effective means for constructing a "team" such that the combined skillset of the team is suitable for the task but each component model does not have all the skills required; and (iii) the lack of effective means for enabling human decision makers to utilise imperfect ML models as assistants or advisers. Because of these reasons, there is less incentive to maintain a large pool of trained ML models that may not be the best for a specific task individually, and the emphasis has been placed on training a "star" model as optimal as possible for each arrival task.The technology of visualization and visual analytics (VIS) can address the aforementioned three "lacks". In many data-intensive applications, VIS can enable decision-makers to observe a large amount of data quickly (e.g., stock market), analyse complex relationships among different data entities (e.g., social network analysis), and make complex judgement based on multiple and sometimes conflicting machine-predictions (e.g., by different epidemiological models). The latest theoretical advance offers an explanation as to what visualization offers users that statistics and algorithms cannot offer. Humans have limited cognitive bandwidth for receiving and reasoning about information. To reduce the amount of information received by humans, statistics and algorithms typically transform a large amount of data to a few variables (e.g., mean and standard deviation) at a higher precision, while visualization presents many more variables at a lower precision (e.g., a line plot of 500 data points in a time series). Because humans can perceive many variables visually at a very low cognitive cost, more cognitive bandwidth can be directed to data-informed reasoning. This explains why financial experts make decisions rarely based only on one or two financial indicators, but also need to observe time series data. Visual analytics is a branch of VIS focusing on combined uses of statistics, algorithms, visualization, and interaction in human decision workflows.In this project, we will develop a new technology to enable human decision makers to benefit from VIS capabilities in their workflows. We address the first aforementioned "lack" by designing and developing a novel VIS-enabled infrastructure where hundreds and thousands of ML models can be stored with their provenance, be tested and profiled automatically and routinely, and be managed as trained model resources by ML model-developers with the aid of VIS capabilities. We address the second "lack" by providing ML model-developers with a VIS-enabled tool for constructing ensemble models (i.e., teams of ML models) by selecting appropriate component models (i.e., team members) from a pool of model resources, and determine an appropriate ensemble strategy (i.e., team structure). Last but not least, we address the third "lack" by providing ML model users (i.e., decision makers who receive low-level predictions or recommendations from ML models) with the VIS capabilities, which allow them to observe quickly the anomalies and conflicts in the low-level predictions made by different models, and when it is helpful, to scrutinise the profile and provenance of these models.

许多大型组织拥有大量训练有素的人力资源。当新任务到达时，管理层通过选择具有不同技能的合适团队成员来构建团队，并为团队安排有效的运营结构。在机器学习（ML）中，模型开发人员通常会为每个任务训练许多模型，然后在丢弃未选择的模型的同时选择执行任务的最佳模型。考虑到保持训练有素的ML模型的成本比雇用一个人低得多，因此浪费了模型资源。这种浪费实践背后的主要原因包括（i）缺乏逮捕大量ML模型的“技能概况”的有效手段； （ii）缺乏构建“团队”的有效手段，因此团队的组合技能适合任务，但每个组件模型都不具有所需的所有技能； （iii）缺乏使人类决策者能够利用不完美的ML模型作为助理或顾问的有效手段。由于这些原因，维持大量受过训练的ML模型的动机较少，这可能不是针对特定任务的最佳选择，并且重点是训练“星”模型在每个到达任务中尽可能最佳的“星”模型。可视化和视觉分析技术（VIS）可以解决上述三个“缺乏”。在许多数据密集型应用程序中，VIS可以使决策者能够快速观察大量数据（例如股票市场），分析不同数据实体之间的复杂关系（例如，社交网络分析），并基于多个机器预测（例如，通过不同的流行病学模型）做出复杂的判断。最新的理论提前提供了一个可视化为用户提供统计和算法无法提供的用户的解释。人类接受和推理信息的认知带宽有限。为了减少人类收到的信息量，统计和算法通常以较高的精度将大量数据转换为几个变量（例如，平均值和标准偏差），而可视化则在较低的精度下呈现了更多的变量（例如，时间序列中500个数据点的一行图）。由于人类可以以非常低的认知成本在视觉上感知许多变量，因此可以将更多的认知带宽引导到数据知识的推理。这就解释了为什么财务专家很少仅基于一个或两个财务指标做出决定，但还需要观察时间序列数据。 Visual Analytics是VIS的一个分支，专注于人类决策工作流程中统计，算法，可视化和互动的联合用途。在该项目中，我们将开发一项新技术，以使人类决策者能够从工作流程中的VIS功能中受益。我们通过设计和开发一种新颖的Visaby基础架构来解决上述第一个“缺乏”，其中可以用它们的出处来存储数百万种ML模型，自动测试和概述，并通过ML模型开发者的帮助，并通过ML模型开发者进行Vis Visibilitions的ML模型。我们通过从模型资源池中选择适当的组件模型（即团队成员），确定适当的组件模型（即团队成员），并确定适当的集合策略（即团队结构），从而解决了第二个“缺乏”工具，以构建合奏模型（即ML模型的团队）来构建合奏模型（即ML模型团队）。最后但并非最不重要的一点是，我们通过提供ML模型用户（即获得低级预测或ML模型的建议的决策者）来解决第三个“缺乏”，这使他们能够快速观察到不同模型做出的低级预测中的异常和冲突，以及在不同模型中进行的，以及何时有助于这些模型和这些型号的概述和外观，并将这些模型范围化。