Collaborative Research: SI2-SSI: Expanding Volunteer Computing

合作研究：SI2-SSI：扩展志愿者计算

基本信息

批准号：
2001752
负责人：
Michael Zentner
金额：
$ 17.25万
依托单位：
University of California-San Diego
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2019
资助国家：
美国
起止时间：
2019-09-01 至 2021-04-30
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=2001752&HistoricalAwards=false
关键词：
Collaborative Research SI2 SSI Expanding

项目摘要

Volunteer computing (VC) uses donated computing time consumer devices such as home computers and smartphones to do scientific computing. It has been shown that VC can provide greater computing power, at lower cost, than conventional approaches such as organizational computing centers and commercial clouds. BOINC is the most common software framework for VC. Essentially, donors of computing time simply have to load BOINC on their computer or smartphone, and then register to donate at the BOINC web site. VC provides "high throughput computing": handling lots of independent jobs, with performance goals based on the rate of job completion rather than completion time for individual jobs. This type of computing (all known as high-throughput computing) is in great demand in most areas of science. Until now, the adoption of VC has been limited by its structure. For example, VC projects (such as Einstein@home and Rosetta@home) are operated by individual research groups, and volunteers must browse and choose from among many such projects. As a result, there are relatively few VC projects, and volunteers are mostly tech-savvy computer enthusiasts. This project aims to solve these problems using two complementary development efforts: First, it will add BOINC-based VC conduits to two major high-performance computing providers: (a) the Texas Advanced Computing Center, a supercomputer center, and (b) nanoHUB, a web portal for nano science that provides computing capabilities.Also, a unified control interface to VC will be developed, tentatively called Science United, where donors can register. The project will benefit thousands of scientists who use these facilities, and it will create technology that makes it easy for other HPC providers to add their own VC back ends. Also, Science United will provide a simpler interface to BOINC volunteers where they will register to support scientific areas, rather than specific projects. Science United will also serve as an allocator of computing power among projects. Thus, new projects will no longer have to do their own marketing and publicity to recruit volunteers. Finally, the creation of a single VC "brand" (i.e Science United) will allow coherent marketing of VC to the public. By creating a huge pool of low-cost computing power that will benefit thousands of scientists, and increasing public awareness of and interest in science, the project plans to establish VC as a central and long-term part of the U.S. scientific cyber infrastructure.Adding VC to an existing HPC facility involves several technical issues, which will be addressed as follows: (1) Packaging science applications (which typically run on Linux cluster nodes) to run on home computers (mostly Windows, some Mac and Linux): the team is developing an approach using VirtualBox and Docker, in which the application and its environment (Linux distribution, libraries, executables) are represented as a set of layers comprising a Docker image, which is then run as a container within a Linux virtual machine on the volunteer device. This has numerous advantages: it reduces the work of packaging applications to near zero; it minimizes network traffic because a given Docker layer is downloaded to a host only once; and it provides a strong security sandbox so that volunteer computers are protected from buggy or malicious applications, (2) File management: Input and output files must be moved between existing private servers and public-facing servers that are accessible to the outside Internet. A file management system will be developed, based on Web RPCs, for this purpose. This system will use content-based naming so that a given file is transferred and stored only once. It also maintains job/file associations so that files can be automatically deleted from the public server when they are no longer needed. (3) Submitting and monitoring jobs: BOINC provides a web interface for efficiently submitting and monitoring large batches of jobs. These were originally developed as part of a system to migrate HTCondor jobs to BOINC. This project is extending it to support the additional requirements of TACC and nanoHUB. Note that these new capabilities are not specific to TACC or nanoHUB: they provide the glue needed to easily add BOINC-based VC to any existing HTC facility. The team is also developing RPC bindings in several languages (Python, C++, PHP). The other component of the project, Science United, is a database-driven web site and an associated web service for the BOINC clients. Science United will control volunteer hosts (i.e. tell them which projects to work for) using BOINC's "Account Manager" mechanism, in which the BOINC client on each host periodically contacts Science United and is told what projects to run. Project servers, not Science United, will distribute jobs and files. Science United will define a set of "keywords" for science areas (physics, biomedicine, environment, etc.) and for location (country, institution). Projects will be labelled with appropriate keywords. Volunteers will have a yes/no/maybe interface for specifying the types of jobs they want to run. Science United will thus provide a mechanism in which a fraction of total computing capacity can be allocated to a project for a given period. Because total capacity changes slowly over time, this allows near-certain guaranteed allocations. Science United will embody a scheduling system that attempts to enforce allocations, honor volunteer preferences, and maximize throughput. Finally, Science United will do detailed accounting of computing. Volunteer hosts will tell Science United how much work (measured by CPU time and FLOPs, GPU time and FLOPs, and number of jobs) they have done for each project. Science United will maintain historical records of this data for volunteers and projects, and current totals with finer granularity (e.g. for each host/project combination). Finally, Science United will provide web interfaces letting volunteers see their contribution status and history, and letting administrators add projects, control allocations, and view accounting data.

志愿者计算 (VC) 使用捐赠的计算时间消费设备（例如家用计算机和智能手机）进行科学计算。事实证明，与组织计算中心和商业云等传统方法相比，VC 可以以更低的成本提供更强的计算能力。 BOINC是VC最常用的软件框架。本质上，计算时间的捐赠者只需在他们的计算机或智能手机上加载 BOINC，然后在 BOINC 网站上注册即可捐赠。 VC 提供“高吞吐量计算”：处理大量独立作业，其性能目标基于作业完成率而不是单个作业的完成时间。这种类型的计算（统称为高通量计算）在大多数科学领域都有很大的需求。到目前为止，VC 的采用一直受到其结构的限制。例如，风险投资项目（例如Einstein@home和Rosetta@home）由各个研究小组运营，志愿者必须从许多此类项目中浏览和选择。因此，风险投资项目相对较少，志愿者大多是精通技术的计算机爱好者。该项目旨在通过两项互补的开发工作来解决这些问题：首先，它将向两个主要的高性能计算提供商添加基于 BOINC 的 VC 管道：(a) 德克萨斯高级计算中心，一个超级计算机中心，以及 (b) nanoHUB ，一个提供计算能力的纳米科学门户网站。此外，还将开发一个统一的 VC 控制界面，暂称为 Science United，捐赠者可以在那里注册。该项目将使使用这些设施的数千名科学家受益，并将创造技术，使其他 HPC 提供商可以轻松添加自己的 VC 后端。此外，Science United 将为 BOINC 志愿者提供一个更简单的界面，他们可以在其中注册以支持科学领域，而不是特定项目。 Science United 还将充当项目之间计算能力的分配者。这样，新项目将不再需要自己做营销和宣传来招募志愿者。最后，创建一个单一的风险投资“品牌”（即科学联合）将使风险投资向公众进行连贯的营销。通过创建一个巨大的低成本计算能力库，让成千上万的科学家受益，并提高公众对科学的认识和兴趣，该项目计划将 VC 建立为美国科学网络基础设施的核心和长期组成部分。对现有 HPC 设施的 VC 涉及几个技术问题，将按如下方式解决： (1) 打包科学应用程序（通常在 Linux 集群节点上运行）以在家用计算机（主要是 Windows，一些 Mac 和 Linux）上运行：团队正在开发一种使用 VirtualBox 的方法Docker，其中应用程序及其环境（Linux 发行版、库、可执行文件）表示为一组包含 Docker 映像的层，然后该映像作为容器在志愿者设备上的 Linux 虚拟机中运行。这有很多优点：它将打包应用程序的工作量减少到接近于零；它最大限度地减少了网络流量，因为给定的 Docker 层仅下载到主机一次；它提供了强大的安全沙箱，以便保护志愿者计算机免受错误或恶意应用程序的侵害。 (2) 文件管理：输入和输出文件必须在现有专用服务器和可通过外部 Internet 访问的面向公众的服务器之间移动。为此，将开发一个基于 Web RPC 的文件管理系统。该系统将使用基于内容的命名，以便给定文件仅传输和存储一次。它还维护作业/文件关联，以便当不再需要文件时可以自动从公共服务器中删除文件。 (3) 提交和监控作业：BOINC 提供了一个 Web 界面，用于高效地提交和监控大批量作业。这些最初是作为将 HTCondor 作业迁移到 BOINC 的系统的一部分而开发的。该项目正在扩展它以支持 TACC 和 nanoHUB 的额外要求。请注意，这些新功能并非 TACC 或 nanoHUB 特有：它们提供了将基于 BOINC 的 VC 轻松添加到任何现有 HTC 设施所需的粘合剂。该团队还在开发多种语言（Python、C++、PHP）的 RPC 绑定。该项目的另一个组成部分 Science United 是一个数据库驱动的网站以及为 BOINC 客户提供的相关 Web 服务。 Science United 将使用 BOINC 的“帐户经理”机制来控制志愿者主机（即告诉他们要为哪些项目工作），其中每个主机上的 BOINC 客户端会定期联系 Science United 并被告知要运行哪些项目。项目服务器（而不是 Science United）将分发作业和文件。 Science United 将为科学领域（物理学、生物医学、环境等）和地点（国家、机构）定义一组“关键词”。项目将使用适当的关键字进行标记。志愿者将有一个是/否/也许的界面来指定他们想要运行的工作类型。因此，科学联合公司将提供一种机制，可以将总计算能力的一小部分分配给给定时期的项目。由于总容量随时间变化缓慢，因此可以实现近乎确定的有保证的分配。 Science United 将实施一个调度系统，试图强制分配、尊重志愿者的偏好并最大化吞吐量。最后，Science United将对计算进行详细的核算。志愿者主持人将告诉 Science United 他们为每个项目完成了多少工作（以 CPU 时间和 FLOPs、GPU 时间和 FLOPs 以及作业数量来衡量）。 Science United 将维护志愿者和项目的这些数据的历史记录，以及更细粒度的当前总计（例如每个主持人/项目组合）。最后，Science United 将提供网络界面，让志愿者查看他们的贡献状态和历史记录，并让管理员添加项目、控制分配和查看会计数据。