Collaborative Research: SI2-SSI: Expanding Volunteer Computing

合作研究：SI2-SSI：扩展志愿者计算

• 批准号: 1664084
• 负责人: Michael Zentner
• 金额: $ 50万
• 依托单位: Purdue University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2017
• 资助国家: 美国
• 起止时间: 2017-05-15 至 2019-11-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1664084&HistoricalAwards=false
• 关键词: Collaborative; Research; SI2; SSI; Expanding

Volunteer computing (VC) uses donated computing time consumer devices such as home computers and smartphones to do scientific computing. It has been shown that VC can provide greater computing power, at lower cost, than conventional approaches such as organizational computing centers and commercial clouds. BOINC is the most common software framework for VC. Essentially, donors of computing time simply have to load BOINC on their computer or smartphone, and then register to donate at the BOINC web site. VC provides "high throughput computing": handling lots of independent jobs, with performance goals based on the rate of job completion rather than completion time for individual jobs. This type of computing (all known as high-throughput computing) is in great demand in most areas of science. Until now, the adoption of VC has been limited by its structure. For example, VC projects (such as Einstein@home and Rosetta@home) are operated by individual research groups, and volunteers must browse and choose from among many such projects. As a result, there are relatively few VC projects, and volunteers are mostly tech-savvy computer enthusiasts. This project aims to solve these problems using two complementary development efforts: First, it will add BOINC-based VC conduits to two major high-performance computing providers: (a) the Texas Advanced Computing Center, a supercomputer center, and (b) nanoHUB, a web portal for nano science that provides computing capabilities.Also, a unified control interface to VC will be developed, tentatively called Science United, where donors can register. The project will benefit thousands of scientists who use these facilities, and it will create technology that makes it easy for other HPC providers to add their own VC back ends. Also, Science United will provide a simpler interface to BOINC volunteers where they will register to support scientific areas, rather than specific projects. Science United will also serve as an allocator of computing power among projects. Thus, new projects will no longer have to do their own marketing and publicity to recruit volunteers. Finally, the creation of a single VC "brand" (i.e Science United) will allow coherent marketing of VC to the public. By creating a huge pool of low-cost computing power that will benefit thousands of scientists, and increasing public awareness of and interest in science, the project plans to establish VC as a central and long-term part of the U.S. scientific cyber infrastructure.Adding VC to an existing HPC facility involves several technical issues, which will be addressed as follows: (1) Packaging science applications (which typically run on Linux cluster nodes) to run on home computers (mostly Windows, some Mac and Linux): the team is developing an approach using VirtualBox and Docker, in which the application and its environment (Linux distribution, libraries, executables) are represented as a set of layers comprising a Docker image, which is then run as a container within a Linux virtual machine on the volunteer device. This has numerous advantages: it reduces the work of packaging applications to near zero; it minimizes network traffic because a given Docker layer is downloaded to a host only once; and it provides a strong security sandbox so that volunteer computers are protected from buggy or malicious applications, (2) File management: Input and output files must be moved between existing private servers and public-facing servers that are accessible to the outside Internet. A file management system will be developed, based on Web RPCs, for this purpose. This system will use content-based naming so that a given file is transferred and stored only once. It also maintains job/file associations so that files can be automatically deleted from the public server when they are no longer needed. (3) Submitting and monitoring jobs: BOINC provides a web interface for efficiently submitting and monitoring large batches of jobs. These were originally developed as part of a system to migrate HTCondor jobs to BOINC. This project is extending it to support the additional requirements of TACC and nanoHUB. Note that these new capabilities are not specific to TACC or nanoHUB: they provide the glue needed to easily add BOINC-based VC to any existing HTC facility. The team is also developing RPC bindings in several languages (Python, C++, PHP). The other component of the project, Science United, is a database-driven web site and an associated web service for the BOINC clients. Science United will control volunteer hosts (i.e. tell them which projects to work for) using BOINC's "Account Manager" mechanism, in which the BOINC client on each host periodically contacts Science United and is told what projects to run. Project servers, not Science United, will distribute jobs and files. Science United will define a set of "keywords" for science areas (physics, biomedicine, environment, etc.) and for location (country, institution). Projects will be labelled with appropriate keywords. Volunteers will have a yes/no/maybe interface for specifying the types of jobs they want to run. Science United will thus provide a mechanism in which a fraction of total computing capacity can be allocated to a project for a given period. Because total capacity changes slowly over time, this allows near-certain guaranteed allocations. Science United will embody a scheduling system that attempts to enforce allocations, honor volunteer preferences, and maximize throughput. Finally, Science United will do detailed accounting of computing. Volunteer hosts will tell Science United how much work (measured by CPU time and FLOPs, GPU time and FLOPs, and number of jobs) they have done for each project. Science United will maintain historical records of this data for volunteers and projects, and current totals with finer granularity (e.g. for each host/project combination). Finally, Science United will provide web interfaces letting volunteers see their contribution status and history, and letting administrators add projects, control allocations, and view accounting data.

志愿者计算（VC）使用捐赠的计算时间消费设备（例如家庭计算机和智能手机）来进行科学计算。已经表明，与组织计算中心和商业云等传统方法相比，VC可以以较低的成本提供更大的计算能力。 BOINC是VC最常见的软件框架。本质上，计算时间的捐助者只需在其计算机或智能手机上加载BOINC，然后在Boinc网站上注册以捐赠。 VC提供“高吞吐量计算”：处理许多独立的工作，其绩效目标基于工作完成率，而不是单个工作的完成时间。在大多数科学领域，这种类型的计算（全部称为高通量计算）的需求非常大。到目前为止，VC的采用受到其结构的限制。例如，VC项目（例如Einstein@Home和Rosetta@Home）由个别研究小组运营，志愿者必须浏览并从许多此类项目中进行选择。结果，VC项目相对较少，志愿者主要是精通技术的计算机爱好者。 该项目旨在通过两项互补的开发工作解决这些问题：首先，它将基于BOINC的VC导管添加到两个主要的高性能计算提供商中：（a）德克萨斯州高级计算中心，超级计算机中心，（b）NanoHub（b）Nano的网站，Nano Science的网站，提供了计算能力。 该项目将使数千名使用这些设施的科学家受益，并将创建技术，使其他HPC提供商可以轻松添加自己的VC后端。此外，科学联合会将为BOINC志愿者提供更简单的界面，在那里他们将注册以支持科学领域，而不是特定的项目。科学联合会还将作为项目之间计算能力的分配。因此，新项目将不再需要进行自己的营销和宣传来招募志愿者。最后，创建单个VC“品牌”（即科学联合会）将使VC连贯地向公众开展营销。 By creating a huge pool of low-cost computing power that will benefit thousands of scientists, and increasing public awareness of and interest in science, the project plans to establish VC as a central and long-term part of the U.S. scientific cyber infrastructure.Adding VC to an existing HPC facility involves several technical issues, which will be addressed as follows: (1) Packaging science applications (which typically run on Linux cluster nodes) to run on home computers （主要是Windows，一些Mac和Linux）：该团队正在使用VirtualBox和Docker开发一种方法，其中应用程序及其环境（Linux Distribution，Libraries，opecutables）表示为包含Docker映像的一组图层，然后在Volununteer设备上以Linux Virtual Machine在Linux虚拟机中运行。这具有许多优势：它将包装应用程序的工作减少到零；它可以最大程度地减少网络流量，因为给定的Docker层仅下载到主机一次。它提供了一个强大的安全沙箱，以便保护志愿者计算机免受货物或恶意应用程序的保护，（2）文件管理：必须在现有的私人服务器和面向公共互联网的公共服务器之间移动输入和输出文件。为此，将基于Web RPC开发文件管理系统。该系统将使用基于内容的命名，以便将给定文件传输并仅存储一次。它还维护作业/文件关联，以便在不再需要公共服务器时可以自动删除文件。 （3）提交和监视工作：BOINC提供了一个Web界面，用于有效地提交和监视大量工作。这些最初是作为将HTCONDOR职位迁移到Boinc的系统的一部分而开发的。该项目正在扩展其以支持TACC和NanoHub的其他要求。请注意，这些新功能并非特定于TACC或纳米HUB：它们提供了轻松将基于BOINC的VC添加到任何现有HTC设施所需的胶水。该团队还以多种语言（Python，C ++，PHP）开发RPC绑定。该项目的另一个组成部分是科学联合会（Science United），是数据库驱动的网站和BOINC客户端的关联的Web服务。科学联合会将使用Boinc的“客户经理”机制来控制志愿者主持人（即告诉他们要工作的项目），在该机制中，每个主机上的BOINC客户端都会定期与Science United联系，并被告知要运行的项目。项目服务器而不是科学联合会将分发工作和文件。科学联合会将为科学领域（物理，生物医学，环境等）和位置（国家，机构）定义一组“关键字”。项目将用适当的关键字标记。志愿者将有一个是/否/也许接口，用于指定他们想要运行的作业类型。因此，科学联合会将提供一种机制，其中可以将总计算能力的一部分分配给给定时期。由于总容量随着时间的流逝而缓慢变化，因此允许几乎确定的保证分配。科学联合会将体现一个计划系统，该系统试图执行分配，荣誉志愿者偏好和最大化吞吐量。最后，科学联合会将对计算进行详细的核算。志愿者主持人将告诉科学曼联，他们为每个项目所做的工作量（通过CPU时间和Flops，GPU时间和失败以及工作数量来衡量）。科学联合会将为志愿者和项目保留此数据的历史记录，以及粒度较好的当前总数（例如，对于每个主机/项目组合）。最后，科学联合会将提供网络接口，让志愿者看到他们的贡献状态和历史，并让管理员添加项目，控制分配和查看会计数据。

项目成果

nanoHUB@home: Expanding nanoHUB through Volunteer Computing

nanoHUB@home：通过志愿者计算扩展 nanoHUB

• 发表时间: 2019
• 期刊: Gateways 2019
• 作者: Haley, Benjamin;Clark, Steven;Denny, Nathan;Desai, Saaketh;Hunt, Marton;Anderson, David
• 通讯作者: Anderson, David