ITR: Robust Large-Scale Distributed Systems

ITR：稳健的大型分布式系统

• 批准号: 0225660
• 负责人: Marinus Kaashoek
• 金额: --
• 依托单位: Massachusetts Institute of Technology
• 依托单位国家: 美国
• 项目类别: Cooperative Agreement
• 财政年份: 2002
• 资助国家: 美国
• 起止时间: 2002-10-01 至 2009-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0225660&HistoricalAwards=false
• 关键词: ITR; Robust; Large; Scale; Distributed

This project is a novel decentralized infrastructure, based on distributed hash tables (DHTs), that will enable a new generation of large-scale distributed applications. The key technology on which we build, DHTs, are robust in the face of failures, attacks and unexpectedly high loads. They are scalable, achieving large system sizes without incurring undue overhead. They are self-configuring, automatically incorporating new nodes without manual intervention or oversight. They simplify distributed programming by providing a clean and flexible interface. And, finally, they provide a shared infrastructure simultaneously usable by many applications.The approach advocated here is a radical departure from both the centralized client-server and the application-specific overlay models of distributed applications. This new approach will not only change the way large-scale distributed systems are built, but could potentially have far reaching societal implications as well. The main challenge in building any distributed system lies in dealing with security, robustness, management, and scaling issues; today each new system must solve these problems for itself, requiring significant hardware investment and sophisticated software design. The shared distributed infrastructure will relieve individual applications of these burdens, thereby greatly reducing the barriers to entry for large-scale distributed services.Our belief that DHTs are the right enabling infrastructure is based on two conjectures: (1) a DHT with application-independent, unconstrained keys and values provides a general purpose interface upon which a wide variety of distributed applications can be built, and (2) distributed applications that make use of the DHT-based infrastructure inherit basic levels of security, robustness, ease of operation, and scaling. Much of the thrust of the proposed research is an exploration of these two conjectures.We will investigate the first conjecture, that the DHT abstraction can support a wide range of applications, by building a variety of DHT-based systems. Our recent work has used DHTs to support such varied applications as distributed file systems, multicast overlay networks, event notification systems, and distributed query processing. DHTs simplify the structure of these systems by providing general-purpose key/value naming rather than imposing structured keys (e.g., hierarchical names in DNS). These systems are early prototypes, but they suggest that DHTs may be as useful to distributed applications as ordinary hash tables are to programs.The second conjecture relies on techniques for creating robust, secure, and self-organizing infrastructures out of many mutually distrustful nodes. Our initial work on robust DHT designs gives us confidence that such techniques are within reach. The bulk of our proposed research will be devoted to the in-depth study of these techniques, with the express aim of producing a sound and coherent design for the infrastructure. To investigate the real-world behavior of our design, we will create a large-scale open testbed for which we will distribute our infrastructure software, some enabling libraries, and a few key compelling applications.In addition to its impact on the creation of distributed applications, our research program will have benefits in education and outreach. Given their current importance, security, robustness, and the design of distributed systems should become central topics in undergraduate computer science education. To this end, we are planning a new interdisciplinary course that will address these issues, and bring them into sharper focus early in the undergraduate course sequence.Our testbed and research agenda is also a good vehicle for encouraging the participation of organizations not traditionally involved in networking and systems research. Participation in the testbed requires little cost (a PC and an Internet connection) and minimal levels of systems expertise and over-sight. Moreover, because the material is closely related to the P2P systems with which many students are familiar, the project might appeal to students who would not normally be attracted to research in this area. Based on this premise, we plan an active outreach program to underrepresented populations at non-research undergraduate institutions.

该项目是一种新型的去中心化基础设施，基于分布式哈希表（DHT），它将支持新一代大规模分布式应用程序。 我们构建的关键技术 DHT 在面对故障、攻击和意外高负载时非常稳健。 它们具有可扩展性，可以实现大型系统规模，而不会产生过多的开销。 它们是自我配置的，自动合并新节点，无需人工干预或监督。它们通过提供干净且灵活的接口来简化分布式编程。 最后，它们提供了可供许多应用程序同时使用的共享基础设施。这里提倡的方法与集中式客户端-服务器和分布式应用程序的特定于应用程序的覆盖模型截然不同。 这种新方法不仅会改变大规模分布式系统的构建方式，而且还可能产生深远的社会影响。 构建任何分布式系统的主要挑战在于处理安全性、稳健性、管理和扩展问题；如今，每个新系统都必须自行解决这些问题，需要大量的硬件投资和复杂的软件设计。 共享的分布式基础设施将减轻单个应用程序的这些负担，从而大大降低大规模分布式服务的进入壁垒。我们认为DHT是正确的使能基础设施是基于两个猜想：（1）与应用程序无关的DHT ，不受约束的键和值提供了一个通用接口，可以在该接口上构建各种分布式应用程序，并且（2）利用基于 DHT 的基础设施的分布式应用程序继承了基本级别的安全性、稳健性、易于操作性和缩放。所提出的研究的大部分主旨是对这两个猜想的探索。我们将研究第一个猜想，即 DHT 抽象可以通过构建各种基于 DHT 的系统来支持广泛的应用。 我们最近的工作使用 DHT 来支持分布式文件系统、多播覆盖网络、事件通知系统和分布式查询处理等各种应用程序。 DHT 通过提供通用键/值命名而不是强加结构化键（例如 DNS 中的分层名称）来简化这些系统的结构。 这些系统是早期原型，但它们表明 DHT 对于分布式应用程序可能就像普通哈希表对于程序一样有用。第二个猜想依赖于从许多相互不信任的节点中创建健壮、安全和自组织基础设施的技术。 我们在稳健的 DHT 设计方面的初步工作让我们相信此类技术是可以实现的。我们提议的研究的大部分将致力于深入研究这些技术，其明确目标是为基础设施提供合理且连贯的设计。为了研究我们设计的真实行为，我们将创建一个大规模的开放测试台，我们将为其分发基础设施软件、一些支持库和一些关键的引人注目的应用程序。应用程序，我们的研究计划将在教育和推广方面受益。鉴于其当前的重要性，安全性、鲁棒性和分布式系统的设计应该成为本科计算机科学教育的中心主题。为此，我们正在计划一门新的跨学科课程来解决这些问题，并在本科课程序列的早期使它们更加引人注目。我们的测试平台和研究议程也是鼓励传统上不参与的组织参与的良好工具。网络和系统研究。 参与测试平台只需要很少的成本（一台电脑和一个互联网连接）以及最低水平的系统专业知识和监督。 此外，由于该材料与许多学生熟悉的 P2P 系统密切相关，因此该项目可能会吸引通常不会被该领域研究吸引的学生。 基于这个前提，我们计划针对非研究型本科机构中代表性不足的人群开展积极的外展计划。