NeTS-NBD-SGER: Map/Reduce for Network Traffic Analysis (MR-Net Sger)

NeTS-NBD-SGER：用于网络流量分析的 Map/Reduce (MR-Net Sger)

• 批准号: 0823774
• 负责人: John Heidemann
• 金额: $ 10.02万
• 依托单位: University of Southern California
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2008
• 资助国家: 美国
• 起止时间: 2008-04-01 至 2010-03-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0823774&HistoricalAwards=false
• 关键词: NeTS; NBD; SGER; Map; Reduce

0823774This project explores a new class of parallel algorithms that process very large network datasets. The goal is to be able to analyze 2.7 billion pings to map the Internet address space, process six months of flow records to understand long-term traffic trends, and search a week?s worth of packet headers to retroactively detect zero-day compromised machines. Each of these tasks requires efficient and economical processing of datasets in sizes from 50GB to several terabytes. This leap in dataset sizes by a factor of 100-1000-fold or more requires fundamentally different ways of handling network data than today?s tcpdump and ethereal on a workstation. Two recent developments make this leap possible. First, Google has demonstrated that the map/reduce abstraction be easily parallelized and run efficiently and cost-effectively over clusters of hundreds of commodity PCs. It is the basis of their web search engine and has prompted at least one open source implementation. Map/reduce is the computation engine of Google?s web search engine, increasingly being used in other applications. Map/reduce is the key to processing huge network datasets. Second, recent programs such as PREDICT make massive network datasets available. PREDICT promises to make available packet header traces, address space scans, netflow records, dark address space traffic, and voice over IP (VoIP) call records from several large ISPs. At USC researchers are collecting packet header traces and address space scans. PREDICT is the key to obtaining huge network datasets. This work is proposes as a SGER because it is both timely and, at this point, highly speculative. The researchers must characterize what network problems can be solved by map/reduce before a full proposal will be credible. This proposal is the key to demonstrating the potential impact of for map/reduce processing of huge network datasets to change our understanding of the Internet.The intellectual merit of this work is to develop a preliminary understanding of how to use map/reduce style processing for network datasets. What algorithms are applicable? What problems parallelize well or poorly? What kind of compute clusters are needed? And more generally, how can networking researchers cope with gigabyte-to-terabyte datasets that are needed to describe a billion-user Internet?The broader merit of this work is that it will lead to answers of both fundamental and practical questions facing the Internet. Considering these datasets, questions include: What does the Internet look like? At what rate is the Internet address space being consumed? How many Internet users connect with dynamic addresses? How can one respond to intrusions effectively? Can new techniques detect low-rate denial-of-service attacks, spam generation in compromised servers? Can one traceback and detect botnets control networks? More important than these specific questions is the broader question of how can one understand properties in a billion-node Internet and the petabyte-per day of traffic that flows over it.

0823774这个项目探讨了一类新的并行算法处理非常大的网络数据集。目的是能够分析27亿个ping来绘制互联网地址空间，处理六个月的流记录以了解长期的交通趋势，并搜索一周价值的数据包标头，以追溯检测零日的折磨机器。这些任务中的每一个都需要从50GB到几个tobytes的大小的数据集进行有效，经济的处理。该数据集大小的飞跃高达100-1000倍或更多，需要与今天的工作站上的TCPDUMP和空灵处理网络数据的根本不同的方法。最近的两个发展使这一飞跃成为可能。首先，Google证明了地图/减少抽象很容易平行，并在数百个商品PC的群集上有效地和成本效益。它是他们的Web搜索引擎的基础，并提示了至少一个开源实现。 MAP/RELAD是Google的Web搜索引擎的计算引擎，越来越多地用于其他应用程序中。地图/减少是处理庞大网络数据集的关键。其次，最近的程序（例如预测）使大规模网络数据集可用。预测有望使数据包标题轨迹，地址空间扫描，NetFlow记录，暗地址空间流量以及IP（VoIP）的语音（VOIP）呼叫记录来自多个大型ISP。在南加州大学，研究人员正在收集数据包标题轨迹并解决空间扫描。预测是获得庞大网络数据集的关键。这项工作是作为SGER提出的，因为它既是及时的，又是高度投机。研究人员必须表征可以通过MAP/减少解决哪些网络问题，然后再提出完整的建议是可信的。该提案是展示庞大网络数据集的地图/处理处理以改变我们对互联网的理解的潜在影响的关键。这项工作的智力优点是对如何使用MAP/减少网络数据集的MAP/减少样式处理的初步了解。什么算法适用？哪些问题可以很好地平行或不良？需要什么样的计算簇？更一般而言，如何网络研究人员如何应对描述十亿用户互联网所需的千兆字节到特比特的数据集？这项工作的更广泛的优点是，它将导致互联网面临的基本问题和实用问题的答案。考虑到这些数据集，问题包括：互联网是什么样的？互联网地址空间以什么速度消耗？有多少互联网用户与动态地址联系在一起？一个人如何有效地应对入侵？新技术是否可以检测到低速拒绝服务攻击，垃圾邮件在受损的服务器中产生？一个追溯和检测僵尸网络控制网络吗？比这些特定问题更重要的是，一个更广泛的问题是，如何了解十亿节点的互联网和每天流量流过的问题。