Execution Modeling and Analytics for Large-scale and Data-intensive Software Applications

大规模数据密集型软件应用程序的执行建模和分析

• 批准号: RGPIN-2018-06312
• 负责人: Shafiq, MuhammadOmair
• 金额: $ 1.68万
• 依托单位: Carleton University
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2019
• 资助国家: 加拿大
• 起止时间: 2019-01-01 至 2020-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=679242
• 关键词: Execution; Modeling; Analytics; Large; scale

Software applications produce logs to keep records of execution. Such logs are analyzed to detect, track any events, errors, faults, or exceptions. Emerging software applications are becoming large-scale, complex, and data-intensive. Such applications are extensively utilized and produce logs as data with massive size, high speed, and wide variety. Managing and analyzing logs at a large-scale becomes challenging due to lack of standards, guidelines, best practices on what and how to log. Currently available solutions rely on limited information and are mostly confined to producing logs as semi-structured text files with descriptions in natural language. Processing of such logs as text files include manual scanning, or usage of tools that carry out basic crawling and multiple traversals. This makes monitoring and management of software applications to be challenging. Key limitations are (1) lack of formal data modeling for logs, (2) lack of guidelines for structuring log data, (3) lack of analytical techniques to process logs, produced by software applications, at a scale of big data. ***This research program aims to discover and carry out advancements in data modeling and analytics of logs for software applications, especially that are large-scale, complex, and data-intensive. We will approach the challenges by building integrated solutions based on data modeling and big data analytics to efficiently collect, organize, and effectively analyze logs. Firstly, data modeling techniques will be built to make logs highly structured, well-expressed, and machine-readable. It will serve as a basis for building standards for logging in software applications. Secondly, specialized, scalable, and effective quantitative and qualitative analytical techniques, integrated with data modeling techniques, will be built to manage, and perform analytics on logs produced by large-scale and data-intensive software applications, while maximally utilizing structured, formally modeled and high expressivity of log data. Long-term goals are to build an integrated big data modeling and analytics framework for logs. It will (1) include data modeling and analytics on logs, and (2) act as an information system for developers, system administrators, and other related staff to facilitate automated analysis, and (3) help in setting up standards and best practices to effectively monitor and manage software applications. ***This is a timely research which will significantly improve monitoring of next-generation large-scale and data-intensive software applications. Impact and usefulness of the research will be demonstrated by applying it to real-life applications. This research program will contribute significantly to the training of highly qualified personnel (HQPs). It will benefit Canada in gaining a leading position in information management and data science where demand such skills is rapidly increasing.

软件应用程序生成日志来保存执行记录。分析此类日志以检测、跟踪任何事件、错误、故障或异常。新兴软件应用程序正变得大规模、复杂且数据密集。此类应用程序被广泛使用，并产生大量、高速且种类繁多的日志数据。由于缺乏关于记录内容和方式的标准、指南、最佳实践，大规模管理和分析日志变得具有挑战性。目前可用的解决方案依赖于有限的信息，并且大多局限于将日志生成为带有自然语言描述的半结构化文本文件。对文本文件等日志的处理包括手动扫描，或者使用进行基本爬取和多次遍历的工具。这使得软件应用程序的监控和管理变得具有挑战性。主要限制是（1）缺乏正式的日志数据建模，（2）缺乏构建日志数据的指南，（3）缺乏以大数据规模处理由软件应用程序生成的日志的分析技术。 ***该研究计划旨在发现并实现软件应用程序的数据建模和日志分析方面的进步，特别是大规模、复杂和数据密集型的软件应用程序。我们将通过构建基于数据建模和大数据分析的集成解决方案来应对挑战，以有效地收集、组织和分析日志。首先，将建立数据建模技术，使日志高度结构化、表达良好且机器可读。它将作为构建软件应用程序登录标准的基础。其次，将建立专门的、可扩展的、有效的定量和定性分析技术，与数据建模技术相结合，对大规模和数据密集型软件应用程序生成的日志进行管理和分析，同时最大限度地利用结构化的、形式化的建模日志数据的表现力高。长期目标是构建一个集成的日志大数据建模和分析框架。它将 (1) 包括对日志的数据建模和分析，(2) 充当开发人员、系统管理员和其他相关人员的信息系统，以促进自动化分析，以及 (3) 帮助建立标准和最佳实践，以实现自动化分析。有效地监控和管理软件应用程序。 ***这是一项及时的研究，将显着改善对下一代大规模数据密集型软件应用程序的监控。该研究的影响和实用性将通过将其应用于现实生活中来证明。该研究计划将为高素质人才（HQP）的培训做出重大贡献。它将有利于加拿大在信息管理和数据科学领域获得领先地位，而这些技能的需求正在迅速增长。