A study of dynamic optimization of data acquisition system using GRID technology

利用GRID技术的数据采集系统动态优化研究

基本信息

批准号：
15540295
负责人：
ITOH Ryosuke
金额：
$ 2.37万
依托单位：
High Energy Accelerator Research Organization (2004)The High Energy Accelerator Research Organization (2003)
依托单位国家：
日本
项目类别：
Grant-in-Aid for Scientific Research (C)
财政年份：
2003
资助国家：
日本
起止时间：
2003 至 2004
项目状态：
已结题

来源：
https://kaken.nii.ac.jp/en/grant/KAKENHI-PROJECT-15540295/
关键词：
High energy physics Data acquisition system GRID データー収集システム Grid 最適化

项目摘要

In a conventional data acquisition system, the function and data flow of event builder and trigger processor are fixed, and there are possibilities of wasting resouces or performance bottleneck depending on the accelerator condition. The purpose of this study is to optimize the usage of the processors and the data flow dynamically by a real time monitoring of the processor load and the data flow rate. In the first year, the monitoring mechanism was developed using a test bench system consisting of 25 PC servers connected via a large GbE switch. The mechanism was realized utilizing Network Shared Memory(NSM) used for the Belle experiment and it was succeeded to make it work. The transfer speed between nodes was also measured using a GRID software called MICH-G2 and was confirmed to be more than 60MB/sec.In the second year, the development of the optimization mechanism of processor utilization and data flow was done. A dynamic change of processing function on a processor was realized by … More modularizing the processing software and replacing them using a dynamic link with the data acquisition framework. Together with a use of large ring buffers and socket connections implemented on each processing node, it was succeeded to change data flow path and/or processing function on a node without interrupting the data acquisition. As the last step, an algorithms to optimize system-wide performance was trying to be developed, however, we have not yet succeeded to develop the mechanism satisfying the real usage in the data acquisition system. A massive switch of the data flow path and processing software is observed to happen at some boundary during a gradual change in the data flow rate, resulting in a waste of resources. Sometimes a short stop of the data acquisition is observed during the switching. Therefore, we need to continue the effort to improve the optimization algorithm with some new technology like the neural-net.The mechanism developed for the monitoring of data flow and processor load was used for the real time reconstruction farm of the Belle experiment and it was reported at the international conference on Computing in High Energy Physics 04. Less

在传统的数据采集系统中，事件构建器和触发处理器的功能和数据流是固定的，并且可能会根据加速器条件浪费资源或性能瓶颈。这项研究的目的是通过实时监视处理器负载和数据流量来优化处理器的使用和数据流动。在第一年，使用大型GBE开关连接的25个PC服务器组成的测试工作台系统开发了监视机制。该机制利用用于Belle实验的网络共享内存（NSM），并成功地使其正常工作。节点之间的传递速度还使用称为MICH-G2的网格软件测量，并被确认为60MB/秒。第二年，已经完成了处理器利用率和数据流的优化机制。通过…更模块化处理软件并使用数据采集框架更换处理软件的处理器上的处理功能的动态更改可以实现。加上在每个处理节点上实现的大环形缓冲区和套接字连接的使用，它成功地更改了节点上的数据流路径和/或处理功能，而无需中断数据采集。作为最后一步，正在尝试开发一种优化全系统性能的算法，但是，我们尚未成功地开发出满足数据采集系统中真实用法的机制。观察到数据流量路径和处理软件的大规模切换在数据流量的年级变化期间发生在某个边界上，从而导致资源浪费。有时在切换过程中观察到数据采集的短暂停止。因此，我们需要继续努力通过一些新技术（例如神经网络）来改善优化算法。用于监视数据流和处理器负载的机制用于Belle实验的实时重建农场，并在高能物理学的国际计算会议上报道了高能物理学04的计算。