eHive-RPC: A Remote Procedure Call Public Interface for eHive

eHive-RPC：eHive 的远程过程调用公共接口

• 批准号: BB/M020398/1
• 负责人: Paul Flicek
• 金额: $ 16.82万
• 依托单位: European Bioinformatics Institute
• 依托单位国家: 英国
• 项目类别: Research Grant
• 财政年份: 2015
• 资助国家: 英国
• 起止时间: 2015 至 无数据
• 项目状态: 已结题
• 来源: https://gtr.ukri.org/projects?ref=BB%2FM020398%2F1
• 关键词: eHive; RPC; Remote; Procedure; Call

Ensembl developed 'eHive' as a production system that manages and optimizes the running of tasks (called 'jobs'), on a compute cluster that may have thousands of Central Processing Units (CPUs). A CPU is the hardware within a computer that carries out the instructions of a computer program by performing the basic input and output operations of the system. Each computer has one or more CPUs.Some compute clusters comprise many thousands of CPUs distributed amongst many computers. With so many computers and CPUs, it is important that jobs are sent to these CPUs in a fair and efficient manner, especially when many users are competing to use the same resources. Clusters usually rely on a central queuing system that holds a list of all the jobs that need to be run and can give individual computers in the cluster explicit instructions about which job to execute. This type of queuing system works well if the jobs each take an hour or more to complete. However, when jobs complete faster than they can be scheduled it creates a processing bottleneck e.g. if a job executes in minutes or less. The usual way to solve the bottleneck is to implement another system on top of the scheduler that 'batches' similar jobs together to make operations more efficient.eHive's novel solution to the issue of job queuing is to move away from this central job scheduling: eHive is a 'distributed' processing system based on 'autonomous agents' with the behavioural structure of honeybees, hence the term 'eHive'. eHive maintains the ability to monitor and track jobs via a central 'blackboard'. Workers are efficiently created on a compute cluster, known as a meadow, with no specific task assigned to them. Once running, each worker contacts the blackboard, is able to find the most suitable kind of job, specializes to claim work and runs multiple jobs of this type in a row. Workers are able to re-specialize to claim other types of jobs once they exhaust their original designation. Each worker regularly updates its status in the blackboard to allow other workers to optimize the overall job distribution.The benefits of eHive are (a) a reduction in the overhead of individual job processing, (b) an increase in the maximum number of tasks that can be running at any one time, (c) an increase in the tolerance to faults in the compute cluster, and (d) the allowance of complicated processes running in parallel.Although eHive was originally designed for the purpose of Ensembl, its functionality is applicable to all data types that have large compute requirements. In this project we aim to transform the possibilities of eHive further, by developing a 'Remote Procedure Call system (RPC) for eHive. This will allow jobs to run on remote clusters as well as local clusters, thereby expanding the use of eHive to multiple compute clusters and cloud computing services. This will enable wider use of eHive within data-intensive fields in the life sciences and beyond.

Ensembl 开发了“eHive”作为生产系统，用于在可能拥有数千个中央处理单元 (CPU) 的计算集群上管理和优化任务（称为“作业”）的运行。 CPU 是计算机内的硬件，它通过执行系统的基本输入和输出操作来执行计算机程序的指令。每台计算机都有一个或多个 CPU。一些计算集群包含分布在许多计算机中的数千个 CPU。对于如此多的计算机和 CPU，以公平且高效的方式将作业发送到这些 CPU 非常重要，尤其是当许多用户竞争使用相同的资源时。集群通常依赖于中央排队系统，该系统保存需要运行的所有作业的列表，并且可以向集群中的各个计算机提供有关执行哪个作业的明确指令。如果每个作业需要一个小时或更长时间才能完成，这种类型的排队系统效果很好。然而，当作业完成的速度快于计划的速度时，就会产生处理瓶颈，例如如果作业在几分钟或更短的时间内执行。解决瓶颈的通常方法是在调度程序之上实现另一个系统，将类似的作业“批处理”在一起，从而提高操作效率。eHive 针对作业排队问题的新颖解决方案是摆脱这种中央作业调度：eHive是一个基于“自主代理”的“分布式”处理系统，具有蜜蜂的行为结构，因此称为“eHive”。 eHive 保持通过中央“黑板”监控和跟踪作业的能力。工作人员是在计算集群（称为草地）上高效创建的，没有分配给他们特定的任务。一旦运行，每个工人都会接触黑板，能够找到最合适的工种，专门领取工种，并连续运行多个该工种的工种。一旦用尽原来的职称，工人就可以重新专业化以从事其他类型的工作。每个工作人员定期更新其在黑板上的状态，以便其他工作人员优化整体工作分配。eHive 的好处是 (a) 减少单个工作处理的开销，(b) 增加可处理的最大任务数可以在任意时间运行，(c) 提高计算集群中的容错能力，以及 (d) 允许并行运行复杂的进程。虽然 eHive 最初是为了 Ensembl 的目的而设计的，但它的功能是适用于所有具有以下特征的数据类型大量的计算需求。在这个项目中，我们的目标是通过为 eHive 开发“远程过程调用系统 (RPC)”来进一步改变 eHive 的可能性。这将允许作业在远程集群和本地集群上运行，从而将 eHive 的使用扩展到多个计算集群和云计算服务。这将使 eHive 在生命科学及其他数据密集型领域得到更广泛的使用。

项目成果

Ensembl 2021.

• DOI: 10.1093/nar/gkaa942
• 发表时间: 2021-01-08
• 期刊: Nucleic acids research
• 影响因子: 14.9
• 作者: Howe KL;Achuthan P;Allen J;Allen J;Alvarez-Jarreta J;Amode MR;Armean IM;Azov AG;Bennett R;Bhai J;Billis K;Boddu S;Charkhchi M;Cummins C;Da Rin Fioretto L;Davidson C;Dodiya K;El Houdaigui B;Fatima R;Gall A;Garcia Giron C;Grego T;Guijarro-Clarke C;Haggerty L;Hemrom A;Hourlier T;Izuogu OG;Juettemann T;Kaikala V;Kay M;Lavidas I;Le T;Lemos D;Gonzalez Martinez J;Marugán JC;Maurel T;McMahon AC;Mohanan S;Moore B;Muffato M;Oheh DN;Paraschas D;Parker A;Parton A;Prosovetskaia I;Sakthivel MP;Salam AIA;Schmitt BM;Schuilenburg H;Sheppard D;Steed E;Szpak M;Szuba M;Taylor K;Thormann A;Threadgold G;Walts B;Winterbottom A;Chakiachvili M;Chaubal A;De Silva N;Flint B;Frankish A;Hunt SE;IIsley GR;Langridge N;Loveland JE;Martin FJ;Mudge JM;Morales J;Perry E;Ruffier M;Tate J;Thybert D;Trevanion SJ;Cunningham F;Yates AD;Zerbino DR;Flicek P
• 通讯作者: Flicek P

Ensembl 2022.

• DOI: 10.1093/nar/gkab1049
• 发表时间: 2022-01-07
• 期刊: Nucleic acids research
• 影响因子: 14.9
• 作者: Cunningham F;Allen JE;Allen J;Alvarez-Jarreta J;Amode MR;Armean IM;Austine-Orimoloye O;Azov AG;Barnes I;Bennett R;Berry A;Bhai J;Bignell A;Billis K;Boddu S;Brooks L;Charkhchi M;Cummins C;Da Rin Fioretto L;Davidson C;Dodiya K;Donaldson S;El Houdaigui B;El Naboulsi T;Fatima R;Giron CG;Genez T;Martinez JG;Guijarro-Clarke C;Gymer A;Hardy M;Hollis Z;Hourlier T;Hunt T;Juettemann T;Kaikala V;Kay M;Lavidas I;Le T;Lemos D;Marugán JC;Mohanan S;Mushtaq A;Naven M;Ogeh DN;Parker A;Parton A;Perry M;Piližota I;Prosovetskaia I;Sakthivel MP;Salam AIA;Schmitt BM;Schuilenburg H;Sheppard D;Pérez-Silva JG;Stark W;Steed E;Sutinen K;Sukumaran R;Sumathipala D;Suner MM;Szpak M;Thormann A;Tricomi FF;Urbina-Gómez D;Veidenberg A;Walsh TA;Walts B;Willhoft N;Winterbottom A;Wass E;Chakiachvili M;Flint B;Frankish A;Giorgetti S;Haggerty L;Hunt SE;IIsley GR;Loveland JE;Martin FJ;Moore B;Mudge JM;Muffato M;Perry E;Ruffier M;Tate J;Thybert D;Trevanion SJ;Dyer S;Harrison PW;Howe KL;Yates AD;Zerbino DR;Flicek P
• 通讯作者: Flicek P

Ensembl 2017.

• DOI: 10.1093/nar/gkw1104
• 发表时间: 2017-01-04
• 期刊: Nucleic acids research
• 影响因子: 14.9
• 作者: Aken BL;Achuthan P;Akanni W;Amode MR;Bernsdorff F;Bhai J;Billis K;Carvalho-Silva D;Cummins C;Clapham P;Gil L;Girón CG;Gordon L;Hourlier T;Hunt SE;Janacek SH;Juettemann T;Keenan S;Laird MR;Lavidas I;Maurel T;McLaren W;Moore B;Murphy DN;Nag R;Newman V;Nuhn M;Ong CK;Parker A;Patricio M;Riat HS;Sheppard D;Sparrow H;Taylor K;Thormann A;Vullo A;Walts B;Wilder SP;Zadissa A;Kostadima M;Martin FJ;Muffato M;Perry E;Ruffier M;Staines DM;Trevanion SJ;Cunningham F;Yates A;Zerbino DR;Flicek P
• 通讯作者: Flicek P

Ensembl 2018.

• DOI: 10.1093/nar/gkx1098
• 发表时间: 2018-01-04
• 期刊: Nucleic acids research
• 影响因子: 14.9
• 作者: Zerbino DR;Achuthan P;Akanni W;Amode MR;Barrell D;Bhai J;Billis K;Cummins C;Gall A;Girón CG;Gil L;Gordon L;Haggerty L;Haskell E;Hourlier T;Izuogu OG;Janacek SH;Juettemann T;To JK;Laird MR;Lavidas I;Liu Z;Loveland JE;Maurel T;McLaren W;Moore B;Mudge J;Murphy DN;Newman V;Nuhn M;Ogeh D;Ong CK;Parker A;Patricio M;Riat HS;Schuilenburg H;Sheppard D;Sparrow H;Taylor K;Thormann A;Vullo A;Walts B;Zadissa A;Frankish A;Hunt SE;Kostadima M;Langridge N;Martin FJ;Muffato M;Perry E;Ruffier M;Staines DM;Trevanion SJ;Aken BL;Cunningham F;Yates A;Flicek P
• 通讯作者: Flicek P

Ensembl 2019.

• DOI: 10.1093/nar/gky1113
• 发表时间: 2019-01-08
• 期刊: Nucleic acids research
• 影响因子: 14.9
• 作者: Cunningham F;Achuthan P;Akanni W;Allen J;Amode MR;Armean IM;Bennett R;Bhai J;Billis K;Boddu S;Cummins C;Davidson C;Dodiya KJ;Gall A;Girón CG;Gil L;Grego T;Haggerty L;Haskell E;Hourlier T;Izuogu OG;Janacek SH;Juettemann T;Kay M;Laird MR;Lavidas I;Liu Z;Loveland JE;Marugán JC;Maurel T;McMahon AC;Moore B;Morales J;Mudge JM;Nuhn M;Ogeh D;Parker A;Parton A;Patricio M;Abdul Salam AI;Schmitt BM;Schuilenburg H;Sheppard D;Sparrow H;Stapleton E;Szuba M;Taylor K;Threadgold G;Thormann A;Vullo A;Walts B;Winterbottom A;Zadissa A;Chakiachvili M;Frankish A;Hunt SE;Kostadima M;Langridge N;Martin FJ;Muffato M;Perry E;Ruffier M;Staines DM;Trevanion SJ;Aken BL;Yates AD;Zerbino DR;Flicek P
• 通讯作者: Flicek P