eHive-RPC: A Remote Procedure Call Public Interface for eHive

eHive-RPC：eHive 的远程过程调用公共接口

• 批准号: BB/M020398/1
• 负责人: Paul Flicek
• 金额: $ 16.82万
• 依托单位: European Bioinformatics Institute
• 依托单位国家: 英国
• 项目类别: Research Grant
• 财政年份: 2015
• 资助国家: 英国
• 起止时间: 2015 至 无数据
• 项目状态: 已结题
• 来源: https://gtr.ukri.org/projects?ref=BB%2FM020398%2F1
• 关键词: eHive; RPC; Remote; Procedure; Call

Ensembl developed 'eHive' as a production system that manages and optimizes the running of tasks (called 'jobs'), on a compute cluster that may have thousands of Central Processing Units (CPUs). A CPU is the hardware within a computer that carries out the instructions of a computer program by performing the basic input and output operations of the system. Each computer has one or more CPUs.Some compute clusters comprise many thousands of CPUs distributed amongst many computers. With so many computers and CPUs, it is important that jobs are sent to these CPUs in a fair and efficient manner, especially when many users are competing to use the same resources. Clusters usually rely on a central queuing system that holds a list of all the jobs that need to be run and can give individual computers in the cluster explicit instructions about which job to execute. This type of queuing system works well if the jobs each take an hour or more to complete. However, when jobs complete faster than they can be scheduled it creates a processing bottleneck e.g. if a job executes in minutes or less. The usual way to solve the bottleneck is to implement another system on top of the scheduler that 'batches' similar jobs together to make operations more efficient.eHive's novel solution to the issue of job queuing is to move away from this central job scheduling: eHive is a 'distributed' processing system based on 'autonomous agents' with the behavioural structure of honeybees, hence the term 'eHive'. eHive maintains the ability to monitor and track jobs via a central 'blackboard'. Workers are efficiently created on a compute cluster, known as a meadow, with no specific task assigned to them. Once running, each worker contacts the blackboard, is able to find the most suitable kind of job, specializes to claim work and runs multiple jobs of this type in a row. Workers are able to re-specialize to claim other types of jobs once they exhaust their original designation. Each worker regularly updates its status in the blackboard to allow other workers to optimize the overall job distribution.The benefits of eHive are (a) a reduction in the overhead of individual job processing, (b) an increase in the maximum number of tasks that can be running at any one time, (c) an increase in the tolerance to faults in the compute cluster, and (d) the allowance of complicated processes running in parallel.Although eHive was originally designed for the purpose of Ensembl, its functionality is applicable to all data types that have large compute requirements. In this project we aim to transform the possibilities of eHive further, by developing a 'Remote Procedure Call system (RPC) for eHive. This will allow jobs to run on remote clusters as well as local clusters, thereby expanding the use of eHive to multiple compute clusters and cloud computing services. This will enable wider use of eHive within data-intensive fields in the life sciences and beyond.

Ensembl开发了“ EHIVE”作为一种生产系统，该系统管理和优化任务的运行（称为“工作”），该系统可能具有数千个中央处理单元（CPU）的计算集群。 CPU是计算机中的硬件，该硬件通过执行系统的基本输入和输出操作来执行计算机程序的说明。每台计算机都有一个或多个CPU。一些计算簇包含许多分布在许多计算机之间的CPU。有了如此多的计算机和CPU，重要的是要以公平有效的方式将作业发送到这些CPU，尤其是当许多用户竞争使用相同资源时。群集通常依靠一个中央排队系统，该系统列出了所有需要运行的作业的列表，并且可以在群集中为单个计算机提供有关执行哪个作业的明确指令。如果每个工作都需要一个小时或更长时间才能完成，则这种排队系统效果很好。但是，当工作完整的速度比他们安排得的快时，它会创建一个处理瓶颈，例如如果工作在几分钟或更短的时间内执行。解决瓶颈的通常方法是在调度程序上实现另一个系统，以“批量”“批量”“将操作”一起提高效率。 Ehive保持通过中央“黑板”监视和跟踪作业的能力。工人是在称为草地的计算集群上有效创建的，没有分配给他们的特定任务。运行后，每个工人都会与黑板联系，能够找到最合适的工作，专门索取工作并连续运行此类型的多个作业。一旦工人耗尽了原始名称，工人就可以重新宣称其他类型的工作。每个工人定期更新在黑板中的状态，以允许其他工人优化整体工作分配。EHIVE的好处是（a）单个工作处理的开销降低，（b）在任何时候可以在任何时间运行的最大任务数量增加，（c）允许的允许范围的允许范围的容忍度增加，以及（d）允许的范围，以及d rumport的范围。 Ensembl的目的，其功能适用于所有具有较大计算要求的数据类型。在这个项目中，我们旨在通过为EHIVE开发“远程程序呼叫系统（RPC）”来进一步改变EHIVE的可能性。这将允许作业在远程群集以及本地群集上运行，从而将EHIVE扩展到多个计算群集和云计算服务。这将使在生命科学及以后的数据密集型领域中更广泛地使用EHIVE。

项目成果

Ensembl 2021.

• DOI: 10.1093/nar/gkaa942
• 发表时间: 2021-01-08
• 期刊: Nucleic acids research
• 影响因子: 14.9
• 作者: Howe KL;Achuthan P;Allen J;Allen J;Alvarez-Jarreta J;Amode MR;Armean IM;Azov AG;Bennett R;Bhai J;Billis K;Boddu S;Charkhchi M;Cummins C;Da Rin Fioretto L;Davidson C;Dodiya K;El Houdaigui B;Fatima R;Gall A;Garcia Giron C;Grego T;Guijarro-Clarke C;Haggerty L;Hemrom A;Hourlier T;Izuogu OG;Juettemann T;Kaikala V;Kay M;Lavidas I;Le T;Lemos D;Gonzalez Martinez J;Marugán JC;Maurel T;McMahon AC;Mohanan S;Moore B;Muffato M;Oheh DN;Paraschas D;Parker A;Parton A;Prosovetskaia I;Sakthivel MP;Salam AIA;Schmitt BM;Schuilenburg H;Sheppard D;Steed E;Szpak M;Szuba M;Taylor K;Thormann A;Threadgold G;Walts B;Winterbottom A;Chakiachvili M;Chaubal A;De Silva N;Flint B;Frankish A;Hunt SE;IIsley GR;Langridge N;Loveland JE;Martin FJ;Mudge JM;Morales J;Perry E;Ruffier M;Tate J;Thybert D;Trevanion SJ;Cunningham F;Yates AD;Zerbino DR;Flicek P
• 通讯作者: Flicek P

Ensembl 2022.

• DOI: 10.1093/nar/gkab1049
• 发表时间: 2022-01-07
• 期刊: Nucleic acids research
• 影响因子: 14.9
• 作者: Cunningham F;Allen JE;Allen J;Alvarez-Jarreta J;Amode MR;Armean IM;Austine-Orimoloye O;Azov AG;Barnes I;Bennett R;Berry A;Bhai J;Bignell A;Billis K;Boddu S;Brooks L;Charkhchi M;Cummins C;Da Rin Fioretto L;Davidson C;Dodiya K;Donaldson S;El Houdaigui B;El Naboulsi T;Fatima R;Giron CG;Genez T;Martinez JG;Guijarro-Clarke C;Gymer A;Hardy M;Hollis Z;Hourlier T;Hunt T;Juettemann T;Kaikala V;Kay M;Lavidas I;Le T;Lemos D;Marugán JC;Mohanan S;Mushtaq A;Naven M;Ogeh DN;Parker A;Parton A;Perry M;Piližota I;Prosovetskaia I;Sakthivel MP;Salam AIA;Schmitt BM;Schuilenburg H;Sheppard D;Pérez-Silva JG;Stark W;Steed E;Sutinen K;Sukumaran R;Sumathipala D;Suner MM;Szpak M;Thormann A;Tricomi FF;Urbina-Gómez D;Veidenberg A;Walsh TA;Walts B;Willhoft N;Winterbottom A;Wass E;Chakiachvili M;Flint B;Frankish A;Giorgetti S;Haggerty L;Hunt SE;IIsley GR;Loveland JE;Martin FJ;Moore B;Mudge JM;Muffato M;Perry E;Ruffier M;Tate J;Thybert D;Trevanion SJ;Dyer S;Harrison PW;Howe KL;Yates AD;Zerbino DR;Flicek P
• 通讯作者: Flicek P

Ensembl 2017.

• DOI: 10.1093/nar/gkw1104
• 发表时间: 2017-01-04
• 期刊: Nucleic acids research
• 影响因子: 14.9
• 作者: Aken BL;Achuthan P;Akanni W;Amode MR;Bernsdorff F;Bhai J;Billis K;Carvalho-Silva D;Cummins C;Clapham P;Gil L;Girón CG;Gordon L;Hourlier T;Hunt SE;Janacek SH;Juettemann T;Keenan S;Laird MR;Lavidas I;Maurel T;McLaren W;Moore B;Murphy DN;Nag R;Newman V;Nuhn M;Ong CK;Parker A;Patricio M;Riat HS;Sheppard D;Sparrow H;Taylor K;Thormann A;Vullo A;Walts B;Wilder SP;Zadissa A;Kostadima M;Martin FJ;Muffato M;Perry E;Ruffier M;Staines DM;Trevanion SJ;Cunningham F;Yates A;Zerbino DR;Flicek P
• 通讯作者: Flicek P

Ensembl 2018.

• DOI: 10.1093/nar/gkx1098
• 发表时间: 2018-01-04
• 期刊: Nucleic acids research
• 影响因子: 14.9
• 作者: Zerbino DR;Achuthan P;Akanni W;Amode MR;Barrell D;Bhai J;Billis K;Cummins C;Gall A;Girón CG;Gil L;Gordon L;Haggerty L;Haskell E;Hourlier T;Izuogu OG;Janacek SH;Juettemann T;To JK;Laird MR;Lavidas I;Liu Z;Loveland JE;Maurel T;McLaren W;Moore B;Mudge J;Murphy DN;Newman V;Nuhn M;Ogeh D;Ong CK;Parker A;Patricio M;Riat HS;Schuilenburg H;Sheppard D;Sparrow H;Taylor K;Thormann A;Vullo A;Walts B;Zadissa A;Frankish A;Hunt SE;Kostadima M;Langridge N;Martin FJ;Muffato M;Perry E;Ruffier M;Staines DM;Trevanion SJ;Aken BL;Cunningham F;Yates A;Flicek P
• 通讯作者: Flicek P

Ensembl 2019.

• DOI: 10.1093/nar/gky1113
• 发表时间: 2019-01-08
• 期刊: Nucleic acids research
• 影响因子: 14.9
• 作者: Cunningham F;Achuthan P;Akanni W;Allen J;Amode MR;Armean IM;Bennett R;Bhai J;Billis K;Boddu S;Cummins C;Davidson C;Dodiya KJ;Gall A;Girón CG;Gil L;Grego T;Haggerty L;Haskell E;Hourlier T;Izuogu OG;Janacek SH;Juettemann T;Kay M;Laird MR;Lavidas I;Liu Z;Loveland JE;Marugán JC;Maurel T;McMahon AC;Moore B;Morales J;Mudge JM;Nuhn M;Ogeh D;Parker A;Parton A;Patricio M;Abdul Salam AI;Schmitt BM;Schuilenburg H;Sheppard D;Sparrow H;Stapleton E;Szuba M;Taylor K;Threadgold G;Thormann A;Vullo A;Walts B;Winterbottom A;Zadissa A;Chakiachvili M;Frankish A;Hunt SE;Kostadima M;Langridge N;Martin FJ;Muffato M;Perry E;Ruffier M;Staines DM;Trevanion SJ;Aken BL;Yates AD;Zerbino DR;Flicek P
• 通讯作者: Flicek P