New YARN non-exclusive resource management scheme through opportunistic idle resource assignment

TY - JOUR

T1 - New YARN non-exclusive resource management scheme through opportunistic idle resource assignment

AU - Yang, Zhengyu

AU - Yao, Yi

AU - Gao, Han

AU - Wang, Jiayin

AU - Mi, Ningfang

AU - Sheng, Bo

N1 - Publisher Copyright: © 2013 IEEE.

PY - 2021/4/1

Y1 - 2021/4/1

N2 - Efficiently managing resources and improving throughput in a large-scale cluster has become a crucial problem with the explosion of data processing applications in recent years. Hadoop YARN and Mesos, as two universal resource management platforms, have been widely adopted in the commodity cluster for co-deploying multiple data processing frameworks, such as Hadoop MapReduce and Apache Spark. However, in the existing resource management, a certain amount of resources are exclusively allocated to a running task and can only be re-assigned after that task is completed. This exclusive mode unfortunately leads to a potential problem that may under-utilize the cluster resources and degrade system performance. To address this issue, we propose a novel opportunistic and efficient resource allocation scheme, named OpERA, which breaks the barriers among the encapsulated resource containers by leveraging the knowledge of actual runtime resource utilizations to re-assign opportunistic available resources to the pending tasks. OpERA avoids incurring severe performance interference to active tasks by further using two approaches to efficiently balances the starvations of reserved tasks and normal queued tasks. We implement and evaluate OpERA in Hadoop YARN v2.5. Our experimental results show that OpERA significantly reduces the average job execution time and increases the resource (CPU and memory) utilizations.

AB - Efficiently managing resources and improving throughput in a large-scale cluster has become a crucial problem with the explosion of data processing applications in recent years. Hadoop YARN and Mesos, as two universal resource management platforms, have been widely adopted in the commodity cluster for co-deploying multiple data processing frameworks, such as Hadoop MapReduce and Apache Spark. However, in the existing resource management, a certain amount of resources are exclusively allocated to a running task and can only be re-assigned after that task is completed. This exclusive mode unfortunately leads to a potential problem that may under-utilize the cluster resources and degrade system performance. To address this issue, we propose a novel opportunistic and efficient resource allocation scheme, named OpERA, which breaks the barriers among the encapsulated resource containers by leveraging the knowledge of actual runtime resource utilizations to re-assign opportunistic available resources to the pending tasks. OpERA avoids incurring severe performance interference to active tasks by further using two approaches to efficiently balances the starvations of reserved tasks and normal queued tasks. We implement and evaluate OpERA in Hadoop YARN v2.5. Our experimental results show that OpERA significantly reduces the average job execution time and increases the resource (CPU and memory) utilizations.

KW - Hadoop YARN

KW - MapReduce scheduling

KW - Resource allocation

KW - opportunistic

KW - reservation

KW - spark

KW - starvation

UR - http://www.scopus.com/inward/record.url?scp=85052663184&partnerID=8YFLogxK

U2 - 10.1109/TCC.2018.2867580

DO - 10.1109/TCC.2018.2867580

M3 - Article

SN - 2168-7161

VL - 9

SP - 696

EP - 709

JO - IEEE Transactions on Cloud Computing

JF - IEEE Transactions on Cloud Computing

IS - 2

M1 - 8450035

ER -