TMaR: a two-stage MapReduce scheduler for heterogeneous environments View Full Text


Ontology type: schema:ScholarlyArticle      Open Access: True


Article Info

DATE

2020-10-07

AUTHORS

Neda Maleki, Hamid Reza Faragardi, Amir Masoud Rahmani, Mauro Conti, Jay Lofstead

ABSTRACT

In the context of MapReduce task scheduling, many algorithms mainly focus on the scheduling of Reduce tasks with the assumption that scheduling of Map tasks is already done. However, in the cloud deployments of MapReduce, the input data is located on remote storage which indicates the importance of the scheduling of Map tasks as well. In this paper, we propose a two-stage Map and Reduce task scheduler for heterogeneous environments, called TMaR. TMaR schedules Map and Reduce tasks on the servers that minimize the task finish time in each stage, respectively. We employ a dynamic partition binder for Reduce tasks in the Reduce stage to lighten the shuffling traffic. Indeed, TMaR minimizes the makespan of a batch of tasks in heterogeneous environments while considering the network traffic. The simulation results demonstrate that TMaR outperforms Hadoop-stock and Hadoop-A in terms of makespan and network traffic and achieves by an average of 29%, 36%, and 14% performance using Wordcount, Sort, and Grep benchmarks. Besides, the power reduction of TMaR is up to 12%. More... »

PAGES

42

References to SciGraph publications

Identifiers

URI

http://scigraph.springernature.com/pub.10.1186/s13673-020-00247-5

DOI

http://dx.doi.org/10.1186/s13673-020-00247-5

DIMENSIONS

https://app.dimensions.ai/details/publication/pub.1131501704


Indexing Status Check whether this publication has been indexed by Scopus and Web Of Science using the SN Indexing Status Tool
Incoming Citations Browse incoming citations for this publication using opencitations.net

JSON-LD is the canonical representation for SciGraph data.

TIP: You can open this SciGraph record using an external JSON-LD service: JSON-LD Playground Google SDTT

[
  {
    "@context": "https://springernature.github.io/scigraph/jsonld/sgcontext.json", 
    "about": [
      {
        "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/08", 
        "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
        "name": "Information and Computing Sciences", 
        "type": "DefinedTerm"
      }, 
      {
        "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/0801", 
        "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
        "name": "Artificial Intelligence and Image Processing", 
        "type": "DefinedTerm"
      }
    ], 
    "author": [
      {
        "affiliation": {
          "alternateName": "Department of Computer Engineering, Science and Research Branch, Islamic Azad University, Tehran, Iran", 
          "id": "http://www.grid.ac/institutes/grid.472472.0", 
          "name": [
            "Department of Computer Engineering, Science and Research Branch, Islamic Azad University, Tehran, Iran"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Maleki", 
        "givenName": "Neda", 
        "id": "sg:person.016635532424.32", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.016635532424.32"
        ], 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "Department of Computer Science and Communication, KTH Royal Institute of Technology, Stockholm, Sweden", 
          "id": "http://www.grid.ac/institutes/grid.5037.1", 
          "name": [
            "Department of Computer Science and Communication, KTH Royal Institute of Technology, Stockholm, Sweden"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Faragardi", 
        "givenName": "Hamid Reza", 
        "id": "sg:person.013645574403.37", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.013645574403.37"
        ], 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "Department of Computer Science, Khazar University, Baku, Azerbaijan", 
          "id": "http://www.grid.ac/institutes/grid.442897.4", 
          "name": [
            "Institute of Research and Development, Duy Tan University, 550000, Da Nang, Vietnam", 
            "Faculty of Information Technology, Duy Tan University, 550000, Da Nang, Vietnam", 
            "Department of Computer Science, Khazar University, Baku, Azerbaijan"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Rahmani", 
        "givenName": "Amir Masoud", 
        "id": "sg:person.010142517353.94", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.010142517353.94"
        ], 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "Department of Mathematics, University of Padua, Padua, Italy", 
          "id": "http://www.grid.ac/institutes/grid.5608.b", 
          "name": [
            "Department of Mathematics, University of Padua, Padua, Italy"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Conti", 
        "givenName": "Mauro", 
        "id": "sg:person.010537540667.47", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.010537540667.47"
        ], 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "Sandia National Laboratories, Albuquerque, NM, USA", 
          "id": "http://www.grid.ac/institutes/grid.474520.0", 
          "name": [
            "Sandia National Laboratories, Albuquerque, NM, USA"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Lofstead", 
        "givenName": "Jay", 
        "id": "sg:person.014545747743.34", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.014545747743.34"
        ], 
        "type": "Person"
      }
    ], 
    "citation": [
      {
        "id": "sg:pub.10.1007/s11590-018-01384-8", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1111370707", 
          "https://doi.org/10.1007/s11590-018-01384-8"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "sg:pub.10.1007/s11227-019-02907-5", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1116997563", 
          "https://doi.org/10.1007/s11227-019-02907-5"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "sg:pub.10.1007/s11227-019-02855-0", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1113624435", 
          "https://doi.org/10.1007/s11227-019-02855-0"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "sg:pub.10.1007/s10586-015-0454-8", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1023182693", 
          "https://doi.org/10.1007/s10586-015-0454-8"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "sg:pub.10.1007/s11227-016-1653-7", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1027165908", 
          "https://doi.org/10.1007/s11227-016-1653-7"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "sg:pub.10.1007/s11227-016-1737-4", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1039060355", 
          "https://doi.org/10.1007/s11227-016-1737-4"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "sg:pub.10.1007/s11042-017-4685-y", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1085127057", 
          "https://doi.org/10.1007/s11042-017-4685-y"
        ], 
        "type": "CreativeWork"
      }
    ], 
    "datePublished": "2020-10-07", 
    "datePublishedReg": "2020-10-07", 
    "description": "In the context of MapReduce task scheduling, many algorithms mainly focus on the scheduling of Reduce tasks with the assumption that scheduling of Map tasks is already done. However, in the cloud deployments of MapReduce, the input data is located on remote storage which indicates the importance of the scheduling of Map tasks as well. In this paper, we propose a two-stage Map and Reduce task scheduler for heterogeneous environments, called TMaR. TMaR schedules Map and Reduce tasks on the servers that minimize the task finish time in each stage, respectively. We employ a dynamic partition binder for Reduce tasks in the Reduce stage to lighten the shuffling traffic. Indeed, TMaR minimizes the makespan of a batch of tasks in heterogeneous environments while considering the network traffic. The simulation results demonstrate that TMaR outperforms Hadoop-stock and Hadoop-A in terms of makespan and network traffic and achieves by an average of 29%, 36%, and 14% performance using Wordcount, Sort, and Grep benchmarks. Besides, the power reduction of TMaR is up to 12%.", 
    "genre": "article", 
    "id": "sg:pub.10.1186/s13673-020-00247-5", 
    "isAccessibleForFree": true, 
    "isPartOf": [
      {
        "id": "sg:journal.1136381", 
        "issn": [
          "2192-1962"
        ], 
        "name": "Human-centric Computing and Information Sciences", 
        "publisher": "Springer Nature", 
        "type": "Periodical"
      }, 
      {
        "issueNumber": "1", 
        "type": "PublicationIssue"
      }, 
      {
        "type": "PublicationVolume", 
        "volumeNumber": "10"
      }
    ], 
    "keywords": [
      "heterogeneous environments", 
      "network traffic", 
      "Reduce tasks", 
      "map tasks", 
      "task finish time", 
      "terms of makespan", 
      "batch of tasks", 
      "cloud deployment", 
      "task scheduling", 
      "task scheduler", 
      "MapReduce scheduler", 
      "remote storage", 
      "reduce stage", 
      "finish time", 
      "scheduling", 
      "input data", 
      "scheduler", 
      "traffic", 
      "task", 
      "makespan", 
      "simulation results", 
      "MapReduce", 
      "Hadoop", 
      "server", 
      "WordCount", 
      "environment", 
      "power reduction", 
      "algorithm", 
      "deployment", 
      "benchmarks", 
      "maps", 
      "performance", 
      "TmaR", 
      "storage", 
      "context", 
      "data", 
      "sort", 
      "terms", 
      "time", 
      "assumption", 
      "stage", 
      "results", 
      "batch", 
      "importance", 
      "average", 
      "reduction", 
      "binder", 
      "paper"
    ], 
    "name": "TMaR: a two-stage MapReduce scheduler for heterogeneous environments", 
    "pagination": "42", 
    "productId": [
      {
        "name": "dimensions_id", 
        "type": "PropertyValue", 
        "value": [
          "pub.1131501704"
        ]
      }, 
      {
        "name": "doi", 
        "type": "PropertyValue", 
        "value": [
          "10.1186/s13673-020-00247-5"
        ]
      }
    ], 
    "sameAs": [
      "https://doi.org/10.1186/s13673-020-00247-5", 
      "https://app.dimensions.ai/details/publication/pub.1131501704"
    ], 
    "sdDataset": "articles", 
    "sdDatePublished": "2022-11-24T21:06", 
    "sdLicense": "https://scigraph.springernature.com/explorer/license/", 
    "sdPublisher": {
      "name": "Springer Nature - SN SciGraph project", 
      "type": "Organization"
    }, 
    "sdSource": "s3://com-springernature-scigraph/baseset/20221124/entities/gbq_results/article/article_856.jsonl", 
    "type": "ScholarlyArticle", 
    "url": "https://doi.org/10.1186/s13673-020-00247-5"
  }
]
 

Download the RDF metadata as:  json-ld nt turtle xml License info

HOW TO GET THIS DATA PROGRAMMATICALLY:

JSON-LD is a popular format for linked data which is fully compatible with JSON.

curl -H 'Accept: application/ld+json' 'https://scigraph.springernature.com/pub.10.1186/s13673-020-00247-5'

N-Triples is a line-based linked data format ideal for batch operations.

curl -H 'Accept: application/n-triples' 'https://scigraph.springernature.com/pub.10.1186/s13673-020-00247-5'

Turtle is a human-readable linked data format.

curl -H 'Accept: text/turtle' 'https://scigraph.springernature.com/pub.10.1186/s13673-020-00247-5'

RDF/XML is a standard XML format for linked data.

curl -H 'Accept: application/rdf+xml' 'https://scigraph.springernature.com/pub.10.1186/s13673-020-00247-5'


 

This table displays all metadata directly associated to this object as RDF triples.

174 TRIPLES      21 PREDICATES      79 URIs      64 LITERALS      6 BLANK NODES

Subject Predicate Object
1 sg:pub.10.1186/s13673-020-00247-5 schema:about anzsrc-for:08
2 anzsrc-for:0801
3 schema:author N5e948eea001841ee89a039292cc026b8
4 schema:citation sg:pub.10.1007/s10586-015-0454-8
5 sg:pub.10.1007/s11042-017-4685-y
6 sg:pub.10.1007/s11227-016-1653-7
7 sg:pub.10.1007/s11227-016-1737-4
8 sg:pub.10.1007/s11227-019-02855-0
9 sg:pub.10.1007/s11227-019-02907-5
10 sg:pub.10.1007/s11590-018-01384-8
11 schema:datePublished 2020-10-07
12 schema:datePublishedReg 2020-10-07
13 schema:description In the context of MapReduce task scheduling, many algorithms mainly focus on the scheduling of Reduce tasks with the assumption that scheduling of Map tasks is already done. However, in the cloud deployments of MapReduce, the input data is located on remote storage which indicates the importance of the scheduling of Map tasks as well. In this paper, we propose a two-stage Map and Reduce task scheduler for heterogeneous environments, called TMaR. TMaR schedules Map and Reduce tasks on the servers that minimize the task finish time in each stage, respectively. We employ a dynamic partition binder for Reduce tasks in the Reduce stage to lighten the shuffling traffic. Indeed, TMaR minimizes the makespan of a batch of tasks in heterogeneous environments while considering the network traffic. The simulation results demonstrate that TMaR outperforms Hadoop-stock and Hadoop-A in terms of makespan and network traffic and achieves by an average of 29%, 36%, and 14% performance using Wordcount, Sort, and Grep benchmarks. Besides, the power reduction of TMaR is up to 12%.
14 schema:genre article
15 schema:isAccessibleForFree true
16 schema:isPartOf N692c1dac3d524cd1914a2cd02ee3dd48
17 Nc2f922a3e8f744e08e14afd821a3682d
18 sg:journal.1136381
19 schema:keywords Hadoop
20 MapReduce
21 MapReduce scheduler
22 Reduce tasks
23 TmaR
24 WordCount
25 algorithm
26 assumption
27 average
28 batch
29 batch of tasks
30 benchmarks
31 binder
32 cloud deployment
33 context
34 data
35 deployment
36 environment
37 finish time
38 heterogeneous environments
39 importance
40 input data
41 makespan
42 map tasks
43 maps
44 network traffic
45 paper
46 performance
47 power reduction
48 reduce stage
49 reduction
50 remote storage
51 results
52 scheduler
53 scheduling
54 server
55 simulation results
56 sort
57 stage
58 storage
59 task
60 task finish time
61 task scheduler
62 task scheduling
63 terms
64 terms of makespan
65 time
66 traffic
67 schema:name TMaR: a two-stage MapReduce scheduler for heterogeneous environments
68 schema:pagination 42
69 schema:productId Nd2922ee5b908451ca9454e43075bc0bd
70 Nfb2df2adafd744b6a034dc298fc1337a
71 schema:sameAs https://app.dimensions.ai/details/publication/pub.1131501704
72 https://doi.org/10.1186/s13673-020-00247-5
73 schema:sdDatePublished 2022-11-24T21:06
74 schema:sdLicense https://scigraph.springernature.com/explorer/license/
75 schema:sdPublisher Nbbb456e7276a412c820375524e46510a
76 schema:url https://doi.org/10.1186/s13673-020-00247-5
77 sgo:license sg:explorer/license/
78 sgo:sdDataset articles
79 rdf:type schema:ScholarlyArticle
80 N031d597480224132a9acabd5bf947515 rdf:first sg:person.010537540667.47
81 rdf:rest N53f963342754455290e161ee7ff1bda6
82 N53f963342754455290e161ee7ff1bda6 rdf:first sg:person.014545747743.34
83 rdf:rest rdf:nil
84 N5e948eea001841ee89a039292cc026b8 rdf:first sg:person.016635532424.32
85 rdf:rest Nbf9589714e34452c88ecdb7cd0cde940
86 N692c1dac3d524cd1914a2cd02ee3dd48 schema:volumeNumber 10
87 rdf:type schema:PublicationVolume
88 Nbbb456e7276a412c820375524e46510a schema:name Springer Nature - SN SciGraph project
89 rdf:type schema:Organization
90 Nbf9589714e34452c88ecdb7cd0cde940 rdf:first sg:person.013645574403.37
91 rdf:rest Nc3788c3baeec4a3bb025109ce66c31a5
92 Nc2f922a3e8f744e08e14afd821a3682d schema:issueNumber 1
93 rdf:type schema:PublicationIssue
94 Nc3788c3baeec4a3bb025109ce66c31a5 rdf:first sg:person.010142517353.94
95 rdf:rest N031d597480224132a9acabd5bf947515
96 Nd2922ee5b908451ca9454e43075bc0bd schema:name doi
97 schema:value 10.1186/s13673-020-00247-5
98 rdf:type schema:PropertyValue
99 Nfb2df2adafd744b6a034dc298fc1337a schema:name dimensions_id
100 schema:value pub.1131501704
101 rdf:type schema:PropertyValue
102 anzsrc-for:08 schema:inDefinedTermSet anzsrc-for:
103 schema:name Information and Computing Sciences
104 rdf:type schema:DefinedTerm
105 anzsrc-for:0801 schema:inDefinedTermSet anzsrc-for:
106 schema:name Artificial Intelligence and Image Processing
107 rdf:type schema:DefinedTerm
108 sg:journal.1136381 schema:issn 2192-1962
109 schema:name Human-centric Computing and Information Sciences
110 schema:publisher Springer Nature
111 rdf:type schema:Periodical
112 sg:person.010142517353.94 schema:affiliation grid-institutes:grid.442897.4
113 schema:familyName Rahmani
114 schema:givenName Amir Masoud
115 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.010142517353.94
116 rdf:type schema:Person
117 sg:person.010537540667.47 schema:affiliation grid-institutes:grid.5608.b
118 schema:familyName Conti
119 schema:givenName Mauro
120 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.010537540667.47
121 rdf:type schema:Person
122 sg:person.013645574403.37 schema:affiliation grid-institutes:grid.5037.1
123 schema:familyName Faragardi
124 schema:givenName Hamid Reza
125 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.013645574403.37
126 rdf:type schema:Person
127 sg:person.014545747743.34 schema:affiliation grid-institutes:grid.474520.0
128 schema:familyName Lofstead
129 schema:givenName Jay
130 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.014545747743.34
131 rdf:type schema:Person
132 sg:person.016635532424.32 schema:affiliation grid-institutes:grid.472472.0
133 schema:familyName Maleki
134 schema:givenName Neda
135 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.016635532424.32
136 rdf:type schema:Person
137 sg:pub.10.1007/s10586-015-0454-8 schema:sameAs https://app.dimensions.ai/details/publication/pub.1023182693
138 https://doi.org/10.1007/s10586-015-0454-8
139 rdf:type schema:CreativeWork
140 sg:pub.10.1007/s11042-017-4685-y schema:sameAs https://app.dimensions.ai/details/publication/pub.1085127057
141 https://doi.org/10.1007/s11042-017-4685-y
142 rdf:type schema:CreativeWork
143 sg:pub.10.1007/s11227-016-1653-7 schema:sameAs https://app.dimensions.ai/details/publication/pub.1027165908
144 https://doi.org/10.1007/s11227-016-1653-7
145 rdf:type schema:CreativeWork
146 sg:pub.10.1007/s11227-016-1737-4 schema:sameAs https://app.dimensions.ai/details/publication/pub.1039060355
147 https://doi.org/10.1007/s11227-016-1737-4
148 rdf:type schema:CreativeWork
149 sg:pub.10.1007/s11227-019-02855-0 schema:sameAs https://app.dimensions.ai/details/publication/pub.1113624435
150 https://doi.org/10.1007/s11227-019-02855-0
151 rdf:type schema:CreativeWork
152 sg:pub.10.1007/s11227-019-02907-5 schema:sameAs https://app.dimensions.ai/details/publication/pub.1116997563
153 https://doi.org/10.1007/s11227-019-02907-5
154 rdf:type schema:CreativeWork
155 sg:pub.10.1007/s11590-018-01384-8 schema:sameAs https://app.dimensions.ai/details/publication/pub.1111370707
156 https://doi.org/10.1007/s11590-018-01384-8
157 rdf:type schema:CreativeWork
158 grid-institutes:grid.442897.4 schema:alternateName Department of Computer Science, Khazar University, Baku, Azerbaijan
159 schema:name Department of Computer Science, Khazar University, Baku, Azerbaijan
160 Faculty of Information Technology, Duy Tan University, 550000, Da Nang, Vietnam
161 Institute of Research and Development, Duy Tan University, 550000, Da Nang, Vietnam
162 rdf:type schema:Organization
163 grid-institutes:grid.472472.0 schema:alternateName Department of Computer Engineering, Science and Research Branch, Islamic Azad University, Tehran, Iran
164 schema:name Department of Computer Engineering, Science and Research Branch, Islamic Azad University, Tehran, Iran
165 rdf:type schema:Organization
166 grid-institutes:grid.474520.0 schema:alternateName Sandia National Laboratories, Albuquerque, NM, USA
167 schema:name Sandia National Laboratories, Albuquerque, NM, USA
168 rdf:type schema:Organization
169 grid-institutes:grid.5037.1 schema:alternateName Department of Computer Science and Communication, KTH Royal Institute of Technology, Stockholm, Sweden
170 schema:name Department of Computer Science and Communication, KTH Royal Institute of Technology, Stockholm, Sweden
171 rdf:type schema:Organization
172 grid-institutes:grid.5608.b schema:alternateName Department of Mathematics, University of Padua, Padua, Italy
173 schema:name Department of Mathematics, University of Padua, Padua, Italy
174 rdf:type schema:Organization
 




Preview window. Press ESC to close (or click here)


...