Multi-objective scheduling of MapReduce jobs in big data processing View Full Text


Ontology type: schema:ScholarlyArticle     


Article Info

DATE

2017-05-03

AUTHORS

Ibrahim Abaker Targio Hashem, Nor Badrul Anuar, Mohsen Marjani, Abdullah Gani, Arun Kumar Sangaiah, Adewole Kayode Sakariyah

ABSTRACT

Data generation has increased drastically over the past few years due to the rapid development of Internet-based technologies. This period has been called the big data era. Big data offer an emerging paradigm shift in data exploration and utilization. The MapReduce computational paradigm is a well-known framework and is considered the main enabler for the distributed and scalable processing of a large amount of data. However, despite recent efforts toward improving the performance of MapReduce, scheduling MapReduce jobs across multiple nodes has been considered a multi-objective optimization problem. This problem can become increasingly complex when virtualized clusters in cloud computing are used to execute a large number of tasks. This study aims to optimize MapReduce job scheduling based on the completion time and cost of cloud service models. First, the problem is formulated as a multi-objective model. The model consists of two objective functions, namely, (i) completion time and (ii) cost minimization. Second, a scheduling algorithm using earliest finish time scheduling that considers resource allocation and job scheduling in the cloud is proposed. Lastly, experimental results show that the proposed scheduler exhibits better performance than other well-known schedulers, such as FIFO and Fair. More... »

PAGES

9979-9994

References to SciGraph publications

Identifiers

URI

http://scigraph.springernature.com/pub.10.1007/s11042-017-4685-y

DOI

http://dx.doi.org/10.1007/s11042-017-4685-y

DIMENSIONS

https://app.dimensions.ai/details/publication/pub.1085127057


Indexing Status Check whether this publication has been indexed by Scopus and Web Of Science using the SN Indexing Status Tool
Incoming Citations Browse incoming citations for this publication using opencitations.net

JSON-LD is the canonical representation for SciGraph data.

TIP: You can open this SciGraph record using an external JSON-LD service: JSON-LD Playground Google SDTT

[
  {
    "@context": "https://springernature.github.io/scigraph/jsonld/sgcontext.json", 
    "about": [
      {
        "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/08", 
        "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
        "name": "Information and Computing Sciences", 
        "type": "DefinedTerm"
      }, 
      {
        "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/0803", 
        "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
        "name": "Computer Software", 
        "type": "DefinedTerm"
      }
    ], 
    "author": [
      {
        "affiliation": {
          "alternateName": "Faculty of Computer Science and Information Technology, University of Malaya, Kuala Lumpur, Malaysia", 
          "id": "http://www.grid.ac/institutes/grid.10347.31", 
          "name": [
            "Faculty of Computer Science and Information Technology, University of Malaya, Kuala Lumpur, Malaysia"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Hashem", 
        "givenName": "Ibrahim Abaker Targio", 
        "id": "sg:person.01035421130.35", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01035421130.35"
        ], 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "Faculty of Computer Science and Information Technology, University of Malaya, Kuala Lumpur, Malaysia", 
          "id": "http://www.grid.ac/institutes/grid.10347.31", 
          "name": [
            "Faculty of Computer Science and Information Technology, University of Malaya, Kuala Lumpur, Malaysia"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Anuar", 
        "givenName": "Nor Badrul", 
        "id": "sg:person.0665142504.27", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.0665142504.27"
        ], 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "Faculty of Computer Science and Information Technology, University of Malaya, Kuala Lumpur, Malaysia", 
          "id": "http://www.grid.ac/institutes/grid.10347.31", 
          "name": [
            "Faculty of Computer Science and Information Technology, University of Malaya, Kuala Lumpur, Malaysia"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Marjani", 
        "givenName": "Mohsen", 
        "id": "sg:person.012750345514.73", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.012750345514.73"
        ], 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "Faculty of Computer Science and Information Technology, University of Malaya, Kuala Lumpur, Malaysia", 
          "id": "http://www.grid.ac/institutes/grid.10347.31", 
          "name": [
            "Faculty of Computer Science and Information Technology, University of Malaya, Kuala Lumpur, Malaysia"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Gani", 
        "givenName": "Abdullah", 
        "id": "sg:person.013623112027.80", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.013623112027.80"
        ], 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "School of Computing Science and Engineering, VIT University, 632014, Vellore, India", 
          "id": "http://www.grid.ac/institutes/grid.412813.d", 
          "name": [
            "School of Computing Science and Engineering, VIT University, 632014, Vellore, India"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Sangaiah", 
        "givenName": "Arun Kumar", 
        "id": "sg:person.01203656177.16", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01203656177.16"
        ], 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "Faculty of Computer Science and Information Technology, University of Malaya, Kuala Lumpur, Malaysia", 
          "id": "http://www.grid.ac/institutes/grid.10347.31", 
          "name": [
            "Faculty of Computer Science and Information Technology, University of Malaya, Kuala Lumpur, Malaysia"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Sakariyah", 
        "givenName": "Adewole Kayode", 
        "type": "Person"
      }
    ], 
    "citation": [
      {
        "id": "sg:pub.10.1007/978-3-642-19294-4_9", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1022039554", 
          "https://doi.org/10.1007/978-3-642-19294-4_9"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "sg:pub.10.1007/s00778-013-0319-9", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1002332096", 
          "https://doi.org/10.1007/s00778-013-0319-9"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "sg:pub.10.1007/s10586-013-0307-2", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1029943907", 
          "https://doi.org/10.1007/s10586-013-0307-2"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "sg:pub.10.1007/s10586-013-0325-0", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1001588828", 
          "https://doi.org/10.1007/s10586-013-0325-0"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "sg:pub.10.1007/s13174-011-0032-0", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1022893214", 
          "https://doi.org/10.1007/s13174-011-0032-0"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "sg:pub.10.1007/s10586-015-0454-8", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1023182693", 
          "https://doi.org/10.1007/s10586-015-0454-8"
        ], 
        "type": "CreativeWork"
      }
    ], 
    "datePublished": "2017-05-03", 
    "datePublishedReg": "2017-05-03", 
    "description": "Data generation has increased drastically over the past few years due to the rapid development of Internet-based technologies. This period has been called the big data era. Big data offer an emerging paradigm shift in data exploration and utilization. The MapReduce computational paradigm is a well-known framework and is considered the main enabler for the distributed and scalable processing of a large amount of data. However, despite recent efforts toward improving the performance of MapReduce, scheduling MapReduce jobs across multiple nodes has been considered a multi-objective optimization problem. This problem can become increasingly complex when virtualized clusters in cloud computing are used to execute a large number of tasks. This study aims to optimize MapReduce job scheduling based on the completion time and cost of cloud service models. First, the problem is formulated as a multi-objective model. The model consists of two objective functions, namely, (i) completion time and (ii) cost minimization. Second, a scheduling algorithm using earliest finish time scheduling that considers resource allocation and job scheduling in the cloud is proposed. Lastly, experimental results show that the proposed scheduler exhibits better performance than other well-known schedulers, such as FIFO and Fair.", 
    "genre": "article", 
    "id": "sg:pub.10.1007/s11042-017-4685-y", 
    "isAccessibleForFree": false, 
    "isPartOf": [
      {
        "id": "sg:journal.1044869", 
        "issn": [
          "1380-7501", 
          "1573-7721"
        ], 
        "name": "Multimedia Tools and Applications", 
        "publisher": "Springer Nature", 
        "type": "Periodical"
      }, 
      {
        "issueNumber": "8", 
        "type": "PublicationIssue"
      }, 
      {
        "type": "PublicationVolume", 
        "volumeNumber": "77"
      }
    ], 
    "keywords": [
      "MapReduce jobs", 
      "job scheduling", 
      "completion time", 
      "cloud service model", 
      "big data processing", 
      "big data era", 
      "performance of MapReduce", 
      "Internet-based technologies", 
      "MapReduce Job Scheduling", 
      "multi-objective scheduling", 
      "multi-objective optimization problem", 
      "cloud computing", 
      "data era", 
      "big data", 
      "computational paradigm", 
      "data exploration", 
      "scheduling algorithm", 
      "scalable processing", 
      "time scheduling", 
      "data generation", 
      "multiple nodes", 
      "service model", 
      "main enablers", 
      "data processing", 
      "scheduling", 
      "multi-objective model", 
      "resource allocation", 
      "optimization problem", 
      "rapid development", 
      "scheduler", 
      "better performance", 
      "objective function", 
      "cost minimization", 
      "experimental results", 
      "large amount", 
      "MapReduce", 
      "computing", 
      "processing", 
      "large number", 
      "paradigm shift", 
      "algorithm", 
      "cloud", 
      "FIFO", 
      "performance", 
      "task", 
      "enablers", 
      "nodes", 
      "jobs", 
      "recent efforts", 
      "technology", 
      "paradigm", 
      "framework", 
      "allocation", 
      "model", 
      "data", 
      "cost", 
      "minimization", 
      "exploration", 
      "time", 
      "utilization", 
      "era", 
      "clusters", 
      "efforts", 
      "Fair", 
      "generation", 
      "number", 
      "amount", 
      "development", 
      "results", 
      "function", 
      "years", 
      "study", 
      "shift", 
      "period", 
      "problem"
    ], 
    "name": "Multi-objective scheduling of MapReduce jobs in big data processing", 
    "pagination": "9979-9994", 
    "productId": [
      {
        "name": "dimensions_id", 
        "type": "PropertyValue", 
        "value": [
          "pub.1085127057"
        ]
      }, 
      {
        "name": "doi", 
        "type": "PropertyValue", 
        "value": [
          "10.1007/s11042-017-4685-y"
        ]
      }
    ], 
    "sameAs": [
      "https://doi.org/10.1007/s11042-017-4685-y", 
      "https://app.dimensions.ai/details/publication/pub.1085127057"
    ], 
    "sdDataset": "articles", 
    "sdDatePublished": "2022-08-04T17:04", 
    "sdLicense": "https://scigraph.springernature.com/explorer/license/", 
    "sdPublisher": {
      "name": "Springer Nature - SN SciGraph project", 
      "type": "Organization"
    }, 
    "sdSource": "s3://com-springernature-scigraph/baseset/20220804/entities/gbq_results/article/article_719.jsonl", 
    "type": "ScholarlyArticle", 
    "url": "https://doi.org/10.1007/s11042-017-4685-y"
  }
]
 

Download the RDF metadata as:  json-ld nt turtle xml License info

HOW TO GET THIS DATA PROGRAMMATICALLY:

JSON-LD is a popular format for linked data which is fully compatible with JSON.

curl -H 'Accept: application/ld+json' 'https://scigraph.springernature.com/pub.10.1007/s11042-017-4685-y'

N-Triples is a line-based linked data format ideal for batch operations.

curl -H 'Accept: application/n-triples' 'https://scigraph.springernature.com/pub.10.1007/s11042-017-4685-y'

Turtle is a human-readable linked data format.

curl -H 'Accept: text/turtle' 'https://scigraph.springernature.com/pub.10.1007/s11042-017-4685-y'

RDF/XML is a standard XML format for linked data.

curl -H 'Accept: application/rdf+xml' 'https://scigraph.springernature.com/pub.10.1007/s11042-017-4685-y'


 

This table displays all metadata directly associated to this object as RDF triples.

193 TRIPLES      21 PREDICATES      105 URIs      91 LITERALS      6 BLANK NODES

Subject Predicate Object
1 sg:pub.10.1007/s11042-017-4685-y schema:about anzsrc-for:08
2 anzsrc-for:0803
3 schema:author Nf862426b2aeb49ed852648e2492edd07
4 schema:citation sg:pub.10.1007/978-3-642-19294-4_9
5 sg:pub.10.1007/s00778-013-0319-9
6 sg:pub.10.1007/s10586-013-0307-2
7 sg:pub.10.1007/s10586-013-0325-0
8 sg:pub.10.1007/s10586-015-0454-8
9 sg:pub.10.1007/s13174-011-0032-0
10 schema:datePublished 2017-05-03
11 schema:datePublishedReg 2017-05-03
12 schema:description Data generation has increased drastically over the past few years due to the rapid development of Internet-based technologies. This period has been called the big data era. Big data offer an emerging paradigm shift in data exploration and utilization. The MapReduce computational paradigm is a well-known framework and is considered the main enabler for the distributed and scalable processing of a large amount of data. However, despite recent efforts toward improving the performance of MapReduce, scheduling MapReduce jobs across multiple nodes has been considered a multi-objective optimization problem. This problem can become increasingly complex when virtualized clusters in cloud computing are used to execute a large number of tasks. This study aims to optimize MapReduce job scheduling based on the completion time and cost of cloud service models. First, the problem is formulated as a multi-objective model. The model consists of two objective functions, namely, (i) completion time and (ii) cost minimization. Second, a scheduling algorithm using earliest finish time scheduling that considers resource allocation and job scheduling in the cloud is proposed. Lastly, experimental results show that the proposed scheduler exhibits better performance than other well-known schedulers, such as FIFO and Fair.
13 schema:genre article
14 schema:isAccessibleForFree false
15 schema:isPartOf N3fc50138f7714e85a923804ff0870619
16 N670035d4745b40c3bf38a3710ed53434
17 sg:journal.1044869
18 schema:keywords FIFO
19 Fair
20 Internet-based technologies
21 MapReduce
22 MapReduce Job Scheduling
23 MapReduce jobs
24 algorithm
25 allocation
26 amount
27 better performance
28 big data
29 big data era
30 big data processing
31 cloud
32 cloud computing
33 cloud service model
34 clusters
35 completion time
36 computational paradigm
37 computing
38 cost
39 cost minimization
40 data
41 data era
42 data exploration
43 data generation
44 data processing
45 development
46 efforts
47 enablers
48 era
49 experimental results
50 exploration
51 framework
52 function
53 generation
54 job scheduling
55 jobs
56 large amount
57 large number
58 main enablers
59 minimization
60 model
61 multi-objective model
62 multi-objective optimization problem
63 multi-objective scheduling
64 multiple nodes
65 nodes
66 number
67 objective function
68 optimization problem
69 paradigm
70 paradigm shift
71 performance
72 performance of MapReduce
73 period
74 problem
75 processing
76 rapid development
77 recent efforts
78 resource allocation
79 results
80 scalable processing
81 scheduler
82 scheduling
83 scheduling algorithm
84 service model
85 shift
86 study
87 task
88 technology
89 time
90 time scheduling
91 utilization
92 years
93 schema:name Multi-objective scheduling of MapReduce jobs in big data processing
94 schema:pagination 9979-9994
95 schema:productId N7fcaf08598304349b106155c77b67514
96 Nbe97d59f7c164b3ca0aad7a52c32aadd
97 schema:sameAs https://app.dimensions.ai/details/publication/pub.1085127057
98 https://doi.org/10.1007/s11042-017-4685-y
99 schema:sdDatePublished 2022-08-04T17:04
100 schema:sdLicense https://scigraph.springernature.com/explorer/license/
101 schema:sdPublisher Ne19b79de1037499e99c428221adddf93
102 schema:url https://doi.org/10.1007/s11042-017-4685-y
103 sgo:license sg:explorer/license/
104 sgo:sdDataset articles
105 rdf:type schema:ScholarlyArticle
106 N0340078e4d0d4270a6f81292103ebd06 schema:affiliation grid-institutes:grid.10347.31
107 schema:familyName Sakariyah
108 schema:givenName Adewole Kayode
109 rdf:type schema:Person
110 N3fc50138f7714e85a923804ff0870619 schema:issueNumber 8
111 rdf:type schema:PublicationIssue
112 N670035d4745b40c3bf38a3710ed53434 schema:volumeNumber 77
113 rdf:type schema:PublicationVolume
114 N7fcaf08598304349b106155c77b67514 schema:name dimensions_id
115 schema:value pub.1085127057
116 rdf:type schema:PropertyValue
117 N8431f8a0b77842a0af03ac968e14f79f rdf:first sg:person.012750345514.73
118 rdf:rest Nac938ce8ca7c4432bd0089508c1d083b
119 N9951b0ba0a4e498eb68bc17bc7464531 rdf:first sg:person.01203656177.16
120 rdf:rest Nb07910a8b5f44445a01fb26dcdac439a
121 N9ac34b0ce3254f109232fa81da2c11db rdf:first sg:person.0665142504.27
122 rdf:rest N8431f8a0b77842a0af03ac968e14f79f
123 Nac938ce8ca7c4432bd0089508c1d083b rdf:first sg:person.013623112027.80
124 rdf:rest N9951b0ba0a4e498eb68bc17bc7464531
125 Nb07910a8b5f44445a01fb26dcdac439a rdf:first N0340078e4d0d4270a6f81292103ebd06
126 rdf:rest rdf:nil
127 Nbe97d59f7c164b3ca0aad7a52c32aadd schema:name doi
128 schema:value 10.1007/s11042-017-4685-y
129 rdf:type schema:PropertyValue
130 Ne19b79de1037499e99c428221adddf93 schema:name Springer Nature - SN SciGraph project
131 rdf:type schema:Organization
132 Nf862426b2aeb49ed852648e2492edd07 rdf:first sg:person.01035421130.35
133 rdf:rest N9ac34b0ce3254f109232fa81da2c11db
134 anzsrc-for:08 schema:inDefinedTermSet anzsrc-for:
135 schema:name Information and Computing Sciences
136 rdf:type schema:DefinedTerm
137 anzsrc-for:0803 schema:inDefinedTermSet anzsrc-for:
138 schema:name Computer Software
139 rdf:type schema:DefinedTerm
140 sg:journal.1044869 schema:issn 1380-7501
141 1573-7721
142 schema:name Multimedia Tools and Applications
143 schema:publisher Springer Nature
144 rdf:type schema:Periodical
145 sg:person.01035421130.35 schema:affiliation grid-institutes:grid.10347.31
146 schema:familyName Hashem
147 schema:givenName Ibrahim Abaker Targio
148 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01035421130.35
149 rdf:type schema:Person
150 sg:person.01203656177.16 schema:affiliation grid-institutes:grid.412813.d
151 schema:familyName Sangaiah
152 schema:givenName Arun Kumar
153 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01203656177.16
154 rdf:type schema:Person
155 sg:person.012750345514.73 schema:affiliation grid-institutes:grid.10347.31
156 schema:familyName Marjani
157 schema:givenName Mohsen
158 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.012750345514.73
159 rdf:type schema:Person
160 sg:person.013623112027.80 schema:affiliation grid-institutes:grid.10347.31
161 schema:familyName Gani
162 schema:givenName Abdullah
163 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.013623112027.80
164 rdf:type schema:Person
165 sg:person.0665142504.27 schema:affiliation grid-institutes:grid.10347.31
166 schema:familyName Anuar
167 schema:givenName Nor Badrul
168 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.0665142504.27
169 rdf:type schema:Person
170 sg:pub.10.1007/978-3-642-19294-4_9 schema:sameAs https://app.dimensions.ai/details/publication/pub.1022039554
171 https://doi.org/10.1007/978-3-642-19294-4_9
172 rdf:type schema:CreativeWork
173 sg:pub.10.1007/s00778-013-0319-9 schema:sameAs https://app.dimensions.ai/details/publication/pub.1002332096
174 https://doi.org/10.1007/s00778-013-0319-9
175 rdf:type schema:CreativeWork
176 sg:pub.10.1007/s10586-013-0307-2 schema:sameAs https://app.dimensions.ai/details/publication/pub.1029943907
177 https://doi.org/10.1007/s10586-013-0307-2
178 rdf:type schema:CreativeWork
179 sg:pub.10.1007/s10586-013-0325-0 schema:sameAs https://app.dimensions.ai/details/publication/pub.1001588828
180 https://doi.org/10.1007/s10586-013-0325-0
181 rdf:type schema:CreativeWork
182 sg:pub.10.1007/s10586-015-0454-8 schema:sameAs https://app.dimensions.ai/details/publication/pub.1023182693
183 https://doi.org/10.1007/s10586-015-0454-8
184 rdf:type schema:CreativeWork
185 sg:pub.10.1007/s13174-011-0032-0 schema:sameAs https://app.dimensions.ai/details/publication/pub.1022893214
186 https://doi.org/10.1007/s13174-011-0032-0
187 rdf:type schema:CreativeWork
188 grid-institutes:grid.10347.31 schema:alternateName Faculty of Computer Science and Information Technology, University of Malaya, Kuala Lumpur, Malaysia
189 schema:name Faculty of Computer Science and Information Technology, University of Malaya, Kuala Lumpur, Malaysia
190 rdf:type schema:Organization
191 grid-institutes:grid.412813.d schema:alternateName School of Computing Science and Engineering, VIT University, 632014, Vellore, India
192 schema:name School of Computing Science and Engineering, VIT University, 632014, Vellore, India
193 rdf:type schema:Organization
 




Preview window. Press ESC to close (or click here)


...