An MPI Implementation of a Self-Submitting Parallel Job Queue View Full Text


Ontology type: schema:ScholarlyArticle      Open Access: True


Article Info

DATE

2012-08

AUTHORS

John M. Neuberger, Nándor Sieben, James W. Swift

ABSTRACT

We present a simple and easy to apply methodology for using high-level self-submitting parallel job queues in an MPI environment. Using C++, we implemented a library of functions, MPQueue, both for testing our concepts and for use in real applications. In particular, we have applied our ideas toward solving computational combinatorics problems and for finding bifurcation diagrams of solutions of partial differential equations (PDE). Our method is general and can be applied in many situations without a lot of programming effort. The key idea is that workers themselves can easily submit new jobs to the currently running job queue. Our applications involve complicated data structures, so we employ serialization to allow data to be effortlessly passed between nodes. Using our library, one can solve large problems in parallel without being an expert in MPI. We demonstrate our methodology and the features of the library with several example programs, and give some results from our current PDE research. We show that our techniques are efficient and effective via overhead and scaling experiments. More... »

PAGES

443-464

References to SciGraph publications

  • 2006. Modernizing the C++ Interface to MPI in RECENT ADVANCES IN PARALLEL VIRTUAL MACHINE AND MESSAGE PASSING INTERFACE
  • Identifiers

    URI

    http://scigraph.springernature.com/pub.10.1007/s10766-012-0195-8

    DOI

    http://dx.doi.org/10.1007/s10766-012-0195-8

    DIMENSIONS

    https://app.dimensions.ai/details/publication/pub.1008746693


    Indexing Status Check whether this publication has been indexed by Scopus and Web Of Science using the SN Indexing Status Tool
    Incoming Citations Browse incoming citations for this publication using opencitations.net

    JSON-LD is the canonical representation for SciGraph data.

    TIP: You can open this SciGraph record using an external JSON-LD service: JSON-LD Playground Google SDTT

    [
      {
        "@context": "https://springernature.github.io/scigraph/jsonld/sgcontext.json", 
        "about": [
          {
            "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/0803", 
            "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
            "name": "Computer Software", 
            "type": "DefinedTerm"
          }, 
          {
            "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/08", 
            "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
            "name": "Information and Computing Sciences", 
            "type": "DefinedTerm"
          }
        ], 
        "author": [
          {
            "affiliation": {
              "alternateName": "Northern Arizona University", 
              "id": "https://www.grid.ac/institutes/grid.261120.6", 
              "name": [
                "Department of Mathematics and Statistics, Northern Arizona University, 86011-5717, Flagstaff, AZ, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Neuberger", 
            "givenName": "John M.", 
            "id": "sg:person.011776401067.11", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.011776401067.11"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "Northern Arizona University", 
              "id": "https://www.grid.ac/institutes/grid.261120.6", 
              "name": [
                "Department of Mathematics and Statistics, Northern Arizona University, 86011-5717, Flagstaff, AZ, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Sieben", 
            "givenName": "N\u00e1ndor", 
            "id": "sg:person.0705654574.72", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.0705654574.72"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "Northern Arizona University", 
              "id": "https://www.grid.ac/institutes/grid.261120.6", 
              "name": [
                "Department of Mathematics and Statistics, Northern Arizona University, 86011-5717, Flagstaff, AZ, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Swift", 
            "givenName": "James W.", 
            "id": "sg:person.010042617041.40", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.010042617041.40"
            ], 
            "type": "Person"
          }
        ], 
        "citation": [
          {
            "id": "https://doi.org/10.1006/jpdc.1998.1450", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1007274167"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/11846802_39", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1011136795", 
              "https://doi.org/10.1007/11846802_39"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/11846802_39", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1011136795", 
              "https://doi.org/10.1007/11846802_39"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1006/jpdc.1996.0107", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1043333435"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1016/j.disc.2007.12.043", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1043659847"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1515/mcma.2011.008", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1052729094"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/99.660313", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1061251961"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1137/050640485", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1062846549"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1142/s0218127409024293", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1062955707"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1177/109434200101500205", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1063976903"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1177/109434200101500205", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1063976903"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1287/ijoc.7.4.365", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1064707557"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/ccgrid.2005.1558672", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1093424724"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/mpidc.1996.534097", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1094381636"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/clustr.2004.1392625", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1094528014"
            ], 
            "type": "CreativeWork"
          }
        ], 
        "datePublished": "2012-08", 
        "datePublishedReg": "2012-08-01", 
        "description": "We present a simple and easy to apply methodology for using high-level self-submitting parallel job queues in an MPI environment. Using C++, we implemented a library of functions, MPQueue, both for testing our concepts and for use in real applications. In particular, we have applied our ideas toward solving computational combinatorics problems and for finding bifurcation diagrams of solutions of partial differential equations (PDE). Our method is general and can be applied in many situations without a lot of programming effort. The key idea is that workers themselves can easily submit new jobs to the currently running job queue. Our applications involve complicated data structures, so we employ serialization to allow data to be effortlessly passed between nodes. Using our library, one can solve large problems in parallel without being an expert in MPI. We demonstrate our methodology and the features of the library with several example programs, and give some results from our current PDE research. We show that our techniques are efficient and effective via overhead and scaling experiments.", 
        "genre": "research_article", 
        "id": "sg:pub.10.1007/s10766-012-0195-8", 
        "inLanguage": [
          "en"
        ], 
        "isAccessibleForFree": true, 
        "isPartOf": [
          {
            "id": "sg:journal.1126246", 
            "issn": [
              "0885-7458", 
              "1573-7640"
            ], 
            "name": "International Journal of Parallel Programming", 
            "type": "Periodical"
          }, 
          {
            "issueNumber": "4", 
            "type": "PublicationIssue"
          }, 
          {
            "type": "PublicationVolume", 
            "volumeNumber": "40"
          }
        ], 
        "name": "An MPI Implementation of a Self-Submitting Parallel Job Queue", 
        "pagination": "443-464", 
        "productId": [
          {
            "name": "readcube_id", 
            "type": "PropertyValue", 
            "value": [
              "29ac3c3a214d159ecf4c858fe481089314527cf99d2febbd4f56217b62f0c38c"
            ]
          }, 
          {
            "name": "doi", 
            "type": "PropertyValue", 
            "value": [
              "10.1007/s10766-012-0195-8"
            ]
          }, 
          {
            "name": "dimensions_id", 
            "type": "PropertyValue", 
            "value": [
              "pub.1008746693"
            ]
          }
        ], 
        "sameAs": [
          "https://doi.org/10.1007/s10766-012-0195-8", 
          "https://app.dimensions.ai/details/publication/pub.1008746693"
        ], 
        "sdDataset": "articles", 
        "sdDatePublished": "2019-04-11T09:54", 
        "sdLicense": "https://scigraph.springernature.com/explorer/license/", 
        "sdPublisher": {
          "name": "Springer Nature - SN SciGraph project", 
          "type": "Organization"
        }, 
        "sdSource": "s3://com-uberresearch-data-dimensions-target-20181106-alternative/cleanup/v134/2549eaecd7973599484d7c17b260dba0a4ecb94b/merge/v9/a6c9fde33151104705d4d7ff012ea9563521a3ce/jats-lookup/v90/0000000347_0000000347/records_89798_00000000.jsonl", 
        "type": "ScholarlyArticle", 
        "url": "https://link.springer.com/10.1007%2Fs10766-012-0195-8"
      }
    ]
     

    Download the RDF metadata as:  json-ld nt turtle xml License info

    HOW TO GET THIS DATA PROGRAMMATICALLY:

    JSON-LD is a popular format for linked data which is fully compatible with JSON.

    curl -H 'Accept: application/ld+json' 'https://scigraph.springernature.com/pub.10.1007/s10766-012-0195-8'

    N-Triples is a line-based linked data format ideal for batch operations.

    curl -H 'Accept: application/n-triples' 'https://scigraph.springernature.com/pub.10.1007/s10766-012-0195-8'

    Turtle is a human-readable linked data format.

    curl -H 'Accept: text/turtle' 'https://scigraph.springernature.com/pub.10.1007/s10766-012-0195-8'

    RDF/XML is a standard XML format for linked data.

    curl -H 'Accept: application/rdf+xml' 'https://scigraph.springernature.com/pub.10.1007/s10766-012-0195-8'


     

    This table displays all metadata directly associated to this object as RDF triples.

    115 TRIPLES      21 PREDICATES      40 URIs      19 LITERALS      7 BLANK NODES

    Subject Predicate Object
    1 sg:pub.10.1007/s10766-012-0195-8 schema:about anzsrc-for:08
    2 anzsrc-for:0803
    3 schema:author N364551d73e0c45718a844d500f6efc1f
    4 schema:citation sg:pub.10.1007/11846802_39
    5 https://doi.org/10.1006/jpdc.1996.0107
    6 https://doi.org/10.1006/jpdc.1998.1450
    7 https://doi.org/10.1016/j.disc.2007.12.043
    8 https://doi.org/10.1109/99.660313
    9 https://doi.org/10.1109/ccgrid.2005.1558672
    10 https://doi.org/10.1109/clustr.2004.1392625
    11 https://doi.org/10.1109/mpidc.1996.534097
    12 https://doi.org/10.1137/050640485
    13 https://doi.org/10.1142/s0218127409024293
    14 https://doi.org/10.1177/109434200101500205
    15 https://doi.org/10.1287/ijoc.7.4.365
    16 https://doi.org/10.1515/mcma.2011.008
    17 schema:datePublished 2012-08
    18 schema:datePublishedReg 2012-08-01
    19 schema:description We present a simple and easy to apply methodology for using high-level self-submitting parallel job queues in an MPI environment. Using C++, we implemented a library of functions, MPQueue, both for testing our concepts and for use in real applications. In particular, we have applied our ideas toward solving computational combinatorics problems and for finding bifurcation diagrams of solutions of partial differential equations (PDE). Our method is general and can be applied in many situations without a lot of programming effort. The key idea is that workers themselves can easily submit new jobs to the currently running job queue. Our applications involve complicated data structures, so we employ serialization to allow data to be effortlessly passed between nodes. Using our library, one can solve large problems in parallel without being an expert in MPI. We demonstrate our methodology and the features of the library with several example programs, and give some results from our current PDE research. We show that our techniques are efficient and effective via overhead and scaling experiments.
    20 schema:genre research_article
    21 schema:inLanguage en
    22 schema:isAccessibleForFree true
    23 schema:isPartOf N9d7485802e6e4461aa21aa69b798f196
    24 Nc3227ac726184c7eb11d7ee00f584f40
    25 sg:journal.1126246
    26 schema:name An MPI Implementation of a Self-Submitting Parallel Job Queue
    27 schema:pagination 443-464
    28 schema:productId N047d2d56a26e456aa6b7575ac3024c57
    29 N8244346d1ba7477582debb417ee66bcc
    30 Ne77ce0c10526476fb8ca55296f0e1d2e
    31 schema:sameAs https://app.dimensions.ai/details/publication/pub.1008746693
    32 https://doi.org/10.1007/s10766-012-0195-8
    33 schema:sdDatePublished 2019-04-11T09:54
    34 schema:sdLicense https://scigraph.springernature.com/explorer/license/
    35 schema:sdPublisher Na86dc4a18d664b07acfaa1909149a96c
    36 schema:url https://link.springer.com/10.1007%2Fs10766-012-0195-8
    37 sgo:license sg:explorer/license/
    38 sgo:sdDataset articles
    39 rdf:type schema:ScholarlyArticle
    40 N047d2d56a26e456aa6b7575ac3024c57 schema:name dimensions_id
    41 schema:value pub.1008746693
    42 rdf:type schema:PropertyValue
    43 N063c706e47eb46a8b67eb15fe8e15186 rdf:first sg:person.010042617041.40
    44 rdf:rest rdf:nil
    45 N364551d73e0c45718a844d500f6efc1f rdf:first sg:person.011776401067.11
    46 rdf:rest N75652c51acde4ed68446288684938892
    47 N75652c51acde4ed68446288684938892 rdf:first sg:person.0705654574.72
    48 rdf:rest N063c706e47eb46a8b67eb15fe8e15186
    49 N8244346d1ba7477582debb417ee66bcc schema:name readcube_id
    50 schema:value 29ac3c3a214d159ecf4c858fe481089314527cf99d2febbd4f56217b62f0c38c
    51 rdf:type schema:PropertyValue
    52 N9d7485802e6e4461aa21aa69b798f196 schema:issueNumber 4
    53 rdf:type schema:PublicationIssue
    54 Na86dc4a18d664b07acfaa1909149a96c schema:name Springer Nature - SN SciGraph project
    55 rdf:type schema:Organization
    56 Nc3227ac726184c7eb11d7ee00f584f40 schema:volumeNumber 40
    57 rdf:type schema:PublicationVolume
    58 Ne77ce0c10526476fb8ca55296f0e1d2e schema:name doi
    59 schema:value 10.1007/s10766-012-0195-8
    60 rdf:type schema:PropertyValue
    61 anzsrc-for:08 schema:inDefinedTermSet anzsrc-for:
    62 schema:name Information and Computing Sciences
    63 rdf:type schema:DefinedTerm
    64 anzsrc-for:0803 schema:inDefinedTermSet anzsrc-for:
    65 schema:name Computer Software
    66 rdf:type schema:DefinedTerm
    67 sg:journal.1126246 schema:issn 0885-7458
    68 1573-7640
    69 schema:name International Journal of Parallel Programming
    70 rdf:type schema:Periodical
    71 sg:person.010042617041.40 schema:affiliation https://www.grid.ac/institutes/grid.261120.6
    72 schema:familyName Swift
    73 schema:givenName James W.
    74 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.010042617041.40
    75 rdf:type schema:Person
    76 sg:person.011776401067.11 schema:affiliation https://www.grid.ac/institutes/grid.261120.6
    77 schema:familyName Neuberger
    78 schema:givenName John M.
    79 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.011776401067.11
    80 rdf:type schema:Person
    81 sg:person.0705654574.72 schema:affiliation https://www.grid.ac/institutes/grid.261120.6
    82 schema:familyName Sieben
    83 schema:givenName Nándor
    84 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.0705654574.72
    85 rdf:type schema:Person
    86 sg:pub.10.1007/11846802_39 schema:sameAs https://app.dimensions.ai/details/publication/pub.1011136795
    87 https://doi.org/10.1007/11846802_39
    88 rdf:type schema:CreativeWork
    89 https://doi.org/10.1006/jpdc.1996.0107 schema:sameAs https://app.dimensions.ai/details/publication/pub.1043333435
    90 rdf:type schema:CreativeWork
    91 https://doi.org/10.1006/jpdc.1998.1450 schema:sameAs https://app.dimensions.ai/details/publication/pub.1007274167
    92 rdf:type schema:CreativeWork
    93 https://doi.org/10.1016/j.disc.2007.12.043 schema:sameAs https://app.dimensions.ai/details/publication/pub.1043659847
    94 rdf:type schema:CreativeWork
    95 https://doi.org/10.1109/99.660313 schema:sameAs https://app.dimensions.ai/details/publication/pub.1061251961
    96 rdf:type schema:CreativeWork
    97 https://doi.org/10.1109/ccgrid.2005.1558672 schema:sameAs https://app.dimensions.ai/details/publication/pub.1093424724
    98 rdf:type schema:CreativeWork
    99 https://doi.org/10.1109/clustr.2004.1392625 schema:sameAs https://app.dimensions.ai/details/publication/pub.1094528014
    100 rdf:type schema:CreativeWork
    101 https://doi.org/10.1109/mpidc.1996.534097 schema:sameAs https://app.dimensions.ai/details/publication/pub.1094381636
    102 rdf:type schema:CreativeWork
    103 https://doi.org/10.1137/050640485 schema:sameAs https://app.dimensions.ai/details/publication/pub.1062846549
    104 rdf:type schema:CreativeWork
    105 https://doi.org/10.1142/s0218127409024293 schema:sameAs https://app.dimensions.ai/details/publication/pub.1062955707
    106 rdf:type schema:CreativeWork
    107 https://doi.org/10.1177/109434200101500205 schema:sameAs https://app.dimensions.ai/details/publication/pub.1063976903
    108 rdf:type schema:CreativeWork
    109 https://doi.org/10.1287/ijoc.7.4.365 schema:sameAs https://app.dimensions.ai/details/publication/pub.1064707557
    110 rdf:type schema:CreativeWork
    111 https://doi.org/10.1515/mcma.2011.008 schema:sameAs https://app.dimensions.ai/details/publication/pub.1052729094
    112 rdf:type schema:CreativeWork
    113 https://www.grid.ac/institutes/grid.261120.6 schema:alternateName Northern Arizona University
    114 schema:name Department of Mathematics and Statistics, Northern Arizona University, 86011-5717, Flagstaff, AZ, USA
    115 rdf:type schema:Organization
     




    Preview window. Press ESC to close (or click here)


    ...