Accelerating the convergence of the back-propagation method View Full Text


Ontology type: schema:ScholarlyArticle     


Article Info

DATE

1988-09

AUTHORS

T. P. Vogl, J. K. Mangis, A. K. Rigler, W. T. Zink, D. L. Alkon

ABSTRACT

The utility of the back-propagation method in establishing suitable weights in a distributed adaptive network has been demonstrated repeatedly. Unfortunately, in many applications, the number of iterations required before convergence can be large. Modifications to the back-propagation algorithm described by Rumelhart et al. (1986) can greatly accelerate convergence. The modifications consist of three changes:1) instead of updating the network weights after each pattern is presented to the network, the network is updated only after the entire repertoire of patterns to be learned has been presented to the network, at which time the algebraic sums of all the weight changes are applied:2) instead of keeping η, the “learning rate” (i.e., the multiplier on the step size) constant, it is varied dynamically so that the algorithm utilizes a near-optimum η, as determined by the local optimization topography; and3) the momentum factor α is set to zero when, as signified by a failure of a step to reduce the total error, the information inherent in prior steps is more likely to be misleading than beneficial. Only after the network takes a useful step, i.e., one that reduces the total error, does α again assume a non-zero value. Considering the selection of weights in neural nets as a problem in classical nonlinear optimization theory, the rationale for algorithms seeking only those weights that produce the globally minimum error is reviewed and rejected. More... »

PAGES

257-263

References to SciGraph publications

  • 1985. A Stochastic Approach to Global Optimization in COMPUTATIONAL MATHEMATICAL PROGRAMMING
  • 1986-01. Convergence of an annealing algorithm in MATHEMATICAL PROGRAMMING
  • 1959-10. On a successive transformation of probability distribution and its application to the analysis of the optimum gradient method in ANNALS OF THE INSTITUTE OF STATISTICAL MATHEMATICS
  • 1983-07. Learning in a marine snail. in SCIENTIFIC AMERICAN
  • Identifiers

    URI

    http://scigraph.springernature.com/pub.10.1007/bf00332914

    DOI

    http://dx.doi.org/10.1007/bf00332914

    DIMENSIONS

    https://app.dimensions.ai/details/publication/pub.1043901229


    Indexing Status Check whether this publication has been indexed by Scopus and Web Of Science using the SN Indexing Status Tool
    Incoming Citations Browse incoming citations for this publication using opencitations.net

    JSON-LD is the canonical representation for SciGraph data.

    TIP: You can open this SciGraph record using an external JSON-LD service: JSON-LD Playground Google SDTT

    [
      {
        "@context": "https://springernature.github.io/scigraph/jsonld/sgcontext.json", 
        "about": [
          {
            "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/0801", 
            "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
            "name": "Artificial Intelligence and Image Processing", 
            "type": "DefinedTerm"
          }, 
          {
            "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/08", 
            "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
            "name": "Information and Computing Sciences", 
            "type": "DefinedTerm"
          }
        ], 
        "author": [
          {
            "affiliation": {
              "name": [
                "Environmental Research Institute of Michigan, 1501 Wilson Boulevard, 22209, Arlington, VA, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Vogl", 
            "givenName": "T. P.", 
            "id": "sg:person.01137660156.00", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01137660156.00"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "name": [
                "Environmental Research Institute of Michigan, 1501 Wilson Boulevard, 22209, Arlington, VA, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Mangis", 
            "givenName": "J. K.", 
            "id": "sg:person.01355251166.56", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01355251166.56"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "University of Missouri", 
              "id": "https://www.grid.ac/institutes/grid.134936.a", 
              "name": [
                "Computer Science Department, University of Missouri, 65401, Rolla, MO, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Rigler", 
            "givenName": "A. K.", 
            "id": "sg:person.01071544756.05", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01071544756.05"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "name": [
                "Environmental Research Institute of Michigan, 1501 Wilson Boulevard, 22209, Arlington, VA, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Zink", 
            "givenName": "W. T.", 
            "id": "sg:person.0677760266.38", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.0677760266.38"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "name": [
                "Neural Systems Section, National Institute of Neurological and Communicative Disorders and Stroke, NIH, 9000 Rockville Pike, 20892, Bethesda, MD, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Alkon", 
            "givenName": "D. L.", 
            "id": "sg:person.0720111551.14", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.0720111551.14"
            ], 
            "type": "Person"
          }
        ], 
        "citation": [
          {
            "id": "sg:pub.10.1007/978-3-642-82450-0_10", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1022559771", 
              "https://doi.org/10.1007/978-3-642-82450-0_10"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1145/322609.322794", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1045073789"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1145/52964.52980", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1049493170"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/bf01582166", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1049532067", 
              "https://doi.org/10.1007/bf01582166"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/bf01831719", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1052300639", 
              "https://doi.org/10.1007/bf01831719"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1038/scientificamerican0783-70", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1056571521", 
              "https://doi.org/10.1038/scientificamerican0783-70"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1126/science.6093258", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1062634105"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1137/1028106", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1062862389"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1137/1028157", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1062862440"
            ], 
            "type": "CreativeWork"
          }
        ], 
        "datePublished": "1988-09", 
        "datePublishedReg": "1988-09-01", 
        "description": "The utility of the back-propagation method in establishing suitable weights in a distributed adaptive network has been demonstrated repeatedly. Unfortunately, in many applications, the number of iterations required before convergence can be large. Modifications to the back-propagation algorithm described by Rumelhart et al. (1986) can greatly accelerate convergence. The modifications consist of three changes:1) instead of updating the network weights after each pattern is presented to the network, the network is updated only after the entire repertoire of patterns to be learned has been presented to the network, at which time the algebraic sums of all the weight changes are applied:2) instead of keeping \u03b7, the \u201clearning rate\u201d (i.e., the multiplier on the step size) constant, it is varied dynamically so that the algorithm utilizes a near-optimum \u03b7, as determined by the local optimization topography; and3) the momentum factor \u03b1 is set to zero when, as signified by a failure of a step to reduce the total error, the information inherent in prior steps is more likely to be misleading than beneficial. Only after the network takes a useful step, i.e., one that reduces the total error, does \u03b1 again assume a non-zero value. Considering the selection of weights in neural nets as a problem in classical nonlinear optimization theory, the rationale for algorithms seeking only those weights that produce the globally minimum error is reviewed and rejected.", 
        "genre": "research_article", 
        "id": "sg:pub.10.1007/bf00332914", 
        "inLanguage": [
          "en"
        ], 
        "isAccessibleForFree": false, 
        "isPartOf": [
          {
            "id": "sg:journal.1081741", 
            "issn": [
              "0340-1200", 
              "1432-0770"
            ], 
            "name": "Biological Cybernetics", 
            "type": "Periodical"
          }, 
          {
            "issueNumber": "4-5", 
            "type": "PublicationIssue"
          }, 
          {
            "type": "PublicationVolume", 
            "volumeNumber": "59"
          }
        ], 
        "name": "Accelerating the convergence of the back-propagation method", 
        "pagination": "257-263", 
        "productId": [
          {
            "name": "doi", 
            "type": "PropertyValue", 
            "value": [
              "10.1007/bf00332914"
            ]
          }, 
          {
            "name": "readcube_id", 
            "type": "PropertyValue", 
            "value": [
              "d11e63ab4607da6de523f4c877c46560b03ef873c63e635a50b90589d29606fe"
            ]
          }, 
          {
            "name": "dimensions_id", 
            "type": "PropertyValue", 
            "value": [
              "pub.1043901229"
            ]
          }
        ], 
        "sameAs": [
          "https://doi.org/10.1007/bf00332914", 
          "https://app.dimensions.ai/details/publication/pub.1043901229"
        ], 
        "sdDataset": "articles", 
        "sdDatePublished": "2019-04-15T08:48", 
        "sdLicense": "https://scigraph.springernature.com/explorer/license/", 
        "sdPublisher": {
          "name": "Springer Nature - SN SciGraph project", 
          "type": "Organization"
        }, 
        "sdSource": "s3://com-uberresearch-data-dimensions-target-20181106-alternative/cleanup/v134/2549eaecd7973599484d7c17b260dba0a4ecb94b/merge/v9/a6c9fde33151104705d4d7ff012ea9563521a3ce/jats-lookup/v90/0000000374_0000000374/records_119717_00000001.jsonl", 
        "type": "ScholarlyArticle", 
        "url": "http://link.springer.com/10.1007/BF00332914"
      }
    ]
     

    Download the RDF metadata as:  json-ld nt turtle xml License info

    HOW TO GET THIS DATA PROGRAMMATICALLY:

    JSON-LD is a popular format for linked data which is fully compatible with JSON.

    curl -H 'Accept: application/ld+json' 'https://scigraph.springernature.com/pub.10.1007/bf00332914'

    N-Triples is a line-based linked data format ideal for batch operations.

    curl -H 'Accept: application/n-triples' 'https://scigraph.springernature.com/pub.10.1007/bf00332914'

    Turtle is a human-readable linked data format.

    curl -H 'Accept: text/turtle' 'https://scigraph.springernature.com/pub.10.1007/bf00332914'

    RDF/XML is a standard XML format for linked data.

    curl -H 'Accept: application/rdf+xml' 'https://scigraph.springernature.com/pub.10.1007/bf00332914'


     

    This table displays all metadata directly associated to this object as RDF triples.

    128 TRIPLES      21 PREDICATES      36 URIs      19 LITERALS      7 BLANK NODES

    Subject Predicate Object
    1 sg:pub.10.1007/bf00332914 schema:about anzsrc-for:08
    2 anzsrc-for:0801
    3 schema:author Nd6c07c40deba44d8a403b6b0d8fa2e6d
    4 schema:citation sg:pub.10.1007/978-3-642-82450-0_10
    5 sg:pub.10.1007/bf01582166
    6 sg:pub.10.1007/bf01831719
    7 sg:pub.10.1038/scientificamerican0783-70
    8 https://doi.org/10.1126/science.6093258
    9 https://doi.org/10.1137/1028106
    10 https://doi.org/10.1137/1028157
    11 https://doi.org/10.1145/322609.322794
    12 https://doi.org/10.1145/52964.52980
    13 schema:datePublished 1988-09
    14 schema:datePublishedReg 1988-09-01
    15 schema:description The utility of the back-propagation method in establishing suitable weights in a distributed adaptive network has been demonstrated repeatedly. Unfortunately, in many applications, the number of iterations required before convergence can be large. Modifications to the back-propagation algorithm described by Rumelhart et al. (1986) can greatly accelerate convergence. The modifications consist of three changes:1) instead of updating the network weights after each pattern is presented to the network, the network is updated only after the entire repertoire of patterns to be learned has been presented to the network, at which time the algebraic sums of all the weight changes are applied:2) instead of keeping η, the “learning rate” (i.e., the multiplier on the step size) constant, it is varied dynamically so that the algorithm utilizes a near-optimum η, as determined by the local optimization topography; and3) the momentum factor α is set to zero when, as signified by a failure of a step to reduce the total error, the information inherent in prior steps is more likely to be misleading than beneficial. Only after the network takes a useful step, i.e., one that reduces the total error, does α again assume a non-zero value. Considering the selection of weights in neural nets as a problem in classical nonlinear optimization theory, the rationale for algorithms seeking only those weights that produce the globally minimum error is reviewed and rejected.
    16 schema:genre research_article
    17 schema:inLanguage en
    18 schema:isAccessibleForFree false
    19 schema:isPartOf N52866f2fb8a7421f8f77aa03bccc512b
    20 Na1134a66903b429bb13ec7a983310434
    21 sg:journal.1081741
    22 schema:name Accelerating the convergence of the back-propagation method
    23 schema:pagination 257-263
    24 schema:productId N0f4361743bbf46a6be8bf5b83144718d
    25 N709bb47cb3f54fbbbd842ba15426e6fc
    26 Ndf712531c6ee46ccb1f73ee64637d8c8
    27 schema:sameAs https://app.dimensions.ai/details/publication/pub.1043901229
    28 https://doi.org/10.1007/bf00332914
    29 schema:sdDatePublished 2019-04-15T08:48
    30 schema:sdLicense https://scigraph.springernature.com/explorer/license/
    31 schema:sdPublisher N8992d0d80b5c42a79ca8a6b358bdcdb9
    32 schema:url http://link.springer.com/10.1007/BF00332914
    33 sgo:license sg:explorer/license/
    34 sgo:sdDataset articles
    35 rdf:type schema:ScholarlyArticle
    36 N0f4361743bbf46a6be8bf5b83144718d schema:name dimensions_id
    37 schema:value pub.1043901229
    38 rdf:type schema:PropertyValue
    39 N284c4174d3a54b248bddf9fb3f2503f3 schema:name Environmental Research Institute of Michigan, 1501 Wilson Boulevard, 22209, Arlington, VA, USA
    40 rdf:type schema:Organization
    41 N52866f2fb8a7421f8f77aa03bccc512b schema:volumeNumber 59
    42 rdf:type schema:PublicationVolume
    43 N5b381f3764f34b9298a2791bef692eaa schema:name Environmental Research Institute of Michigan, 1501 Wilson Boulevard, 22209, Arlington, VA, USA
    44 rdf:type schema:Organization
    45 N709bb47cb3f54fbbbd842ba15426e6fc schema:name doi
    46 schema:value 10.1007/bf00332914
    47 rdf:type schema:PropertyValue
    48 N8992d0d80b5c42a79ca8a6b358bdcdb9 schema:name Springer Nature - SN SciGraph project
    49 rdf:type schema:Organization
    50 N8a151a0e4cc94b8298d8edaaef1c0251 schema:name Neural Systems Section, National Institute of Neurological and Communicative Disorders and Stroke, NIH, 9000 Rockville Pike, 20892, Bethesda, MD, USA
    51 rdf:type schema:Organization
    52 N962dfd52a42241d39d64be55690b7a74 rdf:first sg:person.01355251166.56
    53 rdf:rest Ne4f727da82e04354b343e0051d855827
    54 Na1134a66903b429bb13ec7a983310434 schema:issueNumber 4-5
    55 rdf:type schema:PublicationIssue
    56 Nd6c07c40deba44d8a403b6b0d8fa2e6d rdf:first sg:person.01137660156.00
    57 rdf:rest N962dfd52a42241d39d64be55690b7a74
    58 Ndf712531c6ee46ccb1f73ee64637d8c8 schema:name readcube_id
    59 schema:value d11e63ab4607da6de523f4c877c46560b03ef873c63e635a50b90589d29606fe
    60 rdf:type schema:PropertyValue
    61 Ne181b1a3400c4cafbb4ad2d0f8c3ee61 rdf:first sg:person.0677760266.38
    62 rdf:rest Ne198bc3da7cc461a9bfac3088b6b3983
    63 Ne198bc3da7cc461a9bfac3088b6b3983 rdf:first sg:person.0720111551.14
    64 rdf:rest rdf:nil
    65 Ne4f727da82e04354b343e0051d855827 rdf:first sg:person.01071544756.05
    66 rdf:rest Ne181b1a3400c4cafbb4ad2d0f8c3ee61
    67 Neba2740cde1745afa81f360d42161d32 schema:name Environmental Research Institute of Michigan, 1501 Wilson Boulevard, 22209, Arlington, VA, USA
    68 rdf:type schema:Organization
    69 anzsrc-for:08 schema:inDefinedTermSet anzsrc-for:
    70 schema:name Information and Computing Sciences
    71 rdf:type schema:DefinedTerm
    72 anzsrc-for:0801 schema:inDefinedTermSet anzsrc-for:
    73 schema:name Artificial Intelligence and Image Processing
    74 rdf:type schema:DefinedTerm
    75 sg:journal.1081741 schema:issn 0340-1200
    76 1432-0770
    77 schema:name Biological Cybernetics
    78 rdf:type schema:Periodical
    79 sg:person.01071544756.05 schema:affiliation https://www.grid.ac/institutes/grid.134936.a
    80 schema:familyName Rigler
    81 schema:givenName A. K.
    82 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01071544756.05
    83 rdf:type schema:Person
    84 sg:person.01137660156.00 schema:affiliation Neba2740cde1745afa81f360d42161d32
    85 schema:familyName Vogl
    86 schema:givenName T. P.
    87 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01137660156.00
    88 rdf:type schema:Person
    89 sg:person.01355251166.56 schema:affiliation N5b381f3764f34b9298a2791bef692eaa
    90 schema:familyName Mangis
    91 schema:givenName J. K.
    92 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01355251166.56
    93 rdf:type schema:Person
    94 sg:person.0677760266.38 schema:affiliation N284c4174d3a54b248bddf9fb3f2503f3
    95 schema:familyName Zink
    96 schema:givenName W. T.
    97 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.0677760266.38
    98 rdf:type schema:Person
    99 sg:person.0720111551.14 schema:affiliation N8a151a0e4cc94b8298d8edaaef1c0251
    100 schema:familyName Alkon
    101 schema:givenName D. L.
    102 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.0720111551.14
    103 rdf:type schema:Person
    104 sg:pub.10.1007/978-3-642-82450-0_10 schema:sameAs https://app.dimensions.ai/details/publication/pub.1022559771
    105 https://doi.org/10.1007/978-3-642-82450-0_10
    106 rdf:type schema:CreativeWork
    107 sg:pub.10.1007/bf01582166 schema:sameAs https://app.dimensions.ai/details/publication/pub.1049532067
    108 https://doi.org/10.1007/bf01582166
    109 rdf:type schema:CreativeWork
    110 sg:pub.10.1007/bf01831719 schema:sameAs https://app.dimensions.ai/details/publication/pub.1052300639
    111 https://doi.org/10.1007/bf01831719
    112 rdf:type schema:CreativeWork
    113 sg:pub.10.1038/scientificamerican0783-70 schema:sameAs https://app.dimensions.ai/details/publication/pub.1056571521
    114 https://doi.org/10.1038/scientificamerican0783-70
    115 rdf:type schema:CreativeWork
    116 https://doi.org/10.1126/science.6093258 schema:sameAs https://app.dimensions.ai/details/publication/pub.1062634105
    117 rdf:type schema:CreativeWork
    118 https://doi.org/10.1137/1028106 schema:sameAs https://app.dimensions.ai/details/publication/pub.1062862389
    119 rdf:type schema:CreativeWork
    120 https://doi.org/10.1137/1028157 schema:sameAs https://app.dimensions.ai/details/publication/pub.1062862440
    121 rdf:type schema:CreativeWork
    122 https://doi.org/10.1145/322609.322794 schema:sameAs https://app.dimensions.ai/details/publication/pub.1045073789
    123 rdf:type schema:CreativeWork
    124 https://doi.org/10.1145/52964.52980 schema:sameAs https://app.dimensions.ai/details/publication/pub.1049493170
    125 rdf:type schema:CreativeWork
    126 https://www.grid.ac/institutes/grid.134936.a schema:alternateName University of Missouri
    127 schema:name Computer Science Department, University of Missouri, 65401, Rolla, MO, USA
    128 rdf:type schema:Organization
     




    Preview window. Press ESC to close (or click here)


    ...