Accelerating the convergence of the back-propagation method View Full Text


Ontology type: schema:ScholarlyArticle     


Article Info

DATE

1988-09

AUTHORS

T. P. Vogl, J. K. Mangis, A. K. Rigler, W. T. Zink, D. L. Alkon

ABSTRACT

The utility of the back-propagation method in establishing suitable weights in a distributed adaptive network has been demonstrated repeatedly. Unfortunately, in many applications, the number of iterations required before convergence can be large. Modifications to the back-propagation algorithm described by Rumelhart et al. (1986) can greatly accelerate convergence. The modifications consist of three changes:1) instead of updating the network weights after each pattern is presented to the network, the network is updated only after the entire repertoire of patterns to be learned has been presented to the network, at which time the algebraic sums of all the weight changes are applied:2) instead of keeping η, the “learning rate” (i.e., the multiplier on the step size) constant, it is varied dynamically so that the algorithm utilizes a near-optimum η, as determined by the local optimization topography; and3) the momentum factor α is set to zero when, as signified by a failure of a step to reduce the total error, the information inherent in prior steps is more likely to be misleading than beneficial. Only after the network takes a useful step, i.e., one that reduces the total error, does α again assume a non-zero value. Considering the selection of weights in neural nets as a problem in classical nonlinear optimization theory, the rationale for algorithms seeking only those weights that produce the globally minimum error is reviewed and rejected. More... »

PAGES

257-263

References to SciGraph publications

  • 1985. A Stochastic Approach to Global Optimization in COMPUTATIONAL MATHEMATICAL PROGRAMMING
  • 1986-01. Convergence of an annealing algorithm in MATHEMATICAL PROGRAMMING
  • 1959-10. On a successive transformation of probability distribution and its application to the analysis of the optimum gradient method in ANNALS OF THE INSTITUTE OF STATISTICAL MATHEMATICS
  • 1983-07. Learning in a marine snail. in SCIENTIFIC AMERICAN
  • Identifiers

    URI

    http://scigraph.springernature.com/pub.10.1007/bf00332914

    DOI

    http://dx.doi.org/10.1007/bf00332914

    DIMENSIONS

    https://app.dimensions.ai/details/publication/pub.1043901229


    Indexing Status Check whether this publication has been indexed by Scopus and Web Of Science using the SN Indexing Status Tool
    Incoming Citations Browse incoming citations for this publication using opencitations.net

    JSON-LD is the canonical representation for SciGraph data.

    TIP: You can open this SciGraph record using an external JSON-LD service: JSON-LD Playground Google SDTT

    [
      {
        "@context": "https://springernature.github.io/scigraph/jsonld/sgcontext.json", 
        "about": [
          {
            "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/0801", 
            "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
            "name": "Artificial Intelligence and Image Processing", 
            "type": "DefinedTerm"
          }, 
          {
            "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/08", 
            "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
            "name": "Information and Computing Sciences", 
            "type": "DefinedTerm"
          }
        ], 
        "author": [
          {
            "affiliation": {
              "name": [
                "Environmental Research Institute of Michigan, 1501 Wilson Boulevard, 22209, Arlington, VA, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Vogl", 
            "givenName": "T. P.", 
            "id": "sg:person.01137660156.00", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01137660156.00"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "name": [
                "Environmental Research Institute of Michigan, 1501 Wilson Boulevard, 22209, Arlington, VA, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Mangis", 
            "givenName": "J. K.", 
            "id": "sg:person.01355251166.56", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01355251166.56"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "University of Missouri", 
              "id": "https://www.grid.ac/institutes/grid.134936.a", 
              "name": [
                "Computer Science Department, University of Missouri, 65401, Rolla, MO, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Rigler", 
            "givenName": "A. K.", 
            "id": "sg:person.01071544756.05", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01071544756.05"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "name": [
                "Environmental Research Institute of Michigan, 1501 Wilson Boulevard, 22209, Arlington, VA, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Zink", 
            "givenName": "W. T.", 
            "id": "sg:person.0677760266.38", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.0677760266.38"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "name": [
                "Neural Systems Section, National Institute of Neurological and Communicative Disorders and Stroke, NIH, 9000 Rockville Pike, 20892, Bethesda, MD, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Alkon", 
            "givenName": "D. L.", 
            "id": "sg:person.0720111551.14", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.0720111551.14"
            ], 
            "type": "Person"
          }
        ], 
        "citation": [
          {
            "id": "sg:pub.10.1007/978-3-642-82450-0_10", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1022559771", 
              "https://doi.org/10.1007/978-3-642-82450-0_10"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1145/322609.322794", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1045073789"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1145/52964.52980", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1049493170"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/bf01582166", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1049532067", 
              "https://doi.org/10.1007/bf01582166"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/bf01831719", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1052300639", 
              "https://doi.org/10.1007/bf01831719"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1038/scientificamerican0783-70", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1056571521", 
              "https://doi.org/10.1038/scientificamerican0783-70"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1126/science.6093258", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1062634105"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1137/1028106", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1062862389"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1137/1028157", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1062862440"
            ], 
            "type": "CreativeWork"
          }
        ], 
        "datePublished": "1988-09", 
        "datePublishedReg": "1988-09-01", 
        "description": "The utility of the back-propagation method in establishing suitable weights in a distributed adaptive network has been demonstrated repeatedly. Unfortunately, in many applications, the number of iterations required before convergence can be large. Modifications to the back-propagation algorithm described by Rumelhart et al. (1986) can greatly accelerate convergence. The modifications consist of three changes:1) instead of updating the network weights after each pattern is presented to the network, the network is updated only after the entire repertoire of patterns to be learned has been presented to the network, at which time the algebraic sums of all the weight changes are applied:2) instead of keeping \u03b7, the \u201clearning rate\u201d (i.e., the multiplier on the step size) constant, it is varied dynamically so that the algorithm utilizes a near-optimum \u03b7, as determined by the local optimization topography; and3) the momentum factor \u03b1 is set to zero when, as signified by a failure of a step to reduce the total error, the information inherent in prior steps is more likely to be misleading than beneficial. Only after the network takes a useful step, i.e., one that reduces the total error, does \u03b1 again assume a non-zero value. Considering the selection of weights in neural nets as a problem in classical nonlinear optimization theory, the rationale for algorithms seeking only those weights that produce the globally minimum error is reviewed and rejected.", 
        "genre": "research_article", 
        "id": "sg:pub.10.1007/bf00332914", 
        "inLanguage": [
          "en"
        ], 
        "isAccessibleForFree": false, 
        "isPartOf": [
          {
            "id": "sg:journal.1081741", 
            "issn": [
              "0340-1200", 
              "1432-0770"
            ], 
            "name": "Biological Cybernetics", 
            "type": "Periodical"
          }, 
          {
            "issueNumber": "4-5", 
            "type": "PublicationIssue"
          }, 
          {
            "type": "PublicationVolume", 
            "volumeNumber": "59"
          }
        ], 
        "name": "Accelerating the convergence of the back-propagation method", 
        "pagination": "257-263", 
        "productId": [
          {
            "name": "doi", 
            "type": "PropertyValue", 
            "value": [
              "10.1007/bf00332914"
            ]
          }, 
          {
            "name": "readcube_id", 
            "type": "PropertyValue", 
            "value": [
              "d11e63ab4607da6de523f4c877c46560b03ef873c63e635a50b90589d29606fe"
            ]
          }, 
          {
            "name": "dimensions_id", 
            "type": "PropertyValue", 
            "value": [
              "pub.1043901229"
            ]
          }
        ], 
        "sameAs": [
          "https://doi.org/10.1007/bf00332914", 
          "https://app.dimensions.ai/details/publication/pub.1043901229"
        ], 
        "sdDataset": "articles", 
        "sdDatePublished": "2019-04-15T08:48", 
        "sdLicense": "https://scigraph.springernature.com/explorer/license/", 
        "sdPublisher": {
          "name": "Springer Nature - SN SciGraph project", 
          "type": "Organization"
        }, 
        "sdSource": "s3://com-uberresearch-data-dimensions-target-20181106-alternative/cleanup/v134/2549eaecd7973599484d7c17b260dba0a4ecb94b/merge/v9/a6c9fde33151104705d4d7ff012ea9563521a3ce/jats-lookup/v90/0000000374_0000000374/records_119717_00000001.jsonl", 
        "type": "ScholarlyArticle", 
        "url": "http://link.springer.com/10.1007/BF00332914"
      }
    ]
     

    Download the RDF metadata as:  json-ld nt turtle xml License info

    HOW TO GET THIS DATA PROGRAMMATICALLY:

    JSON-LD is a popular format for linked data which is fully compatible with JSON.

    curl -H 'Accept: application/ld+json' 'https://scigraph.springernature.com/pub.10.1007/bf00332914'

    N-Triples is a line-based linked data format ideal for batch operations.

    curl -H 'Accept: application/n-triples' 'https://scigraph.springernature.com/pub.10.1007/bf00332914'

    Turtle is a human-readable linked data format.

    curl -H 'Accept: text/turtle' 'https://scigraph.springernature.com/pub.10.1007/bf00332914'

    RDF/XML is a standard XML format for linked data.

    curl -H 'Accept: application/rdf+xml' 'https://scigraph.springernature.com/pub.10.1007/bf00332914'


     

    This table displays all metadata directly associated to this object as RDF triples.

    128 TRIPLES      21 PREDICATES      36 URIs      19 LITERALS      7 BLANK NODES

    Subject Predicate Object
    1 sg:pub.10.1007/bf00332914 schema:about anzsrc-for:08
    2 anzsrc-for:0801
    3 schema:author Ndf96a76e2f3e45a7ac2d37fabceb35da
    4 schema:citation sg:pub.10.1007/978-3-642-82450-0_10
    5 sg:pub.10.1007/bf01582166
    6 sg:pub.10.1007/bf01831719
    7 sg:pub.10.1038/scientificamerican0783-70
    8 https://doi.org/10.1126/science.6093258
    9 https://doi.org/10.1137/1028106
    10 https://doi.org/10.1137/1028157
    11 https://doi.org/10.1145/322609.322794
    12 https://doi.org/10.1145/52964.52980
    13 schema:datePublished 1988-09
    14 schema:datePublishedReg 1988-09-01
    15 schema:description The utility of the back-propagation method in establishing suitable weights in a distributed adaptive network has been demonstrated repeatedly. Unfortunately, in many applications, the number of iterations required before convergence can be large. Modifications to the back-propagation algorithm described by Rumelhart et al. (1986) can greatly accelerate convergence. The modifications consist of three changes:1) instead of updating the network weights after each pattern is presented to the network, the network is updated only after the entire repertoire of patterns to be learned has been presented to the network, at which time the algebraic sums of all the weight changes are applied:2) instead of keeping η, the “learning rate” (i.e., the multiplier on the step size) constant, it is varied dynamically so that the algorithm utilizes a near-optimum η, as determined by the local optimization topography; and3) the momentum factor α is set to zero when, as signified by a failure of a step to reduce the total error, the information inherent in prior steps is more likely to be misleading than beneficial. Only after the network takes a useful step, i.e., one that reduces the total error, does α again assume a non-zero value. Considering the selection of weights in neural nets as a problem in classical nonlinear optimization theory, the rationale for algorithms seeking only those weights that produce the globally minimum error is reviewed and rejected.
    16 schema:genre research_article
    17 schema:inLanguage en
    18 schema:isAccessibleForFree false
    19 schema:isPartOf N327425115ed644bcad8fd4cdebb173b1
    20 Nb9b56a1d18e44cd3852d9ce2c69e77ad
    21 sg:journal.1081741
    22 schema:name Accelerating the convergence of the back-propagation method
    23 schema:pagination 257-263
    24 schema:productId N40b9c884545440c58cb38dbbb49f2ad0
    25 N8cb1b323784f468c906c292db265b953
    26 Na0a873e397a54b4ca78040e915b68ace
    27 schema:sameAs https://app.dimensions.ai/details/publication/pub.1043901229
    28 https://doi.org/10.1007/bf00332914
    29 schema:sdDatePublished 2019-04-15T08:48
    30 schema:sdLicense https://scigraph.springernature.com/explorer/license/
    31 schema:sdPublisher N79f28911a57346c68d2e5c5f9cdea845
    32 schema:url http://link.springer.com/10.1007/BF00332914
    33 sgo:license sg:explorer/license/
    34 sgo:sdDataset articles
    35 rdf:type schema:ScholarlyArticle
    36 N0aea916ce9ca407db9e55a1bb5d73006 rdf:first sg:person.01355251166.56
    37 rdf:rest Ned86325b493d408197e6f91c6dbfd5fa
    38 N1e97c9bd6c2c4802879e10c7cfcf56a9 schema:name Environmental Research Institute of Michigan, 1501 Wilson Boulevard, 22209, Arlington, VA, USA
    39 rdf:type schema:Organization
    40 N327425115ed644bcad8fd4cdebb173b1 schema:issueNumber 4-5
    41 rdf:type schema:PublicationIssue
    42 N3400b0f7f8134eab84f11ddabcde9a7e schema:name Environmental Research Institute of Michigan, 1501 Wilson Boulevard, 22209, Arlington, VA, USA
    43 rdf:type schema:Organization
    44 N40b9c884545440c58cb38dbbb49f2ad0 schema:name dimensions_id
    45 schema:value pub.1043901229
    46 rdf:type schema:PropertyValue
    47 N6393ebd6cfec4730aa5aa9632d4005b4 schema:name Environmental Research Institute of Michigan, 1501 Wilson Boulevard, 22209, Arlington, VA, USA
    48 rdf:type schema:Organization
    49 N79f28911a57346c68d2e5c5f9cdea845 schema:name Springer Nature - SN SciGraph project
    50 rdf:type schema:Organization
    51 N81142665848541829d95e1e27c159e9d rdf:first sg:person.0677760266.38
    52 rdf:rest N8222b115a5964c8f87232069b107ce8b
    53 N8222b115a5964c8f87232069b107ce8b rdf:first sg:person.0720111551.14
    54 rdf:rest rdf:nil
    55 N8cb1b323784f468c906c292db265b953 schema:name readcube_id
    56 schema:value d11e63ab4607da6de523f4c877c46560b03ef873c63e635a50b90589d29606fe
    57 rdf:type schema:PropertyValue
    58 Na0a873e397a54b4ca78040e915b68ace schema:name doi
    59 schema:value 10.1007/bf00332914
    60 rdf:type schema:PropertyValue
    61 Nb9b56a1d18e44cd3852d9ce2c69e77ad schema:volumeNumber 59
    62 rdf:type schema:PublicationVolume
    63 Ncc4e86d2fee042b0973999a2edc7a077 schema:name Neural Systems Section, National Institute of Neurological and Communicative Disorders and Stroke, NIH, 9000 Rockville Pike, 20892, Bethesda, MD, USA
    64 rdf:type schema:Organization
    65 Ndf96a76e2f3e45a7ac2d37fabceb35da rdf:first sg:person.01137660156.00
    66 rdf:rest N0aea916ce9ca407db9e55a1bb5d73006
    67 Ned86325b493d408197e6f91c6dbfd5fa rdf:first sg:person.01071544756.05
    68 rdf:rest N81142665848541829d95e1e27c159e9d
    69 anzsrc-for:08 schema:inDefinedTermSet anzsrc-for:
    70 schema:name Information and Computing Sciences
    71 rdf:type schema:DefinedTerm
    72 anzsrc-for:0801 schema:inDefinedTermSet anzsrc-for:
    73 schema:name Artificial Intelligence and Image Processing
    74 rdf:type schema:DefinedTerm
    75 sg:journal.1081741 schema:issn 0340-1200
    76 1432-0770
    77 schema:name Biological Cybernetics
    78 rdf:type schema:Periodical
    79 sg:person.01071544756.05 schema:affiliation https://www.grid.ac/institutes/grid.134936.a
    80 schema:familyName Rigler
    81 schema:givenName A. K.
    82 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01071544756.05
    83 rdf:type schema:Person
    84 sg:person.01137660156.00 schema:affiliation N3400b0f7f8134eab84f11ddabcde9a7e
    85 schema:familyName Vogl
    86 schema:givenName T. P.
    87 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01137660156.00
    88 rdf:type schema:Person
    89 sg:person.01355251166.56 schema:affiliation N6393ebd6cfec4730aa5aa9632d4005b4
    90 schema:familyName Mangis
    91 schema:givenName J. K.
    92 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01355251166.56
    93 rdf:type schema:Person
    94 sg:person.0677760266.38 schema:affiliation N1e97c9bd6c2c4802879e10c7cfcf56a9
    95 schema:familyName Zink
    96 schema:givenName W. T.
    97 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.0677760266.38
    98 rdf:type schema:Person
    99 sg:person.0720111551.14 schema:affiliation Ncc4e86d2fee042b0973999a2edc7a077
    100 schema:familyName Alkon
    101 schema:givenName D. L.
    102 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.0720111551.14
    103 rdf:type schema:Person
    104 sg:pub.10.1007/978-3-642-82450-0_10 schema:sameAs https://app.dimensions.ai/details/publication/pub.1022559771
    105 https://doi.org/10.1007/978-3-642-82450-0_10
    106 rdf:type schema:CreativeWork
    107 sg:pub.10.1007/bf01582166 schema:sameAs https://app.dimensions.ai/details/publication/pub.1049532067
    108 https://doi.org/10.1007/bf01582166
    109 rdf:type schema:CreativeWork
    110 sg:pub.10.1007/bf01831719 schema:sameAs https://app.dimensions.ai/details/publication/pub.1052300639
    111 https://doi.org/10.1007/bf01831719
    112 rdf:type schema:CreativeWork
    113 sg:pub.10.1038/scientificamerican0783-70 schema:sameAs https://app.dimensions.ai/details/publication/pub.1056571521
    114 https://doi.org/10.1038/scientificamerican0783-70
    115 rdf:type schema:CreativeWork
    116 https://doi.org/10.1126/science.6093258 schema:sameAs https://app.dimensions.ai/details/publication/pub.1062634105
    117 rdf:type schema:CreativeWork
    118 https://doi.org/10.1137/1028106 schema:sameAs https://app.dimensions.ai/details/publication/pub.1062862389
    119 rdf:type schema:CreativeWork
    120 https://doi.org/10.1137/1028157 schema:sameAs https://app.dimensions.ai/details/publication/pub.1062862440
    121 rdf:type schema:CreativeWork
    122 https://doi.org/10.1145/322609.322794 schema:sameAs https://app.dimensions.ai/details/publication/pub.1045073789
    123 rdf:type schema:CreativeWork
    124 https://doi.org/10.1145/52964.52980 schema:sameAs https://app.dimensions.ai/details/publication/pub.1049493170
    125 rdf:type schema:CreativeWork
    126 https://www.grid.ac/institutes/grid.134936.a schema:alternateName University of Missouri
    127 schema:name Computer Science Department, University of Missouri, 65401, Rolla, MO, USA
    128 rdf:type schema:Organization
     




    Preview window. Press ESC to close (or click here)


    ...