Solving the Local-Minimum Problem in Training Deep Learning Machines View Full Text


Ontology type: schema:Chapter     


Chapter Info

DATE

2017-10-24

AUTHORS

James Ting-Ho Lo , Yichuan Gui , Yun Peng

ABSTRACT

The local-minimum problem in training deep learning machines (DLMs) has plagued their development. This paper proposes a method to directly solve the problem. Our method is based on convexification of the sum squared error (SSE) criterion through transforming the SSE into a risk averting error (RAE) criterion. To alleviate numerical difficulties, a normalized RAE (NRAE) is employed. The convexity region of the SSE expands as its risk sensitivity index (RSI) increases. Making the best use of the convexity region, our method starts training with a very large RSI, gradually reduces it, and switches to the RAE as soon as the RAE is numerically feasible. After training converges, the resultant DLM is expected to be inside the attraction basin of a global minimum of the SSE. Numerical results are provided to show the effectiveness of the proposed method. More... »

PAGES

166-174

Identifiers

URI

http://scigraph.springernature.com/pub.10.1007/978-3-319-70087-8_18

DOI

http://dx.doi.org/10.1007/978-3-319-70087-8_18

DIMENSIONS

https://app.dimensions.ai/details/publication/pub.1092370213


Indexing Status Check whether this publication has been indexed by Scopus and Web Of Science using the SN Indexing Status Tool
Incoming Citations Browse incoming citations for this publication using opencitations.net

JSON-LD is the canonical representation for SciGraph data.

TIP: You can open this SciGraph record using an external JSON-LD service: JSON-LD Playground Google SDTT

[
  {
    "@context": "https://springernature.github.io/scigraph/jsonld/sgcontext.json", 
    "about": [
      {
        "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/17", 
        "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
        "name": "Psychology and Cognitive Sciences", 
        "type": "DefinedTerm"
      }, 
      {
        "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/1702", 
        "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
        "name": "Cognitive Sciences", 
        "type": "DefinedTerm"
      }
    ], 
    "author": [
      {
        "affiliation": {
          "alternateName": "Department of Mathematics and Statistics, University of Maryland Baltimore County, 21042, Baltimore, MD, USA", 
          "id": "http://www.grid.ac/institutes/grid.266673.0", 
          "name": [
            "Department of Mathematics and Statistics, University of Maryland Baltimore County, 21042, Baltimore, MD, USA"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Lo", 
        "givenName": "James Ting-Ho", 
        "id": "sg:person.013512526631.81", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.013512526631.81"
        ], 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "Department of Computer Science and Electrical Engineering, University of Maryland Baltimore County, 21042, Baltimore, MD, USA", 
          "id": "http://www.grid.ac/institutes/grid.266673.0", 
          "name": [
            "Department of Computer Science and Electrical Engineering, University of Maryland Baltimore County, 21042, Baltimore, MD, USA"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Gui", 
        "givenName": "Yichuan", 
        "id": "sg:person.014111634523.11", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.014111634523.11"
        ], 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "Department of Computer Science and Electrical Engineering, University of Maryland Baltimore County, 21042, Baltimore, MD, USA", 
          "id": "http://www.grid.ac/institutes/grid.266673.0", 
          "name": [
            "Department of Computer Science and Electrical Engineering, University of Maryland Baltimore County, 21042, Baltimore, MD, USA"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Peng", 
        "givenName": "Yun", 
        "id": "sg:person.01136741416.72", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01136741416.72"
        ], 
        "type": "Person"
      }
    ], 
    "datePublished": "2017-10-24", 
    "datePublishedReg": "2017-10-24", 
    "description": "The local-minimum problem in training deep learning machines (DLMs) has plagued their development. This paper proposes a method to directly solve the problem. Our method is based on convexification of the sum squared error (SSE) criterion through transforming the SSE into a risk averting error (RAE) criterion. To alleviate numerical difficulties, a normalized RAE (NRAE) is employed. The convexity region of the SSE expands as its risk sensitivity index (RSI) increases. Making the best use of the convexity region, our method starts training with a very large RSI, gradually reduces it, and switches to the RAE as soon as the RAE is numerically feasible. After training converges, the resultant DLM is expected to be inside the attraction basin of a global minimum of the SSE. Numerical results are provided to show the effectiveness of the proposed method.", 
    "editor": [
      {
        "familyName": "Liu", 
        "givenName": "Derong", 
        "type": "Person"
      }, 
      {
        "familyName": "Xie", 
        "givenName": "Shengli", 
        "type": "Person"
      }, 
      {
        "familyName": "Li", 
        "givenName": "Yuanqing", 
        "type": "Person"
      }, 
      {
        "familyName": "Zhao", 
        "givenName": "Dongbin", 
        "type": "Person"
      }, 
      {
        "familyName": "El-Alfy", 
        "givenName": "El-Sayed M.", 
        "type": "Person"
      }
    ], 
    "genre": "chapter", 
    "id": "sg:pub.10.1007/978-3-319-70087-8_18", 
    "inLanguage": "en", 
    "isAccessibleForFree": false, 
    "isPartOf": {
      "isbn": [
        "978-3-319-70086-1", 
        "978-3-319-70087-8"
      ], 
      "name": "Neural Information Processing", 
      "type": "Book"
    }, 
    "keywords": [
      "deep learning machine", 
      "problem", 
      "development", 
      "difficulties", 
      "RAE", 
      "better use", 
      "effectiveness", 
      "local minima problem", 
      "learning machine", 
      "machine", 
      "paper", 
      "method", 
      "sum", 
      "error criterion", 
      "criteria", 
      "SSE", 
      "risk", 
      "region", 
      "index increases", 
      "increase", 
      "use", 
      "training converges", 
      "basin", 
      "results", 
      "convexification", 
      "numerical difficulties", 
      "convexity region", 
      "RSI", 
      "converges", 
      "attraction basins", 
      "global minimum", 
      "minimum", 
      "numerical results"
    ], 
    "name": "Solving the Local-Minimum Problem in Training Deep Learning Machines", 
    "pagination": "166-174", 
    "productId": [
      {
        "name": "dimensions_id", 
        "type": "PropertyValue", 
        "value": [
          "pub.1092370213"
        ]
      }, 
      {
        "name": "doi", 
        "type": "PropertyValue", 
        "value": [
          "10.1007/978-3-319-70087-8_18"
        ]
      }
    ], 
    "publisher": {
      "name": "Springer Nature", 
      "type": "Organisation"
    }, 
    "sameAs": [
      "https://doi.org/10.1007/978-3-319-70087-8_18", 
      "https://app.dimensions.ai/details/publication/pub.1092370213"
    ], 
    "sdDataset": "chapters", 
    "sdDatePublished": "2022-06-01T22:28", 
    "sdLicense": "https://scigraph.springernature.com/explorer/license/", 
    "sdPublisher": {
      "name": "Springer Nature - SN SciGraph project", 
      "type": "Organization"
    }, 
    "sdSource": "s3://com-springernature-scigraph/baseset/20220601/entities/gbq_results/chapter/chapter_172.jsonl", 
    "type": "Chapter", 
    "url": "https://doi.org/10.1007/978-3-319-70087-8_18"
  }
]
 

Download the RDF metadata as:  json-ld nt turtle xml License info

HOW TO GET THIS DATA PROGRAMMATICALLY:

JSON-LD is a popular format for linked data which is fully compatible with JSON.

curl -H 'Accept: application/ld+json' 'https://scigraph.springernature.com/pub.10.1007/978-3-319-70087-8_18'

N-Triples is a line-based linked data format ideal for batch operations.

curl -H 'Accept: application/n-triples' 'https://scigraph.springernature.com/pub.10.1007/978-3-319-70087-8_18'

Turtle is a human-readable linked data format.

curl -H 'Accept: text/turtle' 'https://scigraph.springernature.com/pub.10.1007/978-3-319-70087-8_18'

RDF/XML is a standard XML format for linked data.

curl -H 'Accept: application/rdf+xml' 'https://scigraph.springernature.com/pub.10.1007/978-3-319-70087-8_18'


 

This table displays all metadata directly associated to this object as RDF triples.

129 TRIPLES      23 PREDICATES      58 URIs      51 LITERALS      7 BLANK NODES

Subject Predicate Object
1 sg:pub.10.1007/978-3-319-70087-8_18 schema:about anzsrc-for:17
2 anzsrc-for:1702
3 schema:author N2fbcfd303eb041edb6803320b56c9f32
4 schema:datePublished 2017-10-24
5 schema:datePublishedReg 2017-10-24
6 schema:description The local-minimum problem in training deep learning machines (DLMs) has plagued their development. This paper proposes a method to directly solve the problem. Our method is based on convexification of the sum squared error (SSE) criterion through transforming the SSE into a risk averting error (RAE) criterion. To alleviate numerical difficulties, a normalized RAE (NRAE) is employed. The convexity region of the SSE expands as its risk sensitivity index (RSI) increases. Making the best use of the convexity region, our method starts training with a very large RSI, gradually reduces it, and switches to the RAE as soon as the RAE is numerically feasible. After training converges, the resultant DLM is expected to be inside the attraction basin of a global minimum of the SSE. Numerical results are provided to show the effectiveness of the proposed method.
7 schema:editor Nbf4bb4e9474a4bb4803c8dc078167a2b
8 schema:genre chapter
9 schema:inLanguage en
10 schema:isAccessibleForFree false
11 schema:isPartOf N812c684b9386466ca840ae921b726868
12 schema:keywords RAE
13 RSI
14 SSE
15 attraction basins
16 basin
17 better use
18 converges
19 convexification
20 convexity region
21 criteria
22 deep learning machine
23 development
24 difficulties
25 effectiveness
26 error criterion
27 global minimum
28 increase
29 index increases
30 learning machine
31 local minima problem
32 machine
33 method
34 minimum
35 numerical difficulties
36 numerical results
37 paper
38 problem
39 region
40 results
41 risk
42 sum
43 training converges
44 use
45 schema:name Solving the Local-Minimum Problem in Training Deep Learning Machines
46 schema:pagination 166-174
47 schema:productId N394663d65255431fbe59c09f968241f7
48 N51dc8b1caf5e41f2a7d0529ed3a4d27d
49 schema:publisher Na170e688460449438dc5c6a2c46507ea
50 schema:sameAs https://app.dimensions.ai/details/publication/pub.1092370213
51 https://doi.org/10.1007/978-3-319-70087-8_18
52 schema:sdDatePublished 2022-06-01T22:28
53 schema:sdLicense https://scigraph.springernature.com/explorer/license/
54 schema:sdPublisher N91cf0ae1ae8c4f208bf445e39696f5be
55 schema:url https://doi.org/10.1007/978-3-319-70087-8_18
56 sgo:license sg:explorer/license/
57 sgo:sdDataset chapters
58 rdf:type schema:Chapter
59 N20ef6996976e428787813a125a128193 schema:familyName Zhao
60 schema:givenName Dongbin
61 rdf:type schema:Person
62 N2682871491ea4da6963f764d8ca536d4 rdf:first Nd42275961f564a21a93aa8a209bca34d
63 rdf:rest N5a10da914df041fd932cc6b88ca171b5
64 N27f335b39387429db01ead9e226a32a1 schema:familyName Liu
65 schema:givenName Derong
66 rdf:type schema:Person
67 N2fbcfd303eb041edb6803320b56c9f32 rdf:first sg:person.013512526631.81
68 rdf:rest Na205d38d24ae4f9a8388a7598d18446c
69 N394663d65255431fbe59c09f968241f7 schema:name dimensions_id
70 schema:value pub.1092370213
71 rdf:type schema:PropertyValue
72 N51dc8b1caf5e41f2a7d0529ed3a4d27d schema:name doi
73 schema:value 10.1007/978-3-319-70087-8_18
74 rdf:type schema:PropertyValue
75 N5a10da914df041fd932cc6b88ca171b5 rdf:first N20ef6996976e428787813a125a128193
76 rdf:rest Nd8dff92ced1e4a959ccec77672295ec3
77 N7c672452bc654fca834a5eff12731c82 rdf:first Nd7a53735f90c490cb3c48d9ef892e66f
78 rdf:rest N2682871491ea4da6963f764d8ca536d4
79 N812c684b9386466ca840ae921b726868 schema:isbn 978-3-319-70086-1
80 978-3-319-70087-8
81 schema:name Neural Information Processing
82 rdf:type schema:Book
83 N91cf0ae1ae8c4f208bf445e39696f5be schema:name Springer Nature - SN SciGraph project
84 rdf:type schema:Organization
85 Na170e688460449438dc5c6a2c46507ea schema:name Springer Nature
86 rdf:type schema:Organisation
87 Na205d38d24ae4f9a8388a7598d18446c rdf:first sg:person.014111634523.11
88 rdf:rest Naf932846b120453f83339f8ba334ae97
89 Naf932846b120453f83339f8ba334ae97 rdf:first sg:person.01136741416.72
90 rdf:rest rdf:nil
91 Nbf4bb4e9474a4bb4803c8dc078167a2b rdf:first N27f335b39387429db01ead9e226a32a1
92 rdf:rest N7c672452bc654fca834a5eff12731c82
93 Nc628c782a9ca47e184702b0592882977 schema:familyName El-Alfy
94 schema:givenName El-Sayed M.
95 rdf:type schema:Person
96 Nd42275961f564a21a93aa8a209bca34d schema:familyName Li
97 schema:givenName Yuanqing
98 rdf:type schema:Person
99 Nd7a53735f90c490cb3c48d9ef892e66f schema:familyName Xie
100 schema:givenName Shengli
101 rdf:type schema:Person
102 Nd8dff92ced1e4a959ccec77672295ec3 rdf:first Nc628c782a9ca47e184702b0592882977
103 rdf:rest rdf:nil
104 anzsrc-for:17 schema:inDefinedTermSet anzsrc-for:
105 schema:name Psychology and Cognitive Sciences
106 rdf:type schema:DefinedTerm
107 anzsrc-for:1702 schema:inDefinedTermSet anzsrc-for:
108 schema:name Cognitive Sciences
109 rdf:type schema:DefinedTerm
110 sg:person.01136741416.72 schema:affiliation grid-institutes:grid.266673.0
111 schema:familyName Peng
112 schema:givenName Yun
113 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01136741416.72
114 rdf:type schema:Person
115 sg:person.013512526631.81 schema:affiliation grid-institutes:grid.266673.0
116 schema:familyName Lo
117 schema:givenName James Ting-Ho
118 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.013512526631.81
119 rdf:type schema:Person
120 sg:person.014111634523.11 schema:affiliation grid-institutes:grid.266673.0
121 schema:familyName Gui
122 schema:givenName Yichuan
123 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.014111634523.11
124 rdf:type schema:Person
125 grid-institutes:grid.266673.0 schema:alternateName Department of Computer Science and Electrical Engineering, University of Maryland Baltimore County, 21042, Baltimore, MD, USA
126 Department of Mathematics and Statistics, University of Maryland Baltimore County, 21042, Baltimore, MD, USA
127 schema:name Department of Computer Science and Electrical Engineering, University of Maryland Baltimore County, 21042, Baltimore, MD, USA
128 Department of Mathematics and Statistics, University of Maryland Baltimore County, 21042, Baltimore, MD, USA
129 rdf:type schema:Organization
 




Preview window. Press ESC to close (or click here)


...