The nature of unsupervised learning in deep neural networks: A new understanding and novel approach View Full Text


Ontology type: schema:ScholarlyArticle     


Article Info

DATE

2016-07

AUTHORS

V. Golovko, A. Kroshchanka, D. Treadwell

ABSTRACT

Over the last decade, the deep neural networks are a hot topic in machine learning. It is breakthrough technology in processing images, video, speech, text and audio. Deep neural network permits us to overcome some limitations of a shallow neural network due to its deep architecture. In this paper we investigate the nature of unsupervised learning in restricted Boltzmann machine. We have proved that maximization of the log-likelihood input data distribution of restricted Boltzmann machine is equivalent to minimizing the cross-entropy and to special case of minimizing the mean squared error. Thus the nature of unsupervised learning is invariant to different training criteria. As a result we propose a new technique called “REBA” for the unsupervised training of deep neural networks. In contrast to Hinton’s conventional approach to the learning of restricted Boltzmann machine, which is based on linear nature of training rule, the proposed technique is founded on nonlinear training rule. We have shown that the classical equations for RBM learning are a special case of the proposed technique. As a result the proposed approach is more universal in contrast to the traditional energy-based model. We demonstrate the performance of the REBA technique using wellknown benchmark problem. The main contribution of this paper is a novel view and new understanding of an unsupervised learning in deep neural networks. More... »

PAGES

127-141

References to SciGraph publications

  • 2008. Nonlinear Principal Component Analysis: Neural Network Models and Applications in PRINCIPAL MANIFOLDS FOR DATA VISUALIZATION AND DIMENSION REDUCTION
  • 2014. A Learning Technique for Deep Belief Neural Networks in NEURAL NETWORKS AND ARTIFICIAL INTELLIGENCE
  • Identifiers

    URI

    http://scigraph.springernature.com/pub.10.3103/s1060992x16030073

    DOI

    http://dx.doi.org/10.3103/s1060992x16030073

    DIMENSIONS

    https://app.dimensions.ai/details/publication/pub.1034013531


    Indexing Status Check whether this publication has been indexed by Scopus and Web Of Science using the SN Indexing Status Tool
    Incoming Citations Browse incoming citations for this publication using opencitations.net

    JSON-LD is the canonical representation for SciGraph data.

    TIP: You can open this SciGraph record using an external JSON-LD service: JSON-LD Playground Google SDTT

    [
      {
        "@context": "https://springernature.github.io/scigraph/jsonld/sgcontext.json", 
        "about": [
          {
            "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/0801", 
            "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
            "name": "Artificial Intelligence and Image Processing", 
            "type": "DefinedTerm"
          }, 
          {
            "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/08", 
            "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
            "name": "Information and Computing Sciences", 
            "type": "DefinedTerm"
          }
        ], 
        "author": [
          {
            "affiliation": {
              "alternateName": "Moscow Engineering Physics Institute", 
              "id": "https://www.grid.ac/institutes/grid.183446.c", 
              "name": [
                "Brest State Technical University, Belarus, Russia", 
                "National Research Nuclear University (MEPHI), Moscow, Russia"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Golovko", 
            "givenName": "V.", 
            "id": "sg:person.07377226005.82", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.07377226005.82"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "Brest State Technical University", 
              "id": "https://www.grid.ac/institutes/grid.448577.d", 
              "name": [
                "Brest State Technical University, Belarus, Russia"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Kroshchanka", 
            "givenName": "A.", 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "name": [
                "5339 Iron Horse Pkwy, 94568, Dublin, CA, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Treadwell", 
            "givenName": "D.", 
            "id": "sg:person.014234303121.70", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.014234303121.70"
            ], 
            "type": "Person"
          }
        ], 
        "citation": [
          {
            "id": "https://doi.org/10.1126/science.1127647", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1004607132"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1162/neco.2006.18.7.1527", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1004707137"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1162/089976602760128018", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1007443228"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-540-73750-6_2", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1039787273", 
              "https://doi.org/10.1007/978-3-540-73750-6_2"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-540-73750-6_2", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1039787273", 
              "https://doi.org/10.1007/978-3-540-73750-6_2"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-319-08201-1_13", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1052672847", 
              "https://doi.org/10.1007/978-3-319-08201-1_13"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/msp.2012.2205597", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1061423808"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/tpami.2013.50", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1061744581"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1561/2200000006", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1068001401"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.21236/ad0241531", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1091822546"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/asru.2011.6163930", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1093464216"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/idaacs.2015.7340725", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1093831162"
            ], 
            "type": "CreativeWork"
          }
        ], 
        "datePublished": "2016-07", 
        "datePublishedReg": "2016-07-01", 
        "description": "Over the last decade, the deep neural networks are a hot topic in machine learning. It is breakthrough technology in processing images, video, speech, text and audio. Deep neural network permits us to overcome some limitations of a shallow neural network due to its deep architecture. In this paper we investigate the nature of unsupervised learning in restricted Boltzmann machine. We have proved that maximization of the log-likelihood input data distribution of restricted Boltzmann machine is equivalent to minimizing the cross-entropy and to special case of minimizing the mean squared error. Thus the nature of unsupervised learning is invariant to different training criteria. As a result we propose a new technique called \u201cREBA\u201d for the unsupervised training of deep neural networks. In contrast to Hinton\u2019s conventional approach to the learning of restricted Boltzmann machine, which is based on linear nature of training rule, the proposed technique is founded on nonlinear training rule. We have shown that the classical equations for RBM learning are a special case of the proposed technique. As a result the proposed approach is more universal in contrast to the traditional energy-based model. We demonstrate the performance of the REBA technique using wellknown benchmark problem. The main contribution of this paper is a novel view and new understanding of an unsupervised learning in deep neural networks.", 
        "genre": "research_article", 
        "id": "sg:pub.10.3103/s1060992x16030073", 
        "inLanguage": [
          "en"
        ], 
        "isAccessibleForFree": false, 
        "isPartOf": [
          {
            "id": "sg:journal.1136839", 
            "issn": [
              "1060-992X", 
              "1934-7898"
            ], 
            "name": "Optical Memory and Neural Networks", 
            "type": "Periodical"
          }, 
          {
            "issueNumber": "3", 
            "type": "PublicationIssue"
          }, 
          {
            "type": "PublicationVolume", 
            "volumeNumber": "25"
          }
        ], 
        "name": "The nature of unsupervised learning in deep neural networks: A new understanding and novel approach", 
        "pagination": "127-141", 
        "productId": [
          {
            "name": "readcube_id", 
            "type": "PropertyValue", 
            "value": [
              "a2088f04307f99d03152b5b85d66a95a2e38ca77ca52bb79bd088ab37eb14530"
            ]
          }, 
          {
            "name": "doi", 
            "type": "PropertyValue", 
            "value": [
              "10.3103/s1060992x16030073"
            ]
          }, 
          {
            "name": "dimensions_id", 
            "type": "PropertyValue", 
            "value": [
              "pub.1034013531"
            ]
          }
        ], 
        "sameAs": [
          "https://doi.org/10.3103/s1060992x16030073", 
          "https://app.dimensions.ai/details/publication/pub.1034013531"
        ], 
        "sdDataset": "articles", 
        "sdDatePublished": "2019-04-11T13:07", 
        "sdLicense": "https://scigraph.springernature.com/explorer/license/", 
        "sdPublisher": {
          "name": "Springer Nature - SN SciGraph project", 
          "type": "Organization"
        }, 
        "sdSource": "s3://com-uberresearch-data-dimensions-target-20181106-alternative/cleanup/v134/2549eaecd7973599484d7c17b260dba0a4ecb94b/merge/v9/a6c9fde33151104705d4d7ff012ea9563521a3ce/jats-lookup/v90/0000000367_0000000367/records_88221_00000000.jsonl", 
        "type": "ScholarlyArticle", 
        "url": "https://link.springer.com/10.3103%2FS1060992X16030073"
      }
    ]
     

    Download the RDF metadata as:  json-ld nt turtle xml License info

    HOW TO GET THIS DATA PROGRAMMATICALLY:

    JSON-LD is a popular format for linked data which is fully compatible with JSON.

    curl -H 'Accept: application/ld+json' 'https://scigraph.springernature.com/pub.10.3103/s1060992x16030073'

    N-Triples is a line-based linked data format ideal for batch operations.

    curl -H 'Accept: application/n-triples' 'https://scigraph.springernature.com/pub.10.3103/s1060992x16030073'

    Turtle is a human-readable linked data format.

    curl -H 'Accept: text/turtle' 'https://scigraph.springernature.com/pub.10.3103/s1060992x16030073'

    RDF/XML is a standard XML format for linked data.

    curl -H 'Accept: application/rdf+xml' 'https://scigraph.springernature.com/pub.10.3103/s1060992x16030073'


     

    This table displays all metadata directly associated to this object as RDF triples.

    115 TRIPLES      21 PREDICATES      38 URIs      19 LITERALS      7 BLANK NODES

    Subject Predicate Object
    1 sg:pub.10.3103/s1060992x16030073 schema:about anzsrc-for:08
    2 anzsrc-for:0801
    3 schema:author Ncc1e70ffef8142d6861f861be8029ef8
    4 schema:citation sg:pub.10.1007/978-3-319-08201-1_13
    5 sg:pub.10.1007/978-3-540-73750-6_2
    6 https://doi.org/10.1109/asru.2011.6163930
    7 https://doi.org/10.1109/idaacs.2015.7340725
    8 https://doi.org/10.1109/msp.2012.2205597
    9 https://doi.org/10.1109/tpami.2013.50
    10 https://doi.org/10.1126/science.1127647
    11 https://doi.org/10.1162/089976602760128018
    12 https://doi.org/10.1162/neco.2006.18.7.1527
    13 https://doi.org/10.1561/2200000006
    14 https://doi.org/10.21236/ad0241531
    15 schema:datePublished 2016-07
    16 schema:datePublishedReg 2016-07-01
    17 schema:description Over the last decade, the deep neural networks are a hot topic in machine learning. It is breakthrough technology in processing images, video, speech, text and audio. Deep neural network permits us to overcome some limitations of a shallow neural network due to its deep architecture. In this paper we investigate the nature of unsupervised learning in restricted Boltzmann machine. We have proved that maximization of the log-likelihood input data distribution of restricted Boltzmann machine is equivalent to minimizing the cross-entropy and to special case of minimizing the mean squared error. Thus the nature of unsupervised learning is invariant to different training criteria. As a result we propose a new technique called “REBA” for the unsupervised training of deep neural networks. In contrast to Hinton’s conventional approach to the learning of restricted Boltzmann machine, which is based on linear nature of training rule, the proposed technique is founded on nonlinear training rule. We have shown that the classical equations for RBM learning are a special case of the proposed technique. As a result the proposed approach is more universal in contrast to the traditional energy-based model. We demonstrate the performance of the REBA technique using wellknown benchmark problem. The main contribution of this paper is a novel view and new understanding of an unsupervised learning in deep neural networks.
    18 schema:genre research_article
    19 schema:inLanguage en
    20 schema:isAccessibleForFree false
    21 schema:isPartOf N630971169bd3403a9bd327b1d5c57c27
    22 N7a90d3bebe9a49aebebe53fa751b2bb6
    23 sg:journal.1136839
    24 schema:name The nature of unsupervised learning in deep neural networks: A new understanding and novel approach
    25 schema:pagination 127-141
    26 schema:productId N010e3d8d91244da1af99e8ce9a502a95
    27 N03e864a1b2094f6da978b12460205eb5
    28 Nea1675daf7f14922bf866482fc24c9de
    29 schema:sameAs https://app.dimensions.ai/details/publication/pub.1034013531
    30 https://doi.org/10.3103/s1060992x16030073
    31 schema:sdDatePublished 2019-04-11T13:07
    32 schema:sdLicense https://scigraph.springernature.com/explorer/license/
    33 schema:sdPublisher N4bd3b5650bcc4695b712b9067daa763e
    34 schema:url https://link.springer.com/10.3103%2FS1060992X16030073
    35 sgo:license sg:explorer/license/
    36 sgo:sdDataset articles
    37 rdf:type schema:ScholarlyArticle
    38 N010e3d8d91244da1af99e8ce9a502a95 schema:name dimensions_id
    39 schema:value pub.1034013531
    40 rdf:type schema:PropertyValue
    41 N03e864a1b2094f6da978b12460205eb5 schema:name doi
    42 schema:value 10.3103/s1060992x16030073
    43 rdf:type schema:PropertyValue
    44 N10d24edc720c4edc92fff5d748995a8a schema:name 5339 Iron Horse Pkwy, 94568, Dublin, CA, USA
    45 rdf:type schema:Organization
    46 N21be2ed7df8e4ef5a319d8c27a02404b schema:affiliation https://www.grid.ac/institutes/grid.448577.d
    47 schema:familyName Kroshchanka
    48 schema:givenName A.
    49 rdf:type schema:Person
    50 N2c907d1b3aa24ed8aacb0c6587bf24ab rdf:first sg:person.014234303121.70
    51 rdf:rest rdf:nil
    52 N4bd3b5650bcc4695b712b9067daa763e schema:name Springer Nature - SN SciGraph project
    53 rdf:type schema:Organization
    54 N630971169bd3403a9bd327b1d5c57c27 schema:issueNumber 3
    55 rdf:type schema:PublicationIssue
    56 N7a90d3bebe9a49aebebe53fa751b2bb6 schema:volumeNumber 25
    57 rdf:type schema:PublicationVolume
    58 Ncc1e70ffef8142d6861f861be8029ef8 rdf:first sg:person.07377226005.82
    59 rdf:rest Nea963521fd7f4039b5c7cf37822e5811
    60 Nea1675daf7f14922bf866482fc24c9de schema:name readcube_id
    61 schema:value a2088f04307f99d03152b5b85d66a95a2e38ca77ca52bb79bd088ab37eb14530
    62 rdf:type schema:PropertyValue
    63 Nea963521fd7f4039b5c7cf37822e5811 rdf:first N21be2ed7df8e4ef5a319d8c27a02404b
    64 rdf:rest N2c907d1b3aa24ed8aacb0c6587bf24ab
    65 anzsrc-for:08 schema:inDefinedTermSet anzsrc-for:
    66 schema:name Information and Computing Sciences
    67 rdf:type schema:DefinedTerm
    68 anzsrc-for:0801 schema:inDefinedTermSet anzsrc-for:
    69 schema:name Artificial Intelligence and Image Processing
    70 rdf:type schema:DefinedTerm
    71 sg:journal.1136839 schema:issn 1060-992X
    72 1934-7898
    73 schema:name Optical Memory and Neural Networks
    74 rdf:type schema:Periodical
    75 sg:person.014234303121.70 schema:affiliation N10d24edc720c4edc92fff5d748995a8a
    76 schema:familyName Treadwell
    77 schema:givenName D.
    78 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.014234303121.70
    79 rdf:type schema:Person
    80 sg:person.07377226005.82 schema:affiliation https://www.grid.ac/institutes/grid.183446.c
    81 schema:familyName Golovko
    82 schema:givenName V.
    83 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.07377226005.82
    84 rdf:type schema:Person
    85 sg:pub.10.1007/978-3-319-08201-1_13 schema:sameAs https://app.dimensions.ai/details/publication/pub.1052672847
    86 https://doi.org/10.1007/978-3-319-08201-1_13
    87 rdf:type schema:CreativeWork
    88 sg:pub.10.1007/978-3-540-73750-6_2 schema:sameAs https://app.dimensions.ai/details/publication/pub.1039787273
    89 https://doi.org/10.1007/978-3-540-73750-6_2
    90 rdf:type schema:CreativeWork
    91 https://doi.org/10.1109/asru.2011.6163930 schema:sameAs https://app.dimensions.ai/details/publication/pub.1093464216
    92 rdf:type schema:CreativeWork
    93 https://doi.org/10.1109/idaacs.2015.7340725 schema:sameAs https://app.dimensions.ai/details/publication/pub.1093831162
    94 rdf:type schema:CreativeWork
    95 https://doi.org/10.1109/msp.2012.2205597 schema:sameAs https://app.dimensions.ai/details/publication/pub.1061423808
    96 rdf:type schema:CreativeWork
    97 https://doi.org/10.1109/tpami.2013.50 schema:sameAs https://app.dimensions.ai/details/publication/pub.1061744581
    98 rdf:type schema:CreativeWork
    99 https://doi.org/10.1126/science.1127647 schema:sameAs https://app.dimensions.ai/details/publication/pub.1004607132
    100 rdf:type schema:CreativeWork
    101 https://doi.org/10.1162/089976602760128018 schema:sameAs https://app.dimensions.ai/details/publication/pub.1007443228
    102 rdf:type schema:CreativeWork
    103 https://doi.org/10.1162/neco.2006.18.7.1527 schema:sameAs https://app.dimensions.ai/details/publication/pub.1004707137
    104 rdf:type schema:CreativeWork
    105 https://doi.org/10.1561/2200000006 schema:sameAs https://app.dimensions.ai/details/publication/pub.1068001401
    106 rdf:type schema:CreativeWork
    107 https://doi.org/10.21236/ad0241531 schema:sameAs https://app.dimensions.ai/details/publication/pub.1091822546
    108 rdf:type schema:CreativeWork
    109 https://www.grid.ac/institutes/grid.183446.c schema:alternateName Moscow Engineering Physics Institute
    110 schema:name Brest State Technical University, Belarus, Russia
    111 National Research Nuclear University (MEPHI), Moscow, Russia
    112 rdf:type schema:Organization
    113 https://www.grid.ac/institutes/grid.448577.d schema:alternateName Brest State Technical University
    114 schema:name Brest State Technical University, Belarus, Russia
    115 rdf:type schema:Organization
     




    Preview window. Press ESC to close (or click here)


    ...