Bilevel Optimization and Machine Learning View Full Text


Ontology type: schema:Chapter      Open Access: True


Chapter Info

DATE

2008

AUTHORS

Kristin P. Bennett , Gautam Kunapuli , Jing Hu , Jong-Shi Pang

ABSTRACT

We examine the interplay of optimization and machine learning. Great progress has been made in machine learning by cleverly reducing machine learning problems to convex optimization problems with one or more hyper-parameters. The availability of powerful convex-programming theory and algorithms has enabled a flood of new research in machine learning models and methods. But many of the steps necessary for successful machine learning models fall outside of the convex machine learning paradigm. Thus we now propose framing machine learning problems as Stackelberg games. The resulting bilevel optimization problem allows for efficient systematic search of large numbers of hyper-parameters. We discuss recent progress in solving these bilevel problems and the many interesting optimization challenges that remain. Finally, we investigate the intriguing possibility of novel machine learning models enabled by bilevel programming. More... »

PAGES

25-47

References to SciGraph publications

  • 2002-01. Nonlinear programming without a penalty function in MATHEMATICAL PROGRAMMING
  • 1994-12. Misclassification minimization in JOURNAL OF GLOBAL OPTIMIZATION
  • 1993-11. Bilinear separation of two sets inn-space in COMPUTATIONAL OPTIMIZATION AND APPLICATIONS
  • 2006-06. Partial Augmented Lagrangian Method and Mathematical Programs with Complementarity Constraints in JOURNAL OF GLOBAL OPTIMIZATION
  • Book

    TITLE

    Computational Intelligence: Research Frontiers

    ISBN

    978-3-540-68858-7
    978-3-540-68860-0

    Identifiers

    URI

    http://scigraph.springernature.com/pub.10.1007/978-3-540-68860-0_2

    DOI

    http://dx.doi.org/10.1007/978-3-540-68860-0_2

    DIMENSIONS

    https://app.dimensions.ai/details/publication/pub.1002853911


    Indexing Status Check whether this publication has been indexed by Scopus and Web Of Science using the SN Indexing Status Tool
    Incoming Citations Browse incoming citations for this publication using opencitations.net

    JSON-LD is the canonical representation for SciGraph data.

    TIP: You can open this SciGraph record using an external JSON-LD service: JSON-LD Playground Google SDTT

    [
      {
        "@context": "https://springernature.github.io/scigraph/jsonld/sgcontext.json", 
        "about": [
          {
            "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/0103", 
            "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
            "name": "Numerical and Computational Mathematics", 
            "type": "DefinedTerm"
          }, 
          {
            "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/01", 
            "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
            "name": "Mathematical Sciences", 
            "type": "DefinedTerm"
          }
        ], 
        "author": [
          {
            "affiliation": {
              "alternateName": "Rensselaer Polytechnic Institute", 
              "id": "https://www.grid.ac/institutes/grid.33647.35", 
              "name": [
                "Dept. of Mathematical Sciences, Rensselaer Polytechnic Institute, Troy, NY, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Bennett", 
            "givenName": "Kristin P.", 
            "id": "sg:person.01003767533.12", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01003767533.12"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "Rensselaer Polytechnic Institute", 
              "id": "https://www.grid.ac/institutes/grid.33647.35", 
              "name": [
                "Dept. of Mathematical Sciences, Rensselaer Polytechnic Institute, Troy, NY, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Kunapuli", 
            "givenName": "Gautam", 
            "id": "sg:person.015554044477.42", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.015554044477.42"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "Rensselaer Polytechnic Institute", 
              "id": "https://www.grid.ac/institutes/grid.33647.35", 
              "name": [
                "Dept. of Mathematical Sciences, Rensselaer Polytechnic Institute, Troy, NY, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Hu", 
            "givenName": "Jing", 
            "id": "sg:person.015145534344.24", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.015145534344.24"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "University of Illinois at Urbana Champaign", 
              "id": "https://www.grid.ac/institutes/grid.35403.31", 
              "name": [
                "Dept. of Industrial and Enterprise Systems Engineering, University of Illinois at Urbana Champaign, Urbana Champaign, IL, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Pang", 
            "givenName": "Jong-Shi", 
            "id": "sg:person.0760734634.57", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.0760734634.57"
            ], 
            "type": "Person"
          }
        ], 
        "citation": [
          {
            "id": "sg:pub.10.1007/s10898-005-3837-1", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1006490833", 
              "https://doi.org/10.1007/s10898-005-3837-1"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/bf01299449", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1014039877", 
              "https://doi.org/10.1007/bf01299449"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/bf01299449", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1014039877", 
              "https://doi.org/10.1007/bf01299449"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1016/s0898-1221(99)90329-2", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1021444634"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1016/s0898-1221(99)90329-2", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1021444634"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1080/10556780410001709439", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1023919916"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/bf01096681", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1030993763", 
              "https://doi.org/10.1007/bf01096681"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/bf01096681", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1030993763", 
              "https://doi.org/10.1007/bf01096681"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/s101070100244", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1046071627", 
              "https://doi.org/10.1007/s101070100244"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1080/0233193031000149894", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1047144486"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1287/opre.21.1.37", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1064728311"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/ijcnn.2006.246935", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1095777461"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1017/cbo9780511983658", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1098667927"
            ], 
            "type": "CreativeWork"
          }
        ], 
        "datePublished": "2008", 
        "datePublishedReg": "2008-01-01", 
        "description": "We examine the interplay of optimization and machine learning. Great progress has been made in machine learning by cleverly reducing machine learning problems to convex optimization problems with one or more hyper-parameters. The availability of powerful convex-programming theory and algorithms has enabled a flood of new research in machine learning models and methods. But many of the steps necessary for successful machine learning models fall outside of the convex machine learning paradigm. Thus we now propose framing machine learning problems as Stackelberg games. The resulting bilevel optimization problem allows for efficient systematic search of large numbers of hyper-parameters. We discuss recent progress in solving these bilevel problems and the many interesting optimization challenges that remain. Finally, we investigate the intriguing possibility of novel machine learning models enabled by bilevel programming.", 
        "editor": [
          {
            "familyName": "Zurada", 
            "givenName": "Jacek M.", 
            "type": "Person"
          }, 
          {
            "familyName": "Yen", 
            "givenName": "Gary G.", 
            "type": "Person"
          }, 
          {
            "familyName": "Wang", 
            "givenName": "Jun", 
            "type": "Person"
          }
        ], 
        "genre": "chapter", 
        "id": "sg:pub.10.1007/978-3-540-68860-0_2", 
        "inLanguage": [
          "en"
        ], 
        "isAccessibleForFree": true, 
        "isPartOf": {
          "isbn": [
            "978-3-540-68858-7", 
            "978-3-540-68860-0"
          ], 
          "name": "Computational Intelligence: Research Frontiers", 
          "type": "Book"
        }, 
        "name": "Bilevel Optimization and Machine Learning", 
        "pagination": "25-47", 
        "productId": [
          {
            "name": "doi", 
            "type": "PropertyValue", 
            "value": [
              "10.1007/978-3-540-68860-0_2"
            ]
          }, 
          {
            "name": "readcube_id", 
            "type": "PropertyValue", 
            "value": [
              "0d9751da787af7f1b525dd0acef4cf9a4b693ff286a61be16bd8fa51365231ac"
            ]
          }, 
          {
            "name": "dimensions_id", 
            "type": "PropertyValue", 
            "value": [
              "pub.1002853911"
            ]
          }
        ], 
        "publisher": {
          "location": "Berlin, Heidelberg", 
          "name": "Springer Berlin Heidelberg", 
          "type": "Organisation"
        }, 
        "sameAs": [
          "https://doi.org/10.1007/978-3-540-68860-0_2", 
          "https://app.dimensions.ai/details/publication/pub.1002853911"
        ], 
        "sdDataset": "chapters", 
        "sdDatePublished": "2019-04-15T21:55", 
        "sdLicense": "https://scigraph.springernature.com/explorer/license/", 
        "sdPublisher": {
          "name": "Springer Nature - SN SciGraph project", 
          "type": "Organization"
        }, 
        "sdSource": "s3://com-uberresearch-data-dimensions-target-20181106-alternative/cleanup/v134/2549eaecd7973599484d7c17b260dba0a4ecb94b/merge/v9/a6c9fde33151104705d4d7ff012ea9563521a3ce/jats-lookup/v90/0000000001_0000000264/records_8693_00000244.jsonl", 
        "type": "Chapter", 
        "url": "http://link.springer.com/10.1007/978-3-540-68860-0_2"
      }
    ]
     

    Download the RDF metadata as:  json-ld nt turtle xml License info

    HOW TO GET THIS DATA PROGRAMMATICALLY:

    JSON-LD is a popular format for linked data which is fully compatible with JSON.

    curl -H 'Accept: application/ld+json' 'https://scigraph.springernature.com/pub.10.1007/978-3-540-68860-0_2'

    N-Triples is a line-based linked data format ideal for batch operations.

    curl -H 'Accept: application/n-triples' 'https://scigraph.springernature.com/pub.10.1007/978-3-540-68860-0_2'

    Turtle is a human-readable linked data format.

    curl -H 'Accept: text/turtle' 'https://scigraph.springernature.com/pub.10.1007/978-3-540-68860-0_2'

    RDF/XML is a standard XML format for linked data.

    curl -H 'Accept: application/rdf+xml' 'https://scigraph.springernature.com/pub.10.1007/978-3-540-68860-0_2'


     

    This table displays all metadata directly associated to this object as RDF triples.

    133 TRIPLES      23 PREDICATES      37 URIs      20 LITERALS      8 BLANK NODES

    Subject Predicate Object
    1 sg:pub.10.1007/978-3-540-68860-0_2 schema:about anzsrc-for:01
    2 anzsrc-for:0103
    3 schema:author Ndd61c7953a754bada3ccc6f01d733bbb
    4 schema:citation sg:pub.10.1007/bf01096681
    5 sg:pub.10.1007/bf01299449
    6 sg:pub.10.1007/s101070100244
    7 sg:pub.10.1007/s10898-005-3837-1
    8 https://doi.org/10.1016/s0898-1221(99)90329-2
    9 https://doi.org/10.1017/cbo9780511983658
    10 https://doi.org/10.1080/0233193031000149894
    11 https://doi.org/10.1080/10556780410001709439
    12 https://doi.org/10.1109/ijcnn.2006.246935
    13 https://doi.org/10.1287/opre.21.1.37
    14 schema:datePublished 2008
    15 schema:datePublishedReg 2008-01-01
    16 schema:description We examine the interplay of optimization and machine learning. Great progress has been made in machine learning by cleverly reducing machine learning problems to convex optimization problems with one or more hyper-parameters. The availability of powerful convex-programming theory and algorithms has enabled a flood of new research in machine learning models and methods. But many of the steps necessary for successful machine learning models fall outside of the convex machine learning paradigm. Thus we now propose framing machine learning problems as Stackelberg games. The resulting bilevel optimization problem allows for efficient systematic search of large numbers of hyper-parameters. We discuss recent progress in solving these bilevel problems and the many interesting optimization challenges that remain. Finally, we investigate the intriguing possibility of novel machine learning models enabled by bilevel programming.
    17 schema:editor Nbab7b3e7d2234ed3a6f692b17057e707
    18 schema:genre chapter
    19 schema:inLanguage en
    20 schema:isAccessibleForFree true
    21 schema:isPartOf N4118ddda05c643149a9f5ad9d0e37eaa
    22 schema:name Bilevel Optimization and Machine Learning
    23 schema:pagination 25-47
    24 schema:productId N6a3887b3c4c8443ab17980de8588f8ec
    25 Nd789e6afb3704a25a8c6fbef1f353a75
    26 Nf3b5ee3ff8d54fe189785be22c4f8ba4
    27 schema:publisher N56f3b28d7edd45c6986d4fe95a439ebe
    28 schema:sameAs https://app.dimensions.ai/details/publication/pub.1002853911
    29 https://doi.org/10.1007/978-3-540-68860-0_2
    30 schema:sdDatePublished 2019-04-15T21:55
    31 schema:sdLicense https://scigraph.springernature.com/explorer/license/
    32 schema:sdPublisher N943c9855dbc94ab1966d86e25396e6b3
    33 schema:url http://link.springer.com/10.1007/978-3-540-68860-0_2
    34 sgo:license sg:explorer/license/
    35 sgo:sdDataset chapters
    36 rdf:type schema:Chapter
    37 N22ba2165f48d4aeaadc37900b5082431 rdf:first sg:person.015145534344.24
    38 rdf:rest N5d376f3008eb4b7883a3302d80e99966
    39 N23aae2e4cf1c4d0aa2420d9b03d7628b schema:familyName Yen
    40 schema:givenName Gary G.
    41 rdf:type schema:Person
    42 N266741aad25f4e50bca3f7a3ca75580a rdf:first N23aae2e4cf1c4d0aa2420d9b03d7628b
    43 rdf:rest N2f0ccc6c3c544072bc30fce4abf36544
    44 N2f0ccc6c3c544072bc30fce4abf36544 rdf:first Nb915f344e404485dba3b6a767a8c5a4b
    45 rdf:rest rdf:nil
    46 N33ef23c0538c4ef995ce35f3a60f5c29 schema:familyName Zurada
    47 schema:givenName Jacek M.
    48 rdf:type schema:Person
    49 N4118ddda05c643149a9f5ad9d0e37eaa schema:isbn 978-3-540-68858-7
    50 978-3-540-68860-0
    51 schema:name Computational Intelligence: Research Frontiers
    52 rdf:type schema:Book
    53 N56f3b28d7edd45c6986d4fe95a439ebe schema:location Berlin, Heidelberg
    54 schema:name Springer Berlin Heidelberg
    55 rdf:type schema:Organisation
    56 N5d376f3008eb4b7883a3302d80e99966 rdf:first sg:person.0760734634.57
    57 rdf:rest rdf:nil
    58 N6a3887b3c4c8443ab17980de8588f8ec schema:name readcube_id
    59 schema:value 0d9751da787af7f1b525dd0acef4cf9a4b693ff286a61be16bd8fa51365231ac
    60 rdf:type schema:PropertyValue
    61 N8252794faf55464aa6ed7c66ee0b2b7d rdf:first sg:person.015554044477.42
    62 rdf:rest N22ba2165f48d4aeaadc37900b5082431
    63 N943c9855dbc94ab1966d86e25396e6b3 schema:name Springer Nature - SN SciGraph project
    64 rdf:type schema:Organization
    65 Nb915f344e404485dba3b6a767a8c5a4b schema:familyName Wang
    66 schema:givenName Jun
    67 rdf:type schema:Person
    68 Nbab7b3e7d2234ed3a6f692b17057e707 rdf:first N33ef23c0538c4ef995ce35f3a60f5c29
    69 rdf:rest N266741aad25f4e50bca3f7a3ca75580a
    70 Nd789e6afb3704a25a8c6fbef1f353a75 schema:name doi
    71 schema:value 10.1007/978-3-540-68860-0_2
    72 rdf:type schema:PropertyValue
    73 Ndd61c7953a754bada3ccc6f01d733bbb rdf:first sg:person.01003767533.12
    74 rdf:rest N8252794faf55464aa6ed7c66ee0b2b7d
    75 Nf3b5ee3ff8d54fe189785be22c4f8ba4 schema:name dimensions_id
    76 schema:value pub.1002853911
    77 rdf:type schema:PropertyValue
    78 anzsrc-for:01 schema:inDefinedTermSet anzsrc-for:
    79 schema:name Mathematical Sciences
    80 rdf:type schema:DefinedTerm
    81 anzsrc-for:0103 schema:inDefinedTermSet anzsrc-for:
    82 schema:name Numerical and Computational Mathematics
    83 rdf:type schema:DefinedTerm
    84 sg:person.01003767533.12 schema:affiliation https://www.grid.ac/institutes/grid.33647.35
    85 schema:familyName Bennett
    86 schema:givenName Kristin P.
    87 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01003767533.12
    88 rdf:type schema:Person
    89 sg:person.015145534344.24 schema:affiliation https://www.grid.ac/institutes/grid.33647.35
    90 schema:familyName Hu
    91 schema:givenName Jing
    92 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.015145534344.24
    93 rdf:type schema:Person
    94 sg:person.015554044477.42 schema:affiliation https://www.grid.ac/institutes/grid.33647.35
    95 schema:familyName Kunapuli
    96 schema:givenName Gautam
    97 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.015554044477.42
    98 rdf:type schema:Person
    99 sg:person.0760734634.57 schema:affiliation https://www.grid.ac/institutes/grid.35403.31
    100 schema:familyName Pang
    101 schema:givenName Jong-Shi
    102 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.0760734634.57
    103 rdf:type schema:Person
    104 sg:pub.10.1007/bf01096681 schema:sameAs https://app.dimensions.ai/details/publication/pub.1030993763
    105 https://doi.org/10.1007/bf01096681
    106 rdf:type schema:CreativeWork
    107 sg:pub.10.1007/bf01299449 schema:sameAs https://app.dimensions.ai/details/publication/pub.1014039877
    108 https://doi.org/10.1007/bf01299449
    109 rdf:type schema:CreativeWork
    110 sg:pub.10.1007/s101070100244 schema:sameAs https://app.dimensions.ai/details/publication/pub.1046071627
    111 https://doi.org/10.1007/s101070100244
    112 rdf:type schema:CreativeWork
    113 sg:pub.10.1007/s10898-005-3837-1 schema:sameAs https://app.dimensions.ai/details/publication/pub.1006490833
    114 https://doi.org/10.1007/s10898-005-3837-1
    115 rdf:type schema:CreativeWork
    116 https://doi.org/10.1016/s0898-1221(99)90329-2 schema:sameAs https://app.dimensions.ai/details/publication/pub.1021444634
    117 rdf:type schema:CreativeWork
    118 https://doi.org/10.1017/cbo9780511983658 schema:sameAs https://app.dimensions.ai/details/publication/pub.1098667927
    119 rdf:type schema:CreativeWork
    120 https://doi.org/10.1080/0233193031000149894 schema:sameAs https://app.dimensions.ai/details/publication/pub.1047144486
    121 rdf:type schema:CreativeWork
    122 https://doi.org/10.1080/10556780410001709439 schema:sameAs https://app.dimensions.ai/details/publication/pub.1023919916
    123 rdf:type schema:CreativeWork
    124 https://doi.org/10.1109/ijcnn.2006.246935 schema:sameAs https://app.dimensions.ai/details/publication/pub.1095777461
    125 rdf:type schema:CreativeWork
    126 https://doi.org/10.1287/opre.21.1.37 schema:sameAs https://app.dimensions.ai/details/publication/pub.1064728311
    127 rdf:type schema:CreativeWork
    128 https://www.grid.ac/institutes/grid.33647.35 schema:alternateName Rensselaer Polytechnic Institute
    129 schema:name Dept. of Mathematical Sciences, Rensselaer Polytechnic Institute, Troy, NY, USA
    130 rdf:type schema:Organization
    131 https://www.grid.ac/institutes/grid.35403.31 schema:alternateName University of Illinois at Urbana Champaign
    132 schema:name Dept. of Industrial and Enterprise Systems Engineering, University of Illinois at Urbana Champaign, Urbana Champaign, IL, USA
    133 rdf:type schema:Organization
     




    Preview window. Press ESC to close (or click here)


    ...