Deep Affect Prediction in-the-Wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond View Full Text


Ontology type: schema:ScholarlyArticle     


Article Info

DATE

2019-02-13

AUTHORS

Dimitrios Kollias, Panagiotis Tzirakis, Mihalis A. Nicolaou, Athanasios Papaioannou, Guoying Zhao, Björn Schuller, Irene Kotsia, Stefanos Zafeiriou

ABSTRACT

Automatic understanding of human affect using visual signals is of great importance in everyday human–machine interactions. Appraising human emotional states, behaviors and reactions displayed in real-world settings, can be accomplished using latent continuous dimensions (e.g., the circumplex model of affect). Valence (i.e., how positive or negative is an emotion) and arousal (i.e., power of the activation of the emotion) constitute popular and effective representations for affect. Nevertheless, the majority of collected datasets this far, although containing naturalistic emotional states, have been captured in highly controlled recording conditions. In this paper, we introduce the Aff-Wild benchmark for training and evaluating affect recognition algorithms. We also report on the results of the First Affect-in-the-wild Challenge (Aff-Wild Challenge) that was recently organized in conjunction with CVPR 2017 on the Aff-Wild database, and was the first ever challenge on the estimation of valence and arousal in-the-wild. Furthermore, we design and extensively train an end-to-end deep neural architecture which performs prediction of continuous emotion dimensions based on visual cues. The proposed deep learning architecture, AffWildNet, includes convolutional and recurrent neural network layers, exploiting the invariant properties of convolutional features, while also modeling temporal dynamics that arise in human behavior via the recurrent layers. The AffWildNet produced state-of-the-art results on the Aff-Wild Challenge. We then exploit the AffWild database for learning features, which can be used as priors for achieving best performances both for dimensional, as well as categorical emotion recognition, using the RECOLA, AFEW-VA and EmotiW 2017 datasets, compared to all other methods designed for the same goal. The database and emotion recognition models are available at http://ibug.doc.ic.ac.uk/resources/first-affect-wild-challenge. More... »

PAGES

1-23

References to SciGraph publications

  • 2014. Face Detection without Bells and Whistles in COMPUTER VISION – ECCV 2014
  • 2011. AVEC 2011–The First International Audio/Visual Emotion Challenge in AFFECTIVE COMPUTING AND INTELLIGENT INTERACTION
  • 2018-04. A Comprehensive Performance Evaluation of Deformable Face Tracking “In-the-Wild” in INTERNATIONAL JOURNAL OF COMPUTER VISION
  • Identifiers

    URI

    http://scigraph.springernature.com/pub.10.1007/s11263-019-01158-4

    DOI

    http://dx.doi.org/10.1007/s11263-019-01158-4

    DIMENSIONS

    https://app.dimensions.ai/details/publication/pub.1112113188


    Indexing Status Check whether this publication has been indexed by Scopus and Web Of Science using the SN Indexing Status Tool
    Incoming Citations Browse incoming citations for this publication using opencitations.net

    JSON-LD is the canonical representation for SciGraph data.

    TIP: You can open this SciGraph record using an external JSON-LD service: JSON-LD Playground Google SDTT

    [
      {
        "@context": "https://springernature.github.io/scigraph/jsonld/sgcontext.json", 
        "about": [
          {
            "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/1701", 
            "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
            "name": "Psychology", 
            "type": "DefinedTerm"
          }, 
          {
            "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/17", 
            "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
            "name": "Psychology and Cognitive Sciences", 
            "type": "DefinedTerm"
          }
        ], 
        "author": [
          {
            "affiliation": {
              "name": [
                "Queens Gate, SW7 2AZ, London, UK"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Kollias", 
            "givenName": "Dimitrios", 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "name": [
                "Queens Gate, SW7 2AZ, London, UK"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Tzirakis", 
            "givenName": "Panagiotis", 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "Goldsmiths University of London", 
              "id": "https://www.grid.ac/institutes/grid.15874.3f", 
              "name": [
                "Queens Gate, SW7 2AZ, London, UK", 
                "Department of Computing, Goldsmiths University of London, SE14 6NW, London, UK"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Nicolaou", 
            "givenName": "Mihalis A.", 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "name": [
                "Queens Gate, SW7 2AZ, London, UK"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Papaioannou", 
            "givenName": "Athanasios", 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "University of Oulu", 
              "id": "https://www.grid.ac/institutes/grid.10858.34", 
              "name": [
                "Queens Gate, SW7 2AZ, London, UK", 
                "Center for Machine Vision and Signal Analysis, University of Oulu, Oulu, Finland"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Zhao", 
            "givenName": "Guoying", 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "name": [
                "Queens Gate, SW7 2AZ, London, UK"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Schuller", 
            "givenName": "Bj\u00f6rn", 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "Middlesex University", 
              "id": "https://www.grid.ac/institutes/grid.15822.3c", 
              "name": [
                "Queens Gate, SW7 2AZ, London, UK", 
                "Department of Computer Science, Middlesex University of London, NW4 4BT, London, UK"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Kotsia", 
            "givenName": "Irene", 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "University of Oulu", 
              "id": "https://www.grid.ac/institutes/grid.10858.34", 
              "name": [
                "Queens Gate, SW7 2AZ, London, UK", 
                "Center for Machine Vision and Signal Analysis, University of Oulu, Oulu, Finland"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Zafeiriou", 
            "givenName": "Stefanos", 
            "type": "Person"
          }
        ], 
        "citation": [
          {
            "id": "https://doi.org/10.1145/2661806.2661807", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1000224183"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1145/2988257.2988258", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1000605546"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1016/j.imavis.2009.08.002", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1001414620"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1145/2388676.2388776", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1005230763"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1145/2522848.2531739", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1006530478"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1037/0022-3514.36.10.1152", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1007148680"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.4018/jse.2012010101", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1008458877"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.4018/jse.2012010101", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1008458877"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1145/2818346.2829994", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1009404629"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-642-24571-8_53", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1009564678", 
              "https://doi.org/10.1007/978-3-642-24571-8_53"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1145/2647868.2654890", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1015409866"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1145/2993148.2997638", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1018892281"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1145/2512530.2512533", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1031012598"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1162/neco.1997.9.8.1735", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1038140272"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1145/2733373.2806408", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1038336158"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1016/s0167-6393(02)00071-7", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1047969151"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1016/s0167-6393(02)00071-7", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1047969151"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-319-10593-2_47", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1049590036", 
              "https://doi.org/10.1007/978-3-319-10593-2_47"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1145/2663204.2666275", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1053634981"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/34.908962", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1061157208"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/t-affc.2011.15", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1061446966"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/t-affc.2011.20", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1061446972"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/t-affc.2011.25", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1061446977"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/t-affc.2011.26", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1061446978"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/tpami.2008.52", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1061743655"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/tpami.2014.2366127", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1061744770"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/tpami.2016.2515606", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1061745016"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.2307/2532051", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1069977481"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1016/j.imavis.2017.02.001", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1083801569"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/s11263-017-0999-5", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1083912225", 
              "https://doi.org/10.1007/s11263-017-0999-5"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/s11263-017-0999-5", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1083912225", 
              "https://doi.org/10.1007/s11263-017-0999-5"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1145/3133944.3133953", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1092335338"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1145/3136755.3143009", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1092684385"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1145/3136755.3143011", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1092684674"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1145/3136755.3143004", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1092684813"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/afgr.2008.4813324", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1093178194"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/fgr.2006.55", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1093271268"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/cvpr.2016.90", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1093359587"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/cvprw.2017.246", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1093571247"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/cvprw.2017.244", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1093830794"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/cvprw.2017.247", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1094008983"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/fgr.2006.6", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1094234214"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/cvprw.2017.248", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1094290594"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/fg.2011.5771462", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1094404119"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/cvprw.2017.245", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1094408213"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/cvprw.2010.5543262", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1094866068"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/icme.2005.1521424", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1094872563"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/iccv.2015.341", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1095300934"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/fg.2013.6553805", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1095646448"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/cvpr.2009.5206848", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1095689025"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.5244/c.30.122", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1096897165"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.5244/c.28.6", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1099426737"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.5244/c.29.41", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1099427264"
            ], 
            "type": "CreativeWork"
          }
        ], 
        "datePublished": "2019-02-13", 
        "datePublishedReg": "2019-02-13", 
        "description": "Automatic understanding of human affect using visual signals is of great importance in everyday human\u2013machine interactions. Appraising human emotional states, behaviors and reactions displayed in real-world settings, can be accomplished using latent continuous dimensions (e.g., the circumplex model of affect). Valence (i.e., how positive or negative is an emotion) and arousal (i.e., power of the activation of the emotion) constitute popular and effective representations for affect. Nevertheless, the majority of collected datasets this far, although containing naturalistic emotional states, have been captured in highly controlled recording conditions. In this paper, we introduce the Aff-Wild benchmark for training and evaluating affect recognition algorithms. We also report on the results of the First Affect-in-the-wild Challenge (Aff-Wild Challenge) that was recently organized in conjunction with CVPR 2017 on the Aff-Wild database, and was the first ever challenge on the estimation of valence and arousal in-the-wild. Furthermore, we design and extensively train an end-to-end deep neural architecture which performs prediction of continuous emotion dimensions based on visual cues. The proposed deep learning architecture, AffWildNet, includes convolutional and recurrent neural network layers, exploiting the invariant properties of convolutional features, while also modeling temporal dynamics that arise in human behavior via the recurrent layers. The AffWildNet produced state-of-the-art results on the Aff-Wild Challenge. We then exploit the AffWild database for learning features, which can be used as priors for achieving best performances both for dimensional, as well as categorical emotion recognition, using the RECOLA, AFEW-VA and EmotiW 2017 datasets, compared to all other methods designed for the same goal. The database and emotion recognition models are available at http://ibug.doc.ic.ac.uk/resources/first-affect-wild-challenge.", 
        "genre": "research_article", 
        "id": "sg:pub.10.1007/s11263-019-01158-4", 
        "inLanguage": [
          "en"
        ], 
        "isAccessibleForFree": false, 
        "isFundedItemOf": [
          {
            "id": "sg:grant.4575735", 
            "type": "MonetaryGrant"
          }
        ], 
        "isPartOf": [
          {
            "id": "sg:journal.1032807", 
            "issn": [
              "0920-5691", 
              "1573-1405"
            ], 
            "name": "International Journal of Computer Vision", 
            "type": "Periodical"
          }
        ], 
        "name": "Deep Affect Prediction in-the-Wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond", 
        "pagination": "1-23", 
        "productId": [
          {
            "name": "readcube_id", 
            "type": "PropertyValue", 
            "value": [
              "e062ab00c3638363a7aaa1a49afe7eff1c6a0f49f78f090ba67347d0b5f62adf"
            ]
          }, 
          {
            "name": "doi", 
            "type": "PropertyValue", 
            "value": [
              "10.1007/s11263-019-01158-4"
            ]
          }, 
          {
            "name": "dimensions_id", 
            "type": "PropertyValue", 
            "value": [
              "pub.1112113188"
            ]
          }
        ], 
        "sameAs": [
          "https://doi.org/10.1007/s11263-019-01158-4", 
          "https://app.dimensions.ai/details/publication/pub.1112113188"
        ], 
        "sdDataset": "articles", 
        "sdDatePublished": "2019-04-11T09:06", 
        "sdLicense": "https://scigraph.springernature.com/explorer/license/", 
        "sdPublisher": {
          "name": "Springer Nature - SN SciGraph project", 
          "type": "Organization"
        }, 
        "sdSource": "s3://com-uberresearch-data-dimensions-target-20181106-alternative/cleanup/v134/2549eaecd7973599484d7c17b260dba0a4ecb94b/merge/v9/a6c9fde33151104705d4d7ff012ea9563521a3ce/jats-lookup/v90/0000000336_0000000336/records_94828_00000000.jsonl", 
        "type": "ScholarlyArticle", 
        "url": "https://link.springer.com/10.1007%2Fs11263-019-01158-4"
      }
    ]
     

    Download the RDF metadata as:  json-ld nt turtle xml License info

    HOW TO GET THIS DATA PROGRAMMATICALLY:

    JSON-LD is a popular format for linked data which is fully compatible with JSON.

    curl -H 'Accept: application/ld+json' 'https://scigraph.springernature.com/pub.10.1007/s11263-019-01158-4'

    N-Triples is a line-based linked data format ideal for batch operations.

    curl -H 'Accept: application/n-triples' 'https://scigraph.springernature.com/pub.10.1007/s11263-019-01158-4'

    Turtle is a human-readable linked data format.

    curl -H 'Accept: text/turtle' 'https://scigraph.springernature.com/pub.10.1007/s11263-019-01158-4'

    RDF/XML is a standard XML format for linked data.

    curl -H 'Accept: application/rdf+xml' 'https://scigraph.springernature.com/pub.10.1007/s11263-019-01158-4'


     

    This table displays all metadata directly associated to this object as RDF triples.

    268 TRIPLES      21 PREDICATES      74 URIs      16 LITERALS      5 BLANK NODES

    Subject Predicate Object
    1 sg:pub.10.1007/s11263-019-01158-4 schema:about anzsrc-for:17
    2 anzsrc-for:1701
    3 schema:author N3fe2e2fca5b84c7295aac22b993cf933
    4 schema:citation sg:pub.10.1007/978-3-319-10593-2_47
    5 sg:pub.10.1007/978-3-642-24571-8_53
    6 sg:pub.10.1007/s11263-017-0999-5
    7 https://doi.org/10.1016/j.imavis.2009.08.002
    8 https://doi.org/10.1016/j.imavis.2017.02.001
    9 https://doi.org/10.1016/s0167-6393(02)00071-7
    10 https://doi.org/10.1037/0022-3514.36.10.1152
    11 https://doi.org/10.1109/34.908962
    12 https://doi.org/10.1109/afgr.2008.4813324
    13 https://doi.org/10.1109/cvpr.2009.5206848
    14 https://doi.org/10.1109/cvpr.2016.90
    15 https://doi.org/10.1109/cvprw.2010.5543262
    16 https://doi.org/10.1109/cvprw.2017.244
    17 https://doi.org/10.1109/cvprw.2017.245
    18 https://doi.org/10.1109/cvprw.2017.246
    19 https://doi.org/10.1109/cvprw.2017.247
    20 https://doi.org/10.1109/cvprw.2017.248
    21 https://doi.org/10.1109/fg.2011.5771462
    22 https://doi.org/10.1109/fg.2013.6553805
    23 https://doi.org/10.1109/fgr.2006.55
    24 https://doi.org/10.1109/fgr.2006.6
    25 https://doi.org/10.1109/iccv.2015.341
    26 https://doi.org/10.1109/icme.2005.1521424
    27 https://doi.org/10.1109/t-affc.2011.15
    28 https://doi.org/10.1109/t-affc.2011.20
    29 https://doi.org/10.1109/t-affc.2011.25
    30 https://doi.org/10.1109/t-affc.2011.26
    31 https://doi.org/10.1109/tpami.2008.52
    32 https://doi.org/10.1109/tpami.2014.2366127
    33 https://doi.org/10.1109/tpami.2016.2515606
    34 https://doi.org/10.1145/2388676.2388776
    35 https://doi.org/10.1145/2512530.2512533
    36 https://doi.org/10.1145/2522848.2531739
    37 https://doi.org/10.1145/2647868.2654890
    38 https://doi.org/10.1145/2661806.2661807
    39 https://doi.org/10.1145/2663204.2666275
    40 https://doi.org/10.1145/2733373.2806408
    41 https://doi.org/10.1145/2818346.2829994
    42 https://doi.org/10.1145/2988257.2988258
    43 https://doi.org/10.1145/2993148.2997638
    44 https://doi.org/10.1145/3133944.3133953
    45 https://doi.org/10.1145/3136755.3143004
    46 https://doi.org/10.1145/3136755.3143009
    47 https://doi.org/10.1145/3136755.3143011
    48 https://doi.org/10.1162/neco.1997.9.8.1735
    49 https://doi.org/10.2307/2532051
    50 https://doi.org/10.4018/jse.2012010101
    51 https://doi.org/10.5244/c.28.6
    52 https://doi.org/10.5244/c.29.41
    53 https://doi.org/10.5244/c.30.122
    54 schema:datePublished 2019-02-13
    55 schema:datePublishedReg 2019-02-13
    56 schema:description Automatic understanding of human affect using visual signals is of great importance in everyday human–machine interactions. Appraising human emotional states, behaviors and reactions displayed in real-world settings, can be accomplished using latent continuous dimensions (e.g., the circumplex model of affect). Valence (i.e., how positive or negative is an emotion) and arousal (i.e., power of the activation of the emotion) constitute popular and effective representations for affect. Nevertheless, the majority of collected datasets this far, although containing naturalistic emotional states, have been captured in highly controlled recording conditions. In this paper, we introduce the Aff-Wild benchmark for training and evaluating affect recognition algorithms. We also report on the results of the First Affect-in-the-wild Challenge (Aff-Wild Challenge) that was recently organized in conjunction with CVPR 2017 on the Aff-Wild database, and was the first ever challenge on the estimation of valence and arousal in-the-wild. Furthermore, we design and extensively train an end-to-end deep neural architecture which performs prediction of continuous emotion dimensions based on visual cues. The proposed deep learning architecture, AffWildNet, includes convolutional and recurrent neural network layers, exploiting the invariant properties of convolutional features, while also modeling temporal dynamics that arise in human behavior via the recurrent layers. The AffWildNet produced state-of-the-art results on the Aff-Wild Challenge. We then exploit the AffWild database for learning features, which can be used as priors for achieving best performances both for dimensional, as well as categorical emotion recognition, using the RECOLA, AFEW-VA and EmotiW 2017 datasets, compared to all other methods designed for the same goal. The database and emotion recognition models are available at http://ibug.doc.ic.ac.uk/resources/first-affect-wild-challenge.
    57 schema:genre research_article
    58 schema:inLanguage en
    59 schema:isAccessibleForFree false
    60 schema:isPartOf sg:journal.1032807
    61 schema:name Deep Affect Prediction in-the-Wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond
    62 schema:pagination 1-23
    63 schema:productId N6e810d11d4984bb1910595bc79428046
    64 Nbf7b58337d1f4f71a8ef7b85de4f7a93
    65 Ned288c3cec444412b3a6391bb47976cb
    66 schema:sameAs https://app.dimensions.ai/details/publication/pub.1112113188
    67 https://doi.org/10.1007/s11263-019-01158-4
    68 schema:sdDatePublished 2019-04-11T09:06
    69 schema:sdLicense https://scigraph.springernature.com/explorer/license/
    70 schema:sdPublisher N0507f89a0ec24a198f1874fad8cd4a80
    71 schema:url https://link.springer.com/10.1007%2Fs11263-019-01158-4
    72 sgo:license sg:explorer/license/
    73 sgo:sdDataset articles
    74 rdf:type schema:ScholarlyArticle
    75 N00ecd313b42746ebb7822ac6a5661958 rdf:first Nb8496cb77b18453a974140fd3fca602d
    76 rdf:rest N6f8e34dd0f8d43d8b53b6d340f472ed5
    77 N0507f89a0ec24a198f1874fad8cd4a80 schema:name Springer Nature - SN SciGraph project
    78 rdf:type schema:Organization
    79 N1a61f9e16c844905b90454d48c314191 rdf:first N3466dbce265345a39ec2f75592d60814
    80 rdf:rest N74da8a94c4bc4a658196a3db9c860605
    81 N2e97a8fcf3de482587a6a06026dbb215 schema:name Queens Gate, SW7 2AZ, London, UK
    82 rdf:type schema:Organization
    83 N3466dbce265345a39ec2f75592d60814 schema:affiliation https://www.grid.ac/institutes/grid.15822.3c
    84 schema:familyName Kotsia
    85 schema:givenName Irene
    86 rdf:type schema:Person
    87 N3818b93c9d9248079fbb2d8297cdc752 schema:name Queens Gate, SW7 2AZ, London, UK
    88 rdf:type schema:Organization
    89 N3fe2e2fca5b84c7295aac22b993cf933 rdf:first N9f869830f1c249348982609c46a30f74
    90 rdf:rest N4bcfb01527c24a0f9b720ca881321cc1
    91 N4bcfb01527c24a0f9b720ca881321cc1 rdf:first Nfdd4fa71d57a417490a5fb9225073b84
    92 rdf:rest Ndec0f2c7b4054e49a024c383d0feb35a
    93 N5186a0308e1c475f94b084d68504be59 schema:affiliation https://www.grid.ac/institutes/grid.15874.3f
    94 schema:familyName Nicolaou
    95 schema:givenName Mihalis A.
    96 rdf:type schema:Person
    97 N562fbaa2ef504045ad27272eef48e99a schema:affiliation N8f43097585c24cc49208499d012e520a
    98 schema:familyName Schuller
    99 schema:givenName Björn
    100 rdf:type schema:Person
    101 N6e810d11d4984bb1910595bc79428046 schema:name doi
    102 schema:value 10.1007/s11263-019-01158-4
    103 rdf:type schema:PropertyValue
    104 N6f8e34dd0f8d43d8b53b6d340f472ed5 rdf:first N562fbaa2ef504045ad27272eef48e99a
    105 rdf:rest N1a61f9e16c844905b90454d48c314191
    106 N74da8a94c4bc4a658196a3db9c860605 rdf:first Na9f04b6b0c594696ba6d9e06a849566e
    107 rdf:rest rdf:nil
    108 N816e5e457ea14d2b8bac1f46ec7f681a schema:name Queens Gate, SW7 2AZ, London, UK
    109 rdf:type schema:Organization
    110 N8f43097585c24cc49208499d012e520a schema:name Queens Gate, SW7 2AZ, London, UK
    111 rdf:type schema:Organization
    112 N9f869830f1c249348982609c46a30f74 schema:affiliation N3818b93c9d9248079fbb2d8297cdc752
    113 schema:familyName Kollias
    114 schema:givenName Dimitrios
    115 rdf:type schema:Person
    116 Na9f04b6b0c594696ba6d9e06a849566e schema:affiliation https://www.grid.ac/institutes/grid.10858.34
    117 schema:familyName Zafeiriou
    118 schema:givenName Stefanos
    119 rdf:type schema:Person
    120 Nb8496cb77b18453a974140fd3fca602d schema:affiliation https://www.grid.ac/institutes/grid.10858.34
    121 schema:familyName Zhao
    122 schema:givenName Guoying
    123 rdf:type schema:Person
    124 Nbf7b58337d1f4f71a8ef7b85de4f7a93 schema:name readcube_id
    125 schema:value e062ab00c3638363a7aaa1a49afe7eff1c6a0f49f78f090ba67347d0b5f62adf
    126 rdf:type schema:PropertyValue
    127 Ndec0f2c7b4054e49a024c383d0feb35a rdf:first N5186a0308e1c475f94b084d68504be59
    128 rdf:rest Nf0a6497a97f94ce9a508f86a5f373317
    129 Ne8f768730ffd4e529840b6e9bff72a42 schema:affiliation N816e5e457ea14d2b8bac1f46ec7f681a
    130 schema:familyName Papaioannou
    131 schema:givenName Athanasios
    132 rdf:type schema:Person
    133 Ned288c3cec444412b3a6391bb47976cb schema:name dimensions_id
    134 schema:value pub.1112113188
    135 rdf:type schema:PropertyValue
    136 Nf0a6497a97f94ce9a508f86a5f373317 rdf:first Ne8f768730ffd4e529840b6e9bff72a42
    137 rdf:rest N00ecd313b42746ebb7822ac6a5661958
    138 Nfdd4fa71d57a417490a5fb9225073b84 schema:affiliation N2e97a8fcf3de482587a6a06026dbb215
    139 schema:familyName Tzirakis
    140 schema:givenName Panagiotis
    141 rdf:type schema:Person
    142 anzsrc-for:17 schema:inDefinedTermSet anzsrc-for:
    143 schema:name Psychology and Cognitive Sciences
    144 rdf:type schema:DefinedTerm
    145 anzsrc-for:1701 schema:inDefinedTermSet anzsrc-for:
    146 schema:name Psychology
    147 rdf:type schema:DefinedTerm
    148 sg:grant.4575735 http://pending.schema.org/fundedItem sg:pub.10.1007/s11263-019-01158-4
    149 rdf:type schema:MonetaryGrant
    150 sg:journal.1032807 schema:issn 0920-5691
    151 1573-1405
    152 schema:name International Journal of Computer Vision
    153 rdf:type schema:Periodical
    154 sg:pub.10.1007/978-3-319-10593-2_47 schema:sameAs https://app.dimensions.ai/details/publication/pub.1049590036
    155 https://doi.org/10.1007/978-3-319-10593-2_47
    156 rdf:type schema:CreativeWork
    157 sg:pub.10.1007/978-3-642-24571-8_53 schema:sameAs https://app.dimensions.ai/details/publication/pub.1009564678
    158 https://doi.org/10.1007/978-3-642-24571-8_53
    159 rdf:type schema:CreativeWork
    160 sg:pub.10.1007/s11263-017-0999-5 schema:sameAs https://app.dimensions.ai/details/publication/pub.1083912225
    161 https://doi.org/10.1007/s11263-017-0999-5
    162 rdf:type schema:CreativeWork
    163 https://doi.org/10.1016/j.imavis.2009.08.002 schema:sameAs https://app.dimensions.ai/details/publication/pub.1001414620
    164 rdf:type schema:CreativeWork
    165 https://doi.org/10.1016/j.imavis.2017.02.001 schema:sameAs https://app.dimensions.ai/details/publication/pub.1083801569
    166 rdf:type schema:CreativeWork
    167 https://doi.org/10.1016/s0167-6393(02)00071-7 schema:sameAs https://app.dimensions.ai/details/publication/pub.1047969151
    168 rdf:type schema:CreativeWork
    169 https://doi.org/10.1037/0022-3514.36.10.1152 schema:sameAs https://app.dimensions.ai/details/publication/pub.1007148680
    170 rdf:type schema:CreativeWork
    171 https://doi.org/10.1109/34.908962 schema:sameAs https://app.dimensions.ai/details/publication/pub.1061157208
    172 rdf:type schema:CreativeWork
    173 https://doi.org/10.1109/afgr.2008.4813324 schema:sameAs https://app.dimensions.ai/details/publication/pub.1093178194
    174 rdf:type schema:CreativeWork
    175 https://doi.org/10.1109/cvpr.2009.5206848 schema:sameAs https://app.dimensions.ai/details/publication/pub.1095689025
    176 rdf:type schema:CreativeWork
    177 https://doi.org/10.1109/cvpr.2016.90 schema:sameAs https://app.dimensions.ai/details/publication/pub.1093359587
    178 rdf:type schema:CreativeWork
    179 https://doi.org/10.1109/cvprw.2010.5543262 schema:sameAs https://app.dimensions.ai/details/publication/pub.1094866068
    180 rdf:type schema:CreativeWork
    181 https://doi.org/10.1109/cvprw.2017.244 schema:sameAs https://app.dimensions.ai/details/publication/pub.1093830794
    182 rdf:type schema:CreativeWork
    183 https://doi.org/10.1109/cvprw.2017.245 schema:sameAs https://app.dimensions.ai/details/publication/pub.1094408213
    184 rdf:type schema:CreativeWork
    185 https://doi.org/10.1109/cvprw.2017.246 schema:sameAs https://app.dimensions.ai/details/publication/pub.1093571247
    186 rdf:type schema:CreativeWork
    187 https://doi.org/10.1109/cvprw.2017.247 schema:sameAs https://app.dimensions.ai/details/publication/pub.1094008983
    188 rdf:type schema:CreativeWork
    189 https://doi.org/10.1109/cvprw.2017.248 schema:sameAs https://app.dimensions.ai/details/publication/pub.1094290594
    190 rdf:type schema:CreativeWork
    191 https://doi.org/10.1109/fg.2011.5771462 schema:sameAs https://app.dimensions.ai/details/publication/pub.1094404119
    192 rdf:type schema:CreativeWork
    193 https://doi.org/10.1109/fg.2013.6553805 schema:sameAs https://app.dimensions.ai/details/publication/pub.1095646448
    194 rdf:type schema:CreativeWork
    195 https://doi.org/10.1109/fgr.2006.55 schema:sameAs https://app.dimensions.ai/details/publication/pub.1093271268
    196 rdf:type schema:CreativeWork
    197 https://doi.org/10.1109/fgr.2006.6 schema:sameAs https://app.dimensions.ai/details/publication/pub.1094234214
    198 rdf:type schema:CreativeWork
    199 https://doi.org/10.1109/iccv.2015.341 schema:sameAs https://app.dimensions.ai/details/publication/pub.1095300934
    200 rdf:type schema:CreativeWork
    201 https://doi.org/10.1109/icme.2005.1521424 schema:sameAs https://app.dimensions.ai/details/publication/pub.1094872563
    202 rdf:type schema:CreativeWork
    203 https://doi.org/10.1109/t-affc.2011.15 schema:sameAs https://app.dimensions.ai/details/publication/pub.1061446966
    204 rdf:type schema:CreativeWork
    205 https://doi.org/10.1109/t-affc.2011.20 schema:sameAs https://app.dimensions.ai/details/publication/pub.1061446972
    206 rdf:type schema:CreativeWork
    207 https://doi.org/10.1109/t-affc.2011.25 schema:sameAs https://app.dimensions.ai/details/publication/pub.1061446977
    208 rdf:type schema:CreativeWork
    209 https://doi.org/10.1109/t-affc.2011.26 schema:sameAs https://app.dimensions.ai/details/publication/pub.1061446978
    210 rdf:type schema:CreativeWork
    211 https://doi.org/10.1109/tpami.2008.52 schema:sameAs https://app.dimensions.ai/details/publication/pub.1061743655
    212 rdf:type schema:CreativeWork
    213 https://doi.org/10.1109/tpami.2014.2366127 schema:sameAs https://app.dimensions.ai/details/publication/pub.1061744770
    214 rdf:type schema:CreativeWork
    215 https://doi.org/10.1109/tpami.2016.2515606 schema:sameAs https://app.dimensions.ai/details/publication/pub.1061745016
    216 rdf:type schema:CreativeWork
    217 https://doi.org/10.1145/2388676.2388776 schema:sameAs https://app.dimensions.ai/details/publication/pub.1005230763
    218 rdf:type schema:CreativeWork
    219 https://doi.org/10.1145/2512530.2512533 schema:sameAs https://app.dimensions.ai/details/publication/pub.1031012598
    220 rdf:type schema:CreativeWork
    221 https://doi.org/10.1145/2522848.2531739 schema:sameAs https://app.dimensions.ai/details/publication/pub.1006530478
    222 rdf:type schema:CreativeWork
    223 https://doi.org/10.1145/2647868.2654890 schema:sameAs https://app.dimensions.ai/details/publication/pub.1015409866
    224 rdf:type schema:CreativeWork
    225 https://doi.org/10.1145/2661806.2661807 schema:sameAs https://app.dimensions.ai/details/publication/pub.1000224183
    226 rdf:type schema:CreativeWork
    227 https://doi.org/10.1145/2663204.2666275 schema:sameAs https://app.dimensions.ai/details/publication/pub.1053634981
    228 rdf:type schema:CreativeWork
    229 https://doi.org/10.1145/2733373.2806408 schema:sameAs https://app.dimensions.ai/details/publication/pub.1038336158
    230 rdf:type schema:CreativeWork
    231 https://doi.org/10.1145/2818346.2829994 schema:sameAs https://app.dimensions.ai/details/publication/pub.1009404629
    232 rdf:type schema:CreativeWork
    233 https://doi.org/10.1145/2988257.2988258 schema:sameAs https://app.dimensions.ai/details/publication/pub.1000605546
    234 rdf:type schema:CreativeWork
    235 https://doi.org/10.1145/2993148.2997638 schema:sameAs https://app.dimensions.ai/details/publication/pub.1018892281
    236 rdf:type schema:CreativeWork
    237 https://doi.org/10.1145/3133944.3133953 schema:sameAs https://app.dimensions.ai/details/publication/pub.1092335338
    238 rdf:type schema:CreativeWork
    239 https://doi.org/10.1145/3136755.3143004 schema:sameAs https://app.dimensions.ai/details/publication/pub.1092684813
    240 rdf:type schema:CreativeWork
    241 https://doi.org/10.1145/3136755.3143009 schema:sameAs https://app.dimensions.ai/details/publication/pub.1092684385
    242 rdf:type schema:CreativeWork
    243 https://doi.org/10.1145/3136755.3143011 schema:sameAs https://app.dimensions.ai/details/publication/pub.1092684674
    244 rdf:type schema:CreativeWork
    245 https://doi.org/10.1162/neco.1997.9.8.1735 schema:sameAs https://app.dimensions.ai/details/publication/pub.1038140272
    246 rdf:type schema:CreativeWork
    247 https://doi.org/10.2307/2532051 schema:sameAs https://app.dimensions.ai/details/publication/pub.1069977481
    248 rdf:type schema:CreativeWork
    249 https://doi.org/10.4018/jse.2012010101 schema:sameAs https://app.dimensions.ai/details/publication/pub.1008458877
    250 rdf:type schema:CreativeWork
    251 https://doi.org/10.5244/c.28.6 schema:sameAs https://app.dimensions.ai/details/publication/pub.1099426737
    252 rdf:type schema:CreativeWork
    253 https://doi.org/10.5244/c.29.41 schema:sameAs https://app.dimensions.ai/details/publication/pub.1099427264
    254 rdf:type schema:CreativeWork
    255 https://doi.org/10.5244/c.30.122 schema:sameAs https://app.dimensions.ai/details/publication/pub.1096897165
    256 rdf:type schema:CreativeWork
    257 https://www.grid.ac/institutes/grid.10858.34 schema:alternateName University of Oulu
    258 schema:name Center for Machine Vision and Signal Analysis, University of Oulu, Oulu, Finland
    259 Queens Gate, SW7 2AZ, London, UK
    260 rdf:type schema:Organization
    261 https://www.grid.ac/institutes/grid.15822.3c schema:alternateName Middlesex University
    262 schema:name Department of Computer Science, Middlesex University of London, NW4 4BT, London, UK
    263 Queens Gate, SW7 2AZ, London, UK
    264 rdf:type schema:Organization
    265 https://www.grid.ac/institutes/grid.15874.3f schema:alternateName Goldsmiths University of London
    266 schema:name Department of Computing, Goldsmiths University of London, SE14 6NW, London, UK
    267 Queens Gate, SW7 2AZ, London, UK
    268 rdf:type schema:Organization
     




    Preview window. Press ESC to close (or click here)


    ...