On the Use of Kappa Coefficients to Measure the Reliability of the Annotation of Non-acted Emotions View Full Text


Ontology type: schema:Chapter     


Chapter Info

DATE

2008

AUTHORS

Zoraida Callejas , Ramón López-Cózar

ABSTRACT

In this paper we study the impact of three main factors on measuring the reliability of the annotation of non-acted emotions: the annotator biases, the similarity between the classified emotions, and the usage of contextual information during the annotation. We employed a corpus collected from real interactions between users and a spoken dialogue system. The user utterances were classified by nine non-expert annotators into four categories. We discuss the problems that the nature of non-acted emotional corpora impose in evaluating the reliability of the annotations using Kappa coefficients. Although deeply affected by the so-called paradoxes of Kappa coefficients, our study shows how taking into account context information and similarity between emotions helps to obtain values closer to the maximum agreement rates attainable, and allow the detection of emotions which are expressed more subtly by the users. More... »

PAGES

221-232

References to SciGraph publications

  • 2005. Real-Life Emotion Representation and Detection in Call Centers Data in AFFECTIVE COMPUTING AND INTELLIGENT INTERACTION
  • Book

    TITLE

    Perception in Multimodal Dialogue Systems

    ISBN

    978-3-540-69368-0
    978-3-540-69369-7

    Identifiers

    URI

    http://scigraph.springernature.com/pub.10.1007/978-3-540-69369-7_25

    DOI

    http://dx.doi.org/10.1007/978-3-540-69369-7_25

    DIMENSIONS

    https://app.dimensions.ai/details/publication/pub.1006280503


    Indexing Status Check whether this publication has been indexed by Scopus and Web Of Science using the SN Indexing Status Tool
    Incoming Citations Browse incoming citations for this publication using opencitations.net

    JSON-LD is the canonical representation for SciGraph data.

    TIP: You can open this SciGraph record using an external JSON-LD service: JSON-LD Playground Google SDTT

    [
      {
        "@context": "https://springernature.github.io/scigraph/jsonld/sgcontext.json", 
        "about": [
          {
            "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/1701", 
            "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
            "name": "Psychology", 
            "type": "DefinedTerm"
          }, 
          {
            "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/17", 
            "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
            "name": "Psychology and Cognitive Sciences", 
            "type": "DefinedTerm"
          }
        ], 
        "author": [
          {
            "affiliation": {
              "name": [
                "Dept. of Languages and Computer Systems, \u00a0, 18071\u00a0Granada, Spain"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Callejas", 
            "givenName": "Zoraida", 
            "id": "sg:person.016307634167.10", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.016307634167.10"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "name": [
                "Dept. of Languages and Computer Systems, \u00a0, 18071\u00a0Granada, Spain"
              ], 
              "type": "Organization"
            }, 
            "familyName": "L\u00f3pez-C\u00f3zar", 
            "givenName": "Ram\u00f3n", 
            "id": "sg:person.013113720067.02", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.013113720067.02"
            ], 
            "type": "Person"
          }
        ], 
        "citation": [
          {
            "id": "sg:pub.10.1007/11573548_95", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1000453836", 
              "https://doi.org/10.1007/11573548_95"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/11573548_95", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1000453836", 
              "https://doi.org/10.1007/11573548_95"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1037/h0031619", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1004984215"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1016/0895-4356(95)00571-4", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1006360632"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1016/0895-4356(90)90158-l", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1007739743"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1016/j.neunet.2005.03.007", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1010392184"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1016/j.neunet.2005.03.007", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1010392184"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1016/j.specom.2006.11.004", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1021697340"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1037/h0077714", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1026751541"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1016/j.specom.2005.09.008", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1046798773"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1016/j.specom.2005.09.008", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1046798773"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/tsa.2004.838534", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1061786254"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.2307/2529310", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1069974986"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.2307/2529886", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1069975476"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.5860/choice.27-6381", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1073298711"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/asru.2003.1318399", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1095196556"
            ], 
            "type": "CreativeWork"
          }
        ], 
        "datePublished": "2008", 
        "datePublishedReg": "2008-01-01", 
        "description": "In this paper we study the impact of three main factors on measuring the reliability of the annotation of non-acted emotions: the annotator biases, the similarity between the classified emotions, and the usage of contextual information during the annotation. We employed a corpus collected from real interactions between users and a spoken dialogue system. The user utterances were classified by nine non-expert annotators into four categories. We discuss the problems that the nature of non-acted emotional corpora impose in evaluating the reliability of the annotations using Kappa coefficients. Although deeply affected by the so-called paradoxes of Kappa coefficients, our study shows how taking into account context information and similarity between emotions helps to obtain values closer to the maximum agreement rates attainable, and allow the detection of emotions which are expressed more subtly by the users.", 
        "editor": [
          {
            "familyName": "Andr\u00e9", 
            "givenName": "Elisabeth", 
            "type": "Person"
          }, 
          {
            "familyName": "Dybkj\u00e6r", 
            "givenName": "Laila", 
            "type": "Person"
          }, 
          {
            "familyName": "Minker", 
            "givenName": "Wolfgang", 
            "type": "Person"
          }, 
          {
            "familyName": "Neumann", 
            "givenName": "Heiko", 
            "type": "Person"
          }, 
          {
            "familyName": "Pieraccini", 
            "givenName": "Roberto", 
            "type": "Person"
          }, 
          {
            "familyName": "Weber", 
            "givenName": "Michael", 
            "type": "Person"
          }
        ], 
        "genre": "chapter", 
        "id": "sg:pub.10.1007/978-3-540-69369-7_25", 
        "inLanguage": [
          "en"
        ], 
        "isAccessibleForFree": false, 
        "isPartOf": {
          "isbn": [
            "978-3-540-69368-0", 
            "978-3-540-69369-7"
          ], 
          "name": "Perception in Multimodal Dialogue Systems", 
          "type": "Book"
        }, 
        "name": "On the Use of Kappa Coefficients to Measure the Reliability of the Annotation of Non-acted Emotions", 
        "pagination": "221-232", 
        "productId": [
          {
            "name": "doi", 
            "type": "PropertyValue", 
            "value": [
              "10.1007/978-3-540-69369-7_25"
            ]
          }, 
          {
            "name": "readcube_id", 
            "type": "PropertyValue", 
            "value": [
              "1dea849871ecc038bd85214a8aa10e7606d9b0f2f571bf5af2adc37b583f85bb"
            ]
          }, 
          {
            "name": "dimensions_id", 
            "type": "PropertyValue", 
            "value": [
              "pub.1006280503"
            ]
          }
        ], 
        "publisher": {
          "location": "Berlin, Heidelberg", 
          "name": "Springer Berlin Heidelberg", 
          "type": "Organisation"
        }, 
        "sameAs": [
          "https://doi.org/10.1007/978-3-540-69369-7_25", 
          "https://app.dimensions.ai/details/publication/pub.1006280503"
        ], 
        "sdDataset": "chapters", 
        "sdDatePublished": "2019-04-15T18:08", 
        "sdLicense": "https://scigraph.springernature.com/explorer/license/", 
        "sdPublisher": {
          "name": "Springer Nature - SN SciGraph project", 
          "type": "Organization"
        }, 
        "sdSource": "s3://com-uberresearch-data-dimensions-target-20181106-alternative/cleanup/v134/2549eaecd7973599484d7c17b260dba0a4ecb94b/merge/v9/a6c9fde33151104705d4d7ff012ea9563521a3ce/jats-lookup/v90/0000000001_0000000264/records_8681_00000247.jsonl", 
        "type": "Chapter", 
        "url": "http://link.springer.com/10.1007/978-3-540-69369-7_25"
      }
    ]
     

    Download the RDF metadata as:  json-ld nt turtle xml License info

    HOW TO GET THIS DATA PROGRAMMATICALLY:

    JSON-LD is a popular format for linked data which is fully compatible with JSON.

    curl -H 'Accept: application/ld+json' 'https://scigraph.springernature.com/pub.10.1007/978-3-540-69369-7_25'

    N-Triples is a line-based linked data format ideal for batch operations.

    curl -H 'Accept: application/n-triples' 'https://scigraph.springernature.com/pub.10.1007/978-3-540-69369-7_25'

    Turtle is a human-readable linked data format.

    curl -H 'Accept: text/turtle' 'https://scigraph.springernature.com/pub.10.1007/978-3-540-69369-7_25'

    RDF/XML is a standard XML format for linked data.

    curl -H 'Accept: application/rdf+xml' 'https://scigraph.springernature.com/pub.10.1007/978-3-540-69369-7_25'


     

    This table displays all metadata directly associated to this object as RDF triples.

    138 TRIPLES      23 PREDICATES      40 URIs      20 LITERALS      8 BLANK NODES

    Subject Predicate Object
    1 sg:pub.10.1007/978-3-540-69369-7_25 schema:about anzsrc-for:17
    2 anzsrc-for:1701
    3 schema:author N02b6b8a4dc7c4fcba07cf09b28c69833
    4 schema:citation sg:pub.10.1007/11573548_95
    5 https://doi.org/10.1016/0895-4356(90)90158-l
    6 https://doi.org/10.1016/0895-4356(95)00571-4
    7 https://doi.org/10.1016/j.neunet.2005.03.007
    8 https://doi.org/10.1016/j.specom.2005.09.008
    9 https://doi.org/10.1016/j.specom.2006.11.004
    10 https://doi.org/10.1037/h0031619
    11 https://doi.org/10.1037/h0077714
    12 https://doi.org/10.1109/asru.2003.1318399
    13 https://doi.org/10.1109/tsa.2004.838534
    14 https://doi.org/10.2307/2529310
    15 https://doi.org/10.2307/2529886
    16 https://doi.org/10.5860/choice.27-6381
    17 schema:datePublished 2008
    18 schema:datePublishedReg 2008-01-01
    19 schema:description In this paper we study the impact of three main factors on measuring the reliability of the annotation of non-acted emotions: the annotator biases, the similarity between the classified emotions, and the usage of contextual information during the annotation. We employed a corpus collected from real interactions between users and a spoken dialogue system. The user utterances were classified by nine non-expert annotators into four categories. We discuss the problems that the nature of non-acted emotional corpora impose in evaluating the reliability of the annotations using Kappa coefficients. Although deeply affected by the so-called paradoxes of Kappa coefficients, our study shows how taking into account context information and similarity between emotions helps to obtain values closer to the maximum agreement rates attainable, and allow the detection of emotions which are expressed more subtly by the users.
    20 schema:editor N93f1f986bef440ee89bbfc70d5f268cd
    21 schema:genre chapter
    22 schema:inLanguage en
    23 schema:isAccessibleForFree false
    24 schema:isPartOf Ne623d98a4ba6424fb31de573805ec21f
    25 schema:name On the Use of Kappa Coefficients to Measure the Reliability of the Annotation of Non-acted Emotions
    26 schema:pagination 221-232
    27 schema:productId N850b59a503144173b85dbf12918fa517
    28 N95938da2b7344e5ab20877deb10a1037
    29 Nd43539fd376d4472b1ccb6c7291bf93c
    30 schema:publisher N29cb3c1b04384aa6b9c5bfb258ecafd4
    31 schema:sameAs https://app.dimensions.ai/details/publication/pub.1006280503
    32 https://doi.org/10.1007/978-3-540-69369-7_25
    33 schema:sdDatePublished 2019-04-15T18:08
    34 schema:sdLicense https://scigraph.springernature.com/explorer/license/
    35 schema:sdPublisher Na376da549f4e42f4be44e35459fdaf53
    36 schema:url http://link.springer.com/10.1007/978-3-540-69369-7_25
    37 sgo:license sg:explorer/license/
    38 sgo:sdDataset chapters
    39 rdf:type schema:Chapter
    40 N02b6b8a4dc7c4fcba07cf09b28c69833 rdf:first sg:person.016307634167.10
    41 rdf:rest N8baa9ed5ba50481c98a69fa0724b7ee4
    42 N111fdbd3ccc14eed97866895b68b94e8 schema:familyName Dybkjær
    43 schema:givenName Laila
    44 rdf:type schema:Person
    45 N29cb3c1b04384aa6b9c5bfb258ecafd4 schema:location Berlin, Heidelberg
    46 schema:name Springer Berlin Heidelberg
    47 rdf:type schema:Organisation
    48 N2a59f9dbc8b84a48841e80b580ffa03f rdf:first Nf64dbc409457431691b10e5fce7b7a9f
    49 rdf:rest Ne46a38b9be2a4e7b99d8b7b7ec87bc64
    50 N550d4034a356461a953f101f349de6e0 schema:familyName Pieraccini
    51 schema:givenName Roberto
    52 rdf:type schema:Person
    53 N6c86cb77aed64b8d81a986080c618e6e rdf:first N550d4034a356461a953f101f349de6e0
    54 rdf:rest N99b12e6ee2a3403ead373569562e4acc
    55 N6e2de433224e48859cd45f2c3eeec55e rdf:first N111fdbd3ccc14eed97866895b68b94e8
    56 rdf:rest N2a59f9dbc8b84a48841e80b580ffa03f
    57 N6f237ac2189d4341b190f6bfc2aeeb62 schema:name Dept. of Languages and Computer Systems,  , 18071 Granada, Spain
    58 rdf:type schema:Organization
    59 N79db4b1c0ebc4970ac1553423d6a2b32 schema:name Dept. of Languages and Computer Systems,  , 18071 Granada, Spain
    60 rdf:type schema:Organization
    61 N850b59a503144173b85dbf12918fa517 schema:name doi
    62 schema:value 10.1007/978-3-540-69369-7_25
    63 rdf:type schema:PropertyValue
    64 N8baa9ed5ba50481c98a69fa0724b7ee4 rdf:first sg:person.013113720067.02
    65 rdf:rest rdf:nil
    66 N93f1f986bef440ee89bbfc70d5f268cd rdf:first N9707717d2b4b4725b73ece2acefa24c6
    67 rdf:rest N6e2de433224e48859cd45f2c3eeec55e
    68 N95938da2b7344e5ab20877deb10a1037 schema:name readcube_id
    69 schema:value 1dea849871ecc038bd85214a8aa10e7606d9b0f2f571bf5af2adc37b583f85bb
    70 rdf:type schema:PropertyValue
    71 N9707717d2b4b4725b73ece2acefa24c6 schema:familyName André
    72 schema:givenName Elisabeth
    73 rdf:type schema:Person
    74 N99b12e6ee2a3403ead373569562e4acc rdf:first Nd329331638164eeea8d72f59aca01f36
    75 rdf:rest rdf:nil
    76 Na376da549f4e42f4be44e35459fdaf53 schema:name Springer Nature - SN SciGraph project
    77 rdf:type schema:Organization
    78 Nd329331638164eeea8d72f59aca01f36 schema:familyName Weber
    79 schema:givenName Michael
    80 rdf:type schema:Person
    81 Nd43539fd376d4472b1ccb6c7291bf93c schema:name dimensions_id
    82 schema:value pub.1006280503
    83 rdf:type schema:PropertyValue
    84 Ne46a38b9be2a4e7b99d8b7b7ec87bc64 rdf:first Nea3b617f38164fc386062b9563d1c8c8
    85 rdf:rest N6c86cb77aed64b8d81a986080c618e6e
    86 Ne623d98a4ba6424fb31de573805ec21f schema:isbn 978-3-540-69368-0
    87 978-3-540-69369-7
    88 schema:name Perception in Multimodal Dialogue Systems
    89 rdf:type schema:Book
    90 Nea3b617f38164fc386062b9563d1c8c8 schema:familyName Neumann
    91 schema:givenName Heiko
    92 rdf:type schema:Person
    93 Nf64dbc409457431691b10e5fce7b7a9f schema:familyName Minker
    94 schema:givenName Wolfgang
    95 rdf:type schema:Person
    96 anzsrc-for:17 schema:inDefinedTermSet anzsrc-for:
    97 schema:name Psychology and Cognitive Sciences
    98 rdf:type schema:DefinedTerm
    99 anzsrc-for:1701 schema:inDefinedTermSet anzsrc-for:
    100 schema:name Psychology
    101 rdf:type schema:DefinedTerm
    102 sg:person.013113720067.02 schema:affiliation N6f237ac2189d4341b190f6bfc2aeeb62
    103 schema:familyName López-Cózar
    104 schema:givenName Ramón
    105 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.013113720067.02
    106 rdf:type schema:Person
    107 sg:person.016307634167.10 schema:affiliation N79db4b1c0ebc4970ac1553423d6a2b32
    108 schema:familyName Callejas
    109 schema:givenName Zoraida
    110 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.016307634167.10
    111 rdf:type schema:Person
    112 sg:pub.10.1007/11573548_95 schema:sameAs https://app.dimensions.ai/details/publication/pub.1000453836
    113 https://doi.org/10.1007/11573548_95
    114 rdf:type schema:CreativeWork
    115 https://doi.org/10.1016/0895-4356(90)90158-l schema:sameAs https://app.dimensions.ai/details/publication/pub.1007739743
    116 rdf:type schema:CreativeWork
    117 https://doi.org/10.1016/0895-4356(95)00571-4 schema:sameAs https://app.dimensions.ai/details/publication/pub.1006360632
    118 rdf:type schema:CreativeWork
    119 https://doi.org/10.1016/j.neunet.2005.03.007 schema:sameAs https://app.dimensions.ai/details/publication/pub.1010392184
    120 rdf:type schema:CreativeWork
    121 https://doi.org/10.1016/j.specom.2005.09.008 schema:sameAs https://app.dimensions.ai/details/publication/pub.1046798773
    122 rdf:type schema:CreativeWork
    123 https://doi.org/10.1016/j.specom.2006.11.004 schema:sameAs https://app.dimensions.ai/details/publication/pub.1021697340
    124 rdf:type schema:CreativeWork
    125 https://doi.org/10.1037/h0031619 schema:sameAs https://app.dimensions.ai/details/publication/pub.1004984215
    126 rdf:type schema:CreativeWork
    127 https://doi.org/10.1037/h0077714 schema:sameAs https://app.dimensions.ai/details/publication/pub.1026751541
    128 rdf:type schema:CreativeWork
    129 https://doi.org/10.1109/asru.2003.1318399 schema:sameAs https://app.dimensions.ai/details/publication/pub.1095196556
    130 rdf:type schema:CreativeWork
    131 https://doi.org/10.1109/tsa.2004.838534 schema:sameAs https://app.dimensions.ai/details/publication/pub.1061786254
    132 rdf:type schema:CreativeWork
    133 https://doi.org/10.2307/2529310 schema:sameAs https://app.dimensions.ai/details/publication/pub.1069974986
    134 rdf:type schema:CreativeWork
    135 https://doi.org/10.2307/2529886 schema:sameAs https://app.dimensions.ai/details/publication/pub.1069975476
    136 rdf:type schema:CreativeWork
    137 https://doi.org/10.5860/choice.27-6381 schema:sameAs https://app.dimensions.ai/details/publication/pub.1073298711
    138 rdf:type schema:CreativeWork
     




    Preview window. Press ESC to close (or click here)


    ...