Semi-formal Evaluation of Conversational Characters View Full Text


Ontology type: schema:Chapter     


Chapter Info

DATE

2009

AUTHORS

Ron Artstein , Sudeep Gandhe , Jillian Gerten , Anton Leuski , David Traum

ABSTRACT

Conversational dialogue systems cannot be evaluated in a fully formal manner, because dialogue is heavily dependent on context and current dialogue theory is not precise enough to specify a target output ahead of time. Instead, we evaluate dialogue systems in a semi-formal manner, using human judges to rate the coherence of a conversational character and correlating these judgments with measures extracted from within the system. We present a series of three evaluations of a single conversational character over the course of a year, demonstrating how this kind of evaluation helps bring about an improvement in overall dialogue coherence. More... »

PAGES

22-35

References to SciGraph publications

Book

TITLE

Languages: From Formal to Natural

ISBN

978-3-642-01747-6
978-3-642-01748-3

Author Affiliations

Identifiers

URI

http://scigraph.springernature.com/pub.10.1007/978-3-642-01748-3_2

DOI

http://dx.doi.org/10.1007/978-3-642-01748-3_2

DIMENSIONS

https://app.dimensions.ai/details/publication/pub.1036975569


Indexing Status Check whether this publication has been indexed by Scopus and Web Of Science using the SN Indexing Status Tool
Incoming Citations Browse incoming citations for this publication using opencitations.net

JSON-LD is the canonical representation for SciGraph data.

TIP: You can open this SciGraph record using an external JSON-LD service: JSON-LD Playground Google SDTT

[
  {
    "@context": "https://springernature.github.io/scigraph/jsonld/sgcontext.json", 
    "about": [
      {
        "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/1701", 
        "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
        "name": "Psychology", 
        "type": "DefinedTerm"
      }, 
      {
        "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/17", 
        "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
        "name": "Psychology and Cognitive Sciences", 
        "type": "DefinedTerm"
      }
    ], 
    "author": [
      {
        "affiliation": {
          "alternateName": "University of Southern California", 
          "id": "https://www.grid.ac/institutes/grid.42505.36", 
          "name": [
            "Institute for Creative Technologies, University of Southern California, 13274 Fiji Way, CA 90292, Marina del Rey, USA"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Artstein", 
        "givenName": "Ron", 
        "id": "sg:person.011476470243.70", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.011476470243.70"
        ], 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "University of Southern California", 
          "id": "https://www.grid.ac/institutes/grid.42505.36", 
          "name": [
            "Institute for Creative Technologies, University of Southern California, 13274 Fiji Way, CA 90292, Marina del Rey, USA"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Gandhe", 
        "givenName": "Sudeep", 
        "id": "sg:person.013667011643.39", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.013667011643.39"
        ], 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "University of Southern California", 
          "id": "https://www.grid.ac/institutes/grid.42505.36", 
          "name": [
            "Institute for Creative Technologies, University of Southern California, 13274 Fiji Way, CA 90292, Marina del Rey, USA"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Gerten", 
        "givenName": "Jillian", 
        "id": "sg:person.016057333243.53", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.016057333243.53"
        ], 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "University of Southern California", 
          "id": "https://www.grid.ac/institutes/grid.42505.36", 
          "name": [
            "Institute for Creative Technologies, University of Southern California, 13274 Fiji Way, CA 90292, Marina del Rey, USA"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Leuski", 
        "givenName": "Anton", 
        "id": "sg:person.015502340645.10", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.015502340645.10"
        ], 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "University of Southern California", 
          "id": "https://www.grid.ac/institutes/grid.42505.36", 
          "name": [
            "Institute for Creative Technologies, University of Southern California, 13274 Fiji Way, CA 90292, Marina del Rey, USA"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Traum", 
        "givenName": "David", 
        "id": "sg:person.013767173315.17", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.013767173315.17"
        ], 
        "type": "Person"
      }
    ], 
    "citation": [
      {
        "id": "sg:pub.10.1007/11821830_10", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1028295289", 
          "https://doi.org/10.1007/11821830_10"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "sg:pub.10.1007/11821830_10", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1028295289", 
          "https://doi.org/10.1007/11821830_10"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "https://doi.org/10.1162/coli.07-034-r2", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1045615725"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "https://doi.org/10.1080/19312450709336664", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1053463751"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "https://doi.org/10.1109/89.817450", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1061242562"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "https://doi.org/10.21236/ada273556", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1091558316"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "https://doi.org/10.3115/1654595.1654600", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1099238187"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "https://doi.org/10.1613/jair.713", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1105579507"
        ], 
        "type": "CreativeWork"
      }
    ], 
    "datePublished": "2009", 
    "datePublishedReg": "2009-01-01", 
    "description": "Conversational dialogue systems cannot be evaluated in a fully formal manner, because dialogue is heavily dependent on context and current dialogue theory is not precise enough to specify a target output ahead of time. Instead, we evaluate dialogue systems in a semi-formal manner, using human judges to rate the coherence of a conversational character and correlating these judgments with measures extracted from within the system. We present a series of three evaluations of a single conversational character over the course of a year, demonstrating how this kind of evaluation helps bring about an improvement in overall dialogue coherence.", 
    "editor": [
      {
        "familyName": "Grumberg", 
        "givenName": "Orna", 
        "type": "Person"
      }, 
      {
        "familyName": "Kaminski", 
        "givenName": "Michael", 
        "type": "Person"
      }, 
      {
        "familyName": "Katz", 
        "givenName": "Shmuel", 
        "type": "Person"
      }, 
      {
        "familyName": "Wintner", 
        "givenName": "Shuly", 
        "type": "Person"
      }
    ], 
    "genre": "chapter", 
    "id": "sg:pub.10.1007/978-3-642-01748-3_2", 
    "inLanguage": [
      "en"
    ], 
    "isAccessibleForFree": false, 
    "isPartOf": {
      "isbn": [
        "978-3-642-01747-6", 
        "978-3-642-01748-3"
      ], 
      "name": "Languages: From Formal to Natural", 
      "type": "Book"
    }, 
    "name": "Semi-formal Evaluation of Conversational Characters", 
    "pagination": "22-35", 
    "productId": [
      {
        "name": "dimensions_id", 
        "type": "PropertyValue", 
        "value": [
          "pub.1036975569"
        ]
      }, 
      {
        "name": "doi", 
        "type": "PropertyValue", 
        "value": [
          "10.1007/978-3-642-01748-3_2"
        ]
      }, 
      {
        "name": "readcube_id", 
        "type": "PropertyValue", 
        "value": [
          "b0235e9fe28cc59f74de320591f0914da2f59d5299220ed83ee6568f8f57dec1"
        ]
      }
    ], 
    "publisher": {
      "location": "Berlin, Heidelberg", 
      "name": "Springer Berlin Heidelberg", 
      "type": "Organisation"
    }, 
    "sameAs": [
      "https://doi.org/10.1007/978-3-642-01748-3_2", 
      "https://app.dimensions.ai/details/publication/pub.1036975569"
    ], 
    "sdDataset": "chapters", 
    "sdDatePublished": "2019-04-16T07:16", 
    "sdLicense": "https://scigraph.springernature.com/explorer/license/", 
    "sdPublisher": {
      "name": "Springer Nature - SN SciGraph project", 
      "type": "Organization"
    }, 
    "sdSource": "s3://com-uberresearch-data-dimensions-target-20181106-alternative/cleanup/v134/2549eaecd7973599484d7c17b260dba0a4ecb94b/merge/v9/a6c9fde33151104705d4d7ff012ea9563521a3ce/jats-lookup/v90/0000000353_0000000353/records_45374_00000000.jsonl", 
    "type": "Chapter", 
    "url": "https://link.springer.com/10.1007%2F978-3-642-01748-3_2"
  }
]
 

Download the RDF metadata as:  json-ld nt turtle xml License info

HOW TO GET THIS DATA PROGRAMMATICALLY:

JSON-LD is a popular format for linked data which is fully compatible with JSON.

curl -H 'Accept: application/ld+json' 'https://scigraph.springernature.com/pub.10.1007/978-3-642-01748-3_2'

N-Triples is a line-based linked data format ideal for batch operations.

curl -H 'Accept: application/n-triples' 'https://scigraph.springernature.com/pub.10.1007/978-3-642-01748-3_2'

Turtle is a human-readable linked data format.

curl -H 'Accept: text/turtle' 'https://scigraph.springernature.com/pub.10.1007/978-3-642-01748-3_2'

RDF/XML is a standard XML format for linked data.

curl -H 'Accept: application/rdf+xml' 'https://scigraph.springernature.com/pub.10.1007/978-3-642-01748-3_2'


 

This table displays all metadata directly associated to this object as RDF triples.

130 TRIPLES      23 PREDICATES      34 URIs      20 LITERALS      8 BLANK NODES

Subject Predicate Object
1 sg:pub.10.1007/978-3-642-01748-3_2 schema:about anzsrc-for:17
2 anzsrc-for:1701
3 schema:author N2642a9718f7f42ffb9902905ba4e9ac1
4 schema:citation sg:pub.10.1007/11821830_10
5 https://doi.org/10.1080/19312450709336664
6 https://doi.org/10.1109/89.817450
7 https://doi.org/10.1162/coli.07-034-r2
8 https://doi.org/10.1613/jair.713
9 https://doi.org/10.21236/ada273556
10 https://doi.org/10.3115/1654595.1654600
11 schema:datePublished 2009
12 schema:datePublishedReg 2009-01-01
13 schema:description Conversational dialogue systems cannot be evaluated in a fully formal manner, because dialogue is heavily dependent on context and current dialogue theory is not precise enough to specify a target output ahead of time. Instead, we evaluate dialogue systems in a semi-formal manner, using human judges to rate the coherence of a conversational character and correlating these judgments with measures extracted from within the system. We present a series of three evaluations of a single conversational character over the course of a year, demonstrating how this kind of evaluation helps bring about an improvement in overall dialogue coherence.
14 schema:editor N29012c9d520b4a1584c08353cd21fcac
15 schema:genre chapter
16 schema:inLanguage en
17 schema:isAccessibleForFree false
18 schema:isPartOf Ndaf87b3f26ae4ebdaf196de746c6064a
19 schema:name Semi-formal Evaluation of Conversational Characters
20 schema:pagination 22-35
21 schema:productId N222edbf8d82a49ba9160236b2db8d63f
22 N29e1053d03f04bcd9fc0673a7d6f1688
23 N557cb9991c8c436fa3f87a51759eca99
24 schema:publisher N98c0bffbbe9c4dd48954d33a9f149eda
25 schema:sameAs https://app.dimensions.ai/details/publication/pub.1036975569
26 https://doi.org/10.1007/978-3-642-01748-3_2
27 schema:sdDatePublished 2019-04-16T07:16
28 schema:sdLicense https://scigraph.springernature.com/explorer/license/
29 schema:sdPublisher N53a22141291e46a6a6f7ddab28b4fa41
30 schema:url https://link.springer.com/10.1007%2F978-3-642-01748-3_2
31 sgo:license sg:explorer/license/
32 sgo:sdDataset chapters
33 rdf:type schema:Chapter
34 N064b5761e7a14a8d8b68e46bbb8d1511 rdf:first N86b762eccb7c4597ab7d45166ad62fef
35 rdf:rest Nc6143eeca06b452b88434c1666a226d8
36 N222edbf8d82a49ba9160236b2db8d63f schema:name doi
37 schema:value 10.1007/978-3-642-01748-3_2
38 rdf:type schema:PropertyValue
39 N2642a9718f7f42ffb9902905ba4e9ac1 rdf:first sg:person.011476470243.70
40 rdf:rest N8c96bbde90754cf0abf9f7f6b5e5a244
41 N29012c9d520b4a1584c08353cd21fcac rdf:first N6c077ac7af164c7e8eff8e45ce920f6e
42 rdf:rest N9089e6d4e6d947608d6ece5bdacaec61
43 N29e1053d03f04bcd9fc0673a7d6f1688 schema:name readcube_id
44 schema:value b0235e9fe28cc59f74de320591f0914da2f59d5299220ed83ee6568f8f57dec1
45 rdf:type schema:PropertyValue
46 N53a22141291e46a6a6f7ddab28b4fa41 schema:name Springer Nature - SN SciGraph project
47 rdf:type schema:Organization
48 N557cb9991c8c436fa3f87a51759eca99 schema:name dimensions_id
49 schema:value pub.1036975569
50 rdf:type schema:PropertyValue
51 N6c077ac7af164c7e8eff8e45ce920f6e schema:familyName Grumberg
52 schema:givenName Orna
53 rdf:type schema:Person
54 N86b762eccb7c4597ab7d45166ad62fef schema:familyName Katz
55 schema:givenName Shmuel
56 rdf:type schema:Person
57 N8c96bbde90754cf0abf9f7f6b5e5a244 rdf:first sg:person.013667011643.39
58 rdf:rest Nd774341257f746d7a95179556a9935a5
59 N9089e6d4e6d947608d6ece5bdacaec61 rdf:first Nf384b3dc884a4a09b1349ca0ad7378b0
60 rdf:rest N064b5761e7a14a8d8b68e46bbb8d1511
61 N98c0bffbbe9c4dd48954d33a9f149eda schema:location Berlin, Heidelberg
62 schema:name Springer Berlin Heidelberg
63 rdf:type schema:Organisation
64 N9994a54c239e40198f9d5f7050a0d101 schema:familyName Wintner
65 schema:givenName Shuly
66 rdf:type schema:Person
67 N99c4fca7d7774934b2efa32a7b08c1d4 rdf:first sg:person.013767173315.17
68 rdf:rest rdf:nil
69 Nc250f9bc929f4139b24d7a77fe3dac76 rdf:first sg:person.015502340645.10
70 rdf:rest N99c4fca7d7774934b2efa32a7b08c1d4
71 Nc6143eeca06b452b88434c1666a226d8 rdf:first N9994a54c239e40198f9d5f7050a0d101
72 rdf:rest rdf:nil
73 Nd774341257f746d7a95179556a9935a5 rdf:first sg:person.016057333243.53
74 rdf:rest Nc250f9bc929f4139b24d7a77fe3dac76
75 Ndaf87b3f26ae4ebdaf196de746c6064a schema:isbn 978-3-642-01747-6
76 978-3-642-01748-3
77 schema:name Languages: From Formal to Natural
78 rdf:type schema:Book
79 Nf384b3dc884a4a09b1349ca0ad7378b0 schema:familyName Kaminski
80 schema:givenName Michael
81 rdf:type schema:Person
82 anzsrc-for:17 schema:inDefinedTermSet anzsrc-for:
83 schema:name Psychology and Cognitive Sciences
84 rdf:type schema:DefinedTerm
85 anzsrc-for:1701 schema:inDefinedTermSet anzsrc-for:
86 schema:name Psychology
87 rdf:type schema:DefinedTerm
88 sg:person.011476470243.70 schema:affiliation https://www.grid.ac/institutes/grid.42505.36
89 schema:familyName Artstein
90 schema:givenName Ron
91 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.011476470243.70
92 rdf:type schema:Person
93 sg:person.013667011643.39 schema:affiliation https://www.grid.ac/institutes/grid.42505.36
94 schema:familyName Gandhe
95 schema:givenName Sudeep
96 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.013667011643.39
97 rdf:type schema:Person
98 sg:person.013767173315.17 schema:affiliation https://www.grid.ac/institutes/grid.42505.36
99 schema:familyName Traum
100 schema:givenName David
101 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.013767173315.17
102 rdf:type schema:Person
103 sg:person.015502340645.10 schema:affiliation https://www.grid.ac/institutes/grid.42505.36
104 schema:familyName Leuski
105 schema:givenName Anton
106 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.015502340645.10
107 rdf:type schema:Person
108 sg:person.016057333243.53 schema:affiliation https://www.grid.ac/institutes/grid.42505.36
109 schema:familyName Gerten
110 schema:givenName Jillian
111 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.016057333243.53
112 rdf:type schema:Person
113 sg:pub.10.1007/11821830_10 schema:sameAs https://app.dimensions.ai/details/publication/pub.1028295289
114 https://doi.org/10.1007/11821830_10
115 rdf:type schema:CreativeWork
116 https://doi.org/10.1080/19312450709336664 schema:sameAs https://app.dimensions.ai/details/publication/pub.1053463751
117 rdf:type schema:CreativeWork
118 https://doi.org/10.1109/89.817450 schema:sameAs https://app.dimensions.ai/details/publication/pub.1061242562
119 rdf:type schema:CreativeWork
120 https://doi.org/10.1162/coli.07-034-r2 schema:sameAs https://app.dimensions.ai/details/publication/pub.1045615725
121 rdf:type schema:CreativeWork
122 https://doi.org/10.1613/jair.713 schema:sameAs https://app.dimensions.ai/details/publication/pub.1105579507
123 rdf:type schema:CreativeWork
124 https://doi.org/10.21236/ada273556 schema:sameAs https://app.dimensions.ai/details/publication/pub.1091558316
125 rdf:type schema:CreativeWork
126 https://doi.org/10.3115/1654595.1654600 schema:sameAs https://app.dimensions.ai/details/publication/pub.1099238187
127 rdf:type schema:CreativeWork
128 https://www.grid.ac/institutes/grid.42505.36 schema:alternateName University of Southern California
129 schema:name Institute for Creative Technologies, University of Southern California, 13274 Fiji Way, CA 90292, Marina del Rey, USA
130 rdf:type schema:Organization
 




Preview window. Press ESC to close (or click here)


...