A novel rater agreement methodology for language transcriptions: evidence from a nonhuman speaker View Full Text


Ontology type: schema:ScholarlyArticle     


Article Info

DATE

2014-07

AUTHORS

Allison B. Kaufman, Erin N. Colbert-White, Robert Rosenthal

ABSTRACT

The ability to measure agreement between two independent observers is vital to any observational study. We use a unique situation, the calculation of inter-rater reliability for transcriptions of a parrot’s speech, to present a novel method of dealing with inter-rater reliability which we believe can be applied to situations in which speech from human subjects may be difficult to transcribe. Challenges encountered included (1) a sparse original agreement matrix which yielded an omnibus measure of inter-rater reliability, (2) “lopsided” 2×2 matrices (i.e. subsets) from the overall matrix and (3) categories used by the transcribers which could not be pre-determined. Our novel approach involved calculating reliability on two levels—that of the corpus and that of the above mentioned smaller subsets of data. Specifically, the technique included the “reverse engineering” of categories, the use of a “null” category when one rater observed a behavior and the other did not, and the use of Fisher’s Exact Test to calculate r-equivalent for the smaller paired subset comparisons. We hope this technique will be useful to those working in similar situations where speech may be difficult to transcribe, such as with small children. More... »

PAGES

2329-2339

References to SciGraph publications

Identifiers

URI

http://scigraph.springernature.com/pub.10.1007/s11135-013-9894-5

DOI

http://dx.doi.org/10.1007/s11135-013-9894-5

DIMENSIONS

https://app.dimensions.ai/details/publication/pub.1051231756


Indexing Status Check whether this publication has been indexed by Scopus and Web Of Science using the SN Indexing Status Tool
Incoming Citations Browse incoming citations for this publication using opencitations.net

JSON-LD is the canonical representation for SciGraph data.

TIP: You can open this SciGraph record using an external JSON-LD service: JSON-LD Playground Google SDTT

[
  {
    "@context": "https://springernature.github.io/scigraph/jsonld/sgcontext.json", 
    "about": [
      {
        "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/1103", 
        "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
        "name": "Clinical Sciences", 
        "type": "DefinedTerm"
      }, 
      {
        "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/11", 
        "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
        "name": "Medical and Health Sciences", 
        "type": "DefinedTerm"
      }
    ], 
    "author": [
      {
        "affiliation": {
          "alternateName": "California State University, San Bernardino", 
          "id": "https://www.grid.ac/institutes/grid.253565.2", 
          "name": [
            "California State University, San Bernardino, 5500 University Parkway, 92407, San Bernardino, CA, USA"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Kaufman", 
        "givenName": "Allison B.", 
        "id": "sg:person.0673562446.80", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.0673562446.80"
        ], 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "University of Puget Sound", 
          "id": "https://www.grid.ac/institutes/grid.267047.0", 
          "name": [
            "Department of Psychology, University of Puget Sound, 1500 N. Warner #1046, 98416, Tacoma, WA, USA"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Colbert-White", 
        "givenName": "Erin N.", 
        "id": "sg:person.01276113332.29", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01276113332.29"
        ], 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "University of California, Riverside", 
          "id": "https://www.grid.ac/institutes/grid.266097.c", 
          "name": [
            "Department of Psychology, University of California, Riverside, 900 University Ave., 92521, Riverside, CA, USA"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Rosenthal", 
        "givenName": "Robert", 
        "type": "Person"
      }
    ], 
    "citation": [
      {
        "id": "sg:pub.10.1007/bf00167757", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1000992588", 
          "https://doi.org/10.1007/bf00167757"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "sg:pub.10.1007/bf00167757", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1000992588", 
          "https://doi.org/10.1007/bf00167757"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "sg:pub.10.1007/bf00180886", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1002210229", 
          "https://doi.org/10.1007/bf00180886"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "sg:pub.10.1007/bf02143958", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1005675555", 
          "https://doi.org/10.1007/bf02143958"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "sg:pub.10.1007/bf02143958", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1005675555", 
          "https://doi.org/10.1007/bf02143958"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "https://doi.org/10.1016/j.anbehav.2009.09.014", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1007138701"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "https://doi.org/10.1037/0022-0663.74.2.166", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1008407883"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "https://doi.org/10.1037/h0076640", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1009190130"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "https://doi.org/10.1037/1082-989x.8.4.492", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1011616675"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "https://doi.org/10.1111/j.2044-8317.1974.tb00535.x", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1021224832"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "https://doi.org/10.1037/a0022097", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1028551848"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "https://doi.org/10.1177/001316446002000104", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1039619716"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "https://doi.org/10.1177/001316446002000104", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1039619716"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "https://doi.org/10.1111/j.2044-8317.1977.tb00728.x", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1048852595"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "https://doi.org/10.1177/01427237030233001", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1053748454"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "https://doi.org/10.1177/01427237030233001", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1053748454"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "https://doi.org/10.1017/s0142716410000032", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1053902178"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "https://doi.org/10.1017/s0142716410000032", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1053902178"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "https://doi.org/10.1086/266577", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1058577190"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "https://doi.org/10.1086/268378", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1058578990"
        ], 
        "type": "CreativeWork"
      }, 
      {
        "id": "https://app.dimensions.ai/details/publication/pub.1081668252", 
        "type": "CreativeWork"
      }, 
      {
        "id": "https://doi.org/10.1093/acprof:oso/9780198529620.003.0005", 
        "sameAs": [
          "https://app.dimensions.ai/details/publication/pub.1089175027"
        ], 
        "type": "CreativeWork"
      }
    ], 
    "datePublished": "2014-07", 
    "datePublishedReg": "2014-07-01", 
    "description": "The ability to measure agreement between two independent observers is vital to any observational study. We use a unique situation, the calculation of inter-rater reliability for transcriptions of a parrot\u2019s speech, to present a novel method of dealing with inter-rater reliability which we believe can be applied to situations in which speech from human subjects may be difficult to transcribe. Challenges encountered included (1) a sparse original agreement matrix which yielded an omnibus measure of inter-rater reliability, (2) \u201clopsided\u201d 2\u00d72 matrices (i.e. subsets) from the overall matrix and (3) categories used by the transcribers which could not be pre-determined. Our novel approach involved calculating reliability on two levels\u2014that of the corpus and that of the above mentioned smaller subsets of data. Specifically, the technique included the \u201creverse engineering\u201d of categories, the use of a \u201cnull\u201d category when one rater observed a behavior and the other did not, and the use of Fisher\u2019s Exact Test to calculate r-equivalent for the smaller paired subset comparisons. We hope this technique will be useful to those working in similar situations where speech may be difficult to transcribe, such as with small children.", 
    "genre": "research_article", 
    "id": "sg:pub.10.1007/s11135-013-9894-5", 
    "inLanguage": [
      "en"
    ], 
    "isAccessibleForFree": false, 
    "isPartOf": [
      {
        "id": "sg:journal.1053888", 
        "issn": [
          "0033-5177", 
          "1573-7845"
        ], 
        "name": "Quality & Quantity", 
        "type": "Periodical"
      }, 
      {
        "issueNumber": "4", 
        "type": "PublicationIssue"
      }, 
      {
        "type": "PublicationVolume", 
        "volumeNumber": "48"
      }
    ], 
    "name": "A novel rater agreement methodology for language transcriptions: evidence from a nonhuman speaker", 
    "pagination": "2329-2339", 
    "productId": [
      {
        "name": "readcube_id", 
        "type": "PropertyValue", 
        "value": [
          "f40bea62adcc774fa5103b3b8dde1f4a9a814c792ac367183974e7f282ea95a0"
        ]
      }, 
      {
        "name": "doi", 
        "type": "PropertyValue", 
        "value": [
          "10.1007/s11135-013-9894-5"
        ]
      }, 
      {
        "name": "dimensions_id", 
        "type": "PropertyValue", 
        "value": [
          "pub.1051231756"
        ]
      }
    ], 
    "sameAs": [
      "https://doi.org/10.1007/s11135-013-9894-5", 
      "https://app.dimensions.ai/details/publication/pub.1051231756"
    ], 
    "sdDataset": "articles", 
    "sdDatePublished": "2019-04-11T00:19", 
    "sdLicense": "https://scigraph.springernature.com/explorer/license/", 
    "sdPublisher": {
      "name": "Springer Nature - SN SciGraph project", 
      "type": "Organization"
    }, 
    "sdSource": "s3://com-uberresearch-data-dimensions-target-20181106-alternative/cleanup/v134/2549eaecd7973599484d7c17b260dba0a4ecb94b/merge/v9/a6c9fde33151104705d4d7ff012ea9563521a3ce/jats-lookup/v90/0000000001_0000000264/records_8695_00000524.jsonl", 
    "type": "ScholarlyArticle", 
    "url": "http://link.springer.com/10.1007%2Fs11135-013-9894-5"
  }
]
 

Download the RDF metadata as:  json-ld nt turtle xml License info

HOW TO GET THIS DATA PROGRAMMATICALLY:

JSON-LD is a popular format for linked data which is fully compatible with JSON.

curl -H 'Accept: application/ld+json' 'https://scigraph.springernature.com/pub.10.1007/s11135-013-9894-5'

N-Triples is a line-based linked data format ideal for batch operations.

curl -H 'Accept: application/n-triples' 'https://scigraph.springernature.com/pub.10.1007/s11135-013-9894-5'

Turtle is a human-readable linked data format.

curl -H 'Accept: text/turtle' 'https://scigraph.springernature.com/pub.10.1007/s11135-013-9894-5'

RDF/XML is a standard XML format for linked data.

curl -H 'Accept: application/rdf+xml' 'https://scigraph.springernature.com/pub.10.1007/s11135-013-9894-5'


 

This table displays all metadata directly associated to this object as RDF triples.

133 TRIPLES      21 PREDICATES      44 URIs      19 LITERALS      7 BLANK NODES

Subject Predicate Object
1 sg:pub.10.1007/s11135-013-9894-5 schema:about anzsrc-for:11
2 anzsrc-for:1103
3 schema:author Nfb71a81cad3c4d65a8f0e834fb563899
4 schema:citation sg:pub.10.1007/bf00167757
5 sg:pub.10.1007/bf00180886
6 sg:pub.10.1007/bf02143958
7 https://app.dimensions.ai/details/publication/pub.1081668252
8 https://doi.org/10.1016/j.anbehav.2009.09.014
9 https://doi.org/10.1017/s0142716410000032
10 https://doi.org/10.1037/0022-0663.74.2.166
11 https://doi.org/10.1037/1082-989x.8.4.492
12 https://doi.org/10.1037/a0022097
13 https://doi.org/10.1037/h0076640
14 https://doi.org/10.1086/266577
15 https://doi.org/10.1086/268378
16 https://doi.org/10.1093/acprof:oso/9780198529620.003.0005
17 https://doi.org/10.1111/j.2044-8317.1974.tb00535.x
18 https://doi.org/10.1111/j.2044-8317.1977.tb00728.x
19 https://doi.org/10.1177/001316446002000104
20 https://doi.org/10.1177/01427237030233001
21 schema:datePublished 2014-07
22 schema:datePublishedReg 2014-07-01
23 schema:description The ability to measure agreement between two independent observers is vital to any observational study. We use a unique situation, the calculation of inter-rater reliability for transcriptions of a parrot’s speech, to present a novel method of dealing with inter-rater reliability which we believe can be applied to situations in which speech from human subjects may be difficult to transcribe. Challenges encountered included (1) a sparse original agreement matrix which yielded an omnibus measure of inter-rater reliability, (2) “lopsided” 2×2 matrices (i.e. subsets) from the overall matrix and (3) categories used by the transcribers which could not be pre-determined. Our novel approach involved calculating reliability on two levels—that of the corpus and that of the above mentioned smaller subsets of data. Specifically, the technique included the “reverse engineering” of categories, the use of a “null” category when one rater observed a behavior and the other did not, and the use of Fisher’s Exact Test to calculate r-equivalent for the smaller paired subset comparisons. We hope this technique will be useful to those working in similar situations where speech may be difficult to transcribe, such as with small children.
24 schema:genre research_article
25 schema:inLanguage en
26 schema:isAccessibleForFree false
27 schema:isPartOf N5bc0b310a8ec49478246fe58e1fed221
28 Na0f66aca29af47d3acff44a30025eec5
29 sg:journal.1053888
30 schema:name A novel rater agreement methodology for language transcriptions: evidence from a nonhuman speaker
31 schema:pagination 2329-2339
32 schema:productId N4e9f08fc665741888fb6852d4a99e169
33 Nbb81201be1cb4a07876010306685a64c
34 Nff4c0a716d1f423a8427c0807aabbbd2
35 schema:sameAs https://app.dimensions.ai/details/publication/pub.1051231756
36 https://doi.org/10.1007/s11135-013-9894-5
37 schema:sdDatePublished 2019-04-11T00:19
38 schema:sdLicense https://scigraph.springernature.com/explorer/license/
39 schema:sdPublisher N09016ceec9464711a45e5bb9aa6a180f
40 schema:url http://link.springer.com/10.1007%2Fs11135-013-9894-5
41 sgo:license sg:explorer/license/
42 sgo:sdDataset articles
43 rdf:type schema:ScholarlyArticle
44 N07f712061e1e44c69a39687c63dbd421 rdf:first N99bdc5e7363a4d9094118cc83fdf671f
45 rdf:rest rdf:nil
46 N09016ceec9464711a45e5bb9aa6a180f schema:name Springer Nature - SN SciGraph project
47 rdf:type schema:Organization
48 N4e9f08fc665741888fb6852d4a99e169 schema:name dimensions_id
49 schema:value pub.1051231756
50 rdf:type schema:PropertyValue
51 N5bc0b310a8ec49478246fe58e1fed221 schema:volumeNumber 48
52 rdf:type schema:PublicationVolume
53 N99bdc5e7363a4d9094118cc83fdf671f schema:affiliation https://www.grid.ac/institutes/grid.266097.c
54 schema:familyName Rosenthal
55 schema:givenName Robert
56 rdf:type schema:Person
57 Na0f66aca29af47d3acff44a30025eec5 schema:issueNumber 4
58 rdf:type schema:PublicationIssue
59 Nbb81201be1cb4a07876010306685a64c schema:name readcube_id
60 schema:value f40bea62adcc774fa5103b3b8dde1f4a9a814c792ac367183974e7f282ea95a0
61 rdf:type schema:PropertyValue
62 Ndb1440458d3f460ea15af8efa70c0045 rdf:first sg:person.01276113332.29
63 rdf:rest N07f712061e1e44c69a39687c63dbd421
64 Nfb71a81cad3c4d65a8f0e834fb563899 rdf:first sg:person.0673562446.80
65 rdf:rest Ndb1440458d3f460ea15af8efa70c0045
66 Nff4c0a716d1f423a8427c0807aabbbd2 schema:name doi
67 schema:value 10.1007/s11135-013-9894-5
68 rdf:type schema:PropertyValue
69 anzsrc-for:11 schema:inDefinedTermSet anzsrc-for:
70 schema:name Medical and Health Sciences
71 rdf:type schema:DefinedTerm
72 anzsrc-for:1103 schema:inDefinedTermSet anzsrc-for:
73 schema:name Clinical Sciences
74 rdf:type schema:DefinedTerm
75 sg:journal.1053888 schema:issn 0033-5177
76 1573-7845
77 schema:name Quality & Quantity
78 rdf:type schema:Periodical
79 sg:person.01276113332.29 schema:affiliation https://www.grid.ac/institutes/grid.267047.0
80 schema:familyName Colbert-White
81 schema:givenName Erin N.
82 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01276113332.29
83 rdf:type schema:Person
84 sg:person.0673562446.80 schema:affiliation https://www.grid.ac/institutes/grid.253565.2
85 schema:familyName Kaufman
86 schema:givenName Allison B.
87 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.0673562446.80
88 rdf:type schema:Person
89 sg:pub.10.1007/bf00167757 schema:sameAs https://app.dimensions.ai/details/publication/pub.1000992588
90 https://doi.org/10.1007/bf00167757
91 rdf:type schema:CreativeWork
92 sg:pub.10.1007/bf00180886 schema:sameAs https://app.dimensions.ai/details/publication/pub.1002210229
93 https://doi.org/10.1007/bf00180886
94 rdf:type schema:CreativeWork
95 sg:pub.10.1007/bf02143958 schema:sameAs https://app.dimensions.ai/details/publication/pub.1005675555
96 https://doi.org/10.1007/bf02143958
97 rdf:type schema:CreativeWork
98 https://app.dimensions.ai/details/publication/pub.1081668252 schema:CreativeWork
99 https://doi.org/10.1016/j.anbehav.2009.09.014 schema:sameAs https://app.dimensions.ai/details/publication/pub.1007138701
100 rdf:type schema:CreativeWork
101 https://doi.org/10.1017/s0142716410000032 schema:sameAs https://app.dimensions.ai/details/publication/pub.1053902178
102 rdf:type schema:CreativeWork
103 https://doi.org/10.1037/0022-0663.74.2.166 schema:sameAs https://app.dimensions.ai/details/publication/pub.1008407883
104 rdf:type schema:CreativeWork
105 https://doi.org/10.1037/1082-989x.8.4.492 schema:sameAs https://app.dimensions.ai/details/publication/pub.1011616675
106 rdf:type schema:CreativeWork
107 https://doi.org/10.1037/a0022097 schema:sameAs https://app.dimensions.ai/details/publication/pub.1028551848
108 rdf:type schema:CreativeWork
109 https://doi.org/10.1037/h0076640 schema:sameAs https://app.dimensions.ai/details/publication/pub.1009190130
110 rdf:type schema:CreativeWork
111 https://doi.org/10.1086/266577 schema:sameAs https://app.dimensions.ai/details/publication/pub.1058577190
112 rdf:type schema:CreativeWork
113 https://doi.org/10.1086/268378 schema:sameAs https://app.dimensions.ai/details/publication/pub.1058578990
114 rdf:type schema:CreativeWork
115 https://doi.org/10.1093/acprof:oso/9780198529620.003.0005 schema:sameAs https://app.dimensions.ai/details/publication/pub.1089175027
116 rdf:type schema:CreativeWork
117 https://doi.org/10.1111/j.2044-8317.1974.tb00535.x schema:sameAs https://app.dimensions.ai/details/publication/pub.1021224832
118 rdf:type schema:CreativeWork
119 https://doi.org/10.1111/j.2044-8317.1977.tb00728.x schema:sameAs https://app.dimensions.ai/details/publication/pub.1048852595
120 rdf:type schema:CreativeWork
121 https://doi.org/10.1177/001316446002000104 schema:sameAs https://app.dimensions.ai/details/publication/pub.1039619716
122 rdf:type schema:CreativeWork
123 https://doi.org/10.1177/01427237030233001 schema:sameAs https://app.dimensions.ai/details/publication/pub.1053748454
124 rdf:type schema:CreativeWork
125 https://www.grid.ac/institutes/grid.253565.2 schema:alternateName California State University, San Bernardino
126 schema:name California State University, San Bernardino, 5500 University Parkway, 92407, San Bernardino, CA, USA
127 rdf:type schema:Organization
128 https://www.grid.ac/institutes/grid.266097.c schema:alternateName University of California, Riverside
129 schema:name Department of Psychology, University of California, Riverside, 900 University Ave., 92521, Riverside, CA, USA
130 rdf:type schema:Organization
131 https://www.grid.ac/institutes/grid.267047.0 schema:alternateName University of Puget Sound
132 schema:name Department of Psychology, University of Puget Sound, 1500 N. Warner #1046, 98416, Tacoma, WA, USA
133 rdf:type schema:Organization
 




Preview window. Press ESC to close (or click here)


...