Variation of Relevance Assessments for Medical Image Retrieval View Full Text


Ontology type: schema:Chapter      Open Access: True


Chapter Info

DATE

2007

AUTHORS

Henning Müller , Paul Clough , Bill Hersh , Antoine Geissbühler

ABSTRACT

Evaluation is crucial for the success of most research domains, and image retrieval is no exception to this. Recently, several benchmarks have been developed for visual information retrieval such as TRECVID, ImageCLEF, and ImagEval to create frameworks for evaluating image retrieval research. An important part of evaluation is the creation of a ground truth or gold standard to evaluate systems against. Much experience has been gained on creating ground truths for textual information retrieval, but for image retrieval these issues require further research. This article will present the process of generating relevance judgements for the medical image retrieval task of ImageCLEF. Many of the problems encountered can be generalised to other image retrieval tasks as well, so the outcome is not limited to the medical domain. Part of the images analysed for relevance were judged by two assessors, and these are analysed with respect to their consistency and potential problems. Our goal is to obtain more information on the ambiguity of the topics developed and generally to keep the variation amongst relevance assessors low. This might partially reduce the subjectivity of system-oriented evaluation, although the evaluation shows that the differences in relevance judgements only have a limited influence on comparative system ranking. A number of outcomes are presented with a goal in mind to create less ambiguous topics for future evaluation campaigns. More... »

PAGES

232-246

Identifiers

URI

http://scigraph.springernature.com/pub.10.1007/978-3-540-71545-0_18

DOI

http://dx.doi.org/10.1007/978-3-540-71545-0_18

DIMENSIONS

https://app.dimensions.ai/details/publication/pub.1027200331


Indexing Status Check whether this publication has been indexed by Scopus and Web Of Science using the SN Indexing Status Tool
Incoming Citations Browse incoming citations for this publication using opencitations.net

JSON-LD is the canonical representation for SciGraph data.

TIP: You can open this SciGraph record using an external JSON-LD service: JSON-LD Playground Google SDTT

[
  {
    "@context": "https://springernature.github.io/scigraph/jsonld/sgcontext.json", 
    "about": [
      {
        "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/08", 
        "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
        "name": "Information and Computing Sciences", 
        "type": "DefinedTerm"
      }, 
      {
        "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/17", 
        "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
        "name": "Psychology and Cognitive Sciences", 
        "type": "DefinedTerm"
      }, 
      {
        "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/0801", 
        "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
        "name": "Artificial Intelligence and Image Processing", 
        "type": "DefinedTerm"
      }, 
      {
        "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/0806", 
        "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
        "name": "Information Systems", 
        "type": "DefinedTerm"
      }, 
      {
        "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/1701", 
        "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
        "name": "Psychology", 
        "type": "DefinedTerm"
      }
    ], 
    "author": [
      {
        "affiliation": {
          "alternateName": "University and Hospitals of Geneva, Medical Informatics, Geneva, Switzerland", 
          "id": "http://www.grid.ac/institutes/grid.150338.c", 
          "name": [
            "University and Hospitals of Geneva, Medical Informatics, Geneva, Switzerland"
          ], 
          "type": "Organization"
        }, 
        "familyName": "M\u00fcller", 
        "givenName": "Henning", 
        "id": "sg:person.07552063233.67", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.07552063233.67"
        ], 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "Department of Information Studies, Sheffield University, Sheffield, UK", 
          "id": "http://www.grid.ac/institutes/grid.11835.3e", 
          "name": [
            "Department of Information Studies, Sheffield University, Sheffield, UK"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Clough", 
        "givenName": "Paul", 
        "id": "sg:person.016305763421.13", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.016305763421.13"
        ], 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "Biomedical Informatics, Oregon Health and Science University, Portland, OR, USA", 
          "id": "http://www.grid.ac/institutes/grid.5288.7", 
          "name": [
            "Biomedical Informatics, Oregon Health and Science University, Portland, OR, USA"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Hersh", 
        "givenName": "Bill", 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "University and Hospitals of Geneva, Medical Informatics, Geneva, Switzerland", 
          "id": "http://www.grid.ac/institutes/grid.150338.c", 
          "name": [
            "University and Hospitals of Geneva, Medical Informatics, Geneva, Switzerland"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Geissb\u00fchler", 
        "givenName": "Antoine", 
        "id": "sg:person.0600360343.20", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.0600360343.20"
        ], 
        "type": "Person"
      }
    ], 
    "datePublished": "2007", 
    "datePublishedReg": "2007-01-01", 
    "description": "Evaluation is crucial for the success of most research domains, and image retrieval is no exception to this. Recently, several benchmarks have been developed for visual information retrieval such as TRECVID, ImageCLEF, and ImagEval to create frameworks for evaluating image retrieval research. An important part of evaluation is the creation of a ground truth or gold standard to evaluate systems against. Much experience has been gained on creating ground truths for textual information retrieval, but for image retrieval these issues require further research. This article will present the process of generating relevance judgements for the medical image retrieval task of ImageCLEF. Many of the problems encountered can be generalised to other image retrieval tasks as well, so the outcome is not limited to the medical domain. Part of the images analysed for relevance were judged by two assessors, and these are analysed with respect to their consistency and potential problems. Our goal is to obtain more information on the ambiguity of the topics developed and generally to keep the variation amongst relevance assessors low. This might partially reduce the subjectivity of system-oriented evaluation, although the evaluation shows that the differences in relevance judgements only have a limited influence on comparative system ranking. A number of outcomes are presented with a goal in mind to create less ambiguous topics for future evaluation campaigns.", 
    "editor": [
      {
        "familyName": "Marchand-Maillet", 
        "givenName": "St\u00e9phane", 
        "type": "Person"
      }, 
      {
        "familyName": "Bruno", 
        "givenName": "Eric", 
        "type": "Person"
      }, 
      {
        "familyName": "N\u00fcrnberger", 
        "givenName": "Andreas", 
        "type": "Person"
      }, 
      {
        "familyName": "Detyniecki", 
        "givenName": "Marcin", 
        "type": "Person"
      }
    ], 
    "genre": "chapter", 
    "id": "sg:pub.10.1007/978-3-540-71545-0_18", 
    "inLanguage": "en", 
    "isAccessibleForFree": true, 
    "isPartOf": {
      "isbn": [
        "978-3-540-71544-3", 
        "978-3-540-71545-0"
      ], 
      "name": "Adaptive Multimedia Retrieval: User, Context, and Feedback", 
      "type": "Book"
    }, 
    "keywords": [
      "image retrieval tasks", 
      "image retrieval", 
      "information retrieval", 
      "retrieval tasks", 
      "relevance judgments", 
      "medical image retrieval task", 
      "ground truth", 
      "system-oriented evaluation", 
      "visual information retrieval", 
      "medical image retrieval", 
      "image retrieval research", 
      "textual information retrieval", 
      "retrieval research", 
      "medical domain", 
      "relevance assessors", 
      "evaluation campaign", 
      "relevance assessments", 
      "research domain", 
      "retrieval", 
      "ImageCLEF", 
      "ambiguous topic", 
      "task", 
      "TRECVID", 
      "important part", 
      "comparative system", 
      "more information", 
      "benchmarks", 
      "system", 
      "images", 
      "domain", 
      "goal", 
      "topic", 
      "framework", 
      "potential problems", 
      "information", 
      "truth", 
      "evaluation", 
      "creation", 
      "issues", 
      "research", 
      "ambiguity", 
      "standards", 
      "consistency", 
      "part", 
      "number", 
      "process", 
      "judgments", 
      "success", 
      "experience", 
      "mind", 
      "further research", 
      "respect", 
      "subjectivity", 
      "article", 
      "gold standard", 
      "relevance", 
      "assessors", 
      "variation", 
      "assessment", 
      "campaign", 
      "number of outcomes", 
      "exception", 
      "outcomes", 
      "limited influence", 
      "influence", 
      "differences", 
      "problem", 
      "most research domains", 
      "ImagEval", 
      "future evaluation campaigns"
    ], 
    "name": "Variation of Relevance Assessments for Medical Image Retrieval", 
    "pagination": "232-246", 
    "productId": [
      {
        "name": "dimensions_id", 
        "type": "PropertyValue", 
        "value": [
          "pub.1027200331"
        ]
      }, 
      {
        "name": "doi", 
        "type": "PropertyValue", 
        "value": [
          "10.1007/978-3-540-71545-0_18"
        ]
      }
    ], 
    "publisher": {
      "name": "Springer Nature", 
      "type": "Organisation"
    }, 
    "sameAs": [
      "https://doi.org/10.1007/978-3-540-71545-0_18", 
      "https://app.dimensions.ai/details/publication/pub.1027200331"
    ], 
    "sdDataset": "chapters", 
    "sdDatePublished": "2021-12-01T20:07", 
    "sdLicense": "https://scigraph.springernature.com/explorer/license/", 
    "sdPublisher": {
      "name": "Springer Nature - SN SciGraph project", 
      "type": "Organization"
    }, 
    "sdSource": "s3://com-springernature-scigraph/baseset/20211201/entities/gbq_results/chapter/chapter_379.jsonl", 
    "type": "Chapter", 
    "url": "https://doi.org/10.1007/978-3-540-71545-0_18"
  }
]
 

Download the RDF metadata as:  json-ld nt turtle xml License info

HOW TO GET THIS DATA PROGRAMMATICALLY:

JSON-LD is a popular format for linked data which is fully compatible with JSON.

curl -H 'Accept: application/ld+json' 'https://scigraph.springernature.com/pub.10.1007/978-3-540-71545-0_18'

N-Triples is a line-based linked data format ideal for batch operations.

curl -H 'Accept: application/n-triples' 'https://scigraph.springernature.com/pub.10.1007/978-3-540-71545-0_18'

Turtle is a human-readable linked data format.

curl -H 'Accept: text/turtle' 'https://scigraph.springernature.com/pub.10.1007/978-3-540-71545-0_18'

RDF/XML is a standard XML format for linked data.

curl -H 'Accept: application/rdf+xml' 'https://scigraph.springernature.com/pub.10.1007/978-3-540-71545-0_18'


 

This table displays all metadata directly associated to this object as RDF triples.

183 TRIPLES      23 PREDICATES      99 URIs      89 LITERALS      7 BLANK NODES

Subject Predicate Object
1 sg:pub.10.1007/978-3-540-71545-0_18 schema:about anzsrc-for:08
2 anzsrc-for:0801
3 anzsrc-for:0806
4 anzsrc-for:17
5 anzsrc-for:1701
6 schema:author N1229b1f3eef4414fb908ccd1d54b80cd
7 schema:datePublished 2007
8 schema:datePublishedReg 2007-01-01
9 schema:description Evaluation is crucial for the success of most research domains, and image retrieval is no exception to this. Recently, several benchmarks have been developed for visual information retrieval such as TRECVID, ImageCLEF, and ImagEval to create frameworks for evaluating image retrieval research. An important part of evaluation is the creation of a ground truth or gold standard to evaluate systems against. Much experience has been gained on creating ground truths for textual information retrieval, but for image retrieval these issues require further research. This article will present the process of generating relevance judgements for the medical image retrieval task of ImageCLEF. Many of the problems encountered can be generalised to other image retrieval tasks as well, so the outcome is not limited to the medical domain. Part of the images analysed for relevance were judged by two assessors, and these are analysed with respect to their consistency and potential problems. Our goal is to obtain more information on the ambiguity of the topics developed and generally to keep the variation amongst relevance assessors low. This might partially reduce the subjectivity of system-oriented evaluation, although the evaluation shows that the differences in relevance judgements only have a limited influence on comparative system ranking. A number of outcomes are presented with a goal in mind to create less ambiguous topics for future evaluation campaigns.
10 schema:editor Ndb98af4b87ee42118af8978c0dec79d0
11 schema:genre chapter
12 schema:inLanguage en
13 schema:isAccessibleForFree true
14 schema:isPartOf N4db02415f2274b738b3fd4eb3c8c7e16
15 schema:keywords ImagEval
16 ImageCLEF
17 TRECVID
18 ambiguity
19 ambiguous topic
20 article
21 assessment
22 assessors
23 benchmarks
24 campaign
25 comparative system
26 consistency
27 creation
28 differences
29 domain
30 evaluation
31 evaluation campaign
32 exception
33 experience
34 framework
35 further research
36 future evaluation campaigns
37 goal
38 gold standard
39 ground truth
40 image retrieval
41 image retrieval research
42 image retrieval tasks
43 images
44 important part
45 influence
46 information
47 information retrieval
48 issues
49 judgments
50 limited influence
51 medical domain
52 medical image retrieval
53 medical image retrieval task
54 mind
55 more information
56 most research domains
57 number
58 number of outcomes
59 outcomes
60 part
61 potential problems
62 problem
63 process
64 relevance
65 relevance assessments
66 relevance assessors
67 relevance judgments
68 research
69 research domain
70 respect
71 retrieval
72 retrieval research
73 retrieval tasks
74 standards
75 subjectivity
76 success
77 system
78 system-oriented evaluation
79 task
80 textual information retrieval
81 topic
82 truth
83 variation
84 visual information retrieval
85 schema:name Variation of Relevance Assessments for Medical Image Retrieval
86 schema:pagination 232-246
87 schema:productId N57ac528215144a48b76add71b5d4215f
88 Na924c4af008149599c8ebee1ee6c9aed
89 schema:publisher N5f00ac0ab430468a8e45e64edb7d018b
90 schema:sameAs https://app.dimensions.ai/details/publication/pub.1027200331
91 https://doi.org/10.1007/978-3-540-71545-0_18
92 schema:sdDatePublished 2021-12-01T20:07
93 schema:sdLicense https://scigraph.springernature.com/explorer/license/
94 schema:sdPublisher N4cad6acbd46c41dfa5c159e7d24b5589
95 schema:url https://doi.org/10.1007/978-3-540-71545-0_18
96 sgo:license sg:explorer/license/
97 sgo:sdDataset chapters
98 rdf:type schema:Chapter
99 N1229b1f3eef4414fb908ccd1d54b80cd rdf:first sg:person.07552063233.67
100 rdf:rest Nb9866a272f774caaafd4fcd1fab9901b
101 N1eb35834520241c398fe352127bcdec8 schema:familyName Detyniecki
102 schema:givenName Marcin
103 rdf:type schema:Person
104 N25dade41829f4e84977359179a33c883 schema:affiliation grid-institutes:grid.5288.7
105 schema:familyName Hersh
106 schema:givenName Bill
107 rdf:type schema:Person
108 N274285906b054a9c8de9f68ed26b9ff9 rdf:first sg:person.0600360343.20
109 rdf:rest rdf:nil
110 N4cad6acbd46c41dfa5c159e7d24b5589 schema:name Springer Nature - SN SciGraph project
111 rdf:type schema:Organization
112 N4db02415f2274b738b3fd4eb3c8c7e16 schema:isbn 978-3-540-71544-3
113 978-3-540-71545-0
114 schema:name Adaptive Multimedia Retrieval: User, Context, and Feedback
115 rdf:type schema:Book
116 N5698a0c7a3704b8699740034ede88f26 rdf:first Neee9e514f9724ba7ba66cefd888870af
117 rdf:rest N6cbf1b4b51724d2fac02947797a4cf2e
118 N57ac528215144a48b76add71b5d4215f schema:name doi
119 schema:value 10.1007/978-3-540-71545-0_18
120 rdf:type schema:PropertyValue
121 N5f00ac0ab430468a8e45e64edb7d018b schema:name Springer Nature
122 rdf:type schema:Organisation
123 N6974f29987a04ec8ae0b61d5d33ad413 rdf:first N79777943fb8a4ba5982bcbb2eea58e82
124 rdf:rest N5698a0c7a3704b8699740034ede88f26
125 N6ae21ab3827e43e4a635318bc8c4f012 rdf:first N25dade41829f4e84977359179a33c883
126 rdf:rest N274285906b054a9c8de9f68ed26b9ff9
127 N6cbf1b4b51724d2fac02947797a4cf2e rdf:first N1eb35834520241c398fe352127bcdec8
128 rdf:rest rdf:nil
129 N79777943fb8a4ba5982bcbb2eea58e82 schema:familyName Bruno
130 schema:givenName Eric
131 rdf:type schema:Person
132 Na924c4af008149599c8ebee1ee6c9aed schema:name dimensions_id
133 schema:value pub.1027200331
134 rdf:type schema:PropertyValue
135 Nb9866a272f774caaafd4fcd1fab9901b rdf:first sg:person.016305763421.13
136 rdf:rest N6ae21ab3827e43e4a635318bc8c4f012
137 Ndb98af4b87ee42118af8978c0dec79d0 rdf:first Ndc26116252574a138ecaebb4fa7351ca
138 rdf:rest N6974f29987a04ec8ae0b61d5d33ad413
139 Ndc26116252574a138ecaebb4fa7351ca schema:familyName Marchand-Maillet
140 schema:givenName Stéphane
141 rdf:type schema:Person
142 Neee9e514f9724ba7ba66cefd888870af schema:familyName Nürnberger
143 schema:givenName Andreas
144 rdf:type schema:Person
145 anzsrc-for:08 schema:inDefinedTermSet anzsrc-for:
146 schema:name Information and Computing Sciences
147 rdf:type schema:DefinedTerm
148 anzsrc-for:0801 schema:inDefinedTermSet anzsrc-for:
149 schema:name Artificial Intelligence and Image Processing
150 rdf:type schema:DefinedTerm
151 anzsrc-for:0806 schema:inDefinedTermSet anzsrc-for:
152 schema:name Information Systems
153 rdf:type schema:DefinedTerm
154 anzsrc-for:17 schema:inDefinedTermSet anzsrc-for:
155 schema:name Psychology and Cognitive Sciences
156 rdf:type schema:DefinedTerm
157 anzsrc-for:1701 schema:inDefinedTermSet anzsrc-for:
158 schema:name Psychology
159 rdf:type schema:DefinedTerm
160 sg:person.016305763421.13 schema:affiliation grid-institutes:grid.11835.3e
161 schema:familyName Clough
162 schema:givenName Paul
163 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.016305763421.13
164 rdf:type schema:Person
165 sg:person.0600360343.20 schema:affiliation grid-institutes:grid.150338.c
166 schema:familyName Geissbühler
167 schema:givenName Antoine
168 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.0600360343.20
169 rdf:type schema:Person
170 sg:person.07552063233.67 schema:affiliation grid-institutes:grid.150338.c
171 schema:familyName Müller
172 schema:givenName Henning
173 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.07552063233.67
174 rdf:type schema:Person
175 grid-institutes:grid.11835.3e schema:alternateName Department of Information Studies, Sheffield University, Sheffield, UK
176 schema:name Department of Information Studies, Sheffield University, Sheffield, UK
177 rdf:type schema:Organization
178 grid-institutes:grid.150338.c schema:alternateName University and Hospitals of Geneva, Medical Informatics, Geneva, Switzerland
179 schema:name University and Hospitals of Geneva, Medical Informatics, Geneva, Switzerland
180 rdf:type schema:Organization
181 grid-institutes:grid.5288.7 schema:alternateName Biomedical Informatics, Oregon Health and Science University, Portland, OR, USA
182 schema:name Biomedical Informatics, Oregon Health and Science University, Portland, OR, USA
183 rdf:type schema:Organization
 




Preview window. Press ESC to close (or click here)


...