Bias and the limits of pooling for large collections View Full Text


Ontology type: schema:ScholarlyArticle      Open Access: True


Article Info

DATE

2007-12

AUTHORS

Chris Buckley, Darrin Dimmick, Ian Soboroff, Ellen Voorhees

ABSTRACT

Modern retrieval test collections are built through a process called pooling in which only a sample of the entire document set is judged for each topic. The idea behind pooling is to find enough relevant documents such that when unjudged documents are assumed to be nonrelevant the resulting judgment set is sufficiently complete and unbiased. Yet a constant-size pool represents an increasingly small percentage of the document set as document sets grow larger, and at some point the assumption of approximately complete judgments must become invalid. This paper shows that the judgment sets produced by traditional pooling when the pools are too small relative to the total document set size can be biased in that they favor relevant documents that contain topic title words. This phenomenon is wholly dependent on the collection size and does not depend on the number of relevant documents for a given topic. We show that the AQUAINT test collection constructed in the recent TREC 2005 workshop exhibits this biased relevance set; it is likely that the test collections based on the much larger GOV2 document set also exhibit the bias. The paper concludes with suggested modifications to traditional pooling and evaluation methodology that may allow very large reusable test collections to be built. More... »

PAGES

491-508

References to SciGraph publications

  • 2003-01. On Collection Size and Retrieval Effectiveness in INFORMATION RETRIEVAL JOURNAL
  • 2006-06. Extreme value theory applied to document retrieval from large collections in INFORMATION RETRIEVAL JOURNAL
  • Identifiers

    URI

    http://scigraph.springernature.com/pub.10.1007/s10791-007-9032-x

    DOI

    http://dx.doi.org/10.1007/s10791-007-9032-x

    DIMENSIONS

    https://app.dimensions.ai/details/publication/pub.1000187118


    Indexing Status Check whether this publication has been indexed by Scopus and Web Of Science using the SN Indexing Status Tool
    Incoming Citations Browse incoming citations for this publication using opencitations.net

    JSON-LD is the canonical representation for SciGraph data.

    TIP: You can open this SciGraph record using an external JSON-LD service: JSON-LD Playground Google SDTT

    [
      {
        "@context": "https://springernature.github.io/scigraph/jsonld/sgcontext.json", 
        "about": [
          {
            "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/2102", 
            "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
            "name": "Curatorial and Related Studies", 
            "type": "DefinedTerm"
          }, 
          {
            "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/21", 
            "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
            "name": "History and Archaeology", 
            "type": "DefinedTerm"
          }
        ], 
        "author": [
          {
            "affiliation": {
              "name": [
                "Sabir Research, Inc., 20878, Gaithersburg, MD, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Buckley", 
            "givenName": "Chris", 
            "id": "sg:person.015660514222.28", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.015660514222.28"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "National Institute of Standards and Technology", 
              "id": "https://www.grid.ac/institutes/grid.94225.38", 
              "name": [
                "Information Technology Laboratory, National Institute of Standards and Technology, 20899-8940, Gaithersburg, MD, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Dimmick", 
            "givenName": "Darrin", 
            "id": "sg:person.011141470653.12", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.011141470653.12"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "National Institute of Standards and Technology", 
              "id": "https://www.grid.ac/institutes/grid.94225.38", 
              "name": [
                "Information Technology Laboratory, National Institute of Standards and Technology, 20899-8940, Gaithersburg, MD, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Soboroff", 
            "givenName": "Ian", 
            "id": "sg:person.014401617547.41", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.014401617547.41"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "National Institute of Standards and Technology", 
              "id": "https://www.grid.ac/institutes/grid.94225.38", 
              "name": [
                "Information Technology Laboratory, National Institute of Standards and Technology, 20899-8940, Gaithersburg, MD, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Voorhees", 
            "givenName": "Ellen", 
            "id": "sg:person.016560273243.63", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.016560273243.63"
            ], 
            "type": "Person"
          }
        ], 
        "citation": [
          {
            "id": "https://doi.org/10.1145/860435.860481", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1006492539"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1145/3166.3197", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1007983120"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1145/564376.564432", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1009001524"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1145/1008992.1009000", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1014166097"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1145/1148170.1148263", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1016420138"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1145/1148170.1148219", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1016626314"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1145/1008992.1009001", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1027915679"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1016/0306-4573(92)90005-k", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1027935191"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1016/0306-4573(92)90005-k", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1027935191"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/s10791-006-0882-4", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1035740417", 
              "https://doi.org/10.1007/s10791-006-0882-4"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/s10791-006-0882-4", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1035740417", 
              "https://doi.org/10.1007/s10791-006-0882-4"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1108/eb050097", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1038847956"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1145/290941.291009", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1039021860"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1023/a:1022904715765", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1044230206", 
              "https://doi.org/10.1023/a:1022904715765"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1145/290941.291014", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1052200729"
            ], 
            "type": "CreativeWork"
          }
        ], 
        "datePublished": "2007-12", 
        "datePublishedReg": "2007-12-01", 
        "description": "Modern retrieval test collections are built through a process called pooling in which only a sample of the entire document set is judged for each topic. The idea behind pooling is to find enough relevant documents such that when unjudged documents are assumed to be nonrelevant the resulting judgment set is sufficiently complete and unbiased. Yet a constant-size pool represents an increasingly small percentage of the document set as document sets grow larger, and at some point the assumption of approximately complete judgments must become invalid. This paper shows that the judgment sets produced by traditional pooling when the pools are too small relative to the total document set size can be biased in that they favor relevant documents that contain topic title words. This phenomenon is wholly dependent on the collection size and does not depend on the number of relevant documents for a given topic. We show that the AQUAINT test collection constructed in the recent TREC 2005 workshop exhibits this biased relevance set; it is likely that the test collections based on the much larger GOV2 document set also exhibit the bias. The paper concludes with suggested modifications to traditional pooling and evaluation methodology that may allow very large reusable test collections to be built.", 
        "genre": "research_article", 
        "id": "sg:pub.10.1007/s10791-007-9032-x", 
        "inLanguage": [
          "en"
        ], 
        "isAccessibleForFree": true, 
        "isPartOf": [
          {
            "id": "sg:journal.1023664", 
            "issn": [
              "1386-4564", 
              "1573-7659"
            ], 
            "name": "Information Retrieval Journal", 
            "type": "Periodical"
          }, 
          {
            "issueNumber": "6", 
            "type": "PublicationIssue"
          }, 
          {
            "type": "PublicationVolume", 
            "volumeNumber": "10"
          }
        ], 
        "name": "Bias and the limits of pooling for large collections", 
        "pagination": "491-508", 
        "productId": [
          {
            "name": "doi", 
            "type": "PropertyValue", 
            "value": [
              "10.1007/s10791-007-9032-x"
            ]
          }, 
          {
            "name": "readcube_id", 
            "type": "PropertyValue", 
            "value": [
              "00e3b3fbaf2108e7274d8033af87554abc35eac972009d01818e75d646cf81b3"
            ]
          }, 
          {
            "name": "dimensions_id", 
            "type": "PropertyValue", 
            "value": [
              "pub.1000187118"
            ]
          }
        ], 
        "sameAs": [
          "https://doi.org/10.1007/s10791-007-9032-x", 
          "https://app.dimensions.ai/details/publication/pub.1000187118"
        ], 
        "sdDataset": "articles", 
        "sdDatePublished": "2019-04-15T09:15", 
        "sdLicense": "https://scigraph.springernature.com/explorer/license/", 
        "sdPublisher": {
          "name": "Springer Nature - SN SciGraph project", 
          "type": "Organization"
        }, 
        "sdSource": "s3://com-uberresearch-data-dimensions-target-20181106-alternative/cleanup/v134/2549eaecd7973599484d7c17b260dba0a4ecb94b/merge/v9/a6c9fde33151104705d4d7ff012ea9563521a3ce/jats-lookup/v90/0000000376_0000000376/records_56171_00000000.jsonl", 
        "type": "ScholarlyArticle", 
        "url": "http://link.springer.com/10.1007%2Fs10791-007-9032-x"
      }
    ]
     

    Download the RDF metadata as:  json-ld nt turtle xml License info

    HOW TO GET THIS DATA PROGRAMMATICALLY:

    JSON-LD is a popular format for linked data which is fully compatible with JSON.

    curl -H 'Accept: application/ld+json' 'https://scigraph.springernature.com/pub.10.1007/s10791-007-9032-x'

    N-Triples is a line-based linked data format ideal for batch operations.

    curl -H 'Accept: application/n-triples' 'https://scigraph.springernature.com/pub.10.1007/s10791-007-9032-x'

    Turtle is a human-readable linked data format.

    curl -H 'Accept: text/turtle' 'https://scigraph.springernature.com/pub.10.1007/s10791-007-9032-x'

    RDF/XML is a standard XML format for linked data.

    curl -H 'Accept: application/rdf+xml' 'https://scigraph.springernature.com/pub.10.1007/s10791-007-9032-x'


     

    This table displays all metadata directly associated to this object as RDF triples.

    125 TRIPLES      21 PREDICATES      40 URIs      19 LITERALS      7 BLANK NODES

    Subject Predicate Object
    1 sg:pub.10.1007/s10791-007-9032-x schema:about anzsrc-for:21
    2 anzsrc-for:2102
    3 schema:author Na2bc49007f20447d85e262451e3b22d9
    4 schema:citation sg:pub.10.1007/s10791-006-0882-4
    5 sg:pub.10.1023/a:1022904715765
    6 https://doi.org/10.1016/0306-4573(92)90005-k
    7 https://doi.org/10.1108/eb050097
    8 https://doi.org/10.1145/1008992.1009000
    9 https://doi.org/10.1145/1008992.1009001
    10 https://doi.org/10.1145/1148170.1148219
    11 https://doi.org/10.1145/1148170.1148263
    12 https://doi.org/10.1145/290941.291009
    13 https://doi.org/10.1145/290941.291014
    14 https://doi.org/10.1145/3166.3197
    15 https://doi.org/10.1145/564376.564432
    16 https://doi.org/10.1145/860435.860481
    17 schema:datePublished 2007-12
    18 schema:datePublishedReg 2007-12-01
    19 schema:description Modern retrieval test collections are built through a process called pooling in which only a sample of the entire document set is judged for each topic. The idea behind pooling is to find enough relevant documents such that when unjudged documents are assumed to be nonrelevant the resulting judgment set is sufficiently complete and unbiased. Yet a constant-size pool represents an increasingly small percentage of the document set as document sets grow larger, and at some point the assumption of approximately complete judgments must become invalid. This paper shows that the judgment sets produced by traditional pooling when the pools are too small relative to the total document set size can be biased in that they favor relevant documents that contain topic title words. This phenomenon is wholly dependent on the collection size and does not depend on the number of relevant documents for a given topic. We show that the AQUAINT test collection constructed in the recent TREC 2005 workshop exhibits this biased relevance set; it is likely that the test collections based on the much larger GOV2 document set also exhibit the bias. The paper concludes with suggested modifications to traditional pooling and evaluation methodology that may allow very large reusable test collections to be built.
    20 schema:genre research_article
    21 schema:inLanguage en
    22 schema:isAccessibleForFree true
    23 schema:isPartOf N18789cee3905453d9976f6ed92e577f8
    24 N21f3edc000bf44d5a38b2c3067093ac1
    25 sg:journal.1023664
    26 schema:name Bias and the limits of pooling for large collections
    27 schema:pagination 491-508
    28 schema:productId N023f92be89d34bbbb7f80c1a41582a95
    29 N3fe8762a2ffe4245810252e7bc456bb0
    30 N76b982cbbe66433587611c9ac7ddcc2e
    31 schema:sameAs https://app.dimensions.ai/details/publication/pub.1000187118
    32 https://doi.org/10.1007/s10791-007-9032-x
    33 schema:sdDatePublished 2019-04-15T09:15
    34 schema:sdLicense https://scigraph.springernature.com/explorer/license/
    35 schema:sdPublisher N0c12da22e95f468daf67e68c4276666a
    36 schema:url http://link.springer.com/10.1007%2Fs10791-007-9032-x
    37 sgo:license sg:explorer/license/
    38 sgo:sdDataset articles
    39 rdf:type schema:ScholarlyArticle
    40 N023f92be89d34bbbb7f80c1a41582a95 schema:name dimensions_id
    41 schema:value pub.1000187118
    42 rdf:type schema:PropertyValue
    43 N0c12da22e95f468daf67e68c4276666a schema:name Springer Nature - SN SciGraph project
    44 rdf:type schema:Organization
    45 N18789cee3905453d9976f6ed92e577f8 schema:issueNumber 6
    46 rdf:type schema:PublicationIssue
    47 N21f3edc000bf44d5a38b2c3067093ac1 schema:volumeNumber 10
    48 rdf:type schema:PublicationVolume
    49 N233ed33fa2f544d992c8d2e91d0b73b1 rdf:first sg:person.014401617547.41
    50 rdf:rest Nfeabdda5d44a47ad843d6fdf802b4bc0
    51 N3fe8762a2ffe4245810252e7bc456bb0 schema:name doi
    52 schema:value 10.1007/s10791-007-9032-x
    53 rdf:type schema:PropertyValue
    54 N55a6380a90f0414e9bf897987b50f309 schema:name Sabir Research, Inc., 20878, Gaithersburg, MD, USA
    55 rdf:type schema:Organization
    56 N75d5b8ce2fba407cbf99b226152ea0fe rdf:first sg:person.011141470653.12
    57 rdf:rest N233ed33fa2f544d992c8d2e91d0b73b1
    58 N76b982cbbe66433587611c9ac7ddcc2e schema:name readcube_id
    59 schema:value 00e3b3fbaf2108e7274d8033af87554abc35eac972009d01818e75d646cf81b3
    60 rdf:type schema:PropertyValue
    61 Na2bc49007f20447d85e262451e3b22d9 rdf:first sg:person.015660514222.28
    62 rdf:rest N75d5b8ce2fba407cbf99b226152ea0fe
    63 Nfeabdda5d44a47ad843d6fdf802b4bc0 rdf:first sg:person.016560273243.63
    64 rdf:rest rdf:nil
    65 anzsrc-for:21 schema:inDefinedTermSet anzsrc-for:
    66 schema:name History and Archaeology
    67 rdf:type schema:DefinedTerm
    68 anzsrc-for:2102 schema:inDefinedTermSet anzsrc-for:
    69 schema:name Curatorial and Related Studies
    70 rdf:type schema:DefinedTerm
    71 sg:journal.1023664 schema:issn 1386-4564
    72 1573-7659
    73 schema:name Information Retrieval Journal
    74 rdf:type schema:Periodical
    75 sg:person.011141470653.12 schema:affiliation https://www.grid.ac/institutes/grid.94225.38
    76 schema:familyName Dimmick
    77 schema:givenName Darrin
    78 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.011141470653.12
    79 rdf:type schema:Person
    80 sg:person.014401617547.41 schema:affiliation https://www.grid.ac/institutes/grid.94225.38
    81 schema:familyName Soboroff
    82 schema:givenName Ian
    83 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.014401617547.41
    84 rdf:type schema:Person
    85 sg:person.015660514222.28 schema:affiliation N55a6380a90f0414e9bf897987b50f309
    86 schema:familyName Buckley
    87 schema:givenName Chris
    88 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.015660514222.28
    89 rdf:type schema:Person
    90 sg:person.016560273243.63 schema:affiliation https://www.grid.ac/institutes/grid.94225.38
    91 schema:familyName Voorhees
    92 schema:givenName Ellen
    93 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.016560273243.63
    94 rdf:type schema:Person
    95 sg:pub.10.1007/s10791-006-0882-4 schema:sameAs https://app.dimensions.ai/details/publication/pub.1035740417
    96 https://doi.org/10.1007/s10791-006-0882-4
    97 rdf:type schema:CreativeWork
    98 sg:pub.10.1023/a:1022904715765 schema:sameAs https://app.dimensions.ai/details/publication/pub.1044230206
    99 https://doi.org/10.1023/a:1022904715765
    100 rdf:type schema:CreativeWork
    101 https://doi.org/10.1016/0306-4573(92)90005-k schema:sameAs https://app.dimensions.ai/details/publication/pub.1027935191
    102 rdf:type schema:CreativeWork
    103 https://doi.org/10.1108/eb050097 schema:sameAs https://app.dimensions.ai/details/publication/pub.1038847956
    104 rdf:type schema:CreativeWork
    105 https://doi.org/10.1145/1008992.1009000 schema:sameAs https://app.dimensions.ai/details/publication/pub.1014166097
    106 rdf:type schema:CreativeWork
    107 https://doi.org/10.1145/1008992.1009001 schema:sameAs https://app.dimensions.ai/details/publication/pub.1027915679
    108 rdf:type schema:CreativeWork
    109 https://doi.org/10.1145/1148170.1148219 schema:sameAs https://app.dimensions.ai/details/publication/pub.1016626314
    110 rdf:type schema:CreativeWork
    111 https://doi.org/10.1145/1148170.1148263 schema:sameAs https://app.dimensions.ai/details/publication/pub.1016420138
    112 rdf:type schema:CreativeWork
    113 https://doi.org/10.1145/290941.291009 schema:sameAs https://app.dimensions.ai/details/publication/pub.1039021860
    114 rdf:type schema:CreativeWork
    115 https://doi.org/10.1145/290941.291014 schema:sameAs https://app.dimensions.ai/details/publication/pub.1052200729
    116 rdf:type schema:CreativeWork
    117 https://doi.org/10.1145/3166.3197 schema:sameAs https://app.dimensions.ai/details/publication/pub.1007983120
    118 rdf:type schema:CreativeWork
    119 https://doi.org/10.1145/564376.564432 schema:sameAs https://app.dimensions.ai/details/publication/pub.1009001524
    120 rdf:type schema:CreativeWork
    121 https://doi.org/10.1145/860435.860481 schema:sameAs https://app.dimensions.ai/details/publication/pub.1006492539
    122 rdf:type schema:CreativeWork
    123 https://www.grid.ac/institutes/grid.94225.38 schema:alternateName National Institute of Standards and Technology
    124 schema:name Information Technology Laboratory, National Institute of Standards and Technology, 20899-8940, Gaithersburg, MD, USA
    125 rdf:type schema:Organization
     




    Preview window. Press ESC to close (or click here)


    ...