Measuring the Wisdom of the Crowd: How Many is Enough? View Full Text


Ontology type: schema:ScholarlyArticle      Open Access: True


Article Info

DATE

2022-04-06

AUTHORS

Volker Walter, Michael Kölle, David Collmar

ABSTRACT

The idea of the wisdom of the crowd is that integrating multiple estimates of a group of individuals provides an outcome that is often better than most of the underlying estimates or even better than the best individual estimate. In this paper, we examine the wisdom of the crowd principle on the example of spatial data collection by paid crowdworkers. We developed a web-based user interface for the collection of vehicles from rasterized shadings derived from 3D point clouds and executed different data collection campaigns on the crowdsourcing marketplace microWorkers. Our main question is: how large must be the crowd in order that the quality of the outcome fulfils the quality requirements of a specific application? To answer this question, we computed precision, recall, F1 score, and geometric quality measures for different crowd sizes. We found that increasing the crowd size improves the quality of the outcome. This improvement is quite large at the beginning and gradually decreases with larger crowd sizes. These findings confirm the wisdom of the crowd principle and help to find an optimum number of the crowd size that is in the end a compromise between data quality, and cost and time required to perform the data collection. More... »

PAGES

269-291

References to SciGraph publications

  • 2021-02-21. Remembering Both the Machine and the Crowd When Sampling Points: Active Learning for Semantic Segmentation of ALS Point Clouds in PATTERN RECOGNITION. ICPR INTERNATIONAL WORKSHOPS AND CHALLENGES
  • 2018-06-18. ThemeRise: a theme-oriented framework for volunteered geographic information applications in OPEN GEOSPATIAL DATA, SOFTWARE AND STANDARDS
  • 2018-02-16. Paid Crowdsourcing as Concept and Content Generator to Enhance Museum Experiences in MUSEUM EXPERIENCE DESIGN
  • 2016-07-02. Learning from crowdsourced labeled data: a survey in ARTIFICIAL INTELLIGENCE REVIEW
  • 2007-08. Citizens as sensors: the world of volunteered geography in GEOJOURNAL
  • 1907-03. Vox Populi in NATURE
  • 2018-10. Implementation, Results, and Problems of Paid Crowd-Based Geospatial Data Collection in PFG – JOURNAL OF PHOTOGRAMMETRY, REMOTE SENSING AND GEOINFORMATION SCIENCE
  • 2015-03-04. Assessment of Logical Consistency in OpenStreetMap Based on the Spatial Similarity Concept in OPENSTREETMAP IN GISCIENCE
  • 2019-05-15. Paid Crowdsourcing, Low Income Contributors, and Subjectivity in ARTIFICIAL INTELLIGENCE APPLICATIONS AND INNOVATIONS
  • Identifiers

    URI

    http://scigraph.springernature.com/pub.10.1007/s41064-022-00202-2

    DOI

    http://dx.doi.org/10.1007/s41064-022-00202-2

    DIMENSIONS

    https://app.dimensions.ai/details/publication/pub.1146912976


    Indexing Status Check whether this publication has been indexed by Scopus and Web Of Science using the SN Indexing Status Tool
    Incoming Citations Browse incoming citations for this publication using opencitations.net

    JSON-LD is the canonical representation for SciGraph data.

    TIP: You can open this SciGraph record using an external JSON-LD service: JSON-LD Playground Google SDTT

    [
      {
        "@context": "https://springernature.github.io/scigraph/jsonld/sgcontext.json", 
        "about": [
          {
            "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/08", 
            "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
            "name": "Information and Computing Sciences", 
            "type": "DefinedTerm"
          }, 
          {
            "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/0806", 
            "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
            "name": "Information Systems", 
            "type": "DefinedTerm"
          }
        ], 
        "author": [
          {
            "affiliation": {
              "alternateName": "Institute for Photogrammetry (ifp), University of Stuttgart, Geschwister-Scholl-Str. 24D, 70174, Stuttgart, Germany", 
              "id": "http://www.grid.ac/institutes/grid.5719.a", 
              "name": [
                "Institute for Photogrammetry (ifp), University of Stuttgart, Geschwister-Scholl-Str. 24D, 70174, Stuttgart, Germany"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Walter", 
            "givenName": "Volker", 
            "id": "sg:person.012367021751.30", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.012367021751.30"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "Institute for Photogrammetry (ifp), University of Stuttgart, Geschwister-Scholl-Str. 24D, 70174, Stuttgart, Germany", 
              "id": "http://www.grid.ac/institutes/grid.5719.a", 
              "name": [
                "Institute for Photogrammetry (ifp), University of Stuttgart, Geschwister-Scholl-Str. 24D, 70174, Stuttgart, Germany"
              ], 
              "type": "Organization"
            }, 
            "familyName": "K\u00f6lle", 
            "givenName": "Michael", 
            "id": "sg:person.015302431607.44", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.015302431607.44"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "Institute for Photogrammetry (ifp), University of Stuttgart, Geschwister-Scholl-Str. 24D, 70174, Stuttgart, Germany", 
              "id": "http://www.grid.ac/institutes/grid.5719.a", 
              "name": [
                "Institute for Photogrammetry (ifp), University of Stuttgart, Geschwister-Scholl-Str. 24D, 70174, Stuttgart, Germany"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Collmar", 
            "givenName": "David", 
            "type": "Person"
          }
        ], 
        "citation": [
          {
            "id": "sg:pub.10.1186/s40965-018-0049-4", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1104245329", 
              "https://doi.org/10.1186/s40965-018-0049-4"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/s10708-007-9111-y", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1014006987", 
              "https://doi.org/10.1007/s10708-007-9111-y"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1038/075450a0", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1035056658", 
              "https://doi.org/10.1038/075450a0"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-319-14280-7_2", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1039444494", 
              "https://doi.org/10.1007/978-3-319-14280-7_2"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-030-68787-8_37", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1135478924", 
              "https://doi.org/10.1007/978-3-030-68787-8_37"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-030-19909-8_20", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1114220932", 
              "https://doi.org/10.1007/978-3-030-19909-8_20"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/s10462-016-9491-9", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1042840118", 
              "https://doi.org/10.1007/s10462-016-9491-9"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/s41064-018-0058-z", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1110507464", 
              "https://doi.org/10.1007/s41064-018-0058-z"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-319-58550-5_7", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1101055990", 
              "https://doi.org/10.1007/978-3-319-58550-5_7"
            ], 
            "type": "CreativeWork"
          }
        ], 
        "datePublished": "2022-04-06", 
        "datePublishedReg": "2022-04-06", 
        "description": "The idea of the wisdom of the crowd is that integrating multiple estimates of a group of individuals provides an outcome that is often better than most of the underlying estimates or even better than the best individual estimate. In this paper, we examine the wisdom of the crowd principle on the example of spatial data collection by paid crowdworkers. We developed a web-based user interface for the collection of vehicles from rasterized shadings derived from 3D point clouds and executed different data collection campaigns on the crowdsourcing marketplace microWorkers. Our main question is: how large must be the crowd in order that the quality of the outcome fulfils the quality requirements of a specific application? To answer this question, we computed precision, recall, F1 score, and geometric quality measures for different crowd sizes. We found that increasing the crowd size improves the quality of the outcome. This improvement is quite large at the beginning and gradually decreases with larger crowd sizes. These findings confirm the wisdom of the crowd principle and help to find an optimum number of the crowd size that is in the end a compromise between data quality, and cost and time required to perform the data collection.", 
        "genre": "article", 
        "id": "sg:pub.10.1007/s41064-022-00202-2", 
        "isAccessibleForFree": true, 
        "isPartOf": [
          {
            "id": "sg:journal.1320463", 
            "issn": [
              "2512-2789", 
              "2512-2819"
            ], 
            "name": "PFG \u2013 Journal of Photogrammetry, Remote Sensing and Geoinformation Science", 
            "publisher": "Springer Nature", 
            "type": "Periodical"
          }, 
          {
            "issueNumber": "3", 
            "type": "PublicationIssue"
          }, 
          {
            "type": "PublicationVolume", 
            "volumeNumber": "90"
          }
        ], 
        "keywords": [
          "web-based user interface", 
          "crowds' principles", 
          "crowd size", 
          "spatial data collection", 
          "user interface", 
          "F1 score", 
          "collection of vehicles", 
          "point clouds", 
          "data collection campaign", 
          "data collection", 
          "data quality", 
          "quality requirements", 
          "crowd", 
          "collection campaign", 
          "specific applications", 
          "quality measures", 
          "optimum number", 
          "multiple estimates", 
          "collection", 
          "cloud", 
          "crowdworkers", 
          "Microworkers", 
          "quality", 
          "requirements", 
          "recall", 
          "vehicles", 
          "interface", 
          "applications", 
          "cost", 
          "precision", 
          "group of individuals", 
          "idea", 
          "principles", 
          "example", 
          "main question", 
          "order", 
          "compromise", 
          "individual estimates", 
          "shading", 
          "wisdom", 
          "improvement", 
          "number", 
          "size", 
          "time", 
          "end", 
          "questions", 
          "estimates", 
          "measures", 
          "campaign", 
          "scores", 
          "individuals", 
          "beginning", 
          "outcomes", 
          "group", 
          "findings", 
          "paper"
        ], 
        "name": "Measuring the Wisdom of the Crowd: How Many is Enough?", 
        "pagination": "269-291", 
        "productId": [
          {
            "name": "dimensions_id", 
            "type": "PropertyValue", 
            "value": [
              "pub.1146912976"
            ]
          }, 
          {
            "name": "doi", 
            "type": "PropertyValue", 
            "value": [
              "10.1007/s41064-022-00202-2"
            ]
          }
        ], 
        "sameAs": [
          "https://doi.org/10.1007/s41064-022-00202-2", 
          "https://app.dimensions.ai/details/publication/pub.1146912976"
        ], 
        "sdDataset": "articles", 
        "sdDatePublished": "2022-12-01T06:45", 
        "sdLicense": "https://scigraph.springernature.com/explorer/license/", 
        "sdPublisher": {
          "name": "Springer Nature - SN SciGraph project", 
          "type": "Organization"
        }, 
        "sdSource": "s3://com-springernature-scigraph/baseset/20221201/entities/gbq_results/article/article_950.jsonl", 
        "type": "ScholarlyArticle", 
        "url": "https://doi.org/10.1007/s41064-022-00202-2"
      }
    ]
     

    Download the RDF metadata as:  json-ld nt turtle xml License info

    HOW TO GET THIS DATA PROGRAMMATICALLY:

    JSON-LD is a popular format for linked data which is fully compatible with JSON.

    curl -H 'Accept: application/ld+json' 'https://scigraph.springernature.com/pub.10.1007/s41064-022-00202-2'

    N-Triples is a line-based linked data format ideal for batch operations.

    curl -H 'Accept: application/n-triples' 'https://scigraph.springernature.com/pub.10.1007/s41064-022-00202-2'

    Turtle is a human-readable linked data format.

    curl -H 'Accept: text/turtle' 'https://scigraph.springernature.com/pub.10.1007/s41064-022-00202-2'

    RDF/XML is a standard XML format for linked data.

    curl -H 'Accept: application/rdf+xml' 'https://scigraph.springernature.com/pub.10.1007/s41064-022-00202-2'


     

    This table displays all metadata directly associated to this object as RDF triples.

    162 TRIPLES      21 PREDICATES      89 URIs      72 LITERALS      6 BLANK NODES

    Subject Predicate Object
    1 sg:pub.10.1007/s41064-022-00202-2 schema:about anzsrc-for:08
    2 anzsrc-for:0806
    3 schema:author Na958f4a5ea6c40c0ae0ee66025f5ee7e
    4 schema:citation sg:pub.10.1007/978-3-030-19909-8_20
    5 sg:pub.10.1007/978-3-030-68787-8_37
    6 sg:pub.10.1007/978-3-319-14280-7_2
    7 sg:pub.10.1007/978-3-319-58550-5_7
    8 sg:pub.10.1007/s10462-016-9491-9
    9 sg:pub.10.1007/s10708-007-9111-y
    10 sg:pub.10.1007/s41064-018-0058-z
    11 sg:pub.10.1038/075450a0
    12 sg:pub.10.1186/s40965-018-0049-4
    13 schema:datePublished 2022-04-06
    14 schema:datePublishedReg 2022-04-06
    15 schema:description The idea of the wisdom of the crowd is that integrating multiple estimates of a group of individuals provides an outcome that is often better than most of the underlying estimates or even better than the best individual estimate. In this paper, we examine the wisdom of the crowd principle on the example of spatial data collection by paid crowdworkers. We developed a web-based user interface for the collection of vehicles from rasterized shadings derived from 3D point clouds and executed different data collection campaigns on the crowdsourcing marketplace microWorkers. Our main question is: how large must be the crowd in order that the quality of the outcome fulfils the quality requirements of a specific application? To answer this question, we computed precision, recall, F1 score, and geometric quality measures for different crowd sizes. We found that increasing the crowd size improves the quality of the outcome. This improvement is quite large at the beginning and gradually decreases with larger crowd sizes. These findings confirm the wisdom of the crowd principle and help to find an optimum number of the crowd size that is in the end a compromise between data quality, and cost and time required to perform the data collection.
    16 schema:genre article
    17 schema:isAccessibleForFree true
    18 schema:isPartOf N7545f567fcb14050b982d4c65a5ad0aa
    19 N8fac12c610c548f58d405075e55573bd
    20 sg:journal.1320463
    21 schema:keywords F1 score
    22 Microworkers
    23 applications
    24 beginning
    25 campaign
    26 cloud
    27 collection
    28 collection campaign
    29 collection of vehicles
    30 compromise
    31 cost
    32 crowd
    33 crowd size
    34 crowds' principles
    35 crowdworkers
    36 data collection
    37 data collection campaign
    38 data quality
    39 end
    40 estimates
    41 example
    42 findings
    43 group
    44 group of individuals
    45 idea
    46 improvement
    47 individual estimates
    48 individuals
    49 interface
    50 main question
    51 measures
    52 multiple estimates
    53 number
    54 optimum number
    55 order
    56 outcomes
    57 paper
    58 point clouds
    59 precision
    60 principles
    61 quality
    62 quality measures
    63 quality requirements
    64 questions
    65 recall
    66 requirements
    67 scores
    68 shading
    69 size
    70 spatial data collection
    71 specific applications
    72 time
    73 user interface
    74 vehicles
    75 web-based user interface
    76 wisdom
    77 schema:name Measuring the Wisdom of the Crowd: How Many is Enough?
    78 schema:pagination 269-291
    79 schema:productId Nd47663ae79f54582ba208a061e4fa93c
    80 Nf634ffd8d7ee43b1a6d04f8deddf3d41
    81 schema:sameAs https://app.dimensions.ai/details/publication/pub.1146912976
    82 https://doi.org/10.1007/s41064-022-00202-2
    83 schema:sdDatePublished 2022-12-01T06:45
    84 schema:sdLicense https://scigraph.springernature.com/explorer/license/
    85 schema:sdPublisher N53bb975f8a8e4b5ba4c32d5775e8118c
    86 schema:url https://doi.org/10.1007/s41064-022-00202-2
    87 sgo:license sg:explorer/license/
    88 sgo:sdDataset articles
    89 rdf:type schema:ScholarlyArticle
    90 N4cf31532072845afbee25a45f6002b0b rdf:first Nbee48240b388419aa9b01effe23c19ed
    91 rdf:rest rdf:nil
    92 N53bb975f8a8e4b5ba4c32d5775e8118c schema:name Springer Nature - SN SciGraph project
    93 rdf:type schema:Organization
    94 N7545f567fcb14050b982d4c65a5ad0aa schema:issueNumber 3
    95 rdf:type schema:PublicationIssue
    96 N8993a90915904f1cbef95c91616aae02 rdf:first sg:person.015302431607.44
    97 rdf:rest N4cf31532072845afbee25a45f6002b0b
    98 N8fac12c610c548f58d405075e55573bd schema:volumeNumber 90
    99 rdf:type schema:PublicationVolume
    100 Na958f4a5ea6c40c0ae0ee66025f5ee7e rdf:first sg:person.012367021751.30
    101 rdf:rest N8993a90915904f1cbef95c91616aae02
    102 Nbee48240b388419aa9b01effe23c19ed schema:affiliation grid-institutes:grid.5719.a
    103 schema:familyName Collmar
    104 schema:givenName David
    105 rdf:type schema:Person
    106 Nd47663ae79f54582ba208a061e4fa93c schema:name doi
    107 schema:value 10.1007/s41064-022-00202-2
    108 rdf:type schema:PropertyValue
    109 Nf634ffd8d7ee43b1a6d04f8deddf3d41 schema:name dimensions_id
    110 schema:value pub.1146912976
    111 rdf:type schema:PropertyValue
    112 anzsrc-for:08 schema:inDefinedTermSet anzsrc-for:
    113 schema:name Information and Computing Sciences
    114 rdf:type schema:DefinedTerm
    115 anzsrc-for:0806 schema:inDefinedTermSet anzsrc-for:
    116 schema:name Information Systems
    117 rdf:type schema:DefinedTerm
    118 sg:journal.1320463 schema:issn 2512-2789
    119 2512-2819
    120 schema:name PFG – Journal of Photogrammetry, Remote Sensing and Geoinformation Science
    121 schema:publisher Springer Nature
    122 rdf:type schema:Periodical
    123 sg:person.012367021751.30 schema:affiliation grid-institutes:grid.5719.a
    124 schema:familyName Walter
    125 schema:givenName Volker
    126 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.012367021751.30
    127 rdf:type schema:Person
    128 sg:person.015302431607.44 schema:affiliation grid-institutes:grid.5719.a
    129 schema:familyName Kölle
    130 schema:givenName Michael
    131 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.015302431607.44
    132 rdf:type schema:Person
    133 sg:pub.10.1007/978-3-030-19909-8_20 schema:sameAs https://app.dimensions.ai/details/publication/pub.1114220932
    134 https://doi.org/10.1007/978-3-030-19909-8_20
    135 rdf:type schema:CreativeWork
    136 sg:pub.10.1007/978-3-030-68787-8_37 schema:sameAs https://app.dimensions.ai/details/publication/pub.1135478924
    137 https://doi.org/10.1007/978-3-030-68787-8_37
    138 rdf:type schema:CreativeWork
    139 sg:pub.10.1007/978-3-319-14280-7_2 schema:sameAs https://app.dimensions.ai/details/publication/pub.1039444494
    140 https://doi.org/10.1007/978-3-319-14280-7_2
    141 rdf:type schema:CreativeWork
    142 sg:pub.10.1007/978-3-319-58550-5_7 schema:sameAs https://app.dimensions.ai/details/publication/pub.1101055990
    143 https://doi.org/10.1007/978-3-319-58550-5_7
    144 rdf:type schema:CreativeWork
    145 sg:pub.10.1007/s10462-016-9491-9 schema:sameAs https://app.dimensions.ai/details/publication/pub.1042840118
    146 https://doi.org/10.1007/s10462-016-9491-9
    147 rdf:type schema:CreativeWork
    148 sg:pub.10.1007/s10708-007-9111-y schema:sameAs https://app.dimensions.ai/details/publication/pub.1014006987
    149 https://doi.org/10.1007/s10708-007-9111-y
    150 rdf:type schema:CreativeWork
    151 sg:pub.10.1007/s41064-018-0058-z schema:sameAs https://app.dimensions.ai/details/publication/pub.1110507464
    152 https://doi.org/10.1007/s41064-018-0058-z
    153 rdf:type schema:CreativeWork
    154 sg:pub.10.1038/075450a0 schema:sameAs https://app.dimensions.ai/details/publication/pub.1035056658
    155 https://doi.org/10.1038/075450a0
    156 rdf:type schema:CreativeWork
    157 sg:pub.10.1186/s40965-018-0049-4 schema:sameAs https://app.dimensions.ai/details/publication/pub.1104245329
    158 https://doi.org/10.1186/s40965-018-0049-4
    159 rdf:type schema:CreativeWork
    160 grid-institutes:grid.5719.a schema:alternateName Institute for Photogrammetry (ifp), University of Stuttgart, Geschwister-Scholl-Str. 24D, 70174, Stuttgart, Germany
    161 schema:name Institute for Photogrammetry (ifp), University of Stuttgart, Geschwister-Scholl-Str. 24D, 70174, Stuttgart, Germany
    162 rdf:type schema:Organization
     




    Preview window. Press ESC to close (or click here)


    ...