Citizen crowds and experts: observer variability in image-based plant phenotyping View Full Text


Ontology type: schema:ScholarlyArticle      Open Access: True


Article Info

DATE

2018-12

AUTHORS

M. Valerio Giuffrida, Feng Chen, Hanno Scharr, Sotirios A. Tsaftaris

ABSTRACT

Background: Image-based plant phenotyping has become a powerful tool in unravelling genotype-environment interactions. The utilization of image analysis and machine learning have become paramount in extracting data stemming from phenotyping experiments. Yet we rely on observer (a human expert) input to perform the phenotyping process. We assume such input to be a 'gold-standard' and use it to evaluate software and algorithms and to train learning-based algorithms. However, we should consider whether any variability among experienced and non-experienced (including plain citizens) observers exists. Here we design a study that measures such variability in an annotation task of an integer-quantifiable phenotype: the leaf count. Results: We compare several experienced and non-experienced observers in annotating leaf counts in images of Arabidopsis Thaliana to measure intra- and inter-observer variability in a controlled study using specially designed annotation tools but also citizens using a distributed citizen-powered web-based platform. In the controlled study observers counted leaves by looking at top-view images, which were taken with low and high resolution optics. We assessed whether the utilization of tools specifically designed for this task can help to reduce such variability. We found that the presence of tools helps to reduce intra-observer variability, and that although intra- and inter-observer variability is present it does not have any effect on longitudinal leaf count trend statistical assessments. We compared the variability of citizen provided annotations (from the web-based platform) and found that plain citizens can provide statistically accurate leaf counts. We also compared a recent machine-learning based leaf counting algorithm and found that while close in performance it is still not within inter-observer variability. Conclusions: While expertise of the observer plays a role, if sufficient statistical power is present, a collection of non-experienced users and even citizens can be included in image-based phenotyping annotation tasks as long they are suitably designed. We hope with these findings that we can re-evaluate the expectations that we have from automated algorithms: as long as they perform within observer variability they can be considered a suitable alternative. In addition, we hope to invigorate an interest in introducing suitably designed tasks on citizen powered platforms not only to obtain useful information (for research) but to help engage the public in this societal important problem. More... »

PAGES

12

References to SciGraph publications

  • 2016-07. Multi-modality imagery database for plant phenotyping in MACHINE VISION AND APPLICATIONS
  • 2013-12. An online database for plant image analysis software tools in PLANT METHODS
  • 2008-05. LabelMe: A Database and Web-Based Tool for Image Annotation in INTERNATIONAL JOURNAL OF COMPUTER VISION
  • 2015. Imaging Methods for Phenotyping of Plant Traits in PHENOMICS IN CROP PLANTS: TRENDS, OPTIONS AND LIMITATIONS
  • 2016. Counting in the Wild in COMPUTER VISION – ECCV 2016
  • 2016. Recurrent Instance Segmentation in COMPUTER VISION – ECCV 2016
  • 2016-05. Leaf segmentation in plant phenotyping: a collation study in MACHINE VISION AND APPLICATIONS
  • 2007-05. Are ordinal rating scales better than percent ratings? a statistical and “psychological” view in EUPHYTICA
  • 2010-09-09. Time to automate identification in NATURE
  • 2017-12. A real-time phenotyping framework using machine learning for plant stress severity rating in soybean in PLANT METHODS
  • Identifiers

    URI

    http://scigraph.springernature.com/pub.10.1186/s13007-018-0278-7

    DOI

    http://dx.doi.org/10.1186/s13007-018-0278-7

    DIMENSIONS

    https://app.dimensions.ai/details/publication/pub.1100920397

    PUBMED

    https://www.ncbi.nlm.nih.gov/pubmed/29449872


    Indexing Status Check whether this publication has been indexed by Scopus and Web Of Science using the SN Indexing Status Tool
    Incoming Citations Browse incoming citations for this publication using opencitations.net

    JSON-LD is the canonical representation for SciGraph data.

    TIP: You can open this SciGraph record using an external JSON-LD service: JSON-LD Playground Google SDTT

    [
      {
        "@context": "https://springernature.github.io/scigraph/jsonld/sgcontext.json", 
        "about": [
          {
            "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/0801", 
            "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
            "name": "Artificial Intelligence and Image Processing", 
            "type": "DefinedTerm"
          }, 
          {
            "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/08", 
            "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
            "name": "Information and Computing Sciences", 
            "type": "DefinedTerm"
          }
        ], 
        "author": [
          {
            "affiliation": {
              "alternateName": "IMT Institute for Advanced Studies Lucca", 
              "id": "https://www.grid.ac/institutes/grid.462365.0", 
              "name": [
                "School of Engineering, Institute of Digital Communications, The University of Edinburgh, EH9 3FB, Edinburgh, UK", 
                "IMT School For Advanced Studies Lucca, Piazza San Francesco, 19, 55100, Lucca, Italy"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Giuffrida", 
            "givenName": "M. Valerio", 
            "id": "sg:person.010717106733.78", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.010717106733.78"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "University of Edinburgh", 
              "id": "https://www.grid.ac/institutes/grid.4305.2", 
              "name": [
                "School of Engineering, Institute of Digital Communications, The University of Edinburgh, EH9 3FB, Edinburgh, UK"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Chen", 
            "givenName": "Feng", 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "Forschungszentrum J\u00fclich", 
              "id": "https://www.grid.ac/institutes/grid.8385.6", 
              "name": [
                "Institute of Bio- and Geosciences (IBG), IBG-2: Plant Sciences, Forschungszentrum J\u00fclich GmbH, Wilhelm-Johnen-Stra\u00dfe, 52425, J\u00fclich, Germany"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Scharr", 
            "givenName": "Hanno", 
            "id": "sg:person.013555443265.15", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.013555443265.15"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "University of Edinburgh", 
              "id": "https://www.grid.ac/institutes/grid.4305.2", 
              "name": [
                "School of Engineering, Institute of Digital Communications, The University of Edinburgh, EH9 3FB, Edinburgh, UK"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Tsaftaris", 
            "givenName": "Sotirios A.", 
            "id": "sg:person.015713074727.37", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.015713074727.37"
            ], 
            "type": "Person"
          }
        ], 
        "citation": [
          {
            "id": "https://doi.org/10.1104/pp.79.1.11", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1000324548"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.3390/s141120078", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1002490891"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1016/j.compag.2016.04.002", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1002674457"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1016/j.compag.2016.04.002", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1002674457"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1016/j.compag.2016.04.002", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1002674457"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/s00138-015-0734-6", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1007589018", 
              "https://doi.org/10.1007/s00138-015-0734-6"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1016/j.ecoinf.2016.02.005", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1009606298"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1111/tpj.13472", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1011817022"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/s00138-015-0737-3", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1013680377", 
              "https://doi.org/10.1007/s00138-015-0737-3"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.3389/fpls.2014.00734", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1014376402"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1080/07352681003617285", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1015916643"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1038/467154a", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1020215414", 
              "https://doi.org/10.1038/467154a"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1038/467154a", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1020215414", 
              "https://doi.org/10.1038/467154a"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-81-322-2226-2_5", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1022322473", 
              "https://doi.org/10.1007/978-81-322-2226-2_5"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1186/1746-4811-9-38", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1023836620", 
              "https://doi.org/10.1186/1746-4811-9-38"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1016/j.patrec.2015.10.013", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1025617980"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/s11263-007-0090-8", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1027534025", 
              "https://doi.org/10.1007/s11263-007-0090-8"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1016/j.tplants.2011.09.005", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1032697097"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1016/s0140-6736(86)90837-8", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1040287180"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1016/s0140-6736(86)90837-8", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1040287180"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1073/pnas.0808843106", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1041235104"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.3390/s16050618", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1042125267"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1093/jxb/50.336.1169", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1047250441"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-319-46478-7_30", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1047637699", 
              "https://doi.org/10.1007/978-3-319-46478-7_30"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1016/j.tplants.2015.10.015", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1047877032"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/s10681-006-9296-z", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1048703841", 
              "https://doi.org/10.1007/s10681-006-9296-z"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1016/j.pbi.2015.02.006", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1050807422"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1016/j.pbi.2015.02.006", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1050807422"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1136/ard.16.4.494", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1052069859"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1111/j.1365-313x.2007.03330.x", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1052404139"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1016/j.tplants.2016.10.002", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1053180734"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-319-46466-4_19", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1053318791", 
              "https://doi.org/10.1007/978-3-319-46466-4_19"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1146/annurev-arplant-050312-120137", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1053427659"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1094/pdis-92-4-0530", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1060093569"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1094/pdis-92-4-0530", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1060093569"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1094/phyto-03-10-0087", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1060099007"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1094/phyto-83-806", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1060109593"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1104/pp.115.3.907", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1060841549"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/msp.2015.2405111", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1061424285"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/tcbb.2015.2404810", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1061541404"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1126/science.1057144", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1062444327"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1186/s13007-017-0173-7", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1084612542", 
              "https://doi.org/10.1186/s13007-017-0173-7"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1186/s13007-017-0173-7", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1084612542", 
              "https://doi.org/10.1186/s13007-017-0173-7"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.5244/c.29.cvppp.1", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1099341135"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.5244/c.29.cvppp.3", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1099341137"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.5244/c.29.cvppp.6", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1099341140"
            ], 
            "type": "CreativeWork"
          }
        ], 
        "datePublished": "2018-12", 
        "datePublishedReg": "2018-12-01", 
        "description": "Background: Image-based plant phenotyping has become a powerful tool in unravelling genotype-environment interactions. The utilization of image analysis and machine learning have become paramount in extracting data stemming from phenotyping experiments. Yet we rely on observer (a human expert) input to perform the phenotyping process. We assume such input to be a 'gold-standard' and use it to evaluate software and algorithms and to train learning-based algorithms. However, we should consider whether any variability among experienced and non-experienced (including plain citizens) observers exists. Here we design a study that measures such variability in an annotation task of an integer-quantifiable phenotype: the leaf count.\nResults: We compare several experienced and non-experienced observers in annotating leaf counts in images of Arabidopsis Thaliana to measure intra- and inter-observer variability in a controlled study using specially designed annotation tools but also citizens using a distributed citizen-powered web-based platform. In the controlled study observers counted leaves by looking at top-view images, which were taken with low and high resolution optics. We assessed whether the utilization of tools specifically designed for this task can help to reduce such variability. We found that the presence of tools helps to reduce intra-observer variability, and that although intra- and inter-observer variability is present it does not have any effect on longitudinal leaf count trend statistical assessments. We compared the variability of citizen provided annotations (from the web-based platform) and found that plain citizens can provide statistically accurate leaf counts. We also compared a recent machine-learning based leaf counting algorithm and found that while close in performance it is still not within inter-observer variability.\nConclusions: While expertise of the observer plays a role, if sufficient statistical power is present, a collection of non-experienced users and even citizens can be included in image-based phenotyping annotation tasks as long they are suitably designed. We hope with these findings that we can re-evaluate the expectations that we have from automated algorithms: as long as they perform within observer variability they can be considered a suitable alternative. In addition, we hope to invigorate an interest in introducing suitably designed tasks on citizen powered platforms not only to obtain useful information (for research) but to help engage the public in this societal important problem.", 
        "genre": "research_article", 
        "id": "sg:pub.10.1186/s13007-018-0278-7", 
        "inLanguage": [
          "en"
        ], 
        "isAccessibleForFree": true, 
        "isFundedItemOf": [
          {
            "id": "sg:grant.5498295", 
            "type": "MonetaryGrant"
          }, 
          {
            "id": "sg:grant.6711524", 
            "type": "MonetaryGrant"
          }, 
          {
            "id": "sg:grant.6502107", 
            "type": "MonetaryGrant"
          }, 
          {
            "id": "sg:grant.4576507", 
            "type": "MonetaryGrant"
          }
        ], 
        "isPartOf": [
          {
            "id": "sg:journal.1035309", 
            "issn": [
              "1746-4811"
            ], 
            "name": "Plant Methods", 
            "type": "Periodical"
          }, 
          {
            "issueNumber": "1", 
            "type": "PublicationIssue"
          }, 
          {
            "type": "PublicationVolume", 
            "volumeNumber": "14"
          }
        ], 
        "name": "Citizen crowds and experts: observer variability in image-based plant phenotyping", 
        "pagination": "12", 
        "productId": [
          {
            "name": "readcube_id", 
            "type": "PropertyValue", 
            "value": [
              "aec3f71d38005c931dfea242dc725c2125391e72d564976bc6ad4e85b4e8229b"
            ]
          }, 
          {
            "name": "pubmed_id", 
            "type": "PropertyValue", 
            "value": [
              "29449872"
            ]
          }, 
          {
            "name": "nlm_unique_id", 
            "type": "PropertyValue", 
            "value": [
              "101245798"
            ]
          }, 
          {
            "name": "doi", 
            "type": "PropertyValue", 
            "value": [
              "10.1186/s13007-018-0278-7"
            ]
          }, 
          {
            "name": "dimensions_id", 
            "type": "PropertyValue", 
            "value": [
              "pub.1100920397"
            ]
          }
        ], 
        "sameAs": [
          "https://doi.org/10.1186/s13007-018-0278-7", 
          "https://app.dimensions.ai/details/publication/pub.1100920397"
        ], 
        "sdDataset": "articles", 
        "sdDatePublished": "2019-04-10T16:38", 
        "sdLicense": "https://scigraph.springernature.com/explorer/license/", 
        "sdPublisher": {
          "name": "Springer Nature - SN SciGraph project", 
          "type": "Organization"
        }, 
        "sdSource": "s3://com-uberresearch-data-dimensions-target-20181106-alternative/cleanup/v134/2549eaecd7973599484d7c17b260dba0a4ecb94b/merge/v9/a6c9fde33151104705d4d7ff012ea9563521a3ce/jats-lookup/v90/0000000001_0000000264/records_8669_00000493.jsonl", 
        "type": "ScholarlyArticle", 
        "url": "http://link.springer.com/10.1186/s13007-018-0278-7"
      }
    ]
     

    Download the RDF metadata as:  json-ld nt turtle xml License info

    HOW TO GET THIS DATA PROGRAMMATICALLY:

    JSON-LD is a popular format for linked data which is fully compatible with JSON.

    curl -H 'Accept: application/ld+json' 'https://scigraph.springernature.com/pub.10.1186/s13007-018-0278-7'

    N-Triples is a line-based linked data format ideal for batch operations.

    curl -H 'Accept: application/n-triples' 'https://scigraph.springernature.com/pub.10.1186/s13007-018-0278-7'

    Turtle is a human-readable linked data format.

    curl -H 'Accept: text/turtle' 'https://scigraph.springernature.com/pub.10.1186/s13007-018-0278-7'

    RDF/XML is a standard XML format for linked data.

    curl -H 'Accept: application/rdf+xml' 'https://scigraph.springernature.com/pub.10.1186/s13007-018-0278-7'


     

    This table displays all metadata directly associated to this object as RDF triples.

    230 TRIPLES      21 PREDICATES      68 URIs      21 LITERALS      9 BLANK NODES

    Subject Predicate Object
    1 sg:pub.10.1186/s13007-018-0278-7 schema:about anzsrc-for:08
    2 anzsrc-for:0801
    3 schema:author Nc957849f3563481fa75aac904d45e935
    4 schema:citation sg:pub.10.1007/978-3-319-46466-4_19
    5 sg:pub.10.1007/978-3-319-46478-7_30
    6 sg:pub.10.1007/978-81-322-2226-2_5
    7 sg:pub.10.1007/s00138-015-0734-6
    8 sg:pub.10.1007/s00138-015-0737-3
    9 sg:pub.10.1007/s10681-006-9296-z
    10 sg:pub.10.1007/s11263-007-0090-8
    11 sg:pub.10.1038/467154a
    12 sg:pub.10.1186/1746-4811-9-38
    13 sg:pub.10.1186/s13007-017-0173-7
    14 https://doi.org/10.1016/j.compag.2016.04.002
    15 https://doi.org/10.1016/j.ecoinf.2016.02.005
    16 https://doi.org/10.1016/j.patrec.2015.10.013
    17 https://doi.org/10.1016/j.pbi.2015.02.006
    18 https://doi.org/10.1016/j.tplants.2011.09.005
    19 https://doi.org/10.1016/j.tplants.2015.10.015
    20 https://doi.org/10.1016/j.tplants.2016.10.002
    21 https://doi.org/10.1016/s0140-6736(86)90837-8
    22 https://doi.org/10.1073/pnas.0808843106
    23 https://doi.org/10.1080/07352681003617285
    24 https://doi.org/10.1093/jxb/50.336.1169
    25 https://doi.org/10.1094/pdis-92-4-0530
    26 https://doi.org/10.1094/phyto-03-10-0087
    27 https://doi.org/10.1094/phyto-83-806
    28 https://doi.org/10.1104/pp.115.3.907
    29 https://doi.org/10.1104/pp.79.1.11
    30 https://doi.org/10.1109/msp.2015.2405111
    31 https://doi.org/10.1109/tcbb.2015.2404810
    32 https://doi.org/10.1111/j.1365-313x.2007.03330.x
    33 https://doi.org/10.1111/tpj.13472
    34 https://doi.org/10.1126/science.1057144
    35 https://doi.org/10.1136/ard.16.4.494
    36 https://doi.org/10.1146/annurev-arplant-050312-120137
    37 https://doi.org/10.3389/fpls.2014.00734
    38 https://doi.org/10.3390/s141120078
    39 https://doi.org/10.3390/s16050618
    40 https://doi.org/10.5244/c.29.cvppp.1
    41 https://doi.org/10.5244/c.29.cvppp.3
    42 https://doi.org/10.5244/c.29.cvppp.6
    43 schema:datePublished 2018-12
    44 schema:datePublishedReg 2018-12-01
    45 schema:description Background: Image-based plant phenotyping has become a powerful tool in unravelling genotype-environment interactions. The utilization of image analysis and machine learning have become paramount in extracting data stemming from phenotyping experiments. Yet we rely on observer (a human expert) input to perform the phenotyping process. We assume such input to be a 'gold-standard' and use it to evaluate software and algorithms and to train learning-based algorithms. However, we should consider whether any variability among experienced and non-experienced (including plain citizens) observers exists. Here we design a study that measures such variability in an annotation task of an integer-quantifiable phenotype: the leaf count. Results: We compare several experienced and non-experienced observers in annotating leaf counts in images of Arabidopsis Thaliana to measure intra- and inter-observer variability in a controlled study using specially designed annotation tools but also citizens using a distributed citizen-powered web-based platform. In the controlled study observers counted leaves by looking at top-view images, which were taken with low and high resolution optics. We assessed whether the utilization of tools specifically designed for this task can help to reduce such variability. We found that the presence of tools helps to reduce intra-observer variability, and that although intra- and inter-observer variability is present it does not have any effect on longitudinal leaf count trend statistical assessments. We compared the variability of citizen provided annotations (from the web-based platform) and found that plain citizens can provide statistically accurate leaf counts. We also compared a recent machine-learning based leaf counting algorithm and found that while close in performance it is still not within inter-observer variability. Conclusions: While expertise of the observer plays a role, if sufficient statistical power is present, a collection of non-experienced users and even citizens can be included in image-based phenotyping annotation tasks as long they are suitably designed. We hope with these findings that we can re-evaluate the expectations that we have from automated algorithms: as long as they perform within observer variability they can be considered a suitable alternative. In addition, we hope to invigorate an interest in introducing suitably designed tasks on citizen powered platforms not only to obtain useful information (for research) but to help engage the public in this societal important problem.
    46 schema:genre research_article
    47 schema:inLanguage en
    48 schema:isAccessibleForFree true
    49 schema:isPartOf N5cf5927a0ba844339c35d94612b38f16
    50 Na3ea69b3401c42349a5837e04e9902cc
    51 sg:journal.1035309
    52 schema:name Citizen crowds and experts: observer variability in image-based plant phenotyping
    53 schema:pagination 12
    54 schema:productId N15e0421a70674b6f9793ba796f7e41cf
    55 N55ee76cfc4544605bed49a7277a776c2
    56 N6e4f345c518f4d528a06741ceb8debcb
    57 N6f0427ea238742e8afee570993371949
    58 Nee79a1ae41244a9eaa1f52876240cbd9
    59 schema:sameAs https://app.dimensions.ai/details/publication/pub.1100920397
    60 https://doi.org/10.1186/s13007-018-0278-7
    61 schema:sdDatePublished 2019-04-10T16:38
    62 schema:sdLicense https://scigraph.springernature.com/explorer/license/
    63 schema:sdPublisher N64b24d6da15d4f558590da5db4633843
    64 schema:url http://link.springer.com/10.1186/s13007-018-0278-7
    65 sgo:license sg:explorer/license/
    66 sgo:sdDataset articles
    67 rdf:type schema:ScholarlyArticle
    68 N15e0421a70674b6f9793ba796f7e41cf schema:name dimensions_id
    69 schema:value pub.1100920397
    70 rdf:type schema:PropertyValue
    71 N178de0a8d2634300badc2ae3baa89f63 rdf:first sg:person.013555443265.15
    72 rdf:rest N2669d0356ee948028b1b0d713e60b6d8
    73 N2669d0356ee948028b1b0d713e60b6d8 rdf:first sg:person.015713074727.37
    74 rdf:rest rdf:nil
    75 N55ee76cfc4544605bed49a7277a776c2 schema:name doi
    76 schema:value 10.1186/s13007-018-0278-7
    77 rdf:type schema:PropertyValue
    78 N5cf5927a0ba844339c35d94612b38f16 schema:volumeNumber 14
    79 rdf:type schema:PublicationVolume
    80 N64b24d6da15d4f558590da5db4633843 schema:name Springer Nature - SN SciGraph project
    81 rdf:type schema:Organization
    82 N6e28ce59f75f4b309222969caa25d2fe rdf:first Ned615cebe7094b10afa54ada7282ac61
    83 rdf:rest N178de0a8d2634300badc2ae3baa89f63
    84 N6e4f345c518f4d528a06741ceb8debcb schema:name pubmed_id
    85 schema:value 29449872
    86 rdf:type schema:PropertyValue
    87 N6f0427ea238742e8afee570993371949 schema:name readcube_id
    88 schema:value aec3f71d38005c931dfea242dc725c2125391e72d564976bc6ad4e85b4e8229b
    89 rdf:type schema:PropertyValue
    90 Na3ea69b3401c42349a5837e04e9902cc schema:issueNumber 1
    91 rdf:type schema:PublicationIssue
    92 Nc957849f3563481fa75aac904d45e935 rdf:first sg:person.010717106733.78
    93 rdf:rest N6e28ce59f75f4b309222969caa25d2fe
    94 Ned615cebe7094b10afa54ada7282ac61 schema:affiliation https://www.grid.ac/institutes/grid.4305.2
    95 schema:familyName Chen
    96 schema:givenName Feng
    97 rdf:type schema:Person
    98 Nee79a1ae41244a9eaa1f52876240cbd9 schema:name nlm_unique_id
    99 schema:value 101245798
    100 rdf:type schema:PropertyValue
    101 anzsrc-for:08 schema:inDefinedTermSet anzsrc-for:
    102 schema:name Information and Computing Sciences
    103 rdf:type schema:DefinedTerm
    104 anzsrc-for:0801 schema:inDefinedTermSet anzsrc-for:
    105 schema:name Artificial Intelligence and Image Processing
    106 rdf:type schema:DefinedTerm
    107 sg:grant.4576507 http://pending.schema.org/fundedItem sg:pub.10.1186/s13007-018-0278-7
    108 rdf:type schema:MonetaryGrant
    109 sg:grant.5498295 http://pending.schema.org/fundedItem sg:pub.10.1186/s13007-018-0278-7
    110 rdf:type schema:MonetaryGrant
    111 sg:grant.6502107 http://pending.schema.org/fundedItem sg:pub.10.1186/s13007-018-0278-7
    112 rdf:type schema:MonetaryGrant
    113 sg:grant.6711524 http://pending.schema.org/fundedItem sg:pub.10.1186/s13007-018-0278-7
    114 rdf:type schema:MonetaryGrant
    115 sg:journal.1035309 schema:issn 1746-4811
    116 schema:name Plant Methods
    117 rdf:type schema:Periodical
    118 sg:person.010717106733.78 schema:affiliation https://www.grid.ac/institutes/grid.462365.0
    119 schema:familyName Giuffrida
    120 schema:givenName M. Valerio
    121 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.010717106733.78
    122 rdf:type schema:Person
    123 sg:person.013555443265.15 schema:affiliation https://www.grid.ac/institutes/grid.8385.6
    124 schema:familyName Scharr
    125 schema:givenName Hanno
    126 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.013555443265.15
    127 rdf:type schema:Person
    128 sg:person.015713074727.37 schema:affiliation https://www.grid.ac/institutes/grid.4305.2
    129 schema:familyName Tsaftaris
    130 schema:givenName Sotirios A.
    131 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.015713074727.37
    132 rdf:type schema:Person
    133 sg:pub.10.1007/978-3-319-46466-4_19 schema:sameAs https://app.dimensions.ai/details/publication/pub.1053318791
    134 https://doi.org/10.1007/978-3-319-46466-4_19
    135 rdf:type schema:CreativeWork
    136 sg:pub.10.1007/978-3-319-46478-7_30 schema:sameAs https://app.dimensions.ai/details/publication/pub.1047637699
    137 https://doi.org/10.1007/978-3-319-46478-7_30
    138 rdf:type schema:CreativeWork
    139 sg:pub.10.1007/978-81-322-2226-2_5 schema:sameAs https://app.dimensions.ai/details/publication/pub.1022322473
    140 https://doi.org/10.1007/978-81-322-2226-2_5
    141 rdf:type schema:CreativeWork
    142 sg:pub.10.1007/s00138-015-0734-6 schema:sameAs https://app.dimensions.ai/details/publication/pub.1007589018
    143 https://doi.org/10.1007/s00138-015-0734-6
    144 rdf:type schema:CreativeWork
    145 sg:pub.10.1007/s00138-015-0737-3 schema:sameAs https://app.dimensions.ai/details/publication/pub.1013680377
    146 https://doi.org/10.1007/s00138-015-0737-3
    147 rdf:type schema:CreativeWork
    148 sg:pub.10.1007/s10681-006-9296-z schema:sameAs https://app.dimensions.ai/details/publication/pub.1048703841
    149 https://doi.org/10.1007/s10681-006-9296-z
    150 rdf:type schema:CreativeWork
    151 sg:pub.10.1007/s11263-007-0090-8 schema:sameAs https://app.dimensions.ai/details/publication/pub.1027534025
    152 https://doi.org/10.1007/s11263-007-0090-8
    153 rdf:type schema:CreativeWork
    154 sg:pub.10.1038/467154a schema:sameAs https://app.dimensions.ai/details/publication/pub.1020215414
    155 https://doi.org/10.1038/467154a
    156 rdf:type schema:CreativeWork
    157 sg:pub.10.1186/1746-4811-9-38 schema:sameAs https://app.dimensions.ai/details/publication/pub.1023836620
    158 https://doi.org/10.1186/1746-4811-9-38
    159 rdf:type schema:CreativeWork
    160 sg:pub.10.1186/s13007-017-0173-7 schema:sameAs https://app.dimensions.ai/details/publication/pub.1084612542
    161 https://doi.org/10.1186/s13007-017-0173-7
    162 rdf:type schema:CreativeWork
    163 https://doi.org/10.1016/j.compag.2016.04.002 schema:sameAs https://app.dimensions.ai/details/publication/pub.1002674457
    164 rdf:type schema:CreativeWork
    165 https://doi.org/10.1016/j.ecoinf.2016.02.005 schema:sameAs https://app.dimensions.ai/details/publication/pub.1009606298
    166 rdf:type schema:CreativeWork
    167 https://doi.org/10.1016/j.patrec.2015.10.013 schema:sameAs https://app.dimensions.ai/details/publication/pub.1025617980
    168 rdf:type schema:CreativeWork
    169 https://doi.org/10.1016/j.pbi.2015.02.006 schema:sameAs https://app.dimensions.ai/details/publication/pub.1050807422
    170 rdf:type schema:CreativeWork
    171 https://doi.org/10.1016/j.tplants.2011.09.005 schema:sameAs https://app.dimensions.ai/details/publication/pub.1032697097
    172 rdf:type schema:CreativeWork
    173 https://doi.org/10.1016/j.tplants.2015.10.015 schema:sameAs https://app.dimensions.ai/details/publication/pub.1047877032
    174 rdf:type schema:CreativeWork
    175 https://doi.org/10.1016/j.tplants.2016.10.002 schema:sameAs https://app.dimensions.ai/details/publication/pub.1053180734
    176 rdf:type schema:CreativeWork
    177 https://doi.org/10.1016/s0140-6736(86)90837-8 schema:sameAs https://app.dimensions.ai/details/publication/pub.1040287180
    178 rdf:type schema:CreativeWork
    179 https://doi.org/10.1073/pnas.0808843106 schema:sameAs https://app.dimensions.ai/details/publication/pub.1041235104
    180 rdf:type schema:CreativeWork
    181 https://doi.org/10.1080/07352681003617285 schema:sameAs https://app.dimensions.ai/details/publication/pub.1015916643
    182 rdf:type schema:CreativeWork
    183 https://doi.org/10.1093/jxb/50.336.1169 schema:sameAs https://app.dimensions.ai/details/publication/pub.1047250441
    184 rdf:type schema:CreativeWork
    185 https://doi.org/10.1094/pdis-92-4-0530 schema:sameAs https://app.dimensions.ai/details/publication/pub.1060093569
    186 rdf:type schema:CreativeWork
    187 https://doi.org/10.1094/phyto-03-10-0087 schema:sameAs https://app.dimensions.ai/details/publication/pub.1060099007
    188 rdf:type schema:CreativeWork
    189 https://doi.org/10.1094/phyto-83-806 schema:sameAs https://app.dimensions.ai/details/publication/pub.1060109593
    190 rdf:type schema:CreativeWork
    191 https://doi.org/10.1104/pp.115.3.907 schema:sameAs https://app.dimensions.ai/details/publication/pub.1060841549
    192 rdf:type schema:CreativeWork
    193 https://doi.org/10.1104/pp.79.1.11 schema:sameAs https://app.dimensions.ai/details/publication/pub.1000324548
    194 rdf:type schema:CreativeWork
    195 https://doi.org/10.1109/msp.2015.2405111 schema:sameAs https://app.dimensions.ai/details/publication/pub.1061424285
    196 rdf:type schema:CreativeWork
    197 https://doi.org/10.1109/tcbb.2015.2404810 schema:sameAs https://app.dimensions.ai/details/publication/pub.1061541404
    198 rdf:type schema:CreativeWork
    199 https://doi.org/10.1111/j.1365-313x.2007.03330.x schema:sameAs https://app.dimensions.ai/details/publication/pub.1052404139
    200 rdf:type schema:CreativeWork
    201 https://doi.org/10.1111/tpj.13472 schema:sameAs https://app.dimensions.ai/details/publication/pub.1011817022
    202 rdf:type schema:CreativeWork
    203 https://doi.org/10.1126/science.1057144 schema:sameAs https://app.dimensions.ai/details/publication/pub.1062444327
    204 rdf:type schema:CreativeWork
    205 https://doi.org/10.1136/ard.16.4.494 schema:sameAs https://app.dimensions.ai/details/publication/pub.1052069859
    206 rdf:type schema:CreativeWork
    207 https://doi.org/10.1146/annurev-arplant-050312-120137 schema:sameAs https://app.dimensions.ai/details/publication/pub.1053427659
    208 rdf:type schema:CreativeWork
    209 https://doi.org/10.3389/fpls.2014.00734 schema:sameAs https://app.dimensions.ai/details/publication/pub.1014376402
    210 rdf:type schema:CreativeWork
    211 https://doi.org/10.3390/s141120078 schema:sameAs https://app.dimensions.ai/details/publication/pub.1002490891
    212 rdf:type schema:CreativeWork
    213 https://doi.org/10.3390/s16050618 schema:sameAs https://app.dimensions.ai/details/publication/pub.1042125267
    214 rdf:type schema:CreativeWork
    215 https://doi.org/10.5244/c.29.cvppp.1 schema:sameAs https://app.dimensions.ai/details/publication/pub.1099341135
    216 rdf:type schema:CreativeWork
    217 https://doi.org/10.5244/c.29.cvppp.3 schema:sameAs https://app.dimensions.ai/details/publication/pub.1099341137
    218 rdf:type schema:CreativeWork
    219 https://doi.org/10.5244/c.29.cvppp.6 schema:sameAs https://app.dimensions.ai/details/publication/pub.1099341140
    220 rdf:type schema:CreativeWork
    221 https://www.grid.ac/institutes/grid.4305.2 schema:alternateName University of Edinburgh
    222 schema:name School of Engineering, Institute of Digital Communications, The University of Edinburgh, EH9 3FB, Edinburgh, UK
    223 rdf:type schema:Organization
    224 https://www.grid.ac/institutes/grid.462365.0 schema:alternateName IMT Institute for Advanced Studies Lucca
    225 schema:name IMT School For Advanced Studies Lucca, Piazza San Francesco, 19, 55100, Lucca, Italy
    226 School of Engineering, Institute of Digital Communications, The University of Edinburgh, EH9 3FB, Edinburgh, UK
    227 rdf:type schema:Organization
    228 https://www.grid.ac/institutes/grid.8385.6 schema:alternateName Forschungszentrum Jülich
    229 schema:name Institute of Bio- and Geosciences (IBG), IBG-2: Plant Sciences, Forschungszentrum Jülich GmbH, Wilhelm-Johnen-Straße, 52425, Jülich, Germany
    230 rdf:type schema:Organization
     




    Preview window. Press ESC to close (or click here)


    ...