The Pascal Visual Object Classes Challenge: A Retrospective View Full Text


Ontology type: schema:ScholarlyArticle      Open Access: True


Article Info

DATE

2015-01

AUTHORS

Mark Everingham, S. M. Ali Eslami, Luc Van Gool, Christopher K. I. Williams, John Winn, Andrew Zisserman

ABSTRACT

The Pascal Visual Object Classes (VOC) challenge consists of two components: (i) a publicly available dataset of images together with ground truth annotation and standardised evaluation software; and (ii) an annual competition and workshop. There are five challenges: classification, detection, segmentation, action classification, and person layout. In this paper we provide a review of the challenge from 2008–2012. The paper is intended for two audiences: algorithm designers, researchers who want to see what the state of the art is, as measured by performance on the VOC datasets, along with the limitations and weak points of the current generation of algorithms; and, challenge designers, who want to see what we as organisers have learnt from the process and our recommendations for the organisation of future challenges. To analyse the performance of submitted algorithms on the VOC datasets we introduce a number of novel evaluation methods: a bootstrapping method for determining whether differences in the performance of two algorithms are significant or not; a normalised average precision so that performance can be compared across classes with different proportions of positive instances; a clustering method for visualising the performance across multiple algorithms so that the hard and easy images can be identified; and the use of a joint classifier over the submitted algorithms in order to measure their complementarity and combined performance. We also analyse the community’s progress through time using the methods of Hoiem et al. (Proceedings of European Conference on Computer Vision, 2012) to identify the types of occurring errors. We conclude the paper with an appraisal of the aspects of the challenge that worked well, and those that could be improved in future challenges. More... »

PAGES

98-136

References to SciGraph publications

  • 2004-11. Distinctive Image Features from Scale-Invariant Keypoints in INTERNATIONAL JOURNAL OF COMPUTER VISION
  • 2012-05. Modulating Shape Features by Color Attention for Object Recognition in INTERNATIONAL JOURNAL OF COMPUTER VISION
  • 2004-05. Robust Real-Time Face Detection in INTERNATIONAL JOURNAL OF COMPUTER VISION
  • 2012. Segmentation over Detection by Coupled Global and Local Sparse Representations in COMPUTER VISION – ECCV 2012
  • 2008-05. LabelMe: A Database and Web-Based Tool for Image Annotation in INTERNATIONAL JOURNAL OF COMPUTER VISION
  • 2012. Object Reading: Text Recognition for Object Recognition in COMPUTER VISION – ECCV 2012. WORKSHOPS AND DEMONSTRATIONS
  • 2012. Semantic Segmentation with Second-Order Pooling in COMPUTER VISION – ECCV 2012
  • 2004. All of Statistics, A Concise Course in Statistical Inference in NONE
  • 2012. On Recognizing Actions in Still Images via Multiple Features in COMPUTER VISION – ECCV 2012. WORKSHOPS AND DEMONSTRATIONS
  • 2014. Visualizing and Understanding Convolutional Networks in COMPUTER VISION – ECCV 2014
  • 2012. Object-Centric Spatial Pooling for Image Classification in COMPUTER VISION – ECCV 2012
  • 2013-09. Selective Search for Object Recognition in INTERNATIONAL JOURNAL OF COMPUTER VISION
  • 2012. Diagnosing Error in Object Detectors in COMPUTER VISION – ECCV 2012
  • 2010-06. The Pascal Visual Object Classes (VOC) Challenge in INTERNATIONAL JOURNAL OF COMPUTER VISION
  • Identifiers

    URI

    http://scigraph.springernature.com/pub.10.1007/s11263-014-0733-5

    DOI

    http://dx.doi.org/10.1007/s11263-014-0733-5

    DIMENSIONS

    https://app.dimensions.ai/details/publication/pub.1017073734


    Indexing Status Check whether this publication has been indexed by Scopus and Web Of Science using the SN Indexing Status Tool
    Incoming Citations Browse incoming citations for this publication using opencitations.net

    JSON-LD is the canonical representation for SciGraph data.

    TIP: You can open this SciGraph record using an external JSON-LD service: JSON-LD Playground Google SDTT

    [
      {
        "@context": "https://springernature.github.io/scigraph/jsonld/sgcontext.json", 
        "about": [
          {
            "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/0801", 
            "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
            "name": "Artificial Intelligence and Image Processing", 
            "type": "DefinedTerm"
          }, 
          {
            "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/08", 
            "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
            "name": "Information and Computing Sciences", 
            "type": "DefinedTerm"
          }
        ], 
        "author": [
          {
            "affiliation": {
              "alternateName": "University of Leeds", 
              "id": "https://www.grid.ac/institutes/grid.9909.9", 
              "name": [
                "University of Leeds, Leeds, UK"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Everingham", 
            "givenName": "Mark", 
            "id": "sg:person.012363753235.37", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.012363753235.37"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "Microsoft Research (United Kingdom)", 
              "id": "https://www.grid.ac/institutes/grid.24488.32", 
              "name": [
                "Microsoft Research, Cambridge, UK"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Eslami", 
            "givenName": "S. M. Ali", 
            "id": "sg:person.011434125503.42", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.011434125503.42"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "Swiss Federal Institute of Technology in Zurich", 
              "id": "https://www.grid.ac/institutes/grid.5801.c", 
              "name": [
                "KU Leuven, Leuven, Belgium", 
                "ETH, Zurich, Switzerland"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Van Gool", 
            "givenName": "Luc", 
            "id": "sg:person.0616213477.88", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.0616213477.88"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "University of Edinburgh", 
              "id": "https://www.grid.ac/institutes/grid.4305.2", 
              "name": [
                "University of Edinburgh, Edinburgh, UK"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Williams", 
            "givenName": "Christopher K. I.", 
            "id": "sg:person.07643001615.73", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.07643001615.73"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "Microsoft Research (United Kingdom)", 
              "id": "https://www.grid.ac/institutes/grid.24488.32", 
              "name": [
                "Microsoft Research, Cambridge, UK"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Winn", 
            "givenName": "John", 
            "id": "sg:person.01221574626.63", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01221574626.63"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "University of Oxford", 
              "id": "https://www.grid.ac/institutes/grid.4991.5", 
              "name": [
                "University of Oxford, Oxford, UK"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Zisserman", 
            "givenName": "Andrew", 
            "id": "sg:person.012270111307.09", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.012270111307.09"
            ], 
            "type": "Person"
          }
        ], 
        "citation": [
          {
            "id": "sg:pub.10.1023/b:visi.0000013087.49260.fb", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1001944608", 
              "https://doi.org/10.1023/b:visi.0000013087.49260.fb"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-642-33786-4_32", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1002182143", 
              "https://doi.org/10.1007/978-3-642-33786-4_32"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-642-33885-4_46", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1003958533", 
              "https://doi.org/10.1007/978-3-642-33885-4_46"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1145/1961189.1961199", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1013637525"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/s11263-009-0275-4", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1014796149", 
              "https://doi.org/10.1007/s11263-009-0275-4"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/s11263-009-0275-4", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1014796149", 
              "https://doi.org/10.1007/s11263-009-0275-4"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/s11263-007-0090-8", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1027534025", 
              "https://doi.org/10.1007/s11263-007-0090-8"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-642-33712-3_25", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1027572846", 
              "https://doi.org/10.1007/978-3-642-33712-3_25"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-642-33709-3_1", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1029182106", 
              "https://doi.org/10.1007/978-3-642-33709-3_1"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/s11263-011-0495-2", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1031328704", 
              "https://doi.org/10.1007/s11263-011-0495-2"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-319-10590-1_53", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1032233097", 
              "https://doi.org/10.1007/978-3-319-10590-1_53"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/s11263-013-0620-5", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1033900312", 
              "https://doi.org/10.1007/s11263-013-0620-5"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/tpami.2012.204", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1035667356"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1016/j.asoc.2012.12.013", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1038782578"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-642-33885-4_27", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1045347976", 
              "https://doi.org/10.1007/978-3-642-33885-4_27"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-642-33715-4_48", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1050957536", 
              "https://doi.org/10.1007/978-3-642-33715-4_48"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1023/b:visi.0000029664.99615.94", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1052687286", 
              "https://doi.org/10.1023/b:visi.0000029664.99615.94"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-0-387-21736-9", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1053160163", 
              "https://doi.org/10.1007/978-0-387-21736-9"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-0-387-21736-9", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1053160163", 
              "https://doi.org/10.1007/978-0-387-21736-9"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1093/biomet/91.3.743", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1059421368"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/tpami.2009.167", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1061743745"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/iccv.2011.6126486", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1093330382"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/cvpr.2011.5995330", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1093409457"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/cvpr.2005.177", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1093997066"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/cvpr.2014.222", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1094012327"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/cvpr.2011.5995347", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1094016389"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/cvpr.2009.5206772", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1094211775"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/cvpr.2013.428", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1094300906"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/cvpr.2013.428", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1094300906"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/cvpr.2006.68", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1094512911"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/cvpr.2011.5995368", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1094538483"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/cvpr.2010.5540226", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1094653828"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/cvpr.2014.81", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1094727707"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/cvpr.2013.424", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1094917736"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/cvpr.2013.424", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1094917736"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/iccv.2009.5459183", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1095015498"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/cvpr.2010.5540096", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1095033380"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/cvpr.2009.5206757", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1095180230"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.5244/c.26.82", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1099383252"
            ], 
            "type": "CreativeWork"
          }
        ], 
        "datePublished": "2015-01", 
        "datePublishedReg": "2015-01-01", 
        "description": "The Pascal Visual Object Classes (VOC) challenge consists of two components: (i) a publicly available dataset of images together with ground truth annotation and standardised evaluation software; and (ii) an annual competition and workshop. There are five challenges: classification, detection, segmentation, action classification, and person layout. In this paper we provide a review of the challenge from 2008\u20132012. The paper is intended for two audiences: algorithm designers, researchers who want to see what the state of the art is, as measured by performance on the VOC datasets, along with the limitations and weak points of the current generation of algorithms; and, challenge designers, who want to see what we as organisers have learnt from the process and our recommendations for the organisation of future challenges. To analyse the performance of submitted algorithms on the VOC datasets we introduce a number of novel evaluation methods: a bootstrapping method for determining whether differences in the performance of two algorithms are significant or not; a normalised average precision so that performance can be compared across classes with different proportions of positive instances; a clustering method for visualising the performance across multiple algorithms so that the hard and easy images can be identified; and the use of a joint classifier over the submitted algorithms in order to measure their complementarity and combined performance. We also analyse the community\u2019s progress through time using the methods of Hoiem et al. (Proceedings of European Conference on Computer Vision, 2012) to identify the types of occurring errors. We conclude the paper with an appraisal of the aspects of the challenge that worked well, and those that could be improved in future challenges.", 
        "genre": "research_article", 
        "id": "sg:pub.10.1007/s11263-014-0733-5", 
        "inLanguage": [
          "en"
        ], 
        "isAccessibleForFree": true, 
        "isFundedItemOf": [
          {
            "id": "sg:grant.3773089", 
            "type": "MonetaryGrant"
          }
        ], 
        "isPartOf": [
          {
            "id": "sg:journal.1032807", 
            "issn": [
              "0920-5691", 
              "1573-1405"
            ], 
            "name": "International Journal of Computer Vision", 
            "type": "Periodical"
          }, 
          {
            "issueNumber": "1", 
            "type": "PublicationIssue"
          }, 
          {
            "type": "PublicationVolume", 
            "volumeNumber": "111"
          }
        ], 
        "name": "The Pascal Visual Object Classes Challenge: A Retrospective", 
        "pagination": "98-136", 
        "productId": [
          {
            "name": "readcube_id", 
            "type": "PropertyValue", 
            "value": [
              "51589b37f97e65098a651ae660d7394590f9549fb895db92dc7cc2b69a6d6fbf"
            ]
          }, 
          {
            "name": "doi", 
            "type": "PropertyValue", 
            "value": [
              "10.1007/s11263-014-0733-5"
            ]
          }, 
          {
            "name": "dimensions_id", 
            "type": "PropertyValue", 
            "value": [
              "pub.1017073734"
            ]
          }
        ], 
        "sameAs": [
          "https://doi.org/10.1007/s11263-014-0733-5", 
          "https://app.dimensions.ai/details/publication/pub.1017073734"
        ], 
        "sdDataset": "articles", 
        "sdDatePublished": "2019-04-10T18:22", 
        "sdLicense": "https://scigraph.springernature.com/explorer/license/", 
        "sdPublisher": {
          "name": "Springer Nature - SN SciGraph project", 
          "type": "Organization"
        }, 
        "sdSource": "s3://com-uberresearch-data-dimensions-target-20181106-alternative/cleanup/v134/2549eaecd7973599484d7c17b260dba0a4ecb94b/merge/v9/a6c9fde33151104705d4d7ff012ea9563521a3ce/jats-lookup/v90/0000000001_0000000264/records_8675_00000521.jsonl", 
        "type": "ScholarlyArticle", 
        "url": "http://link.springer.com/10.1007%2Fs11263-014-0733-5"
      }
    ]
     

    Download the RDF metadata as:  json-ld nt turtle xml License info

    HOW TO GET THIS DATA PROGRAMMATICALLY:

    JSON-LD is a popular format for linked data which is fully compatible with JSON.

    curl -H 'Accept: application/ld+json' 'https://scigraph.springernature.com/pub.10.1007/s11263-014-0733-5'

    N-Triples is a line-based linked data format ideal for batch operations.

    curl -H 'Accept: application/n-triples' 'https://scigraph.springernature.com/pub.10.1007/s11263-014-0733-5'

    Turtle is a human-readable linked data format.

    curl -H 'Accept: text/turtle' 'https://scigraph.springernature.com/pub.10.1007/s11263-014-0733-5'

    RDF/XML is a standard XML format for linked data.

    curl -H 'Accept: application/rdf+xml' 'https://scigraph.springernature.com/pub.10.1007/s11263-014-0733-5'


     

    This table displays all metadata directly associated to this object as RDF triples.

    230 TRIPLES      21 PREDICATES      62 URIs      19 LITERALS      7 BLANK NODES

    Subject Predicate Object
    1 sg:pub.10.1007/s11263-014-0733-5 schema:about anzsrc-for:08
    2 anzsrc-for:0801
    3 schema:author N848352d8bd38425b9a59bc3d87b66443
    4 schema:citation sg:pub.10.1007/978-0-387-21736-9
    5 sg:pub.10.1007/978-3-319-10590-1_53
    6 sg:pub.10.1007/978-3-642-33709-3_1
    7 sg:pub.10.1007/978-3-642-33712-3_25
    8 sg:pub.10.1007/978-3-642-33715-4_48
    9 sg:pub.10.1007/978-3-642-33786-4_32
    10 sg:pub.10.1007/978-3-642-33885-4_27
    11 sg:pub.10.1007/978-3-642-33885-4_46
    12 sg:pub.10.1007/s11263-007-0090-8
    13 sg:pub.10.1007/s11263-009-0275-4
    14 sg:pub.10.1007/s11263-011-0495-2
    15 sg:pub.10.1007/s11263-013-0620-5
    16 sg:pub.10.1023/b:visi.0000013087.49260.fb
    17 sg:pub.10.1023/b:visi.0000029664.99615.94
    18 https://doi.org/10.1016/j.asoc.2012.12.013
    19 https://doi.org/10.1093/biomet/91.3.743
    20 https://doi.org/10.1109/cvpr.2005.177
    21 https://doi.org/10.1109/cvpr.2006.68
    22 https://doi.org/10.1109/cvpr.2009.5206757
    23 https://doi.org/10.1109/cvpr.2009.5206772
    24 https://doi.org/10.1109/cvpr.2010.5540096
    25 https://doi.org/10.1109/cvpr.2010.5540226
    26 https://doi.org/10.1109/cvpr.2011.5995330
    27 https://doi.org/10.1109/cvpr.2011.5995347
    28 https://doi.org/10.1109/cvpr.2011.5995368
    29 https://doi.org/10.1109/cvpr.2013.424
    30 https://doi.org/10.1109/cvpr.2013.428
    31 https://doi.org/10.1109/cvpr.2014.222
    32 https://doi.org/10.1109/cvpr.2014.81
    33 https://doi.org/10.1109/iccv.2009.5459183
    34 https://doi.org/10.1109/iccv.2011.6126486
    35 https://doi.org/10.1109/tpami.2009.167
    36 https://doi.org/10.1109/tpami.2012.204
    37 https://doi.org/10.1145/1961189.1961199
    38 https://doi.org/10.5244/c.26.82
    39 schema:datePublished 2015-01
    40 schema:datePublishedReg 2015-01-01
    41 schema:description The Pascal Visual Object Classes (VOC) challenge consists of two components: (i) a publicly available dataset of images together with ground truth annotation and standardised evaluation software; and (ii) an annual competition and workshop. There are five challenges: classification, detection, segmentation, action classification, and person layout. In this paper we provide a review of the challenge from 2008–2012. The paper is intended for two audiences: algorithm designers, researchers who want to see what the state of the art is, as measured by performance on the VOC datasets, along with the limitations and weak points of the current generation of algorithms; and, challenge designers, who want to see what we as organisers have learnt from the process and our recommendations for the organisation of future challenges. To analyse the performance of submitted algorithms on the VOC datasets we introduce a number of novel evaluation methods: a bootstrapping method for determining whether differences in the performance of two algorithms are significant or not; a normalised average precision so that performance can be compared across classes with different proportions of positive instances; a clustering method for visualising the performance across multiple algorithms so that the hard and easy images can be identified; and the use of a joint classifier over the submitted algorithms in order to measure their complementarity and combined performance. We also analyse the community’s progress through time using the methods of Hoiem et al. (Proceedings of European Conference on Computer Vision, 2012) to identify the types of occurring errors. We conclude the paper with an appraisal of the aspects of the challenge that worked well, and those that could be improved in future challenges.
    42 schema:genre research_article
    43 schema:inLanguage en
    44 schema:isAccessibleForFree true
    45 schema:isPartOf N1a6c1bd3eaae46c59d0986725e5d52cc
    46 Nb01345f4caeb4b81b4f98e03ea856599
    47 sg:journal.1032807
    48 schema:name The Pascal Visual Object Classes Challenge: A Retrospective
    49 schema:pagination 98-136
    50 schema:productId N1136e76be818495698434e7373041dbf
    51 N4eaff29f7afb46c49f09af9a8480eb6b
    52 Na52753d7ebf146bab3a14c7048e3a9b3
    53 schema:sameAs https://app.dimensions.ai/details/publication/pub.1017073734
    54 https://doi.org/10.1007/s11263-014-0733-5
    55 schema:sdDatePublished 2019-04-10T18:22
    56 schema:sdLicense https://scigraph.springernature.com/explorer/license/
    57 schema:sdPublisher Nfcda9fa5e76f4ea7b137cfbcef5e3250
    58 schema:url http://link.springer.com/10.1007%2Fs11263-014-0733-5
    59 sgo:license sg:explorer/license/
    60 sgo:sdDataset articles
    61 rdf:type schema:ScholarlyArticle
    62 N1136e76be818495698434e7373041dbf schema:name dimensions_id
    63 schema:value pub.1017073734
    64 rdf:type schema:PropertyValue
    65 N1a6c1bd3eaae46c59d0986725e5d52cc schema:issueNumber 1
    66 rdf:type schema:PublicationIssue
    67 N27a3fd262fc441c9bfbcad5a3edae7ce rdf:first sg:person.011434125503.42
    68 rdf:rest Nadc8b397e79145f19ab93a8acec3ce53
    69 N41a7c313594f420eb329baa18d0d6a36 rdf:first sg:person.012270111307.09
    70 rdf:rest rdf:nil
    71 N4eaff29f7afb46c49f09af9a8480eb6b schema:name readcube_id
    72 schema:value 51589b37f97e65098a651ae660d7394590f9549fb895db92dc7cc2b69a6d6fbf
    73 rdf:type schema:PropertyValue
    74 N569f01ec21d044dda409dd4980ecae35 rdf:first sg:person.01221574626.63
    75 rdf:rest N41a7c313594f420eb329baa18d0d6a36
    76 N848352d8bd38425b9a59bc3d87b66443 rdf:first sg:person.012363753235.37
    77 rdf:rest N27a3fd262fc441c9bfbcad5a3edae7ce
    78 Na52753d7ebf146bab3a14c7048e3a9b3 schema:name doi
    79 schema:value 10.1007/s11263-014-0733-5
    80 rdf:type schema:PropertyValue
    81 Nadc8b397e79145f19ab93a8acec3ce53 rdf:first sg:person.0616213477.88
    82 rdf:rest Nf1c4f0107324489f8963ffc04c5867e0
    83 Nb01345f4caeb4b81b4f98e03ea856599 schema:volumeNumber 111
    84 rdf:type schema:PublicationVolume
    85 Nf1c4f0107324489f8963ffc04c5867e0 rdf:first sg:person.07643001615.73
    86 rdf:rest N569f01ec21d044dda409dd4980ecae35
    87 Nfcda9fa5e76f4ea7b137cfbcef5e3250 schema:name Springer Nature - SN SciGraph project
    88 rdf:type schema:Organization
    89 anzsrc-for:08 schema:inDefinedTermSet anzsrc-for:
    90 schema:name Information and Computing Sciences
    91 rdf:type schema:DefinedTerm
    92 anzsrc-for:0801 schema:inDefinedTermSet anzsrc-for:
    93 schema:name Artificial Intelligence and Image Processing
    94 rdf:type schema:DefinedTerm
    95 sg:grant.3773089 http://pending.schema.org/fundedItem sg:pub.10.1007/s11263-014-0733-5
    96 rdf:type schema:MonetaryGrant
    97 sg:journal.1032807 schema:issn 0920-5691
    98 1573-1405
    99 schema:name International Journal of Computer Vision
    100 rdf:type schema:Periodical
    101 sg:person.011434125503.42 schema:affiliation https://www.grid.ac/institutes/grid.24488.32
    102 schema:familyName Eslami
    103 schema:givenName S. M. Ali
    104 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.011434125503.42
    105 rdf:type schema:Person
    106 sg:person.01221574626.63 schema:affiliation https://www.grid.ac/institutes/grid.24488.32
    107 schema:familyName Winn
    108 schema:givenName John
    109 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01221574626.63
    110 rdf:type schema:Person
    111 sg:person.012270111307.09 schema:affiliation https://www.grid.ac/institutes/grid.4991.5
    112 schema:familyName Zisserman
    113 schema:givenName Andrew
    114 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.012270111307.09
    115 rdf:type schema:Person
    116 sg:person.012363753235.37 schema:affiliation https://www.grid.ac/institutes/grid.9909.9
    117 schema:familyName Everingham
    118 schema:givenName Mark
    119 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.012363753235.37
    120 rdf:type schema:Person
    121 sg:person.0616213477.88 schema:affiliation https://www.grid.ac/institutes/grid.5801.c
    122 schema:familyName Van Gool
    123 schema:givenName Luc
    124 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.0616213477.88
    125 rdf:type schema:Person
    126 sg:person.07643001615.73 schema:affiliation https://www.grid.ac/institutes/grid.4305.2
    127 schema:familyName Williams
    128 schema:givenName Christopher K. I.
    129 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.07643001615.73
    130 rdf:type schema:Person
    131 sg:pub.10.1007/978-0-387-21736-9 schema:sameAs https://app.dimensions.ai/details/publication/pub.1053160163
    132 https://doi.org/10.1007/978-0-387-21736-9
    133 rdf:type schema:CreativeWork
    134 sg:pub.10.1007/978-3-319-10590-1_53 schema:sameAs https://app.dimensions.ai/details/publication/pub.1032233097
    135 https://doi.org/10.1007/978-3-319-10590-1_53
    136 rdf:type schema:CreativeWork
    137 sg:pub.10.1007/978-3-642-33709-3_1 schema:sameAs https://app.dimensions.ai/details/publication/pub.1029182106
    138 https://doi.org/10.1007/978-3-642-33709-3_1
    139 rdf:type schema:CreativeWork
    140 sg:pub.10.1007/978-3-642-33712-3_25 schema:sameAs https://app.dimensions.ai/details/publication/pub.1027572846
    141 https://doi.org/10.1007/978-3-642-33712-3_25
    142 rdf:type schema:CreativeWork
    143 sg:pub.10.1007/978-3-642-33715-4_48 schema:sameAs https://app.dimensions.ai/details/publication/pub.1050957536
    144 https://doi.org/10.1007/978-3-642-33715-4_48
    145 rdf:type schema:CreativeWork
    146 sg:pub.10.1007/978-3-642-33786-4_32 schema:sameAs https://app.dimensions.ai/details/publication/pub.1002182143
    147 https://doi.org/10.1007/978-3-642-33786-4_32
    148 rdf:type schema:CreativeWork
    149 sg:pub.10.1007/978-3-642-33885-4_27 schema:sameAs https://app.dimensions.ai/details/publication/pub.1045347976
    150 https://doi.org/10.1007/978-3-642-33885-4_27
    151 rdf:type schema:CreativeWork
    152 sg:pub.10.1007/978-3-642-33885-4_46 schema:sameAs https://app.dimensions.ai/details/publication/pub.1003958533
    153 https://doi.org/10.1007/978-3-642-33885-4_46
    154 rdf:type schema:CreativeWork
    155 sg:pub.10.1007/s11263-007-0090-8 schema:sameAs https://app.dimensions.ai/details/publication/pub.1027534025
    156 https://doi.org/10.1007/s11263-007-0090-8
    157 rdf:type schema:CreativeWork
    158 sg:pub.10.1007/s11263-009-0275-4 schema:sameAs https://app.dimensions.ai/details/publication/pub.1014796149
    159 https://doi.org/10.1007/s11263-009-0275-4
    160 rdf:type schema:CreativeWork
    161 sg:pub.10.1007/s11263-011-0495-2 schema:sameAs https://app.dimensions.ai/details/publication/pub.1031328704
    162 https://doi.org/10.1007/s11263-011-0495-2
    163 rdf:type schema:CreativeWork
    164 sg:pub.10.1007/s11263-013-0620-5 schema:sameAs https://app.dimensions.ai/details/publication/pub.1033900312
    165 https://doi.org/10.1007/s11263-013-0620-5
    166 rdf:type schema:CreativeWork
    167 sg:pub.10.1023/b:visi.0000013087.49260.fb schema:sameAs https://app.dimensions.ai/details/publication/pub.1001944608
    168 https://doi.org/10.1023/b:visi.0000013087.49260.fb
    169 rdf:type schema:CreativeWork
    170 sg:pub.10.1023/b:visi.0000029664.99615.94 schema:sameAs https://app.dimensions.ai/details/publication/pub.1052687286
    171 https://doi.org/10.1023/b:visi.0000029664.99615.94
    172 rdf:type schema:CreativeWork
    173 https://doi.org/10.1016/j.asoc.2012.12.013 schema:sameAs https://app.dimensions.ai/details/publication/pub.1038782578
    174 rdf:type schema:CreativeWork
    175 https://doi.org/10.1093/biomet/91.3.743 schema:sameAs https://app.dimensions.ai/details/publication/pub.1059421368
    176 rdf:type schema:CreativeWork
    177 https://doi.org/10.1109/cvpr.2005.177 schema:sameAs https://app.dimensions.ai/details/publication/pub.1093997066
    178 rdf:type schema:CreativeWork
    179 https://doi.org/10.1109/cvpr.2006.68 schema:sameAs https://app.dimensions.ai/details/publication/pub.1094512911
    180 rdf:type schema:CreativeWork
    181 https://doi.org/10.1109/cvpr.2009.5206757 schema:sameAs https://app.dimensions.ai/details/publication/pub.1095180230
    182 rdf:type schema:CreativeWork
    183 https://doi.org/10.1109/cvpr.2009.5206772 schema:sameAs https://app.dimensions.ai/details/publication/pub.1094211775
    184 rdf:type schema:CreativeWork
    185 https://doi.org/10.1109/cvpr.2010.5540096 schema:sameAs https://app.dimensions.ai/details/publication/pub.1095033380
    186 rdf:type schema:CreativeWork
    187 https://doi.org/10.1109/cvpr.2010.5540226 schema:sameAs https://app.dimensions.ai/details/publication/pub.1094653828
    188 rdf:type schema:CreativeWork
    189 https://doi.org/10.1109/cvpr.2011.5995330 schema:sameAs https://app.dimensions.ai/details/publication/pub.1093409457
    190 rdf:type schema:CreativeWork
    191 https://doi.org/10.1109/cvpr.2011.5995347 schema:sameAs https://app.dimensions.ai/details/publication/pub.1094016389
    192 rdf:type schema:CreativeWork
    193 https://doi.org/10.1109/cvpr.2011.5995368 schema:sameAs https://app.dimensions.ai/details/publication/pub.1094538483
    194 rdf:type schema:CreativeWork
    195 https://doi.org/10.1109/cvpr.2013.424 schema:sameAs https://app.dimensions.ai/details/publication/pub.1094917736
    196 rdf:type schema:CreativeWork
    197 https://doi.org/10.1109/cvpr.2013.428 schema:sameAs https://app.dimensions.ai/details/publication/pub.1094300906
    198 rdf:type schema:CreativeWork
    199 https://doi.org/10.1109/cvpr.2014.222 schema:sameAs https://app.dimensions.ai/details/publication/pub.1094012327
    200 rdf:type schema:CreativeWork
    201 https://doi.org/10.1109/cvpr.2014.81 schema:sameAs https://app.dimensions.ai/details/publication/pub.1094727707
    202 rdf:type schema:CreativeWork
    203 https://doi.org/10.1109/iccv.2009.5459183 schema:sameAs https://app.dimensions.ai/details/publication/pub.1095015498
    204 rdf:type schema:CreativeWork
    205 https://doi.org/10.1109/iccv.2011.6126486 schema:sameAs https://app.dimensions.ai/details/publication/pub.1093330382
    206 rdf:type schema:CreativeWork
    207 https://doi.org/10.1109/tpami.2009.167 schema:sameAs https://app.dimensions.ai/details/publication/pub.1061743745
    208 rdf:type schema:CreativeWork
    209 https://doi.org/10.1109/tpami.2012.204 schema:sameAs https://app.dimensions.ai/details/publication/pub.1035667356
    210 rdf:type schema:CreativeWork
    211 https://doi.org/10.1145/1961189.1961199 schema:sameAs https://app.dimensions.ai/details/publication/pub.1013637525
    212 rdf:type schema:CreativeWork
    213 https://doi.org/10.5244/c.26.82 schema:sameAs https://app.dimensions.ai/details/publication/pub.1099383252
    214 rdf:type schema:CreativeWork
    215 https://www.grid.ac/institutes/grid.24488.32 schema:alternateName Microsoft Research (United Kingdom)
    216 schema:name Microsoft Research, Cambridge, UK
    217 rdf:type schema:Organization
    218 https://www.grid.ac/institutes/grid.4305.2 schema:alternateName University of Edinburgh
    219 schema:name University of Edinburgh, Edinburgh, UK
    220 rdf:type schema:Organization
    221 https://www.grid.ac/institutes/grid.4991.5 schema:alternateName University of Oxford
    222 schema:name University of Oxford, Oxford, UK
    223 rdf:type schema:Organization
    224 https://www.grid.ac/institutes/grid.5801.c schema:alternateName Swiss Federal Institute of Technology in Zurich
    225 schema:name ETH, Zurich, Switzerland
    226 KU Leuven, Leuven, Belgium
    227 rdf:type schema:Organization
    228 https://www.grid.ac/institutes/grid.9909.9 schema:alternateName University of Leeds
    229 schema:name University of Leeds, Leeds, UK
    230 rdf:type schema:Organization
     




    Preview window. Press ESC to close (or click here)


    ...