Playing for Data: Ground Truth from Computer Games View Full Text


Ontology type: schema:Chapter      Open Access: True


Chapter Info

DATE

2016-09-17

AUTHORS

Stephan R. Richter , Vibhav Vineet , Stefan Roth , Vladlen Koltun

ABSTRACT

Recent progress in computer vision has been driven by high-capacity models trained on large datasets. Unfortunately, creating large datasets with pixel-level labels has been extremely costly due to the amount of human effort required. In this paper, we present an approach to rapidly creating pixel-accurate semantic label maps for images extracted from modern computer games. Although the source code and the internal operation of commercial games are inaccessible, we show that associations between image patches can be reconstructed from the communication between the game and the graphics hardware. This enables rapid propagation of semantic labels within and across images synthesized by the game, with no access to the source code or the content. We validate the presented approach by producing dense pixel-level semantic annotations for 25 thousand images synthesized by a photorealistic open-world computer game. Experiments on semantic segmentation datasets show that using the acquired data to supplement real-world images significantly increases accuracy and that the acquired data enables reducing the amount of hand-labeled real-world data: models trained with game data and just 13\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tfrac{1}{3}$$\end{document} of the CamVid training set outperform models trained on the complete CamVid training set. More... »

PAGES

102-118

Book

TITLE

Computer Vision – ECCV 2016

ISBN

978-3-319-46474-9
978-3-319-46475-6

Identifiers

URI

http://scigraph.springernature.com/pub.10.1007/978-3-319-46475-6_7

DOI

http://dx.doi.org/10.1007/978-3-319-46475-6_7

DIMENSIONS

https://app.dimensions.ai/details/publication/pub.1025415319


Indexing Status Check whether this publication has been indexed by Scopus and Web Of Science using the SN Indexing Status Tool
Incoming Citations Browse incoming citations for this publication using opencitations.net

JSON-LD is the canonical representation for SciGraph data.

TIP: You can open this SciGraph record using an external JSON-LD service: JSON-LD Playground Google SDTT

[
  {
    "@context": "https://springernature.github.io/scigraph/jsonld/sgcontext.json", 
    "about": [
      {
        "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/08", 
        "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
        "name": "Information and Computing Sciences", 
        "type": "DefinedTerm"
      }, 
      {
        "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/0801", 
        "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
        "name": "Artificial Intelligence and Image Processing", 
        "type": "DefinedTerm"
      }, 
      {
        "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/0806", 
        "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
        "name": "Information Systems", 
        "type": "DefinedTerm"
      }
    ], 
    "author": [
      {
        "affiliation": {
          "alternateName": "TU Darmstadt, Darmstadt, Germany", 
          "id": "http://www.grid.ac/institutes/grid.6546.1", 
          "name": [
            "TU Darmstadt, Darmstadt, Germany"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Richter", 
        "givenName": "Stephan R.", 
        "id": "sg:person.015470044641.22", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.015470044641.22"
        ], 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "Intel Labs, Santa Clara, USA", 
          "id": "http://www.grid.ac/institutes/grid.419318.6", 
          "name": [
            "Intel Labs, Santa Clara, USA"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Vineet", 
        "givenName": "Vibhav", 
        "id": "sg:person.015423043165.65", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.015423043165.65"
        ], 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "TU Darmstadt, Darmstadt, Germany", 
          "id": "http://www.grid.ac/institutes/grid.6546.1", 
          "name": [
            "TU Darmstadt, Darmstadt, Germany"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Roth", 
        "givenName": "Stefan", 
        "id": "sg:person.011116631605.20", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.011116631605.20"
        ], 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "Intel Labs, Santa Clara, USA", 
          "id": "http://www.grid.ac/institutes/grid.419318.6", 
          "name": [
            "Intel Labs, Santa Clara, USA"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Koltun", 
        "givenName": "Vladlen", 
        "id": "sg:person.015565367417.08", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.015565367417.08"
        ], 
        "type": "Person"
      }
    ], 
    "datePublished": "2016-09-17", 
    "datePublishedReg": "2016-09-17", 
    "description": "Recent progress in computer vision has been driven by high-capacity models trained on large datasets. Unfortunately, creating large datasets with pixel-level labels has been extremely costly due to the amount of human effort required. In this paper, we present an approach to rapidly creating pixel-accurate semantic label maps for images extracted from modern computer games. Although the source code and the internal operation of commercial games are inaccessible, we show that associations between image patches can be reconstructed from the communication between the game and the graphics hardware. This enables rapid propagation of semantic labels within and across images synthesized by the game, with no access to the source code or the content. We validate the presented approach by producing dense pixel-level semantic annotations for 25 thousand images synthesized by a photorealistic open-world computer game. Experiments on semantic segmentation datasets show that using the acquired data to supplement real-world images significantly increases accuracy and that the acquired data enables reducing the amount of hand-labeled real-world data: models trained with game data and just 13\\documentclass[12pt]{minimal}\n\t\t\t\t\\usepackage{amsmath}\n\t\t\t\t\\usepackage{wasysym}\n\t\t\t\t\\usepackage{amsfonts}\n\t\t\t\t\\usepackage{amssymb}\n\t\t\t\t\\usepackage{amsbsy}\n\t\t\t\t\\usepackage{mathrsfs}\n\t\t\t\t\\usepackage{upgreek}\n\t\t\t\t\\setlength{\\oddsidemargin}{-69pt}\n\t\t\t\t\\begin{document}$$\\tfrac{1}{3}$$\\end{document} of the CamVid training set outperform models trained on the complete CamVid training set.", 
    "editor": [
      {
        "familyName": "Leibe", 
        "givenName": "Bastian", 
        "type": "Person"
      }, 
      {
        "familyName": "Matas", 
        "givenName": "Jiri", 
        "type": "Person"
      }, 
      {
        "familyName": "Sebe", 
        "givenName": "Nicu", 
        "type": "Person"
      }, 
      {
        "familyName": "Welling", 
        "givenName": "Max", 
        "type": "Person"
      }
    ], 
    "genre": "chapter", 
    "id": "sg:pub.10.1007/978-3-319-46475-6_7", 
    "isAccessibleForFree": true, 
    "isPartOf": {
      "isbn": [
        "978-3-319-46474-9", 
        "978-3-319-46475-6"
      ], 
      "name": "Computer Vision \u2013 ECCV 2016", 
      "type": "Book"
    }, 
    "keywords": [
      "computer games", 
      "source code", 
      "large datasets", 
      "pixel-level labels", 
      "semantic segmentation datasets", 
      "modern computer games", 
      "real-world images", 
      "semantic label map", 
      "high-capacity models", 
      "real-world data", 
      "computer vision", 
      "semantic annotation", 
      "semantic labels", 
      "graphics hardware", 
      "segmentation dataset", 
      "human effort", 
      "image patches", 
      "game data", 
      "label maps", 
      "commercial games", 
      "ground truth", 
      "training set", 
      "data enables", 
      "outperform models", 
      "dataset", 
      "internal operations", 
      "images", 
      "game", 
      "code", 
      "labels", 
      "hardware", 
      "annotation", 
      "vision", 
      "communication", 
      "accuracy", 
      "data", 
      "model", 
      "set", 
      "access", 
      "enables", 
      "maps", 
      "operation", 
      "training", 
      "truth", 
      "amount", 
      "experiments", 
      "efforts", 
      "rapid propagation", 
      "progress", 
      "patches", 
      "recent progress", 
      "propagation", 
      "content", 
      "approach", 
      "association", 
      "paper"
    ], 
    "name": "Playing for Data: Ground Truth from Computer Games", 
    "pagination": "102-118", 
    "productId": [
      {
        "name": "dimensions_id", 
        "type": "PropertyValue", 
        "value": [
          "pub.1025415319"
        ]
      }, 
      {
        "name": "doi", 
        "type": "PropertyValue", 
        "value": [
          "10.1007/978-3-319-46475-6_7"
        ]
      }
    ], 
    "publisher": {
      "name": "Springer Nature", 
      "type": "Organisation"
    }, 
    "sameAs": [
      "https://doi.org/10.1007/978-3-319-46475-6_7", 
      "https://app.dimensions.ai/details/publication/pub.1025415319"
    ], 
    "sdDataset": "chapters", 
    "sdDatePublished": "2022-12-01T06:50", 
    "sdLicense": "https://scigraph.springernature.com/explorer/license/", 
    "sdPublisher": {
      "name": "Springer Nature - SN SciGraph project", 
      "type": "Organization"
    }, 
    "sdSource": "s3://com-springernature-scigraph/baseset/20221201/entities/gbq_results/chapter/chapter_307.jsonl", 
    "type": "Chapter", 
    "url": "https://doi.org/10.1007/978-3-319-46475-6_7"
  }
]
 

Download the RDF metadata as:  json-ld nt turtle xml License info

HOW TO GET THIS DATA PROGRAMMATICALLY:

JSON-LD is a popular format for linked data which is fully compatible with JSON.

curl -H 'Accept: application/ld+json' 'https://scigraph.springernature.com/pub.10.1007/978-3-319-46475-6_7'

N-Triples is a line-based linked data format ideal for batch operations.

curl -H 'Accept: application/n-triples' 'https://scigraph.springernature.com/pub.10.1007/978-3-319-46475-6_7'

Turtle is a human-readable linked data format.

curl -H 'Accept: text/turtle' 'https://scigraph.springernature.com/pub.10.1007/978-3-319-46475-6_7'

RDF/XML is a standard XML format for linked data.

curl -H 'Accept: application/rdf+xml' 'https://scigraph.springernature.com/pub.10.1007/978-3-319-46475-6_7'


 

This table displays all metadata directly associated to this object as RDF triples.

158 TRIPLES      22 PREDICATES      81 URIs      73 LITERALS      7 BLANK NODES

Subject Predicate Object
1 sg:pub.10.1007/978-3-319-46475-6_7 schema:about anzsrc-for:08
2 anzsrc-for:0801
3 anzsrc-for:0806
4 schema:author N339b7a879eed43d89271020dce91c3aa
5 schema:datePublished 2016-09-17
6 schema:datePublishedReg 2016-09-17
7 schema:description Recent progress in computer vision has been driven by high-capacity models trained on large datasets. Unfortunately, creating large datasets with pixel-level labels has been extremely costly due to the amount of human effort required. In this paper, we present an approach to rapidly creating pixel-accurate semantic label maps for images extracted from modern computer games. Although the source code and the internal operation of commercial games are inaccessible, we show that associations between image patches can be reconstructed from the communication between the game and the graphics hardware. This enables rapid propagation of semantic labels within and across images synthesized by the game, with no access to the source code or the content. We validate the presented approach by producing dense pixel-level semantic annotations for 25 thousand images synthesized by a photorealistic open-world computer game. Experiments on semantic segmentation datasets show that using the acquired data to supplement real-world images significantly increases accuracy and that the acquired data enables reducing the amount of hand-labeled real-world data: models trained with game data and just 13\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tfrac{1}{3}$$\end{document} of the CamVid training set outperform models trained on the complete CamVid training set.
8 schema:editor Nffc67e5e795b418b9241eae1f1a1d7d8
9 schema:genre chapter
10 schema:isAccessibleForFree true
11 schema:isPartOf Nd8c5bc76518e48e3b9960b9f8dc2b8a6
12 schema:keywords access
13 accuracy
14 amount
15 annotation
16 approach
17 association
18 code
19 commercial games
20 communication
21 computer games
22 computer vision
23 content
24 data
25 data enables
26 dataset
27 efforts
28 enables
29 experiments
30 game
31 game data
32 graphics hardware
33 ground truth
34 hardware
35 high-capacity models
36 human effort
37 image patches
38 images
39 internal operations
40 label maps
41 labels
42 large datasets
43 maps
44 model
45 modern computer games
46 operation
47 outperform models
48 paper
49 patches
50 pixel-level labels
51 progress
52 propagation
53 rapid propagation
54 real-world data
55 real-world images
56 recent progress
57 segmentation dataset
58 semantic annotation
59 semantic label map
60 semantic labels
61 semantic segmentation datasets
62 set
63 source code
64 training
65 training set
66 truth
67 vision
68 schema:name Playing for Data: Ground Truth from Computer Games
69 schema:pagination 102-118
70 schema:productId N7c1a1716610b47e98a3ea3b69ef057be
71 Nae4a7bee085b4918ad00cdb50237afe5
72 schema:publisher N96d3af66584342619c676f522c599dcb
73 schema:sameAs https://app.dimensions.ai/details/publication/pub.1025415319
74 https://doi.org/10.1007/978-3-319-46475-6_7
75 schema:sdDatePublished 2022-12-01T06:50
76 schema:sdLicense https://scigraph.springernature.com/explorer/license/
77 schema:sdPublisher N2804059e8b1742c88331a05b45d96bb5
78 schema:url https://doi.org/10.1007/978-3-319-46475-6_7
79 sgo:license sg:explorer/license/
80 sgo:sdDataset chapters
81 rdf:type schema:Chapter
82 N2804059e8b1742c88331a05b45d96bb5 schema:name Springer Nature - SN SciGraph project
83 rdf:type schema:Organization
84 N2afb13a464ac449fa52f03714b7cd751 schema:familyName Welling
85 schema:givenName Max
86 rdf:type schema:Person
87 N339b7a879eed43d89271020dce91c3aa rdf:first sg:person.015470044641.22
88 rdf:rest Nb89f84e7159547418e6037c1a5958e3c
89 N3e91127147774683837fd5bb8d53f844 schema:familyName Matas
90 schema:givenName Jiri
91 rdf:type schema:Person
92 N4939636c5a6e4cd1b3203f2a23f5a07b rdf:first N2afb13a464ac449fa52f03714b7cd751
93 rdf:rest rdf:nil
94 N53bf433c62944de0893b954a8abc8019 rdf:first sg:person.015565367417.08
95 rdf:rest rdf:nil
96 N717c1eb3f84f4da596c7e4f1ecee8d6b rdf:first sg:person.011116631605.20
97 rdf:rest N53bf433c62944de0893b954a8abc8019
98 N7c1a1716610b47e98a3ea3b69ef057be schema:name dimensions_id
99 schema:value pub.1025415319
100 rdf:type schema:PropertyValue
101 N8f28113941614f39a96ad47856123ea9 rdf:first N9c5cbc1a4b8a4fc0a805245389a8d335
102 rdf:rest N4939636c5a6e4cd1b3203f2a23f5a07b
103 N96d3af66584342619c676f522c599dcb schema:name Springer Nature
104 rdf:type schema:Organisation
105 N9c5cbc1a4b8a4fc0a805245389a8d335 schema:familyName Sebe
106 schema:givenName Nicu
107 rdf:type schema:Person
108 Nae4a7bee085b4918ad00cdb50237afe5 schema:name doi
109 schema:value 10.1007/978-3-319-46475-6_7
110 rdf:type schema:PropertyValue
111 Nb89f84e7159547418e6037c1a5958e3c rdf:first sg:person.015423043165.65
112 rdf:rest N717c1eb3f84f4da596c7e4f1ecee8d6b
113 Nc751c1288539460ab4fd495a9d80483d schema:familyName Leibe
114 schema:givenName Bastian
115 rdf:type schema:Person
116 Nd8c5bc76518e48e3b9960b9f8dc2b8a6 schema:isbn 978-3-319-46474-9
117 978-3-319-46475-6
118 schema:name Computer Vision – ECCV 2016
119 rdf:type schema:Book
120 Nfea6a56816cc470fb4bf04dce532bdcf rdf:first N3e91127147774683837fd5bb8d53f844
121 rdf:rest N8f28113941614f39a96ad47856123ea9
122 Nffc67e5e795b418b9241eae1f1a1d7d8 rdf:first Nc751c1288539460ab4fd495a9d80483d
123 rdf:rest Nfea6a56816cc470fb4bf04dce532bdcf
124 anzsrc-for:08 schema:inDefinedTermSet anzsrc-for:
125 schema:name Information and Computing Sciences
126 rdf:type schema:DefinedTerm
127 anzsrc-for:0801 schema:inDefinedTermSet anzsrc-for:
128 schema:name Artificial Intelligence and Image Processing
129 rdf:type schema:DefinedTerm
130 anzsrc-for:0806 schema:inDefinedTermSet anzsrc-for:
131 schema:name Information Systems
132 rdf:type schema:DefinedTerm
133 sg:person.011116631605.20 schema:affiliation grid-institutes:grid.6546.1
134 schema:familyName Roth
135 schema:givenName Stefan
136 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.011116631605.20
137 rdf:type schema:Person
138 sg:person.015423043165.65 schema:affiliation grid-institutes:grid.419318.6
139 schema:familyName Vineet
140 schema:givenName Vibhav
141 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.015423043165.65
142 rdf:type schema:Person
143 sg:person.015470044641.22 schema:affiliation grid-institutes:grid.6546.1
144 schema:familyName Richter
145 schema:givenName Stephan R.
146 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.015470044641.22
147 rdf:type schema:Person
148 sg:person.015565367417.08 schema:affiliation grid-institutes:grid.419318.6
149 schema:familyName Koltun
150 schema:givenName Vladlen
151 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.015565367417.08
152 rdf:type schema:Person
153 grid-institutes:grid.419318.6 schema:alternateName Intel Labs, Santa Clara, USA
154 schema:name Intel Labs, Santa Clara, USA
155 rdf:type schema:Organization
156 grid-institutes:grid.6546.1 schema:alternateName TU Darmstadt, Darmstadt, Germany
157 schema:name TU Darmstadt, Darmstadt, Germany
158 rdf:type schema:Organization
 




Preview window. Press ESC to close (or click here)


...