Answering Visual What-If Questions: From Actions to Predicted Scene Descriptions View Full Text


Ontology type: schema:Chapter      Open Access: True


Chapter Info

DATE

2019-01-23

AUTHORS

Misha Wagner , Hector Basevi , Rakshith Shetty , Wenbin Li , Mateusz Malinowski , Mario Fritz , Aleš Leonardis

ABSTRACT

In-depth scene descriptions and question answering tasks have greatly increased the scope of today’s definition of scene understanding. While such tasks are in principle open ended, current formulations primarily focus on describing only the current state of the scenes under consideration. In contrast, in this paper, we focus on the future states of the scenes which are also conditioned on actions. We posit this as a question answering task, where an answer has to be given about a future scene state, given observations of the current scene, and a question that includes a hypothetical action. Our solution is a hybrid model which integrates a physics engine into a question answering architecture in order to anticipate future scene states resulting from object-object interactions caused by an action. We demonstrate first results on this challenging new problem and compare to baselines, where we outperform fully data-driven end-to-end learning approaches. More... »

PAGES

521-537

Identifiers

URI

http://scigraph.springernature.com/pub.10.1007/978-3-030-11009-3_32

DOI

http://dx.doi.org/10.1007/978-3-030-11009-3_32

DIMENSIONS

https://app.dimensions.ai/details/publication/pub.1111703329


Indexing Status Check whether this publication has been indexed by Scopus and Web Of Science using the SN Indexing Status Tool
Incoming Citations Browse incoming citations for this publication using opencitations.net

JSON-LD is the canonical representation for SciGraph data.

TIP: You can open this SciGraph record using an external JSON-LD service: JSON-LD Playground Google SDTT

[
  {
    "@context": "https://springernature.github.io/scigraph/jsonld/sgcontext.json", 
    "about": [
      {
        "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/17", 
        "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
        "name": "Psychology and Cognitive Sciences", 
        "type": "DefinedTerm"
      }, 
      {
        "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/1701", 
        "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
        "name": "Psychology", 
        "type": "DefinedTerm"
      }
    ], 
    "author": [
      {
        "affiliation": {
          "alternateName": "University of Birmingham, Birmingham, UK", 
          "id": "http://www.grid.ac/institutes/grid.6572.6", 
          "name": [
            "University of Birmingham, Birmingham, UK"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Wagner", 
        "givenName": "Misha", 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "University of Birmingham, Birmingham, UK", 
          "id": "http://www.grid.ac/institutes/grid.6572.6", 
          "name": [
            "University of Birmingham, Birmingham, UK"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Basevi", 
        "givenName": "Hector", 
        "id": "sg:person.0676377155.31", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.0676377155.31"
        ], 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "Max Planck Institute for Informatics, Saarland Informatics Campus, Saarbr\u00fccken, Germany", 
          "id": "http://www.grid.ac/institutes/grid.419528.3", 
          "name": [
            "Max Planck Institute for Informatics, Saarland Informatics Campus, Saarbr\u00fccken, Germany"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Shetty", 
        "givenName": "Rakshith", 
        "id": "sg:person.012536402327.77", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.012536402327.77"
        ], 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "Max Planck Institute for Informatics, Saarland Informatics Campus, Saarbr\u00fccken, Germany", 
          "id": "http://www.grid.ac/institutes/grid.419528.3", 
          "name": [
            "Max Planck Institute for Informatics, Saarland Informatics Campus, Saarbr\u00fccken, Germany"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Li", 
        "givenName": "Wenbin", 
        "id": "sg:person.011262202211.32", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.011262202211.32"
        ], 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "Max Planck Institute for Informatics, Saarland Informatics Campus, Saarbr\u00fccken, Germany", 
          "id": "http://www.grid.ac/institutes/grid.419528.3", 
          "name": [
            "Max Planck Institute for Informatics, Saarland Informatics Campus, Saarbr\u00fccken, Germany"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Malinowski", 
        "givenName": "Mateusz", 
        "id": "sg:person.07716544521.15", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.07716544521.15"
        ], 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "CISPA Helmholtz Center i.G., Saarland Informatics Campus, Saarbr\u00fccken, Germany", 
          "id": "http://www.grid.ac/institutes/grid.507511.7", 
          "name": [
            "CISPA Helmholtz Center i.G., Saarland Informatics Campus, Saarbr\u00fccken, Germany"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Fritz", 
        "givenName": "Mario", 
        "id": "sg:person.013361072755.17", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.013361072755.17"
        ], 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "University of Birmingham, Birmingham, UK", 
          "id": "http://www.grid.ac/institutes/grid.6572.6", 
          "name": [
            "University of Birmingham, Birmingham, UK"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Leonardis", 
        "givenName": "Ale\u0161", 
        "id": "sg:person.012526151717.64", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.012526151717.64"
        ], 
        "type": "Person"
      }
    ], 
    "datePublished": "2019-01-23", 
    "datePublishedReg": "2019-01-23", 
    "description": "In-depth scene descriptions and question answering tasks have greatly increased the scope of today\u2019s definition of scene understanding. While such tasks are in principle open ended, current formulations primarily focus on describing only the current state of the scenes under consideration. In contrast, in this paper, we focus on the future states of the scenes which are also conditioned on actions. We posit this as a question answering task, where an answer has to be given about a future scene state, given observations of the current scene, and a question that includes a hypothetical action. Our solution is a hybrid model which integrates a physics engine into a question answering architecture in order to anticipate future scene states resulting from object-object interactions caused by an action. We demonstrate first results on this challenging new problem and compare to baselines, where we outperform fully data-driven end-to-end learning approaches.", 
    "editor": [
      {
        "familyName": "Leal-Taix\u00e9", 
        "givenName": "Laura", 
        "type": "Person"
      }, 
      {
        "familyName": "Roth", 
        "givenName": "Stefan", 
        "type": "Person"
      }
    ], 
    "genre": "chapter", 
    "id": "sg:pub.10.1007/978-3-030-11009-3_32", 
    "isAccessibleForFree": true, 
    "isPartOf": {
      "isbn": [
        "978-3-030-11008-6", 
        "978-3-030-11009-3"
      ], 
      "name": "Computer Vision \u2013 ECCV 2018 Workshops", 
      "type": "Book"
    }, 
    "keywords": [
      "scene description", 
      "scene state", 
      "data-driven end", 
      "end learning approach", 
      "object-object interactions", 
      "Question Answering task", 
      "scene understanding", 
      "physics engine", 
      "challenging new problem", 
      "learning approach", 
      "such tasks", 
      "current scene", 
      "scene", 
      "task", 
      "hybrid model", 
      "new problems", 
      "future state", 
      "current state", 
      "hypothetical action", 
      "architecture", 
      "engine", 
      "first results", 
      "definition", 
      "description", 
      "state", 
      "solution", 
      "scope", 
      "answers", 
      "model", 
      "order", 
      "questions", 
      "end", 
      "action", 
      "consideration", 
      "results", 
      "formulation", 
      "interaction", 
      "understanding", 
      "current formulation", 
      "baseline", 
      "observations", 
      "today\u2019s definition", 
      "contrast", 
      "paper", 
      "approach", 
      "problem"
    ], 
    "name": "Answering Visual What-If Questions: From Actions to Predicted Scene Descriptions", 
    "pagination": "521-537", 
    "productId": [
      {
        "name": "dimensions_id", 
        "type": "PropertyValue", 
        "value": [
          "pub.1111703329"
        ]
      }, 
      {
        "name": "doi", 
        "type": "PropertyValue", 
        "value": [
          "10.1007/978-3-030-11009-3_32"
        ]
      }
    ], 
    "publisher": {
      "name": "Springer Nature", 
      "type": "Organisation"
    }, 
    "sameAs": [
      "https://doi.org/10.1007/978-3-030-11009-3_32", 
      "https://app.dimensions.ai/details/publication/pub.1111703329"
    ], 
    "sdDataset": "chapters", 
    "sdDatePublished": "2022-12-01T06:55", 
    "sdLicense": "https://scigraph.springernature.com/explorer/license/", 
    "sdPublisher": {
      "name": "Springer Nature - SN SciGraph project", 
      "type": "Organization"
    }, 
    "sdSource": "s3://com-springernature-scigraph/baseset/20221201/entities/gbq_results/chapter/chapter_71.jsonl", 
    "type": "Chapter", 
    "url": "https://doi.org/10.1007/978-3-030-11009-3_32"
  }
]
 

Download the RDF metadata as:  json-ld nt turtle xml License info

HOW TO GET THIS DATA PROGRAMMATICALLY:

JSON-LD is a popular format for linked data which is fully compatible with JSON.

curl -H 'Accept: application/ld+json' 'https://scigraph.springernature.com/pub.10.1007/978-3-030-11009-3_32'

N-Triples is a line-based linked data format ideal for batch operations.

curl -H 'Accept: application/n-triples' 'https://scigraph.springernature.com/pub.10.1007/978-3-030-11009-3_32'

Turtle is a human-readable linked data format.

curl -H 'Accept: text/turtle' 'https://scigraph.springernature.com/pub.10.1007/978-3-030-11009-3_32'

RDF/XML is a standard XML format for linked data.

curl -H 'Accept: application/rdf+xml' 'https://scigraph.springernature.com/pub.10.1007/978-3-030-11009-3_32'


 

This table displays all metadata directly associated to this object as RDF triples.

157 TRIPLES      22 PREDICATES      70 URIs      63 LITERALS      7 BLANK NODES

Subject Predicate Object
1 sg:pub.10.1007/978-3-030-11009-3_32 schema:about anzsrc-for:17
2 anzsrc-for:1701
3 schema:author N88d4c85e0b1a4178a31d3ab7a5e703b7
4 schema:datePublished 2019-01-23
5 schema:datePublishedReg 2019-01-23
6 schema:description In-depth scene descriptions and question answering tasks have greatly increased the scope of today’s definition of scene understanding. While such tasks are in principle open ended, current formulations primarily focus on describing only the current state of the scenes under consideration. In contrast, in this paper, we focus on the future states of the scenes which are also conditioned on actions. We posit this as a question answering task, where an answer has to be given about a future scene state, given observations of the current scene, and a question that includes a hypothetical action. Our solution is a hybrid model which integrates a physics engine into a question answering architecture in order to anticipate future scene states resulting from object-object interactions caused by an action. We demonstrate first results on this challenging new problem and compare to baselines, where we outperform fully data-driven end-to-end learning approaches.
7 schema:editor Nd9fa46c49d5d47f383ed97a71b0bc26d
8 schema:genre chapter
9 schema:isAccessibleForFree true
10 schema:isPartOf Nc12f2c57d13a4ca9a6d9c8ee45b2c591
11 schema:keywords Question Answering task
12 action
13 answers
14 approach
15 architecture
16 baseline
17 challenging new problem
18 consideration
19 contrast
20 current formulation
21 current scene
22 current state
23 data-driven end
24 definition
25 description
26 end
27 end learning approach
28 engine
29 first results
30 formulation
31 future state
32 hybrid model
33 hypothetical action
34 interaction
35 learning approach
36 model
37 new problems
38 object-object interactions
39 observations
40 order
41 paper
42 physics engine
43 problem
44 questions
45 results
46 scene
47 scene description
48 scene state
49 scene understanding
50 scope
51 solution
52 state
53 such tasks
54 task
55 today’s definition
56 understanding
57 schema:name Answering Visual What-If Questions: From Actions to Predicted Scene Descriptions
58 schema:pagination 521-537
59 schema:productId N47b2257f592341bfb5f59d50dc875ecd
60 N66bc275284d14b4eab3cab97974905d2
61 schema:publisher Neca11e599f40451cb31e6ace1afa6f27
62 schema:sameAs https://app.dimensions.ai/details/publication/pub.1111703329
63 https://doi.org/10.1007/978-3-030-11009-3_32
64 schema:sdDatePublished 2022-12-01T06:55
65 schema:sdLicense https://scigraph.springernature.com/explorer/license/
66 schema:sdPublisher Ne9de82193ddf4bbd8c6cd2dc6af12c5b
67 schema:url https://doi.org/10.1007/978-3-030-11009-3_32
68 sgo:license sg:explorer/license/
69 sgo:sdDataset chapters
70 rdf:type schema:Chapter
71 N144a2a7931ad4074b90188d98e0692ae rdf:first sg:person.011262202211.32
72 rdf:rest N52c15a3db39c4a0b869fc685a3c8b1ea
73 N45f2a5dda044418ba6a82434de97052e schema:familyName Leal-Taixé
74 schema:givenName Laura
75 rdf:type schema:Person
76 N47b2257f592341bfb5f59d50dc875ecd schema:name dimensions_id
77 schema:value pub.1111703329
78 rdf:type schema:PropertyValue
79 N52c15a3db39c4a0b869fc685a3c8b1ea rdf:first sg:person.07716544521.15
80 rdf:rest Ne09344ac977b4095845a8aa0fc756d52
81 N6291fd4c1d354b50baa40976c5ed7c50 rdf:first sg:person.0676377155.31
82 rdf:rest Nc9e7c102289242fc90f5b0729138d0aa
83 N66bc275284d14b4eab3cab97974905d2 schema:name doi
84 schema:value 10.1007/978-3-030-11009-3_32
85 rdf:type schema:PropertyValue
86 N772ae347433c44f6904e3ce0e7fafd0c schema:affiliation grid-institutes:grid.6572.6
87 schema:familyName Wagner
88 schema:givenName Misha
89 rdf:type schema:Person
90 N7cbb791f525b4637bc1f32f3ce8bfc50 rdf:first N9df44a6f3ee041f49d09660f44c4e088
91 rdf:rest rdf:nil
92 N88d4c85e0b1a4178a31d3ab7a5e703b7 rdf:first N772ae347433c44f6904e3ce0e7fafd0c
93 rdf:rest N6291fd4c1d354b50baa40976c5ed7c50
94 N9df44a6f3ee041f49d09660f44c4e088 schema:familyName Roth
95 schema:givenName Stefan
96 rdf:type schema:Person
97 Nc12f2c57d13a4ca9a6d9c8ee45b2c591 schema:isbn 978-3-030-11008-6
98 978-3-030-11009-3
99 schema:name Computer Vision – ECCV 2018 Workshops
100 rdf:type schema:Book
101 Nc9e7c102289242fc90f5b0729138d0aa rdf:first sg:person.012536402327.77
102 rdf:rest N144a2a7931ad4074b90188d98e0692ae
103 Nd9fa46c49d5d47f383ed97a71b0bc26d rdf:first N45f2a5dda044418ba6a82434de97052e
104 rdf:rest N7cbb791f525b4637bc1f32f3ce8bfc50
105 Ne09344ac977b4095845a8aa0fc756d52 rdf:first sg:person.013361072755.17
106 rdf:rest Ne7380358ad7a4fa9b70bb1f5a2985ba2
107 Ne7380358ad7a4fa9b70bb1f5a2985ba2 rdf:first sg:person.012526151717.64
108 rdf:rest rdf:nil
109 Ne9de82193ddf4bbd8c6cd2dc6af12c5b schema:name Springer Nature - SN SciGraph project
110 rdf:type schema:Organization
111 Neca11e599f40451cb31e6ace1afa6f27 schema:name Springer Nature
112 rdf:type schema:Organisation
113 anzsrc-for:17 schema:inDefinedTermSet anzsrc-for:
114 schema:name Psychology and Cognitive Sciences
115 rdf:type schema:DefinedTerm
116 anzsrc-for:1701 schema:inDefinedTermSet anzsrc-for:
117 schema:name Psychology
118 rdf:type schema:DefinedTerm
119 sg:person.011262202211.32 schema:affiliation grid-institutes:grid.419528.3
120 schema:familyName Li
121 schema:givenName Wenbin
122 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.011262202211.32
123 rdf:type schema:Person
124 sg:person.012526151717.64 schema:affiliation grid-institutes:grid.6572.6
125 schema:familyName Leonardis
126 schema:givenName Aleš
127 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.012526151717.64
128 rdf:type schema:Person
129 sg:person.012536402327.77 schema:affiliation grid-institutes:grid.419528.3
130 schema:familyName Shetty
131 schema:givenName Rakshith
132 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.012536402327.77
133 rdf:type schema:Person
134 sg:person.013361072755.17 schema:affiliation grid-institutes:grid.507511.7
135 schema:familyName Fritz
136 schema:givenName Mario
137 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.013361072755.17
138 rdf:type schema:Person
139 sg:person.0676377155.31 schema:affiliation grid-institutes:grid.6572.6
140 schema:familyName Basevi
141 schema:givenName Hector
142 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.0676377155.31
143 rdf:type schema:Person
144 sg:person.07716544521.15 schema:affiliation grid-institutes:grid.419528.3
145 schema:familyName Malinowski
146 schema:givenName Mateusz
147 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.07716544521.15
148 rdf:type schema:Person
149 grid-institutes:grid.419528.3 schema:alternateName Max Planck Institute for Informatics, Saarland Informatics Campus, Saarbrücken, Germany
150 schema:name Max Planck Institute for Informatics, Saarland Informatics Campus, Saarbrücken, Germany
151 rdf:type schema:Organization
152 grid-institutes:grid.507511.7 schema:alternateName CISPA Helmholtz Center i.G., Saarland Informatics Campus, Saarbrücken, Germany
153 schema:name CISPA Helmholtz Center i.G., Saarland Informatics Campus, Saarbrücken, Germany
154 rdf:type schema:Organization
155 grid-institutes:grid.6572.6 schema:alternateName University of Birmingham, Birmingham, UK
156 schema:name University of Birmingham, Birmingham, UK
157 rdf:type schema:Organization
 




Preview window. Press ESC to close (or click here)


...