Configurable 3D Scene Synthesis and 2D Image Rendering with Per-pixel Ground Truth Using Stochastic Grammars View Full Text


Ontology type: schema:ScholarlyArticle      Open Access: True


Article Info

DATE

2018-06-30

AUTHORS

Chenfanfu Jiang, Siyuan Qi, Yixin Zhu, Siyuan Huang, Jenny Lin, Lap-Fai Yu, Demetri Terzopoulos, Song-Chun Zhu

ABSTRACT

We propose a systematic learning-based approach to the generation of massive quantities of synthetic 3D scenes and arbitrary numbers of photorealistic 2D images thereof, with associated ground truth information, for the purposes of training, benchmarking, and diagnosing learning-based computer vision and robotics algorithms. In particular, we devise a learning-based pipeline of algorithms capable of automatically generating and rendering a potentially infinite variety of indoor scenes by using a stochastic grammar, represented as an attributed Spatial And-Or Graph, in conjunction with state-of-the-art physics-based rendering. Our pipeline is capable of synthesizing scene layouts with high diversity, and it is configurable inasmuch as it enables the precise customization and control of important attributes of the generated scenes. It renders photorealistic RGB images of the generated scenes while automatically synthesizing detailed, per-pixel ground truth data, including visible surface depth and normal, object identity, and material information (detailed to object parts), as well as environments (e.g., illuminations and camera viewpoints). We demonstrate the value of our synthesized dataset, by improving performance in certain machine-learning-based scene understanding tasks—depth and surface normal prediction, semantic segmentation, reconstruction, etc.—and by providing benchmarks for and diagnostics of trained models by modifying object attributes and scene properties in a controllable manner. More... »

PAGES

920-941

References to SciGraph publications

  • 2015-11-03. FlowCap: 2D Human Pose from Optical Flow in PATTERN RECOGNITION
  • 2008. Sample Selection Bias Correction Theory in ALGORITHMIC LEARNING THEORY
  • 2016-09-17. Generative Image Modeling Using Style and Structure Adversarial Networks in COMPUTER VISION – ECCV 2016
  • 2012. Indoor Segmentation and Support Inference from RGBD Images in COMPUTER VISION – ECCV 2012
  • 2016. Multi-band Polarization Imaging and Applications in NONE
  • 2015-01-28. Scene Understanding by Reasoning Stability and Safety in INTERNATIONAL JOURNAL OF COMPUTER VISION
  • 2016-09-17. A Multi-scale CNN for Affordance Segmentation in RGB Images in COMPUTER VISION – ECCV 2016
  • 2014-06-25. The Pascal Visual Object Classes Challenge: A Retrospective in INTERNATIONAL JOURNAL OF COMPUTER VISION
  • 2016-09-17. Playing for Data: Ground Truth from Computer Games in COMPUTER VISION – ECCV 2016
  • 2014. Physically Grounded Spatio-temporal Object Affordances in COMPUTER VISION – ECCV 2014
  • 2014. Microsoft COCO: Common Objects in Context in COMPUTER VISION – ECCV 2014
  • 2017-09-13. From Virtual to Real World Visual Perception Using Domain Adaptation—The DPM as Example in DOMAIN ADAPTATION IN COMPUTER VISION APPLICATIONS
  • 2014. Video Text Detection in NONE
  • 2017. Domain Adaptation in Computer Vision Applications in NONE
  • 2014. Reasoning about Object Affordances in a Knowledge Base Representation in COMPUTER VISION – ECCV 2014
  • 2014-11-12. Indoor Scene Understanding with Geometric and Semantic Contexts in INTERNATIONAL JOURNAL OF COMPUTER VISION
  • 2015. Integrated Imaging and Vision Techniques for Industrial Inspection, Advances and Applications in NONE
  • 2014. Sliding Shapes for 3D Object Detection in Depth Images in COMPUTER VISION – ECCV 2014
  • 2016-11-24. How Useful Is Photo-Realistic Rendering for Visual Learning? in COMPUTER VISION – ECCV 2016 WORKSHOPS
  • 2014. OpenDR: An Approximate Differentiable Renderer in COMPUTER VISION – ECCV 2014
  • 2016-09-17. Marker-Less 3D Human Motion Capture with Monocular Image Sequence and Height-Maps in COMPUTER VISION – ECCV 2016
  • 2016-11-24. UnrealCV: Connecting Computer Vision to Unreal Engine in COMPUTER VISION – ECCV 2016 WORKSHOPS
  • Identifiers

    URI

    http://scigraph.springernature.com/pub.10.1007/s11263-018-1103-5

    DOI

    http://dx.doi.org/10.1007/s11263-018-1103-5

    DIMENSIONS

    https://app.dimensions.ai/details/publication/pub.1105210299


    Indexing Status Check whether this publication has been indexed by Scopus and Web Of Science using the SN Indexing Status Tool
    Incoming Citations Browse incoming citations for this publication using opencitations.net

    JSON-LD is the canonical representation for SciGraph data.

    TIP: You can open this SciGraph record using an external JSON-LD service: JSON-LD Playground Google SDTT

    [
      {
        "@context": "https://springernature.github.io/scigraph/jsonld/sgcontext.json", 
        "about": [
          {
            "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/08", 
            "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
            "name": "Information and Computing Sciences", 
            "type": "DefinedTerm"
          }, 
          {
            "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/0801", 
            "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
            "name": "Artificial Intelligence and Image Processing", 
            "type": "DefinedTerm"
          }
        ], 
        "author": [
          {
            "affiliation": {
              "alternateName": "SIG Center for Computer Graphics, University of Pennsylvania, Philadelphia, USA", 
              "id": "http://www.grid.ac/institutes/grid.25879.31", 
              "name": [
                "SIG Center for Computer Graphics, University of Pennsylvania, Philadelphia, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Jiang", 
            "givenName": "Chenfanfu", 
            "id": "sg:person.01323172313.41", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01323172313.41"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "UCLA Center for Vision, Cognition, Learning and Autonomy, University of California, Los Angeles, Los Angeles, USA", 
              "id": "http://www.grid.ac/institutes/grid.19006.3e", 
              "name": [
                "UCLA Center for Vision, Cognition, Learning and Autonomy, University of California, Los Angeles, Los Angeles, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Qi", 
            "givenName": "Siyuan", 
            "id": "sg:person.012512654354.22", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.012512654354.22"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "UCLA Center for Vision, Cognition, Learning and Autonomy, University of California, Los Angeles, Los Angeles, USA", 
              "id": "http://www.grid.ac/institutes/grid.19006.3e", 
              "name": [
                "UCLA Center for Vision, Cognition, Learning and Autonomy, University of California, Los Angeles, Los Angeles, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Zhu", 
            "givenName": "Yixin", 
            "id": "sg:person.07662331163.43", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.07662331163.43"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "UCLA Center for Vision, Cognition, Learning and Autonomy, University of California, Los Angeles, Los Angeles, USA", 
              "id": "http://www.grid.ac/institutes/grid.19006.3e", 
              "name": [
                "UCLA Center for Vision, Cognition, Learning and Autonomy, University of California, Los Angeles, Los Angeles, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Huang", 
            "givenName": "Siyuan", 
            "id": "sg:person.013200243570.22", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.013200243570.22"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "UCLA Center for Vision, Cognition, Learning and Autonomy, University of California, Los Angeles, Los Angeles, USA", 
              "id": "http://www.grid.ac/institutes/grid.19006.3e", 
              "name": [
                "UCLA Center for Vision, Cognition, Learning and Autonomy, University of California, Los Angeles, Los Angeles, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Lin", 
            "givenName": "Jenny", 
            "id": "sg:person.011203112510.68", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.011203112510.68"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "Graphics and Virtual Environments Laboratory, University of Massachusetts Boston, Boston, USA", 
              "id": "http://www.grid.ac/institutes/grid.266685.9", 
              "name": [
                "Graphics and Virtual Environments Laboratory, University of Massachusetts Boston, Boston, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Yu", 
            "givenName": "Lap-Fai", 
            "id": "sg:person.013767210740.42", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.013767210740.42"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "UCLA Computer Graphics and Vision Laboratory, University of California, Los Angeles, Los Angeles, USA", 
              "id": "http://www.grid.ac/institutes/grid.19006.3e", 
              "name": [
                "UCLA Computer Graphics and Vision Laboratory, University of California, Los Angeles, Los Angeles, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Terzopoulos", 
            "givenName": "Demetri", 
            "id": "sg:person.016347323445.35", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.016347323445.35"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "UCLA Center for Vision, Cognition, Learning and Autonomy, University of California, Los Angeles, Los Angeles, USA", 
              "id": "http://www.grid.ac/institutes/grid.19006.3e", 
              "name": [
                "UCLA Center for Vision, Cognition, Learning and Autonomy, University of California, Los Angeles, Los Angeles, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Zhu", 
            "givenName": "Song-Chun", 
            "id": "sg:person.012430434515.36", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.012430434515.36"
            ], 
            "type": "Person"
          }
        ], 
        "citation": [
          {
            "id": "sg:pub.10.1007/978-3-319-10599-4_41", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1046056033", 
              "https://doi.org/10.1007/978-3-319-10599-4_41"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-540-87987-9_8", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1012636020", 
              "https://doi.org/10.1007/978-3-540-87987-9_8"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-319-46475-6_7", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1025415319", 
              "https://doi.org/10.1007/978-3-319-46475-6_7"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-319-58347-1_13", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1091588219", 
              "https://doi.org/10.1007/978-3-319-58347-1_13"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/s11263-014-0779-4", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1009995058", 
              "https://doi.org/10.1007/s11263-014-0779-4"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-1-4471-6741-9", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1021729328", 
              "https://doi.org/10.1007/978-1-4471-6741-9"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-319-10578-9_54", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1013612828", 
              "https://doi.org/10.1007/978-3-319-10578-9_54"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/s11263-014-0795-4", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1051031938", 
              "https://doi.org/10.1007/s11263-014-0795-4"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-319-46493-0_12", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1045724022", 
              "https://doi.org/10.1007/978-3-319-46493-0_12"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-319-10602-1_48", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1045321436", 
              "https://doi.org/10.1007/978-3-319-10602-1_48"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-319-46493-0_2", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1049164757", 
              "https://doi.org/10.1007/978-3-319-46493-0_2"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-319-10584-0_11", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1017659317", 
              "https://doi.org/10.1007/978-3-319-10584-0_11"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-642-33715-4_54", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1053469442", 
              "https://doi.org/10.1007/978-3-642-33715-4_54"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-662-49373-1", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1052826178", 
              "https://doi.org/10.1007/978-3-662-49373-1"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-319-24947-6_34", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1000861733", 
              "https://doi.org/10.1007/978-3-319-24947-6_34"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/s11263-014-0733-5", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1017073734", 
              "https://doi.org/10.1007/s11263-014-0733-5"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-319-46493-0_20", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1034474844", 
              "https://doi.org/10.1007/978-3-319-46493-0_20"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-319-49409-8_18", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1090663472", 
              "https://doi.org/10.1007/978-3-319-49409-8_18"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-319-58347-1", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1091572224", 
              "https://doi.org/10.1007/978-3-319-58347-1"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-1-4471-6515-6", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1045307848", 
              "https://doi.org/10.1007/978-1-4471-6515-6"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-319-10605-2_27", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1018385677", 
              "https://doi.org/10.1007/978-3-319-10605-2_27"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-319-49409-8_75", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1086290585", 
              "https://doi.org/10.1007/978-3-319-49409-8_75"
            ], 
            "type": "CreativeWork"
          }
        ], 
        "datePublished": "2018-06-30", 
        "datePublishedReg": "2018-06-30", 
        "description": "We propose a systematic learning-based approach to the generation of massive quantities of synthetic 3D scenes and arbitrary numbers of photorealistic 2D images thereof, with associated ground truth information, for the purposes of training, benchmarking, and diagnosing learning-based computer vision and robotics algorithms. In particular, we devise a learning-based pipeline of algorithms capable of automatically generating and rendering a potentially infinite variety of indoor scenes by using a stochastic grammar, represented as an attributed Spatial And-Or Graph, in conjunction with state-of-the-art physics-based rendering. Our pipeline is capable of synthesizing scene layouts with high diversity, and it is configurable inasmuch as it enables the precise customization and control of important attributes of the generated scenes. It renders photorealistic RGB images of the generated scenes while automatically synthesizing detailed, per-pixel ground truth data, including visible surface depth and normal, object identity, and material information (detailed to object parts), as well as environments (e.g., illuminations and camera viewpoints). We demonstrate the value of our synthesized dataset, by improving performance in certain machine-learning-based scene understanding tasks\u2014depth and surface normal prediction, semantic segmentation, reconstruction, etc.\u2014and by providing benchmarks for and diagnostics of trained models by modifying object attributes and scene properties in a controllable manner.", 
        "genre": "article", 
        "id": "sg:pub.10.1007/s11263-018-1103-5", 
        "inLanguage": "en", 
        "isAccessibleForFree": true, 
        "isFundedItemOf": [
          {
            "id": "sg:grant.7132711", 
            "type": "MonetaryGrant"
          }
        ], 
        "isPartOf": [
          {
            "id": "sg:journal.1032807", 
            "issn": [
              "0920-5691", 
              "1573-1405"
            ], 
            "name": "International Journal of Computer Vision", 
            "publisher": "Springer Nature", 
            "type": "Periodical"
          }, 
          {
            "issueNumber": "9", 
            "type": "PublicationIssue"
          }, 
          {
            "type": "PublicationVolume", 
            "volumeNumber": "126"
          }
        ], 
        "keywords": [
          "associated ground truth information", 
          "learning-based approach", 
          "synthetic 3D scenes", 
          "surface normal prediction", 
          "stochastic grammars", 
          "ground truth information", 
          "ground truth data", 
          "computer vision", 
          "robotic algorithms", 
          "scene understanding", 
          "semantic segmentation", 
          "indoor scenes", 
          "scene synthesis", 
          "RGB images", 
          "scene layout", 
          "truth information", 
          "object attributes", 
          "ground truth", 
          "purpose of training", 
          "scene properties", 
          "truth data", 
          "normal prediction", 
          "scene", 
          "massive quantities", 
          "algorithm", 
          "precise customization", 
          "images", 
          "material information", 
          "arbitrary number", 
          "pipeline", 
          "important attributes", 
          "segmentation", 
          "information", 
          "rendering", 
          "customization", 
          "attributes", 
          "grammar", 
          "datasets", 
          "graph", 
          "benchmarks", 
          "vision", 
          "layout", 
          "environment", 
          "surface depth", 
          "Spatial", 
          "performance", 
          "infinite variety", 
          "reconstruction", 
          "truth", 
          "training", 
          "model", 
          "generation", 
          "data", 
          "prediction", 
          "number", 
          "variety", 
          "controllable manner", 
          "manner", 
          "purpose", 
          "control", 
          "state", 
          "conjunction", 
          "diagnostics", 
          "diversity", 
          "understanding", 
          "identity", 
          "quantity", 
          "high diversity", 
          "values", 
          "properties", 
          "depth", 
          "synthesis", 
          "approach", 
          "systematic learning-based approach", 
          "photorealistic 2D images", 
          "learning-based computer vision", 
          "learning-based pipeline", 
          "attributed Spatial", 
          "art physics-based rendering", 
          "physics-based rendering", 
          "photorealistic RGB images", 
          "pixel ground truth data", 
          "visible surface depth", 
          "Configurable 3D Scene Synthesis", 
          "pixel Ground Truth"
        ], 
        "name": "Configurable 3D Scene Synthesis and 2D Image Rendering with Per-pixel Ground Truth Using Stochastic Grammars", 
        "pagination": "920-941", 
        "productId": [
          {
            "name": "dimensions_id", 
            "type": "PropertyValue", 
            "value": [
              "pub.1105210299"
            ]
          }, 
          {
            "name": "doi", 
            "type": "PropertyValue", 
            "value": [
              "10.1007/s11263-018-1103-5"
            ]
          }
        ], 
        "sameAs": [
          "https://doi.org/10.1007/s11263-018-1103-5", 
          "https://app.dimensions.ai/details/publication/pub.1105210299"
        ], 
        "sdDataset": "articles", 
        "sdDatePublished": "2022-01-01T18:49", 
        "sdLicense": "https://scigraph.springernature.com/explorer/license/", 
        "sdPublisher": {
          "name": "Springer Nature - SN SciGraph project", 
          "type": "Organization"
        }, 
        "sdSource": "s3://com-springernature-scigraph/baseset/20220101/entities/gbq_results/article/article_777.jsonl", 
        "type": "ScholarlyArticle", 
        "url": "https://doi.org/10.1007/s11263-018-1103-5"
      }
    ]
     

    Download the RDF metadata as:  json-ld nt turtle xml License info

    HOW TO GET THIS DATA PROGRAMMATICALLY:

    JSON-LD is a popular format for linked data which is fully compatible with JSON.

    curl -H 'Accept: application/ld+json' 'https://scigraph.springernature.com/pub.10.1007/s11263-018-1103-5'

    N-Triples is a line-based linked data format ideal for batch operations.

    curl -H 'Accept: application/n-triples' 'https://scigraph.springernature.com/pub.10.1007/s11263-018-1103-5'

    Turtle is a human-readable linked data format.

    curl -H 'Accept: text/turtle' 'https://scigraph.springernature.com/pub.10.1007/s11263-018-1103-5'

    RDF/XML is a standard XML format for linked data.

    curl -H 'Accept: application/rdf+xml' 'https://scigraph.springernature.com/pub.10.1007/s11263-018-1103-5'


     

    This table displays all metadata directly associated to this object as RDF triples.

    290 TRIPLES      22 PREDICATES      132 URIs      102 LITERALS      6 BLANK NODES

    Subject Predicate Object
    1 sg:pub.10.1007/s11263-018-1103-5 schema:about anzsrc-for:08
    2 anzsrc-for:0801
    3 schema:author N880ab50d5bf44fe8b81ded0d16dbd492
    4 schema:citation sg:pub.10.1007/978-1-4471-6515-6
    5 sg:pub.10.1007/978-1-4471-6741-9
    6 sg:pub.10.1007/978-3-319-10578-9_54
    7 sg:pub.10.1007/978-3-319-10584-0_11
    8 sg:pub.10.1007/978-3-319-10599-4_41
    9 sg:pub.10.1007/978-3-319-10602-1_48
    10 sg:pub.10.1007/978-3-319-10605-2_27
    11 sg:pub.10.1007/978-3-319-24947-6_34
    12 sg:pub.10.1007/978-3-319-46475-6_7
    13 sg:pub.10.1007/978-3-319-46493-0_12
    14 sg:pub.10.1007/978-3-319-46493-0_2
    15 sg:pub.10.1007/978-3-319-46493-0_20
    16 sg:pub.10.1007/978-3-319-49409-8_18
    17 sg:pub.10.1007/978-3-319-49409-8_75
    18 sg:pub.10.1007/978-3-319-58347-1
    19 sg:pub.10.1007/978-3-319-58347-1_13
    20 sg:pub.10.1007/978-3-540-87987-9_8
    21 sg:pub.10.1007/978-3-642-33715-4_54
    22 sg:pub.10.1007/978-3-662-49373-1
    23 sg:pub.10.1007/s11263-014-0733-5
    24 sg:pub.10.1007/s11263-014-0779-4
    25 sg:pub.10.1007/s11263-014-0795-4
    26 schema:datePublished 2018-06-30
    27 schema:datePublishedReg 2018-06-30
    28 schema:description We propose a systematic learning-based approach to the generation of massive quantities of synthetic 3D scenes and arbitrary numbers of photorealistic 2D images thereof, with associated ground truth information, for the purposes of training, benchmarking, and diagnosing learning-based computer vision and robotics algorithms. In particular, we devise a learning-based pipeline of algorithms capable of automatically generating and rendering a potentially infinite variety of indoor scenes by using a stochastic grammar, represented as an attributed Spatial And-Or Graph, in conjunction with state-of-the-art physics-based rendering. Our pipeline is capable of synthesizing scene layouts with high diversity, and it is configurable inasmuch as it enables the precise customization and control of important attributes of the generated scenes. It renders photorealistic RGB images of the generated scenes while automatically synthesizing detailed, per-pixel ground truth data, including visible surface depth and normal, object identity, and material information (detailed to object parts), as well as environments (e.g., illuminations and camera viewpoints). We demonstrate the value of our synthesized dataset, by improving performance in certain machine-learning-based scene understanding tasks—depth and surface normal prediction, semantic segmentation, reconstruction, etc.—and by providing benchmarks for and diagnostics of trained models by modifying object attributes and scene properties in a controllable manner.
    29 schema:genre article
    30 schema:inLanguage en
    31 schema:isAccessibleForFree true
    32 schema:isPartOf N5746eb40440c44aea56f34a9183062e5
    33 N66dbe397897446439469177cffca1317
    34 sg:journal.1032807
    35 schema:keywords Configurable 3D Scene Synthesis
    36 RGB images
    37 Spatial
    38 algorithm
    39 approach
    40 arbitrary number
    41 art physics-based rendering
    42 associated ground truth information
    43 attributed Spatial
    44 attributes
    45 benchmarks
    46 computer vision
    47 conjunction
    48 control
    49 controllable manner
    50 customization
    51 data
    52 datasets
    53 depth
    54 diagnostics
    55 diversity
    56 environment
    57 generation
    58 grammar
    59 graph
    60 ground truth
    61 ground truth data
    62 ground truth information
    63 high diversity
    64 identity
    65 images
    66 important attributes
    67 indoor scenes
    68 infinite variety
    69 information
    70 layout
    71 learning-based approach
    72 learning-based computer vision
    73 learning-based pipeline
    74 manner
    75 massive quantities
    76 material information
    77 model
    78 normal prediction
    79 number
    80 object attributes
    81 performance
    82 photorealistic 2D images
    83 photorealistic RGB images
    84 physics-based rendering
    85 pipeline
    86 pixel Ground Truth
    87 pixel ground truth data
    88 precise customization
    89 prediction
    90 properties
    91 purpose
    92 purpose of training
    93 quantity
    94 reconstruction
    95 rendering
    96 robotic algorithms
    97 scene
    98 scene layout
    99 scene properties
    100 scene synthesis
    101 scene understanding
    102 segmentation
    103 semantic segmentation
    104 state
    105 stochastic grammars
    106 surface depth
    107 surface normal prediction
    108 synthesis
    109 synthetic 3D scenes
    110 systematic learning-based approach
    111 training
    112 truth
    113 truth data
    114 truth information
    115 understanding
    116 values
    117 variety
    118 visible surface depth
    119 vision
    120 schema:name Configurable 3D Scene Synthesis and 2D Image Rendering with Per-pixel Ground Truth Using Stochastic Grammars
    121 schema:pagination 920-941
    122 schema:productId N917f26db9db84ffe9f1645377ee887f2
    123 Nec75e62e1aad480eb1cb0904e7bddb33
    124 schema:sameAs https://app.dimensions.ai/details/publication/pub.1105210299
    125 https://doi.org/10.1007/s11263-018-1103-5
    126 schema:sdDatePublished 2022-01-01T18:49
    127 schema:sdLicense https://scigraph.springernature.com/explorer/license/
    128 schema:sdPublisher Nd79006d3c5c24f8787e7d17adcdac911
    129 schema:url https://doi.org/10.1007/s11263-018-1103-5
    130 sgo:license sg:explorer/license/
    131 sgo:sdDataset articles
    132 rdf:type schema:ScholarlyArticle
    133 N20a91ae683c54d93b0873bb0cd24db55 rdf:first sg:person.07662331163.43
    134 rdf:rest N33cf33baa3d04b888db5c1fb99369dd9
    135 N22a424243797425686e50ec3f6f6ba99 rdf:first sg:person.013767210740.42
    136 rdf:rest Nc9d3cc578e4d4576828e0124858760d7
    137 N33cf33baa3d04b888db5c1fb99369dd9 rdf:first sg:person.013200243570.22
    138 rdf:rest N3c3ccd24ad5849c280da72b43174206a
    139 N3c3ccd24ad5849c280da72b43174206a rdf:first sg:person.011203112510.68
    140 rdf:rest N22a424243797425686e50ec3f6f6ba99
    141 N5746eb40440c44aea56f34a9183062e5 schema:issueNumber 9
    142 rdf:type schema:PublicationIssue
    143 N66dbe397897446439469177cffca1317 schema:volumeNumber 126
    144 rdf:type schema:PublicationVolume
    145 N880ab50d5bf44fe8b81ded0d16dbd492 rdf:first sg:person.01323172313.41
    146 rdf:rest Na1548a7982e6401c8b7d2a1bd9880e1a
    147 N917f26db9db84ffe9f1645377ee887f2 schema:name doi
    148 schema:value 10.1007/s11263-018-1103-5
    149 rdf:type schema:PropertyValue
    150 Na1548a7982e6401c8b7d2a1bd9880e1a rdf:first sg:person.012512654354.22
    151 rdf:rest N20a91ae683c54d93b0873bb0cd24db55
    152 Na1747d95850149f2a4a58b85cac132b9 rdf:first sg:person.012430434515.36
    153 rdf:rest rdf:nil
    154 Nc9d3cc578e4d4576828e0124858760d7 rdf:first sg:person.016347323445.35
    155 rdf:rest Na1747d95850149f2a4a58b85cac132b9
    156 Nd79006d3c5c24f8787e7d17adcdac911 schema:name Springer Nature - SN SciGraph project
    157 rdf:type schema:Organization
    158 Nec75e62e1aad480eb1cb0904e7bddb33 schema:name dimensions_id
    159 schema:value pub.1105210299
    160 rdf:type schema:PropertyValue
    161 anzsrc-for:08 schema:inDefinedTermSet anzsrc-for:
    162 schema:name Information and Computing Sciences
    163 rdf:type schema:DefinedTerm
    164 anzsrc-for:0801 schema:inDefinedTermSet anzsrc-for:
    165 schema:name Artificial Intelligence and Image Processing
    166 rdf:type schema:DefinedTerm
    167 sg:grant.7132711 http://pending.schema.org/fundedItem sg:pub.10.1007/s11263-018-1103-5
    168 rdf:type schema:MonetaryGrant
    169 sg:journal.1032807 schema:issn 0920-5691
    170 1573-1405
    171 schema:name International Journal of Computer Vision
    172 schema:publisher Springer Nature
    173 rdf:type schema:Periodical
    174 sg:person.011203112510.68 schema:affiliation grid-institutes:grid.19006.3e
    175 schema:familyName Lin
    176 schema:givenName Jenny
    177 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.011203112510.68
    178 rdf:type schema:Person
    179 sg:person.012430434515.36 schema:affiliation grid-institutes:grid.19006.3e
    180 schema:familyName Zhu
    181 schema:givenName Song-Chun
    182 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.012430434515.36
    183 rdf:type schema:Person
    184 sg:person.012512654354.22 schema:affiliation grid-institutes:grid.19006.3e
    185 schema:familyName Qi
    186 schema:givenName Siyuan
    187 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.012512654354.22
    188 rdf:type schema:Person
    189 sg:person.013200243570.22 schema:affiliation grid-institutes:grid.19006.3e
    190 schema:familyName Huang
    191 schema:givenName Siyuan
    192 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.013200243570.22
    193 rdf:type schema:Person
    194 sg:person.01323172313.41 schema:affiliation grid-institutes:grid.25879.31
    195 schema:familyName Jiang
    196 schema:givenName Chenfanfu
    197 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01323172313.41
    198 rdf:type schema:Person
    199 sg:person.013767210740.42 schema:affiliation grid-institutes:grid.266685.9
    200 schema:familyName Yu
    201 schema:givenName Lap-Fai
    202 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.013767210740.42
    203 rdf:type schema:Person
    204 sg:person.016347323445.35 schema:affiliation grid-institutes:grid.19006.3e
    205 schema:familyName Terzopoulos
    206 schema:givenName Demetri
    207 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.016347323445.35
    208 rdf:type schema:Person
    209 sg:person.07662331163.43 schema:affiliation grid-institutes:grid.19006.3e
    210 schema:familyName Zhu
    211 schema:givenName Yixin
    212 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.07662331163.43
    213 rdf:type schema:Person
    214 sg:pub.10.1007/978-1-4471-6515-6 schema:sameAs https://app.dimensions.ai/details/publication/pub.1045307848
    215 https://doi.org/10.1007/978-1-4471-6515-6
    216 rdf:type schema:CreativeWork
    217 sg:pub.10.1007/978-1-4471-6741-9 schema:sameAs https://app.dimensions.ai/details/publication/pub.1021729328
    218 https://doi.org/10.1007/978-1-4471-6741-9
    219 rdf:type schema:CreativeWork
    220 sg:pub.10.1007/978-3-319-10578-9_54 schema:sameAs https://app.dimensions.ai/details/publication/pub.1013612828
    221 https://doi.org/10.1007/978-3-319-10578-9_54
    222 rdf:type schema:CreativeWork
    223 sg:pub.10.1007/978-3-319-10584-0_11 schema:sameAs https://app.dimensions.ai/details/publication/pub.1017659317
    224 https://doi.org/10.1007/978-3-319-10584-0_11
    225 rdf:type schema:CreativeWork
    226 sg:pub.10.1007/978-3-319-10599-4_41 schema:sameAs https://app.dimensions.ai/details/publication/pub.1046056033
    227 https://doi.org/10.1007/978-3-319-10599-4_41
    228 rdf:type schema:CreativeWork
    229 sg:pub.10.1007/978-3-319-10602-1_48 schema:sameAs https://app.dimensions.ai/details/publication/pub.1045321436
    230 https://doi.org/10.1007/978-3-319-10602-1_48
    231 rdf:type schema:CreativeWork
    232 sg:pub.10.1007/978-3-319-10605-2_27 schema:sameAs https://app.dimensions.ai/details/publication/pub.1018385677
    233 https://doi.org/10.1007/978-3-319-10605-2_27
    234 rdf:type schema:CreativeWork
    235 sg:pub.10.1007/978-3-319-24947-6_34 schema:sameAs https://app.dimensions.ai/details/publication/pub.1000861733
    236 https://doi.org/10.1007/978-3-319-24947-6_34
    237 rdf:type schema:CreativeWork
    238 sg:pub.10.1007/978-3-319-46475-6_7 schema:sameAs https://app.dimensions.ai/details/publication/pub.1025415319
    239 https://doi.org/10.1007/978-3-319-46475-6_7
    240 rdf:type schema:CreativeWork
    241 sg:pub.10.1007/978-3-319-46493-0_12 schema:sameAs https://app.dimensions.ai/details/publication/pub.1045724022
    242 https://doi.org/10.1007/978-3-319-46493-0_12
    243 rdf:type schema:CreativeWork
    244 sg:pub.10.1007/978-3-319-46493-0_2 schema:sameAs https://app.dimensions.ai/details/publication/pub.1049164757
    245 https://doi.org/10.1007/978-3-319-46493-0_2
    246 rdf:type schema:CreativeWork
    247 sg:pub.10.1007/978-3-319-46493-0_20 schema:sameAs https://app.dimensions.ai/details/publication/pub.1034474844
    248 https://doi.org/10.1007/978-3-319-46493-0_20
    249 rdf:type schema:CreativeWork
    250 sg:pub.10.1007/978-3-319-49409-8_18 schema:sameAs https://app.dimensions.ai/details/publication/pub.1090663472
    251 https://doi.org/10.1007/978-3-319-49409-8_18
    252 rdf:type schema:CreativeWork
    253 sg:pub.10.1007/978-3-319-49409-8_75 schema:sameAs https://app.dimensions.ai/details/publication/pub.1086290585
    254 https://doi.org/10.1007/978-3-319-49409-8_75
    255 rdf:type schema:CreativeWork
    256 sg:pub.10.1007/978-3-319-58347-1 schema:sameAs https://app.dimensions.ai/details/publication/pub.1091572224
    257 https://doi.org/10.1007/978-3-319-58347-1
    258 rdf:type schema:CreativeWork
    259 sg:pub.10.1007/978-3-319-58347-1_13 schema:sameAs https://app.dimensions.ai/details/publication/pub.1091588219
    260 https://doi.org/10.1007/978-3-319-58347-1_13
    261 rdf:type schema:CreativeWork
    262 sg:pub.10.1007/978-3-540-87987-9_8 schema:sameAs https://app.dimensions.ai/details/publication/pub.1012636020
    263 https://doi.org/10.1007/978-3-540-87987-9_8
    264 rdf:type schema:CreativeWork
    265 sg:pub.10.1007/978-3-642-33715-4_54 schema:sameAs https://app.dimensions.ai/details/publication/pub.1053469442
    266 https://doi.org/10.1007/978-3-642-33715-4_54
    267 rdf:type schema:CreativeWork
    268 sg:pub.10.1007/978-3-662-49373-1 schema:sameAs https://app.dimensions.ai/details/publication/pub.1052826178
    269 https://doi.org/10.1007/978-3-662-49373-1
    270 rdf:type schema:CreativeWork
    271 sg:pub.10.1007/s11263-014-0733-5 schema:sameAs https://app.dimensions.ai/details/publication/pub.1017073734
    272 https://doi.org/10.1007/s11263-014-0733-5
    273 rdf:type schema:CreativeWork
    274 sg:pub.10.1007/s11263-014-0779-4 schema:sameAs https://app.dimensions.ai/details/publication/pub.1009995058
    275 https://doi.org/10.1007/s11263-014-0779-4
    276 rdf:type schema:CreativeWork
    277 sg:pub.10.1007/s11263-014-0795-4 schema:sameAs https://app.dimensions.ai/details/publication/pub.1051031938
    278 https://doi.org/10.1007/s11263-014-0795-4
    279 rdf:type schema:CreativeWork
    280 grid-institutes:grid.19006.3e schema:alternateName UCLA Center for Vision, Cognition, Learning and Autonomy, University of California, Los Angeles, Los Angeles, USA
    281 UCLA Computer Graphics and Vision Laboratory, University of California, Los Angeles, Los Angeles, USA
    282 schema:name UCLA Center for Vision, Cognition, Learning and Autonomy, University of California, Los Angeles, Los Angeles, USA
    283 UCLA Computer Graphics and Vision Laboratory, University of California, Los Angeles, Los Angeles, USA
    284 rdf:type schema:Organization
    285 grid-institutes:grid.25879.31 schema:alternateName SIG Center for Computer Graphics, University of Pennsylvania, Philadelphia, USA
    286 schema:name SIG Center for Computer Graphics, University of Pennsylvania, Philadelphia, USA
    287 rdf:type schema:Organization
    288 grid-institutes:grid.266685.9 schema:alternateName Graphics and Virtual Environments Laboratory, University of Massachusetts Boston, Boston, USA
    289 schema:name Graphics and Virtual Environments Laboratory, University of Massachusetts Boston, Boston, USA
    290 rdf:type schema:Organization
     




    Preview window. Press ESC to close (or click here)


    ...