DeepTAM: Deep Tracking and Mapping with Convolutional Neural Networks View Full Text


Ontology type: schema:ScholarlyArticle     


Article Info

DATE

2019-09-03

AUTHORS

Huizhong Zhou, Benjamin Ummenhofer, Thomas Brox

ABSTRACT

We present a system for dense keyframe-based camera tracking and depth map estimation that is entirely learned. For tracking, we estimate small pose increments between the current camera image and a synthetic viewpoint. This formulation significantly simplifies the learning problem and alleviates the dataset bias for camera motions. Further, we show that generating a large number of pose hypotheses leads to more accurate predictions. For mapping, we accumulate information in a cost volume centered at the current depth estimate. The mapping network then combines the cost volume and the keyframe image to update the depth prediction, thereby effectively making use of depth measurements and image-based priors. Our approach yields state-of-the-art results with few images and is robust with respect to noisy camera poses. We demonstrate that the performance of our 6 DOF tracking competes with RGB-D tracking algorithms.We compare favorably against strong classic and deep learning powered dense depth algorithms. More... »

PAGES

756-769

References to SciGraph publications

  • 2004-11. Distinctive Image Features from Scale-Invariant Keypoints in INTERNATIONAL JOURNAL OF COMPUTER VISION
  • 2014. LSD-SLAM: Large-Scale Direct Monocular SLAM in COMPUTER VISION – ECCV 2014
  • 2014-11-21. Indoor Scene Understanding with RGB-D Images: Bottom-up Segmentation, Object Detection and Semantic Segmentation in INTERNATIONAL JOURNAL OF COMPUTER VISION
  • 2016-09-17. Pixelwise View Selection for Unstructured Multi-View Stereo in COMPUTER VISION – ECCV 2016
  • 2019-05-26. Learning Deeply Supervised Good Features to Match for Dense Monocular Reconstruction in COMPUTER VISION – ACCV 2018
  • 2018-10-06. DeepTAM: Deep Tracking and Mapping in COMPUTER VISION – ECCV 2018
  • 2006-05-01. Efficient Belief Propagation for Early Vision in INTERNATIONAL JOURNAL OF COMPUTER VISION
  • Identifiers

    URI

    http://scigraph.springernature.com/pub.10.1007/s11263-019-01221-0

    DOI

    http://dx.doi.org/10.1007/s11263-019-01221-0

    DIMENSIONS

    https://app.dimensions.ai/details/publication/pub.1120795419


    Indexing Status Check whether this publication has been indexed by Scopus and Web Of Science using the SN Indexing Status Tool
    Incoming Citations Browse incoming citations for this publication using opencitations.net

    JSON-LD is the canonical representation for SciGraph data.

    TIP: You can open this SciGraph record using an external JSON-LD service: JSON-LD Playground Google SDTT

    [
      {
        "@context": "https://springernature.github.io/scigraph/jsonld/sgcontext.json", 
        "about": [
          {
            "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/08", 
            "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
            "name": "Information and Computing Sciences", 
            "type": "DefinedTerm"
          }, 
          {
            "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/0801", 
            "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
            "name": "Artificial Intelligence and Image Processing", 
            "type": "DefinedTerm"
          }
        ], 
        "author": [
          {
            "affiliation": {
              "alternateName": "University of Freiburg, Freiburg, Germany", 
              "id": "http://www.grid.ac/institutes/grid.5963.9", 
              "name": [
                "University of Freiburg, Freiburg, Germany"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Zhou", 
            "givenName": "Huizhong", 
            "id": "sg:person.015337214304.05", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.015337214304.05"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "Intel Labs, Munich, Germany", 
              "id": "http://www.grid.ac/institutes/None", 
              "name": [
                "Intel Labs, Munich, Germany"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Ummenhofer", 
            "givenName": "Benjamin", 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "University of Freiburg, Freiburg, Germany", 
              "id": "http://www.grid.ac/institutes/grid.5963.9", 
              "name": [
                "University of Freiburg, Freiburg, Germany"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Brox", 
            "givenName": "Thomas", 
            "id": "sg:person.012443225372.65", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.012443225372.65"
            ], 
            "type": "Person"
          }
        ], 
        "citation": [
          {
            "id": "sg:pub.10.1007/978-3-030-01270-0_50", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1107454859", 
              "https://doi.org/10.1007/978-3-030-01270-0_50"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/s11263-014-0777-6", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1042985411", 
              "https://doi.org/10.1007/s11263-014-0777-6"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/s11263-006-7899-4", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1004925763", 
              "https://doi.org/10.1007/s11263-006-7899-4"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-319-10605-2_54", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1003628531", 
              "https://doi.org/10.1007/978-3-319-10605-2_54"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-030-20873-8_39", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1115676037", 
              "https://doi.org/10.1007/978-3-030-20873-8_39"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1023/b:visi.0000029664.99615.94", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1052687286", 
              "https://doi.org/10.1023/b:visi.0000029664.99615.94"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-319-46487-9_31", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1045172788", 
              "https://doi.org/10.1007/978-3-319-46487-9_31"
            ], 
            "type": "CreativeWork"
          }
        ], 
        "datePublished": "2019-09-03", 
        "datePublishedReg": "2019-09-03", 
        "description": "We present a system for dense keyframe-based camera tracking and depth map estimation that is entirely learned. For tracking, we estimate small pose increments between the current camera image and a synthetic viewpoint. This formulation significantly simplifies the learning problem and alleviates the dataset bias for camera motions. Further, we show that generating a large number of pose hypotheses leads to more accurate predictions. For mapping, we accumulate information in a cost volume centered at the current depth estimate. The mapping network then combines the cost volume and the keyframe image to update the depth prediction, thereby effectively making use of depth measurements and image-based priors. Our approach yields state-of-the-art results with few images and is robust with respect to noisy camera poses. We demonstrate that the performance of our 6\u00a0DOF tracking competes with RGB-D tracking algorithms.We compare favorably against strong classic and deep learning powered dense depth algorithms.", 
        "genre": "article", 
        "id": "sg:pub.10.1007/s11263-019-01221-0", 
        "isAccessibleForFree": false, 
        "isPartOf": [
          {
            "id": "sg:journal.1032807", 
            "issn": [
              "0920-5691", 
              "1573-1405"
            ], 
            "name": "International Journal of Computer Vision", 
            "publisher": "Springer Nature", 
            "type": "Periodical"
          }, 
          {
            "issueNumber": "3", 
            "type": "PublicationIssue"
          }, 
          {
            "type": "PublicationVolume", 
            "volumeNumber": "128"
          }
        ], 
        "keywords": [
          "cost volume", 
          "convolutional neural network", 
          "current camera image", 
          "depth map estimation", 
          "image-based priors", 
          "current depth estimate", 
          "camera tracking", 
          "camera pose", 
          "deep learning", 
          "pose hypotheses", 
          "camera motion", 
          "keyframe images", 
          "dataset bias", 
          "Deep Tracking", 
          "art results", 
          "neural network", 
          "camera images", 
          "learning problem", 
          "mapping network", 
          "depth prediction", 
          "tracking algorithm", 
          "map estimation", 
          "more accurate predictions", 
          "tracking", 
          "synthetic viewpoint", 
          "algorithm", 
          "images", 
          "network", 
          "depth estimates", 
          "depth algorithm", 
          "accurate prediction", 
          "large number", 
          "depth measurements", 
          "pose", 
          "learning", 
          "mapping", 
          "priors", 
          "information", 
          "prediction", 
          "performance", 
          "system", 
          "viewpoint", 
          "estimation", 
          "number", 
          "motion", 
          "use", 
          "state", 
          "results", 
          "measurements", 
          "formulation", 
          "volume", 
          "respect", 
          "increment", 
          "problem", 
          "competes", 
          "approach", 
          "estimates", 
          "bias", 
          "hypothesis"
        ], 
        "name": "DeepTAM: Deep Tracking and Mapping with Convolutional Neural Networks", 
        "pagination": "756-769", 
        "productId": [
          {
            "name": "dimensions_id", 
            "type": "PropertyValue", 
            "value": [
              "pub.1120795419"
            ]
          }, 
          {
            "name": "doi", 
            "type": "PropertyValue", 
            "value": [
              "10.1007/s11263-019-01221-0"
            ]
          }
        ], 
        "sameAs": [
          "https://doi.org/10.1007/s11263-019-01221-0", 
          "https://app.dimensions.ai/details/publication/pub.1120795419"
        ], 
        "sdDataset": "articles", 
        "sdDatePublished": "2022-10-01T06:46", 
        "sdLicense": "https://scigraph.springernature.com/explorer/license/", 
        "sdPublisher": {
          "name": "Springer Nature - SN SciGraph project", 
          "type": "Organization"
        }, 
        "sdSource": "s3://com-springernature-scigraph/baseset/20221001/entities/gbq_results/article/article_813.jsonl", 
        "type": "ScholarlyArticle", 
        "url": "https://doi.org/10.1007/s11263-019-01221-0"
      }
    ]
     

    Download the RDF metadata as:  json-ld nt turtle xml License info

    HOW TO GET THIS DATA PROGRAMMATICALLY:

    JSON-LD is a popular format for linked data which is fully compatible with JSON.

    curl -H 'Accept: application/ld+json' 'https://scigraph.springernature.com/pub.10.1007/s11263-019-01221-0'

    N-Triples is a line-based linked data format ideal for batch operations.

    curl -H 'Accept: application/n-triples' 'https://scigraph.springernature.com/pub.10.1007/s11263-019-01221-0'

    Turtle is a human-readable linked data format.

    curl -H 'Accept: text/turtle' 'https://scigraph.springernature.com/pub.10.1007/s11263-019-01221-0'

    RDF/XML is a standard XML format for linked data.

    curl -H 'Accept: application/rdf+xml' 'https://scigraph.springernature.com/pub.10.1007/s11263-019-01221-0'


     

    This table displays all metadata directly associated to this object as RDF triples.

    160 TRIPLES      21 PREDICATES      90 URIs      75 LITERALS      6 BLANK NODES

    Subject Predicate Object
    1 sg:pub.10.1007/s11263-019-01221-0 schema:about anzsrc-for:08
    2 anzsrc-for:0801
    3 schema:author Nc303987a2c4945baa0014945bf307f9a
    4 schema:citation sg:pub.10.1007/978-3-030-01270-0_50
    5 sg:pub.10.1007/978-3-030-20873-8_39
    6 sg:pub.10.1007/978-3-319-10605-2_54
    7 sg:pub.10.1007/978-3-319-46487-9_31
    8 sg:pub.10.1007/s11263-006-7899-4
    9 sg:pub.10.1007/s11263-014-0777-6
    10 sg:pub.10.1023/b:visi.0000029664.99615.94
    11 schema:datePublished 2019-09-03
    12 schema:datePublishedReg 2019-09-03
    13 schema:description We present a system for dense keyframe-based camera tracking and depth map estimation that is entirely learned. For tracking, we estimate small pose increments between the current camera image and a synthetic viewpoint. This formulation significantly simplifies the learning problem and alleviates the dataset bias for camera motions. Further, we show that generating a large number of pose hypotheses leads to more accurate predictions. For mapping, we accumulate information in a cost volume centered at the current depth estimate. The mapping network then combines the cost volume and the keyframe image to update the depth prediction, thereby effectively making use of depth measurements and image-based priors. Our approach yields state-of-the-art results with few images and is robust with respect to noisy camera poses. We demonstrate that the performance of our 6 DOF tracking competes with RGB-D tracking algorithms.We compare favorably against strong classic and deep learning powered dense depth algorithms.
    14 schema:genre article
    15 schema:isAccessibleForFree false
    16 schema:isPartOf N397ba14aaf744201a4842fda98a7b18e
    17 N957c6ab2681646e2bff6a5273f8b12ea
    18 sg:journal.1032807
    19 schema:keywords Deep Tracking
    20 accurate prediction
    21 algorithm
    22 approach
    23 art results
    24 bias
    25 camera images
    26 camera motion
    27 camera pose
    28 camera tracking
    29 competes
    30 convolutional neural network
    31 cost volume
    32 current camera image
    33 current depth estimate
    34 dataset bias
    35 deep learning
    36 depth algorithm
    37 depth estimates
    38 depth map estimation
    39 depth measurements
    40 depth prediction
    41 estimates
    42 estimation
    43 formulation
    44 hypothesis
    45 image-based priors
    46 images
    47 increment
    48 information
    49 keyframe images
    50 large number
    51 learning
    52 learning problem
    53 map estimation
    54 mapping
    55 mapping network
    56 measurements
    57 more accurate predictions
    58 motion
    59 network
    60 neural network
    61 number
    62 performance
    63 pose
    64 pose hypotheses
    65 prediction
    66 priors
    67 problem
    68 respect
    69 results
    70 state
    71 synthetic viewpoint
    72 system
    73 tracking
    74 tracking algorithm
    75 use
    76 viewpoint
    77 volume
    78 schema:name DeepTAM: Deep Tracking and Mapping with Convolutional Neural Networks
    79 schema:pagination 756-769
    80 schema:productId N6c0c330545854278a957a1bba58f3748
    81 N73c08654aa8141a3bf30873cd5bc4a65
    82 schema:sameAs https://app.dimensions.ai/details/publication/pub.1120795419
    83 https://doi.org/10.1007/s11263-019-01221-0
    84 schema:sdDatePublished 2022-10-01T06:46
    85 schema:sdLicense https://scigraph.springernature.com/explorer/license/
    86 schema:sdPublisher N79fc0e6576384d9b919ac8e20949b737
    87 schema:url https://doi.org/10.1007/s11263-019-01221-0
    88 sgo:license sg:explorer/license/
    89 sgo:sdDataset articles
    90 rdf:type schema:ScholarlyArticle
    91 N397ba14aaf744201a4842fda98a7b18e schema:issueNumber 3
    92 rdf:type schema:PublicationIssue
    93 N3e503e7c55e14803b199b7e87bbdc15a rdf:first N809574b95ff94028b8c7de30ae0cf25c
    94 rdf:rest Nd53dc8f0eddc49d6bc287a7ceaf13877
    95 N6c0c330545854278a957a1bba58f3748 schema:name doi
    96 schema:value 10.1007/s11263-019-01221-0
    97 rdf:type schema:PropertyValue
    98 N73c08654aa8141a3bf30873cd5bc4a65 schema:name dimensions_id
    99 schema:value pub.1120795419
    100 rdf:type schema:PropertyValue
    101 N79fc0e6576384d9b919ac8e20949b737 schema:name Springer Nature - SN SciGraph project
    102 rdf:type schema:Organization
    103 N809574b95ff94028b8c7de30ae0cf25c schema:affiliation grid-institutes:None
    104 schema:familyName Ummenhofer
    105 schema:givenName Benjamin
    106 rdf:type schema:Person
    107 N957c6ab2681646e2bff6a5273f8b12ea schema:volumeNumber 128
    108 rdf:type schema:PublicationVolume
    109 Nc303987a2c4945baa0014945bf307f9a rdf:first sg:person.015337214304.05
    110 rdf:rest N3e503e7c55e14803b199b7e87bbdc15a
    111 Nd53dc8f0eddc49d6bc287a7ceaf13877 rdf:first sg:person.012443225372.65
    112 rdf:rest rdf:nil
    113 anzsrc-for:08 schema:inDefinedTermSet anzsrc-for:
    114 schema:name Information and Computing Sciences
    115 rdf:type schema:DefinedTerm
    116 anzsrc-for:0801 schema:inDefinedTermSet anzsrc-for:
    117 schema:name Artificial Intelligence and Image Processing
    118 rdf:type schema:DefinedTerm
    119 sg:journal.1032807 schema:issn 0920-5691
    120 1573-1405
    121 schema:name International Journal of Computer Vision
    122 schema:publisher Springer Nature
    123 rdf:type schema:Periodical
    124 sg:person.012443225372.65 schema:affiliation grid-institutes:grid.5963.9
    125 schema:familyName Brox
    126 schema:givenName Thomas
    127 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.012443225372.65
    128 rdf:type schema:Person
    129 sg:person.015337214304.05 schema:affiliation grid-institutes:grid.5963.9
    130 schema:familyName Zhou
    131 schema:givenName Huizhong
    132 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.015337214304.05
    133 rdf:type schema:Person
    134 sg:pub.10.1007/978-3-030-01270-0_50 schema:sameAs https://app.dimensions.ai/details/publication/pub.1107454859
    135 https://doi.org/10.1007/978-3-030-01270-0_50
    136 rdf:type schema:CreativeWork
    137 sg:pub.10.1007/978-3-030-20873-8_39 schema:sameAs https://app.dimensions.ai/details/publication/pub.1115676037
    138 https://doi.org/10.1007/978-3-030-20873-8_39
    139 rdf:type schema:CreativeWork
    140 sg:pub.10.1007/978-3-319-10605-2_54 schema:sameAs https://app.dimensions.ai/details/publication/pub.1003628531
    141 https://doi.org/10.1007/978-3-319-10605-2_54
    142 rdf:type schema:CreativeWork
    143 sg:pub.10.1007/978-3-319-46487-9_31 schema:sameAs https://app.dimensions.ai/details/publication/pub.1045172788
    144 https://doi.org/10.1007/978-3-319-46487-9_31
    145 rdf:type schema:CreativeWork
    146 sg:pub.10.1007/s11263-006-7899-4 schema:sameAs https://app.dimensions.ai/details/publication/pub.1004925763
    147 https://doi.org/10.1007/s11263-006-7899-4
    148 rdf:type schema:CreativeWork
    149 sg:pub.10.1007/s11263-014-0777-6 schema:sameAs https://app.dimensions.ai/details/publication/pub.1042985411
    150 https://doi.org/10.1007/s11263-014-0777-6
    151 rdf:type schema:CreativeWork
    152 sg:pub.10.1023/b:visi.0000029664.99615.94 schema:sameAs https://app.dimensions.ai/details/publication/pub.1052687286
    153 https://doi.org/10.1023/b:visi.0000029664.99615.94
    154 rdf:type schema:CreativeWork
    155 grid-institutes:None schema:alternateName Intel Labs, Munich, Germany
    156 schema:name Intel Labs, Munich, Germany
    157 rdf:type schema:Organization
    158 grid-institutes:grid.5963.9 schema:alternateName University of Freiburg, Freiburg, Germany
    159 schema:name University of Freiburg, Freiburg, Germany
    160 rdf:type schema:Organization
     




    Preview window. Press ESC to close (or click here)


    ...