End-to-End Learning of Video Super-Resolution with Motion Compensation View Full Text


Ontology type: schema:Chapter      Open Access: True


Chapter Info

DATE

2017-08-15

AUTHORS

Osama Makansi , Eddy Ilg , Thomas Brox

ABSTRACT

Learning approaches have shown great success in the task of super-resolving an image given a low resolution input. Video super-resolution aims for exploiting additionally the information from multiple images. Typically, the images are related via optical flow and consecutive image warping. In this paper, we provide an end-to-end video super-resolution network that, in contrast to previous works, includes the estimation of optical flow in the overall network architecture. We analyze the usage of optical flow for video super-resolution and find that common off-the-shelf image warping does not allow video super-resolution to benefit much from optical flow. We rather propose an operation for motion compensation that performs warping from low to high resolution directly. We show that with this network configuration, video super-resolution can benefit from optical flow and we obtain state-of-the-art results on the popular test sets. We also show that the processing of whole images rather than independent patches is responsible for a large increase in accuracy. More... »

PAGES

203-214

Identifiers

URI

http://scigraph.springernature.com/pub.10.1007/978-3-319-66709-6_17

DOI

http://dx.doi.org/10.1007/978-3-319-66709-6_17

DIMENSIONS

https://app.dimensions.ai/details/publication/pub.1091259702


Indexing Status Check whether this publication has been indexed by Scopus and Web Of Science using the SN Indexing Status Tool
Incoming Citations Browse incoming citations for this publication using opencitations.net

JSON-LD is the canonical representation for SciGraph data.

TIP: You can open this SciGraph record using an external JSON-LD service: JSON-LD Playground Google SDTT

[
  {
    "@context": "https://springernature.github.io/scigraph/jsonld/sgcontext.json", 
    "about": [
      {
        "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/08", 
        "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
        "name": "Information and Computing Sciences", 
        "type": "DefinedTerm"
      }, 
      {
        "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/10", 
        "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
        "name": "Technology", 
        "type": "DefinedTerm"
      }, 
      {
        "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/0801", 
        "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
        "name": "Artificial Intelligence and Image Processing", 
        "type": "DefinedTerm"
      }, 
      {
        "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/1005", 
        "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
        "name": "Communications Technologies", 
        "type": "DefinedTerm"
      }
    ], 
    "author": [
      {
        "affiliation": {
          "alternateName": "Department of Computer Science, University of Freiburg, Freiburg im Breisgau, Germany", 
          "id": "http://www.grid.ac/institutes/grid.5963.9", 
          "name": [
            "Department of Computer Science, University of Freiburg, Freiburg im Breisgau, Germany"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Makansi", 
        "givenName": "Osama", 
        "id": "sg:person.013221150447.86", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.013221150447.86"
        ], 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "Department of Computer Science, University of Freiburg, Freiburg im Breisgau, Germany", 
          "id": "http://www.grid.ac/institutes/grid.5963.9", 
          "name": [
            "Department of Computer Science, University of Freiburg, Freiburg im Breisgau, Germany"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Ilg", 
        "givenName": "Eddy", 
        "id": "sg:person.014016531047.11", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.014016531047.11"
        ], 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "Department of Computer Science, University of Freiburg, Freiburg im Breisgau, Germany", 
          "id": "http://www.grid.ac/institutes/grid.5963.9", 
          "name": [
            "Department of Computer Science, University of Freiburg, Freiburg im Breisgau, Germany"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Brox", 
        "givenName": "Thomas", 
        "id": "sg:person.012443225372.65", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.012443225372.65"
        ], 
        "type": "Person"
      }
    ], 
    "datePublished": "2017-08-15", 
    "datePublishedReg": "2017-08-15", 
    "description": "Learning approaches have shown great success in the task of super-resolving an image given a low resolution input. Video super-resolution aims for exploiting additionally the information from multiple images. Typically, the images are related via optical flow and consecutive image warping. In this paper, we provide an end-to-end video super-resolution network that, in contrast to previous works, includes the estimation of optical flow in the overall network architecture. We analyze the usage of optical flow for video super-resolution and find that common off-the-shelf image warping does not allow video super-resolution to benefit much from optical flow. We rather propose an operation for motion compensation that performs warping from low to high resolution directly. We show that with this network configuration, video super-resolution can benefit from optical flow and we obtain state-of-the-art results on the popular test sets. We also show that the processing of whole images rather than independent patches is responsible for a large increase in accuracy.", 
    "editor": [
      {
        "familyName": "Roth", 
        "givenName": "Volker", 
        "type": "Person"
      }, 
      {
        "familyName": "Vetter", 
        "givenName": "Thomas", 
        "type": "Person"
      }
    ], 
    "genre": "chapter", 
    "id": "sg:pub.10.1007/978-3-319-66709-6_17", 
    "isAccessibleForFree": true, 
    "isPartOf": {
      "isbn": [
        "978-3-319-66708-9", 
        "978-3-319-66709-6"
      ], 
      "name": "Pattern Recognition", 
      "type": "Book"
    }, 
    "keywords": [
      "optical flow", 
      "image warping", 
      "motion compensation", 
      "super-resolution network", 
      "Video Super-Resolution", 
      "overall network architecture", 
      "low-resolution input", 
      "end learning", 
      "network architecture", 
      "art results", 
      "whole image", 
      "learning approach", 
      "multiple images", 
      "Super-Resolution", 
      "network configuration", 
      "video", 
      "test set", 
      "great success", 
      "images", 
      "warping", 
      "independent patches", 
      "previous work", 
      "high resolution", 
      "architecture", 
      "network", 
      "task", 
      "learning", 
      "usage", 
      "accuracy", 
      "processing", 
      "information", 
      "set", 
      "input", 
      "operation", 
      "patches", 
      "estimation", 
      "end", 
      "compensation", 
      "configuration", 
      "work", 
      "resolution", 
      "paper", 
      "success", 
      "state", 
      "flow", 
      "results", 
      "approach", 
      "increase", 
      "contrast", 
      "large increase"
    ], 
    "name": "End-to-End Learning of Video Super-Resolution with Motion Compensation", 
    "pagination": "203-214", 
    "productId": [
      {
        "name": "dimensions_id", 
        "type": "PropertyValue", 
        "value": [
          "pub.1091259702"
        ]
      }, 
      {
        "name": "doi", 
        "type": "PropertyValue", 
        "value": [
          "10.1007/978-3-319-66709-6_17"
        ]
      }
    ], 
    "publisher": {
      "name": "Springer Nature", 
      "type": "Organisation"
    }, 
    "sameAs": [
      "https://doi.org/10.1007/978-3-319-66709-6_17", 
      "https://app.dimensions.ai/details/publication/pub.1091259702"
    ], 
    "sdDataset": "chapters", 
    "sdDatePublished": "2022-09-02T16:12", 
    "sdLicense": "https://scigraph.springernature.com/explorer/license/", 
    "sdPublisher": {
      "name": "Springer Nature - SN SciGraph project", 
      "type": "Organization"
    }, 
    "sdSource": "s3://com-springernature-scigraph/baseset/20220902/entities/gbq_results/chapter/chapter_243.jsonl", 
    "type": "Chapter", 
    "url": "https://doi.org/10.1007/978-3-319-66709-6_17"
  }
]
 

Download the RDF metadata as:  json-ld nt turtle xml License info

HOW TO GET THIS DATA PROGRAMMATICALLY:

JSON-LD is a popular format for linked data which is fully compatible with JSON.

curl -H 'Accept: application/ld+json' 'https://scigraph.springernature.com/pub.10.1007/978-3-319-66709-6_17'

N-Triples is a line-based linked data format ideal for batch operations.

curl -H 'Accept: application/n-triples' 'https://scigraph.springernature.com/pub.10.1007/978-3-319-66709-6_17'

Turtle is a human-readable linked data format.

curl -H 'Accept: text/turtle' 'https://scigraph.springernature.com/pub.10.1007/978-3-319-66709-6_17'

RDF/XML is a standard XML format for linked data.

curl -H 'Accept: application/rdf+xml' 'https://scigraph.springernature.com/pub.10.1007/978-3-319-66709-6_17'


 

This table displays all metadata directly associated to this object as RDF triples.

136 TRIPLES      22 PREDICATES      76 URIs      67 LITERALS      7 BLANK NODES

Subject Predicate Object
1 sg:pub.10.1007/978-3-319-66709-6_17 schema:about anzsrc-for:08
2 anzsrc-for:0801
3 anzsrc-for:10
4 anzsrc-for:1005
5 schema:author Na1408fa6d274465b97ec3c846a3e1120
6 schema:datePublished 2017-08-15
7 schema:datePublishedReg 2017-08-15
8 schema:description Learning approaches have shown great success in the task of super-resolving an image given a low resolution input. Video super-resolution aims for exploiting additionally the information from multiple images. Typically, the images are related via optical flow and consecutive image warping. In this paper, we provide an end-to-end video super-resolution network that, in contrast to previous works, includes the estimation of optical flow in the overall network architecture. We analyze the usage of optical flow for video super-resolution and find that common off-the-shelf image warping does not allow video super-resolution to benefit much from optical flow. We rather propose an operation for motion compensation that performs warping from low to high resolution directly. We show that with this network configuration, video super-resolution can benefit from optical flow and we obtain state-of-the-art results on the popular test sets. We also show that the processing of whole images rather than independent patches is responsible for a large increase in accuracy.
9 schema:editor N3b41aee96e1145098fbf8014d953f3ae
10 schema:genre chapter
11 schema:isAccessibleForFree true
12 schema:isPartOf Na556f2ac192044418a98fe99e4fd6314
13 schema:keywords Super-Resolution
14 Video Super-Resolution
15 accuracy
16 approach
17 architecture
18 art results
19 compensation
20 configuration
21 contrast
22 end
23 end learning
24 estimation
25 flow
26 great success
27 high resolution
28 image warping
29 images
30 increase
31 independent patches
32 information
33 input
34 large increase
35 learning
36 learning approach
37 low-resolution input
38 motion compensation
39 multiple images
40 network
41 network architecture
42 network configuration
43 operation
44 optical flow
45 overall network architecture
46 paper
47 patches
48 previous work
49 processing
50 resolution
51 results
52 set
53 state
54 success
55 super-resolution network
56 task
57 test set
58 usage
59 video
60 warping
61 whole image
62 work
63 schema:name End-to-End Learning of Video Super-Resolution with Motion Compensation
64 schema:pagination 203-214
65 schema:productId N13595eef24344208981a1b3052362eb2
66 N257690f782a34a5993ab6a8fafa78870
67 schema:publisher N6e5af282c7cd4a5986d90c52abd8ef9f
68 schema:sameAs https://app.dimensions.ai/details/publication/pub.1091259702
69 https://doi.org/10.1007/978-3-319-66709-6_17
70 schema:sdDatePublished 2022-09-02T16:12
71 schema:sdLicense https://scigraph.springernature.com/explorer/license/
72 schema:sdPublisher N6593f605ffef46ec8401c013b3061a91
73 schema:url https://doi.org/10.1007/978-3-319-66709-6_17
74 sgo:license sg:explorer/license/
75 sgo:sdDataset chapters
76 rdf:type schema:Chapter
77 N13595eef24344208981a1b3052362eb2 schema:name dimensions_id
78 schema:value pub.1091259702
79 rdf:type schema:PropertyValue
80 N257690f782a34a5993ab6a8fafa78870 schema:name doi
81 schema:value 10.1007/978-3-319-66709-6_17
82 rdf:type schema:PropertyValue
83 N3441ecdf2c3641c78481472603f9ed93 rdf:first sg:person.012443225372.65
84 rdf:rest rdf:nil
85 N36ab1b6ad0724d90bce52ed42df57cf6 schema:familyName Vetter
86 schema:givenName Thomas
87 rdf:type schema:Person
88 N3b41aee96e1145098fbf8014d953f3ae rdf:first N3b557c6f21bc4efe990049f07720960e
89 rdf:rest N44df478e0d4f47b492f79e465602c07a
90 N3b557c6f21bc4efe990049f07720960e schema:familyName Roth
91 schema:givenName Volker
92 rdf:type schema:Person
93 N44df478e0d4f47b492f79e465602c07a rdf:first N36ab1b6ad0724d90bce52ed42df57cf6
94 rdf:rest rdf:nil
95 N6593f605ffef46ec8401c013b3061a91 schema:name Springer Nature - SN SciGraph project
96 rdf:type schema:Organization
97 N6e5af282c7cd4a5986d90c52abd8ef9f schema:name Springer Nature
98 rdf:type schema:Organisation
99 N7a63162372144a7ca96f160295587f6d rdf:first sg:person.014016531047.11
100 rdf:rest N3441ecdf2c3641c78481472603f9ed93
101 Na1408fa6d274465b97ec3c846a3e1120 rdf:first sg:person.013221150447.86
102 rdf:rest N7a63162372144a7ca96f160295587f6d
103 Na556f2ac192044418a98fe99e4fd6314 schema:isbn 978-3-319-66708-9
104 978-3-319-66709-6
105 schema:name Pattern Recognition
106 rdf:type schema:Book
107 anzsrc-for:08 schema:inDefinedTermSet anzsrc-for:
108 schema:name Information and Computing Sciences
109 rdf:type schema:DefinedTerm
110 anzsrc-for:0801 schema:inDefinedTermSet anzsrc-for:
111 schema:name Artificial Intelligence and Image Processing
112 rdf:type schema:DefinedTerm
113 anzsrc-for:10 schema:inDefinedTermSet anzsrc-for:
114 schema:name Technology
115 rdf:type schema:DefinedTerm
116 anzsrc-for:1005 schema:inDefinedTermSet anzsrc-for:
117 schema:name Communications Technologies
118 rdf:type schema:DefinedTerm
119 sg:person.012443225372.65 schema:affiliation grid-institutes:grid.5963.9
120 schema:familyName Brox
121 schema:givenName Thomas
122 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.012443225372.65
123 rdf:type schema:Person
124 sg:person.013221150447.86 schema:affiliation grid-institutes:grid.5963.9
125 schema:familyName Makansi
126 schema:givenName Osama
127 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.013221150447.86
128 rdf:type schema:Person
129 sg:person.014016531047.11 schema:affiliation grid-institutes:grid.5963.9
130 schema:familyName Ilg
131 schema:givenName Eddy
132 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.014016531047.11
133 rdf:type schema:Person
134 grid-institutes:grid.5963.9 schema:alternateName Department of Computer Science, University of Freiburg, Freiburg im Breisgau, Germany
135 schema:name Department of Computer Science, University of Freiburg, Freiburg im Breisgau, Germany
136 rdf:type schema:Organization
 




Preview window. Press ESC to close (or click here)


...