End-to-End Learning of Video Super-Resolution with Motion Compensation View Full Text


Ontology type: schema:Chapter      Open Access: True


Chapter Info

DATE

2017-08-15

AUTHORS

Osama Makansi , Eddy Ilg , Thomas Brox

ABSTRACT

Learning approaches have shown great success in the task of super-resolving an image given a low resolution input. Video super-resolution aims for exploiting additionally the information from multiple images. Typically, the images are related via optical flow and consecutive image warping. In this paper, we provide an end-to-end video super-resolution network that, in contrast to previous works, includes the estimation of optical flow in the overall network architecture. We analyze the usage of optical flow for video super-resolution and find that common off-the-shelf image warping does not allow video super-resolution to benefit much from optical flow. We rather propose an operation for motion compensation that performs warping from low to high resolution directly. We show that with this network configuration, video super-resolution can benefit from optical flow and we obtain state-of-the-art results on the popular test sets. We also show that the processing of whole images rather than independent patches is responsible for a large increase in accuracy. More... »

PAGES

203-214

Identifiers

URI

http://scigraph.springernature.com/pub.10.1007/978-3-319-66709-6_17

DOI

http://dx.doi.org/10.1007/978-3-319-66709-6_17

DIMENSIONS

https://app.dimensions.ai/details/publication/pub.1091259702


Indexing Status Check whether this publication has been indexed by Scopus and Web Of Science using the SN Indexing Status Tool
Incoming Citations Browse incoming citations for this publication using opencitations.net

JSON-LD is the canonical representation for SciGraph data.

TIP: You can open this SciGraph record using an external JSON-LD service: JSON-LD Playground Google SDTT

[
  {
    "@context": "https://springernature.github.io/scigraph/jsonld/sgcontext.json", 
    "about": [
      {
        "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/08", 
        "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
        "name": "Information and Computing Sciences", 
        "type": "DefinedTerm"
      }, 
      {
        "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/10", 
        "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
        "name": "Technology", 
        "type": "DefinedTerm"
      }, 
      {
        "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/0801", 
        "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
        "name": "Artificial Intelligence and Image Processing", 
        "type": "DefinedTerm"
      }, 
      {
        "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/1005", 
        "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
        "name": "Communications Technologies", 
        "type": "DefinedTerm"
      }
    ], 
    "author": [
      {
        "affiliation": {
          "alternateName": "Department of Computer Science, University of Freiburg, Freiburg im Breisgau, Germany", 
          "id": "http://www.grid.ac/institutes/grid.5963.9", 
          "name": [
            "Department of Computer Science, University of Freiburg, Freiburg im Breisgau, Germany"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Makansi", 
        "givenName": "Osama", 
        "id": "sg:person.013221150447.86", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.013221150447.86"
        ], 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "Department of Computer Science, University of Freiburg, Freiburg im Breisgau, Germany", 
          "id": "http://www.grid.ac/institutes/grid.5963.9", 
          "name": [
            "Department of Computer Science, University of Freiburg, Freiburg im Breisgau, Germany"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Ilg", 
        "givenName": "Eddy", 
        "id": "sg:person.014016531047.11", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.014016531047.11"
        ], 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "Department of Computer Science, University of Freiburg, Freiburg im Breisgau, Germany", 
          "id": "http://www.grid.ac/institutes/grid.5963.9", 
          "name": [
            "Department of Computer Science, University of Freiburg, Freiburg im Breisgau, Germany"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Brox", 
        "givenName": "Thomas", 
        "id": "sg:person.012443225372.65", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.012443225372.65"
        ], 
        "type": "Person"
      }
    ], 
    "datePublished": "2017-08-15", 
    "datePublishedReg": "2017-08-15", 
    "description": "Learning approaches have shown great success in the task of super-resolving an image given a low resolution input. Video super-resolution aims for exploiting additionally the information from multiple images. Typically, the images are related via optical flow and consecutive image warping. In this paper, we provide an end-to-end video super-resolution network that, in contrast to previous works, includes the estimation of optical flow in the overall network architecture. We analyze the usage of optical flow for video super-resolution and find that common off-the-shelf image warping does not allow video super-resolution to benefit much from optical flow. We rather propose an operation for motion compensation that performs warping from low to high resolution directly. We show that with this network configuration, video super-resolution can benefit from optical flow and we obtain state-of-the-art results on the popular test sets. We also show that the processing of whole images rather than independent patches is responsible for a large increase in accuracy.", 
    "editor": [
      {
        "familyName": "Roth", 
        "givenName": "Volker", 
        "type": "Person"
      }, 
      {
        "familyName": "Vetter", 
        "givenName": "Thomas", 
        "type": "Person"
      }
    ], 
    "genre": "chapter", 
    "id": "sg:pub.10.1007/978-3-319-66709-6_17", 
    "isAccessibleForFree": true, 
    "isPartOf": {
      "isbn": [
        "978-3-319-66708-9", 
        "978-3-319-66709-6"
      ], 
      "name": "Pattern Recognition", 
      "type": "Book"
    }, 
    "keywords": [
      "optical flow", 
      "image warping", 
      "motion compensation", 
      "super-resolution network", 
      "Video Super-Resolution", 
      "overall network architecture", 
      "low-resolution input", 
      "end learning", 
      "network architecture", 
      "art results", 
      "whole image", 
      "learning approach", 
      "multiple images", 
      "Super-Resolution", 
      "network configuration", 
      "video", 
      "test set", 
      "great success", 
      "images", 
      "warping", 
      "independent patches", 
      "previous work", 
      "high resolution", 
      "architecture", 
      "network", 
      "task", 
      "learning", 
      "usage", 
      "accuracy", 
      "processing", 
      "information", 
      "set", 
      "input", 
      "operation", 
      "patches", 
      "estimation", 
      "end", 
      "compensation", 
      "configuration", 
      "work", 
      "resolution", 
      "paper", 
      "success", 
      "state", 
      "flow", 
      "results", 
      "approach", 
      "increase", 
      "contrast", 
      "large increase"
    ], 
    "name": "End-to-End Learning of Video Super-Resolution with Motion Compensation", 
    "pagination": "203-214", 
    "productId": [
      {
        "name": "dimensions_id", 
        "type": "PropertyValue", 
        "value": [
          "pub.1091259702"
        ]
      }, 
      {
        "name": "doi", 
        "type": "PropertyValue", 
        "value": [
          "10.1007/978-3-319-66709-6_17"
        ]
      }
    ], 
    "publisher": {
      "name": "Springer Nature", 
      "type": "Organisation"
    }, 
    "sameAs": [
      "https://doi.org/10.1007/978-3-319-66709-6_17", 
      "https://app.dimensions.ai/details/publication/pub.1091259702"
    ], 
    "sdDataset": "chapters", 
    "sdDatePublished": "2022-12-01T06:51", 
    "sdLicense": "https://scigraph.springernature.com/explorer/license/", 
    "sdPublisher": {
      "name": "Springer Nature - SN SciGraph project", 
      "type": "Organization"
    }, 
    "sdSource": "s3://com-springernature-scigraph/baseset/20221201/entities/gbq_results/chapter/chapter_324.jsonl", 
    "type": "Chapter", 
    "url": "https://doi.org/10.1007/978-3-319-66709-6_17"
  }
]
 

Download the RDF metadata as:  json-ld nt turtle xml License info

HOW TO GET THIS DATA PROGRAMMATICALLY:

JSON-LD is a popular format for linked data which is fully compatible with JSON.

curl -H 'Accept: application/ld+json' 'https://scigraph.springernature.com/pub.10.1007/978-3-319-66709-6_17'

N-Triples is a line-based linked data format ideal for batch operations.

curl -H 'Accept: application/n-triples' 'https://scigraph.springernature.com/pub.10.1007/978-3-319-66709-6_17'

Turtle is a human-readable linked data format.

curl -H 'Accept: text/turtle' 'https://scigraph.springernature.com/pub.10.1007/978-3-319-66709-6_17'

RDF/XML is a standard XML format for linked data.

curl -H 'Accept: application/rdf+xml' 'https://scigraph.springernature.com/pub.10.1007/978-3-319-66709-6_17'


 

This table displays all metadata directly associated to this object as RDF triples.

136 TRIPLES      22 PREDICATES      76 URIs      67 LITERALS      7 BLANK NODES

Subject Predicate Object
1 sg:pub.10.1007/978-3-319-66709-6_17 schema:about anzsrc-for:08
2 anzsrc-for:0801
3 anzsrc-for:10
4 anzsrc-for:1005
5 schema:author Nba3439568f7e4d8fb90cf2f3d857a393
6 schema:datePublished 2017-08-15
7 schema:datePublishedReg 2017-08-15
8 schema:description Learning approaches have shown great success in the task of super-resolving an image given a low resolution input. Video super-resolution aims for exploiting additionally the information from multiple images. Typically, the images are related via optical flow and consecutive image warping. In this paper, we provide an end-to-end video super-resolution network that, in contrast to previous works, includes the estimation of optical flow in the overall network architecture. We analyze the usage of optical flow for video super-resolution and find that common off-the-shelf image warping does not allow video super-resolution to benefit much from optical flow. We rather propose an operation for motion compensation that performs warping from low to high resolution directly. We show that with this network configuration, video super-resolution can benefit from optical flow and we obtain state-of-the-art results on the popular test sets. We also show that the processing of whole images rather than independent patches is responsible for a large increase in accuracy.
9 schema:editor N48a4ced1b1bc4f1ba9d9f5b7440cb369
10 schema:genre chapter
11 schema:isAccessibleForFree true
12 schema:isPartOf N786884b3da6a43e59de54f4048964109
13 schema:keywords Super-Resolution
14 Video Super-Resolution
15 accuracy
16 approach
17 architecture
18 art results
19 compensation
20 configuration
21 contrast
22 end
23 end learning
24 estimation
25 flow
26 great success
27 high resolution
28 image warping
29 images
30 increase
31 independent patches
32 information
33 input
34 large increase
35 learning
36 learning approach
37 low-resolution input
38 motion compensation
39 multiple images
40 network
41 network architecture
42 network configuration
43 operation
44 optical flow
45 overall network architecture
46 paper
47 patches
48 previous work
49 processing
50 resolution
51 results
52 set
53 state
54 success
55 super-resolution network
56 task
57 test set
58 usage
59 video
60 warping
61 whole image
62 work
63 schema:name End-to-End Learning of Video Super-Resolution with Motion Compensation
64 schema:pagination 203-214
65 schema:productId N91e84de048fb452da825bac48d264303
66 N94abe7cda31c4dfeab19c5ba992ba26b
67 schema:publisher Nf69cf6cdb2ec47ad88eb40f27febdbcb
68 schema:sameAs https://app.dimensions.ai/details/publication/pub.1091259702
69 https://doi.org/10.1007/978-3-319-66709-6_17
70 schema:sdDatePublished 2022-12-01T06:51
71 schema:sdLicense https://scigraph.springernature.com/explorer/license/
72 schema:sdPublisher N454583d134cb4b3284c7a164016a3313
73 schema:url https://doi.org/10.1007/978-3-319-66709-6_17
74 sgo:license sg:explorer/license/
75 sgo:sdDataset chapters
76 rdf:type schema:Chapter
77 N1b7efa0e7b6c495aa4285bbbb65dcd3a schema:familyName Vetter
78 schema:givenName Thomas
79 rdf:type schema:Person
80 N1c086a3e046042399fcc61cdfe8430af rdf:first sg:person.014016531047.11
81 rdf:rest N54f8e37760fb4b0aae3d3aed32db963d
82 N33fe3917c6504d0288f5e947b629b12b schema:familyName Roth
83 schema:givenName Volker
84 rdf:type schema:Person
85 N454583d134cb4b3284c7a164016a3313 schema:name Springer Nature - SN SciGraph project
86 rdf:type schema:Organization
87 N48a4ced1b1bc4f1ba9d9f5b7440cb369 rdf:first N33fe3917c6504d0288f5e947b629b12b
88 rdf:rest N98c38cc45d0842028b6ae213746fc735
89 N54f8e37760fb4b0aae3d3aed32db963d rdf:first sg:person.012443225372.65
90 rdf:rest rdf:nil
91 N786884b3da6a43e59de54f4048964109 schema:isbn 978-3-319-66708-9
92 978-3-319-66709-6
93 schema:name Pattern Recognition
94 rdf:type schema:Book
95 N91e84de048fb452da825bac48d264303 schema:name doi
96 schema:value 10.1007/978-3-319-66709-6_17
97 rdf:type schema:PropertyValue
98 N94abe7cda31c4dfeab19c5ba992ba26b schema:name dimensions_id
99 schema:value pub.1091259702
100 rdf:type schema:PropertyValue
101 N98c38cc45d0842028b6ae213746fc735 rdf:first N1b7efa0e7b6c495aa4285bbbb65dcd3a
102 rdf:rest rdf:nil
103 Nba3439568f7e4d8fb90cf2f3d857a393 rdf:first sg:person.013221150447.86
104 rdf:rest N1c086a3e046042399fcc61cdfe8430af
105 Nf69cf6cdb2ec47ad88eb40f27febdbcb schema:name Springer Nature
106 rdf:type schema:Organisation
107 anzsrc-for:08 schema:inDefinedTermSet anzsrc-for:
108 schema:name Information and Computing Sciences
109 rdf:type schema:DefinedTerm
110 anzsrc-for:0801 schema:inDefinedTermSet anzsrc-for:
111 schema:name Artificial Intelligence and Image Processing
112 rdf:type schema:DefinedTerm
113 anzsrc-for:10 schema:inDefinedTermSet anzsrc-for:
114 schema:name Technology
115 rdf:type schema:DefinedTerm
116 anzsrc-for:1005 schema:inDefinedTermSet anzsrc-for:
117 schema:name Communications Technologies
118 rdf:type schema:DefinedTerm
119 sg:person.012443225372.65 schema:affiliation grid-institutes:grid.5963.9
120 schema:familyName Brox
121 schema:givenName Thomas
122 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.012443225372.65
123 rdf:type schema:Person
124 sg:person.013221150447.86 schema:affiliation grid-institutes:grid.5963.9
125 schema:familyName Makansi
126 schema:givenName Osama
127 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.013221150447.86
128 rdf:type schema:Person
129 sg:person.014016531047.11 schema:affiliation grid-institutes:grid.5963.9
130 schema:familyName Ilg
131 schema:givenName Eddy
132 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.014016531047.11
133 rdf:type schema:Person
134 grid-institutes:grid.5963.9 schema:alternateName Department of Computer Science, University of Freiburg, Freiburg im Breisgau, Germany
135 schema:name Department of Computer Science, University of Freiburg, Freiburg im Breisgau, Germany
136 rdf:type schema:Organization
 




Preview window. Press ESC to close (or click here)


...