Artistic Style Transfer for Videos and Spherical Images View Full Text


Ontology type: schema:ScholarlyArticle      Open Access: True


Article Info

DATE

2018-04-21

AUTHORS

Manuel Ruder, Alexey Dosovitskiy, Thomas Brox

ABSTRACT

Manually re-drawing an image in a certain artistic style takes a professional artist a long time. Doing this for a video sequence single-handedly is beyond imagination. We present two computational approaches that transfer the style from one image (for example, a painting) to a whole video sequence. In our first approach, we adapt to videos the original image style transfer technique by Gatys et al. based on energy minimization. We introduce new ways of initialization and new loss functions to generate consistent and stable stylized video sequences even in cases with large motion and strong occlusion. Our second approach formulates video stylization as a learning problem. We propose a deep network architecture and training procedures that allow us to stylize arbitrary-length videos in a consistent and stable way, and nearly in real time. We show that the proposed methods clearly outperform simpler baselines both qualitatively and quantitatively. Finally, we propose a way to adapt these approaches also to 360∘\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$^\circ $$\end{document} images and videos as they emerge with recent virtual reality hardware. More... »

PAGES

1199-1219

References to SciGraph publications

  • 2010. Dense Point Trajectories by GPU-Accelerated Large Displacement Optical Flow in COMPUTER VISION – ECCV 2010
  • 2016-09-17. Perceptual Losses for Real-Time Style Transfer and Super-Resolution in COMPUTER VISION – ECCV 2016
  • 2016-08-27. Artistic Style Transfer for Videos in PATTERN RECOGNITION
  • 2016-09-17. Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks in COMPUTER VISION – ECCV 2016
  • 2012. A Naturalistic Open Source Movie for Optical Flow Evaluation in COMPUTER VISION – ECCV 2012
  • 2014. Microsoft COCO: Common Objects in Context in COMPUTER VISION – ECCV 2014
  • Identifiers

    URI

    http://scigraph.springernature.com/pub.10.1007/s11263-018-1089-z

    DOI

    http://dx.doi.org/10.1007/s11263-018-1089-z

    DIMENSIONS

    https://app.dimensions.ai/details/publication/pub.1103506006


    Indexing Status Check whether this publication has been indexed by Scopus and Web Of Science using the SN Indexing Status Tool
    Incoming Citations Browse incoming citations for this publication using opencitations.net

    JSON-LD is the canonical representation for SciGraph data.

    TIP: You can open this SciGraph record using an external JSON-LD service: JSON-LD Playground Google SDTT

    [
      {
        "@context": "https://springernature.github.io/scigraph/jsonld/sgcontext.json", 
        "about": [
          {
            "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/08", 
            "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
            "name": "Information and Computing Sciences", 
            "type": "DefinedTerm"
          }, 
          {
            "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/0801", 
            "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
            "name": "Artificial Intelligence and Image Processing", 
            "type": "DefinedTerm"
          }
        ], 
        "author": [
          {
            "affiliation": {
              "alternateName": "Department of Computer Science, BIOSS Centre for Biological Signalling Studies, University of Freiburg, Freiburg, Germany", 
              "id": "http://www.grid.ac/institutes/grid.5963.9", 
              "name": [
                "Department of Computer Science, BIOSS Centre for Biological Signalling Studies, University of Freiburg, Freiburg, Germany"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Ruder", 
            "givenName": "Manuel", 
            "id": "sg:person.010615500061.50", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.010615500061.50"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "Department of Computer Science, BIOSS Centre for Biological Signalling Studies, University of Freiburg, Freiburg, Germany", 
              "id": "http://www.grid.ac/institutes/grid.5963.9", 
              "name": [
                "Department of Computer Science, BIOSS Centre for Biological Signalling Studies, University of Freiburg, Freiburg, Germany"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Dosovitskiy", 
            "givenName": "Alexey", 
            "id": "sg:person.011726376703.15", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.011726376703.15"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "Department of Computer Science, BIOSS Centre for Biological Signalling Studies, University of Freiburg, Freiburg, Germany", 
              "id": "http://www.grid.ac/institutes/grid.5963.9", 
              "name": [
                "Department of Computer Science, BIOSS Centre for Biological Signalling Studies, University of Freiburg, Freiburg, Germany"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Brox", 
            "givenName": "Thomas", 
            "id": "sg:person.012443225372.65", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.012443225372.65"
            ], 
            "type": "Person"
          }
        ], 
        "citation": [
          {
            "id": "sg:pub.10.1007/978-3-642-33783-3_44", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1004909083", 
              "https://doi.org/10.1007/978-3-642-33783-3_44"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-642-15549-9_32", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1043037058", 
              "https://doi.org/10.1007/978-3-642-15549-9_32"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-319-46487-9_43", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1000850552", 
              "https://doi.org/10.1007/978-3-319-46487-9_43"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-319-10602-1_48", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1045321436", 
              "https://doi.org/10.1007/978-3-319-10602-1_48"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-319-46475-6_43", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1018034649", 
              "https://doi.org/10.1007/978-3-319-46475-6_43"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-319-45886-1_3", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1044711914", 
              "https://doi.org/10.1007/978-3-319-45886-1_3"
            ], 
            "type": "CreativeWork"
          }
        ], 
        "datePublished": "2018-04-21", 
        "datePublishedReg": "2018-04-21", 
        "description": "Manually re-drawing an image in a certain artistic style takes a professional artist a long time. Doing this for a video sequence single-handedly is beyond imagination. We present two computational approaches that transfer the style from one image (for example, a painting) to a whole video sequence. In our first approach, we adapt to videos the original image style transfer technique by Gatys et al. based on energy minimization. We introduce new ways of initialization and new loss functions to generate consistent and stable stylized video sequences even in cases with large motion and strong occlusion. Our second approach formulates video stylization as a learning problem. We propose a deep network architecture and training procedures that allow us to stylize arbitrary-length videos in a consistent and stable way, and nearly in real time. We show that the proposed methods clearly outperform simpler baselines both qualitatively and quantitatively. Finally, we propose a way to adapt these approaches also to 360\u2218\\documentclass[12pt]{minimal}\n\t\t\t\t\\usepackage{amsmath}\n\t\t\t\t\\usepackage{wasysym}\n\t\t\t\t\\usepackage{amsfonts}\n\t\t\t\t\\usepackage{amssymb}\n\t\t\t\t\\usepackage{amsbsy}\n\t\t\t\t\\usepackage{mathrsfs}\n\t\t\t\t\\usepackage{upgreek}\n\t\t\t\t\\setlength{\\oddsidemargin}{-69pt}\n\t\t\t\t\\begin{document}$$^\\circ $$\\end{document} images and videos as they emerge with recent virtual reality hardware.", 
        "genre": "article", 
        "id": "sg:pub.10.1007/s11263-018-1089-z", 
        "isAccessibleForFree": true, 
        "isPartOf": [
          {
            "id": "sg:journal.1032807", 
            "issn": [
              "0920-5691", 
              "1573-1405"
            ], 
            "name": "International Journal of Computer Vision", 
            "publisher": "Springer Nature", 
            "type": "Periodical"
          }, 
          {
            "issueNumber": "11", 
            "type": "PublicationIssue"
          }, 
          {
            "type": "PublicationVolume", 
            "volumeNumber": "126"
          }
        ], 
        "keywords": [
          "video sequences", 
          "whole video sequence", 
          "deep network architecture", 
          "virtual reality hardware", 
          "Gatys et al", 
          "style transfer techniques", 
          "certain artistic style", 
          "new loss function", 
          "artistic style transfer", 
          "network architecture", 
          "style transfer", 
          "learning problem", 
          "simple baseline", 
          "spherical images", 
          "strong occlusions", 
          "loss function", 
          "video", 
          "real time", 
          "training procedure", 
          "images", 
          "large motion", 
          "first approach", 
          "artistic style", 
          "second approach", 
          "computational approach", 
          "hardware", 
          "new way", 
          "architecture", 
          "energy minimization", 
          "initialization", 
          "stylization", 
          "professional artists", 
          "way", 
          "stable way", 
          "minimization", 
          "style", 
          "sequence", 
          "long time", 
          "technique", 
          "time", 
          "method", 
          "motion", 
          "et al", 
          "transfer technique", 
          "occlusion", 
          "artists", 
          "function", 
          "procedure", 
          "cases", 
          "transfer", 
          "imagination", 
          "baseline", 
          "al", 
          "approach", 
          "problem"
        ], 
        "name": "Artistic Style Transfer for Videos and Spherical Images", 
        "pagination": "1199-1219", 
        "productId": [
          {
            "name": "dimensions_id", 
            "type": "PropertyValue", 
            "value": [
              "pub.1103506006"
            ]
          }, 
          {
            "name": "doi", 
            "type": "PropertyValue", 
            "value": [
              "10.1007/s11263-018-1089-z"
            ]
          }
        ], 
        "sameAs": [
          "https://doi.org/10.1007/s11263-018-1089-z", 
          "https://app.dimensions.ai/details/publication/pub.1103506006"
        ], 
        "sdDataset": "articles", 
        "sdDatePublished": "2022-09-02T16:02", 
        "sdLicense": "https://scigraph.springernature.com/explorer/license/", 
        "sdPublisher": {
          "name": "Springer Nature - SN SciGraph project", 
          "type": "Organization"
        }, 
        "sdSource": "s3://com-springernature-scigraph/baseset/20220902/entities/gbq_results/article/article_767.jsonl", 
        "type": "ScholarlyArticle", 
        "url": "https://doi.org/10.1007/s11263-018-1089-z"
      }
    ]
     

    Download the RDF metadata as:  json-ld nt turtle xml License info

    HOW TO GET THIS DATA PROGRAMMATICALLY:

    JSON-LD is a popular format for linked data which is fully compatible with JSON.

    curl -H 'Accept: application/ld+json' 'https://scigraph.springernature.com/pub.10.1007/s11263-018-1089-z'

    N-Triples is a line-based linked data format ideal for batch operations.

    curl -H 'Accept: application/n-triples' 'https://scigraph.springernature.com/pub.10.1007/s11263-018-1089-z'

    Turtle is a human-readable linked data format.

    curl -H 'Accept: text/turtle' 'https://scigraph.springernature.com/pub.10.1007/s11263-018-1089-z'

    RDF/XML is a standard XML format for linked data.

    curl -H 'Accept: application/rdf+xml' 'https://scigraph.springernature.com/pub.10.1007/s11263-018-1089-z'


     

    This table displays all metadata directly associated to this object as RDF triples.

    150 TRIPLES      21 PREDICATES      85 URIs      71 LITERALS      6 BLANK NODES

    Subject Predicate Object
    1 sg:pub.10.1007/s11263-018-1089-z schema:about anzsrc-for:08
    2 anzsrc-for:0801
    3 schema:author N7e02304396054bc4bc3509a4b4aec12d
    4 schema:citation sg:pub.10.1007/978-3-319-10602-1_48
    5 sg:pub.10.1007/978-3-319-45886-1_3
    6 sg:pub.10.1007/978-3-319-46475-6_43
    7 sg:pub.10.1007/978-3-319-46487-9_43
    8 sg:pub.10.1007/978-3-642-15549-9_32
    9 sg:pub.10.1007/978-3-642-33783-3_44
    10 schema:datePublished 2018-04-21
    11 schema:datePublishedReg 2018-04-21
    12 schema:description Manually re-drawing an image in a certain artistic style takes a professional artist a long time. Doing this for a video sequence single-handedly is beyond imagination. We present two computational approaches that transfer the style from one image (for example, a painting) to a whole video sequence. In our first approach, we adapt to videos the original image style transfer technique by Gatys et al. based on energy minimization. We introduce new ways of initialization and new loss functions to generate consistent and stable stylized video sequences even in cases with large motion and strong occlusion. Our second approach formulates video stylization as a learning problem. We propose a deep network architecture and training procedures that allow us to stylize arbitrary-length videos in a consistent and stable way, and nearly in real time. We show that the proposed methods clearly outperform simpler baselines both qualitatively and quantitatively. Finally, we propose a way to adapt these approaches also to 360∘\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$^\circ $$\end{document} images and videos as they emerge with recent virtual reality hardware.
    13 schema:genre article
    14 schema:isAccessibleForFree true
    15 schema:isPartOf N04d3ba10edf441a7bcbf47cb5595ce9f
    16 Nb6800e02981343d3a4f5e71fc7ef4a15
    17 sg:journal.1032807
    18 schema:keywords Gatys et al
    19 al
    20 approach
    21 architecture
    22 artistic style
    23 artistic style transfer
    24 artists
    25 baseline
    26 cases
    27 certain artistic style
    28 computational approach
    29 deep network architecture
    30 energy minimization
    31 et al
    32 first approach
    33 function
    34 hardware
    35 images
    36 imagination
    37 initialization
    38 large motion
    39 learning problem
    40 long time
    41 loss function
    42 method
    43 minimization
    44 motion
    45 network architecture
    46 new loss function
    47 new way
    48 occlusion
    49 problem
    50 procedure
    51 professional artists
    52 real time
    53 second approach
    54 sequence
    55 simple baseline
    56 spherical images
    57 stable way
    58 strong occlusions
    59 style
    60 style transfer
    61 style transfer techniques
    62 stylization
    63 technique
    64 time
    65 training procedure
    66 transfer
    67 transfer technique
    68 video
    69 video sequences
    70 virtual reality hardware
    71 way
    72 whole video sequence
    73 schema:name Artistic Style Transfer for Videos and Spherical Images
    74 schema:pagination 1199-1219
    75 schema:productId N24041e24dc0e4647ab5d5e2c289df796
    76 N8cd5f00f6c3c4d72a40acdb92dee1a0a
    77 schema:sameAs https://app.dimensions.ai/details/publication/pub.1103506006
    78 https://doi.org/10.1007/s11263-018-1089-z
    79 schema:sdDatePublished 2022-09-02T16:02
    80 schema:sdLicense https://scigraph.springernature.com/explorer/license/
    81 schema:sdPublisher Nbe638462b1784bb594555ad51b118c84
    82 schema:url https://doi.org/10.1007/s11263-018-1089-z
    83 sgo:license sg:explorer/license/
    84 sgo:sdDataset articles
    85 rdf:type schema:ScholarlyArticle
    86 N02959c22eb30450eb16e575c4b1b1bbb rdf:first sg:person.012443225372.65
    87 rdf:rest rdf:nil
    88 N04d3ba10edf441a7bcbf47cb5595ce9f schema:issueNumber 11
    89 rdf:type schema:PublicationIssue
    90 N24041e24dc0e4647ab5d5e2c289df796 schema:name dimensions_id
    91 schema:value pub.1103506006
    92 rdf:type schema:PropertyValue
    93 N468c91bc61c54c29aa2e45b8bcec0e1c rdf:first sg:person.011726376703.15
    94 rdf:rest N02959c22eb30450eb16e575c4b1b1bbb
    95 N7e02304396054bc4bc3509a4b4aec12d rdf:first sg:person.010615500061.50
    96 rdf:rest N468c91bc61c54c29aa2e45b8bcec0e1c
    97 N8cd5f00f6c3c4d72a40acdb92dee1a0a schema:name doi
    98 schema:value 10.1007/s11263-018-1089-z
    99 rdf:type schema:PropertyValue
    100 Nb6800e02981343d3a4f5e71fc7ef4a15 schema:volumeNumber 126
    101 rdf:type schema:PublicationVolume
    102 Nbe638462b1784bb594555ad51b118c84 schema:name Springer Nature - SN SciGraph project
    103 rdf:type schema:Organization
    104 anzsrc-for:08 schema:inDefinedTermSet anzsrc-for:
    105 schema:name Information and Computing Sciences
    106 rdf:type schema:DefinedTerm
    107 anzsrc-for:0801 schema:inDefinedTermSet anzsrc-for:
    108 schema:name Artificial Intelligence and Image Processing
    109 rdf:type schema:DefinedTerm
    110 sg:journal.1032807 schema:issn 0920-5691
    111 1573-1405
    112 schema:name International Journal of Computer Vision
    113 schema:publisher Springer Nature
    114 rdf:type schema:Periodical
    115 sg:person.010615500061.50 schema:affiliation grid-institutes:grid.5963.9
    116 schema:familyName Ruder
    117 schema:givenName Manuel
    118 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.010615500061.50
    119 rdf:type schema:Person
    120 sg:person.011726376703.15 schema:affiliation grid-institutes:grid.5963.9
    121 schema:familyName Dosovitskiy
    122 schema:givenName Alexey
    123 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.011726376703.15
    124 rdf:type schema:Person
    125 sg:person.012443225372.65 schema:affiliation grid-institutes:grid.5963.9
    126 schema:familyName Brox
    127 schema:givenName Thomas
    128 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.012443225372.65
    129 rdf:type schema:Person
    130 sg:pub.10.1007/978-3-319-10602-1_48 schema:sameAs https://app.dimensions.ai/details/publication/pub.1045321436
    131 https://doi.org/10.1007/978-3-319-10602-1_48
    132 rdf:type schema:CreativeWork
    133 sg:pub.10.1007/978-3-319-45886-1_3 schema:sameAs https://app.dimensions.ai/details/publication/pub.1044711914
    134 https://doi.org/10.1007/978-3-319-45886-1_3
    135 rdf:type schema:CreativeWork
    136 sg:pub.10.1007/978-3-319-46475-6_43 schema:sameAs https://app.dimensions.ai/details/publication/pub.1018034649
    137 https://doi.org/10.1007/978-3-319-46475-6_43
    138 rdf:type schema:CreativeWork
    139 sg:pub.10.1007/978-3-319-46487-9_43 schema:sameAs https://app.dimensions.ai/details/publication/pub.1000850552
    140 https://doi.org/10.1007/978-3-319-46487-9_43
    141 rdf:type schema:CreativeWork
    142 sg:pub.10.1007/978-3-642-15549-9_32 schema:sameAs https://app.dimensions.ai/details/publication/pub.1043037058
    143 https://doi.org/10.1007/978-3-642-15549-9_32
    144 rdf:type schema:CreativeWork
    145 sg:pub.10.1007/978-3-642-33783-3_44 schema:sameAs https://app.dimensions.ai/details/publication/pub.1004909083
    146 https://doi.org/10.1007/978-3-642-33783-3_44
    147 rdf:type schema:CreativeWork
    148 grid-institutes:grid.5963.9 schema:alternateName Department of Computer Science, BIOSS Centre for Biological Signalling Studies, University of Freiburg, Freiburg, Germany
    149 schema:name Department of Computer Science, BIOSS Centre for Biological Signalling Studies, University of Freiburg, Freiburg, Germany
    150 rdf:type schema:Organization
     




    Preview window. Press ESC to close (or click here)


    ...