Dynamic turbulence mitigation for long-range imaging in the presence of large moving objects View Full Text


Ontology type: schema:ScholarlyArticle      Open Access: True


Article Info

DATE

2019-12

AUTHORS

Robert Nieuwenhuizen, Judith Dijk, Klamer Schutte

ABSTRACT

Long-range imaging with visible or infrared observation systems is typically hampered by atmospheric turbulence. Software-based turbulence mitigation methods aim to stabilize and sharpen such recorded image sequences based on the image data only. Although successful restoration has been achieved on static scenes in the past, a significant challenge remains in accounting for moving objects such that they remain visible as moving objects in the output. Here, we investigate a new approach for turbulence mitigation on background as well as large moving objects under moderate turbulence conditions. In our method, we apply and compare different optical flow algorithms to locally estimate both the apparent and true object motion in image sequences and subsequently apply dynamic super-resolution, image sharpening, and newly developed local stabilization methods to the aligned images. We assess the use of these stabilization methods as well as a new method for occlusion compensation for these conditions. The proposed methods are qualitatively evaluated on several visible light recordings of real-world scenes. We demonstrate that our methods achieve a similar image quality on background elements as our prior methods for static scenes, but at the same time obtain a substantial improvement in image quality and reduction in image artifacts on moving objects. In addition, we show that our stabilization and occlusion compensation methods can be robustly used for turbulence mitigation in imagery featuring complex backgrounds and occlusion effects, without compromising the performance in less challenging conditions. More... »

PAGES

2

References to SciGraph publications

  • 2007-10. Real-time stabilization of long range observation system turbulent video in JOURNAL OF REAL-TIME IMAGE PROCESSING
  • 2008. Atmospheric Turbulence Restoration by Diffeomorphic Image Registration and Blind Deconvolution in ADVANCED CONCEPTS FOR INTELLIGENT VISION SYSTEMS
  • 2007. A Duality Based Approach for Realtime TV-L1 Optical Flow in PATTERN RECOGNITION
  • 2011-03. A Database and Evaluation Methodology for Optical Flow in INTERNATIONAL JOURNAL OF COMPUTER VISION
  • Identifiers

    URI

    http://scigraph.springernature.com/pub.10.1186/s13640-018-0380-9

    DOI

    http://dx.doi.org/10.1186/s13640-018-0380-9

    DIMENSIONS

    https://app.dimensions.ai/details/publication/pub.1111105176

    PUBMED

    https://www.ncbi.nlm.nih.gov/pubmed/30873210


    Indexing Status Check whether this publication has been indexed by Scopus and Web Of Science using the SN Indexing Status Tool
    Incoming Citations Browse incoming citations for this publication using opencitations.net

    JSON-LD is the canonical representation for SciGraph data.

    TIP: You can open this SciGraph record using an external JSON-LD service: JSON-LD Playground Google SDTT

    [
      {
        "@context": "https://springernature.github.io/scigraph/jsonld/sgcontext.json", 
        "about": [
          {
            "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/0801", 
            "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
            "name": "Artificial Intelligence and Image Processing", 
            "type": "DefinedTerm"
          }, 
          {
            "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/08", 
            "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
            "name": "Information and Computing Sciences", 
            "type": "DefinedTerm"
          }
        ], 
        "author": [
          {
            "affiliation": {
              "alternateName": "Netherlands Organisation for Applied Scientific Research", 
              "id": "https://www.grid.ac/institutes/grid.4858.1", 
              "name": [
                "TNO, PO Box 96864, 2509 JG, The Hague, The Netherlands"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Nieuwenhuizen", 
            "givenName": "Robert", 
            "id": "sg:person.01302047704.06", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01302047704.06"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "Netherlands Organisation for Applied Scientific Research", 
              "id": "https://www.grid.ac/institutes/grid.4858.1", 
              "name": [
                "TNO, PO Box 96864, 2509 JG, The Hague, The Netherlands"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Dijk", 
            "givenName": "Judith", 
            "id": "sg:person.013063441715.59", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.013063441715.59"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "Netherlands Organisation for Applied Scientific Research", 
              "id": "https://www.grid.ac/institutes/grid.4858.1", 
              "name": [
                "TNO, PO Box 96864, 2509 JG, The Hague, The Netherlands"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Schutte", 
            "givenName": "Klamer", 
            "id": "sg:person.011676611345.85", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.011676611345.85"
            ], 
            "type": "Person"
          }
        ], 
        "citation": [
          {
            "id": "sg:pub.10.1007/978-3-540-74936-3_22", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1016508230", 
              "https://doi.org/10.1007/978-3-540-74936-3_22"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-540-74936-3_22", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1016508230", 
              "https://doi.org/10.1007/978-3-540-74936-3_22"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1364/oe.23.005091", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1030246437"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-540-88458-3_36", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1033569074", 
              "https://doi.org/10.1007/978-3-540-88458-3_36"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/978-3-540-88458-3_36", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1033569074", 
              "https://doi.org/10.1007/978-3-540-88458-3_36"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/s11263-010-0390-2", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1034215603", 
              "https://doi.org/10.1007/s11263-010-0390-2"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1016/0004-3702(81)90024-2", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1039049535"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1016/j.sigpro.2008.10.012", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1040519414"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "sg:pub.10.1007/s11554-007-0037-x", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1046013605", 
              "https://doi.org/10.1007/s11554-007-0037-x"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/tpami.2012.82", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1061744413"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1109/tpami.2012.97", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1061744422"
            ], 
            "type": "CreativeWork"
          }, 
          {
            "id": "https://doi.org/10.1364/ao.53.001181", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1065130243"
            ], 
            "type": "CreativeWork"
          }
        ], 
        "datePublished": "2019-12", 
        "datePublishedReg": "2019-12-01", 
        "description": "Long-range imaging with visible or infrared observation systems is typically hampered by atmospheric turbulence. Software-based turbulence mitigation methods aim to stabilize and sharpen such recorded image sequences based on the image data only. Although successful restoration has been achieved on static scenes in the past, a significant challenge remains in accounting for moving objects such that they remain visible as moving objects in the output. Here, we investigate a new approach for turbulence mitigation on background as well as large moving objects under moderate turbulence conditions. In our method, we apply and compare different optical flow algorithms to locally estimate both the apparent and true object motion in image sequences and subsequently apply dynamic super-resolution, image sharpening, and newly developed local stabilization methods to the aligned images. We assess the use of these stabilization methods as well as a new method for occlusion compensation for these conditions. The proposed methods are qualitatively evaluated on several visible light recordings of real-world scenes. We demonstrate that our methods achieve a similar image quality on background elements as our prior methods for static scenes, but at the same time obtain a substantial improvement in image quality and reduction in image artifacts on moving objects. In addition, we show that our stabilization and occlusion compensation methods can be robustly used for turbulence mitigation in imagery featuring complex backgrounds and occlusion effects, without compromising the performance in less challenging conditions.", 
        "genre": "research_article", 
        "id": "sg:pub.10.1186/s13640-018-0380-9", 
        "inLanguage": [
          "en"
        ], 
        "isAccessibleForFree": true, 
        "isPartOf": [
          {
            "id": "sg:journal.1038561", 
            "issn": [
              "1687-5176", 
              "1687-5281"
            ], 
            "name": "EURASIP Journal on Image and Video Processing", 
            "type": "Periodical"
          }, 
          {
            "issueNumber": "1", 
            "type": "PublicationIssue"
          }, 
          {
            "type": "PublicationVolume", 
            "volumeNumber": "2019"
          }
        ], 
        "name": "Dynamic turbulence mitigation for long-range imaging in the presence of large moving objects", 
        "pagination": "2", 
        "productId": [
          {
            "name": "readcube_id", 
            "type": "PropertyValue", 
            "value": [
              "79d68ed7675675dc92dffbd265344dae3dfdbed620c7f87624c8478d57458f28"
            ]
          }, 
          {
            "name": "pubmed_id", 
            "type": "PropertyValue", 
            "value": [
              "30873210"
            ]
          }, 
          {
            "name": "nlm_unique_id", 
            "type": "PropertyValue", 
            "value": [
              "101304025"
            ]
          }, 
          {
            "name": "doi", 
            "type": "PropertyValue", 
            "value": [
              "10.1186/s13640-018-0380-9"
            ]
          }, 
          {
            "name": "dimensions_id", 
            "type": "PropertyValue", 
            "value": [
              "pub.1111105176"
            ]
          }
        ], 
        "sameAs": [
          "https://doi.org/10.1186/s13640-018-0380-9", 
          "https://app.dimensions.ai/details/publication/pub.1111105176"
        ], 
        "sdDataset": "articles", 
        "sdDatePublished": "2019-04-11T13:20", 
        "sdLicense": "https://scigraph.springernature.com/explorer/license/", 
        "sdPublisher": {
          "name": "Springer Nature - SN SciGraph project", 
          "type": "Organization"
        }, 
        "sdSource": "s3://com-uberresearch-data-dimensions-target-20181106-alternative/cleanup/v134/2549eaecd7973599484d7c17b260dba0a4ecb94b/merge/v9/a6c9fde33151104705d4d7ff012ea9563521a3ce/jats-lookup/v90/0000000368_0000000368/records_78964_00000001.jsonl", 
        "type": "ScholarlyArticle", 
        "url": "https://link.springer.com/10.1186%2Fs13640-018-0380-9"
      }
    ]
     

    Download the RDF metadata as:  json-ld nt turtle xml License info

    HOW TO GET THIS DATA PROGRAMMATICALLY:

    JSON-LD is a popular format for linked data which is fully compatible with JSON.

    curl -H 'Accept: application/ld+json' 'https://scigraph.springernature.com/pub.10.1186/s13640-018-0380-9'

    N-Triples is a line-based linked data format ideal for batch operations.

    curl -H 'Accept: application/n-triples' 'https://scigraph.springernature.com/pub.10.1186/s13640-018-0380-9'

    Turtle is a human-readable linked data format.

    curl -H 'Accept: text/turtle' 'https://scigraph.springernature.com/pub.10.1186/s13640-018-0380-9'

    RDF/XML is a standard XML format for linked data.

    curl -H 'Accept: application/rdf+xml' 'https://scigraph.springernature.com/pub.10.1186/s13640-018-0380-9'


     

    This table displays all metadata directly associated to this object as RDF triples.

    117 TRIPLES      21 PREDICATES      39 URIs      21 LITERALS      9 BLANK NODES

    Subject Predicate Object
    1 sg:pub.10.1186/s13640-018-0380-9 schema:about anzsrc-for:08
    2 anzsrc-for:0801
    3 schema:author N924e6805f4ef496f960ec6612a31cf5c
    4 schema:citation sg:pub.10.1007/978-3-540-74936-3_22
    5 sg:pub.10.1007/978-3-540-88458-3_36
    6 sg:pub.10.1007/s11263-010-0390-2
    7 sg:pub.10.1007/s11554-007-0037-x
    8 https://doi.org/10.1016/0004-3702(81)90024-2
    9 https://doi.org/10.1016/j.sigpro.2008.10.012
    10 https://doi.org/10.1109/tpami.2012.82
    11 https://doi.org/10.1109/tpami.2012.97
    12 https://doi.org/10.1364/ao.53.001181
    13 https://doi.org/10.1364/oe.23.005091
    14 schema:datePublished 2019-12
    15 schema:datePublishedReg 2019-12-01
    16 schema:description Long-range imaging with visible or infrared observation systems is typically hampered by atmospheric turbulence. Software-based turbulence mitigation methods aim to stabilize and sharpen such recorded image sequences based on the image data only. Although successful restoration has been achieved on static scenes in the past, a significant challenge remains in accounting for moving objects such that they remain visible as moving objects in the output. Here, we investigate a new approach for turbulence mitigation on background as well as large moving objects under moderate turbulence conditions. In our method, we apply and compare different optical flow algorithms to locally estimate both the apparent and true object motion in image sequences and subsequently apply dynamic super-resolution, image sharpening, and newly developed local stabilization methods to the aligned images. We assess the use of these stabilization methods as well as a new method for occlusion compensation for these conditions. The proposed methods are qualitatively evaluated on several visible light recordings of real-world scenes. We demonstrate that our methods achieve a similar image quality on background elements as our prior methods for static scenes, but at the same time obtain a substantial improvement in image quality and reduction in image artifacts on moving objects. In addition, we show that our stabilization and occlusion compensation methods can be robustly used for turbulence mitigation in imagery featuring complex backgrounds and occlusion effects, without compromising the performance in less challenging conditions.
    17 schema:genre research_article
    18 schema:inLanguage en
    19 schema:isAccessibleForFree true
    20 schema:isPartOf Nc6cae6af1ceb4fdc847f955799645e46
    21 Nd3dce5d4726c49c5b4fcf158995870ea
    22 sg:journal.1038561
    23 schema:name Dynamic turbulence mitigation for long-range imaging in the presence of large moving objects
    24 schema:pagination 2
    25 schema:productId N007228775971467eab5bd152f2a73549
    26 N0e90e87b5c7d495d879f9c7882d87000
    27 N760e38c3c592405bbe85f409cac0d1f6
    28 N801b54c2076946d1ab209771078bb489
    29 Nfe70e4898382459e8422904d68789000
    30 schema:sameAs https://app.dimensions.ai/details/publication/pub.1111105176
    31 https://doi.org/10.1186/s13640-018-0380-9
    32 schema:sdDatePublished 2019-04-11T13:20
    33 schema:sdLicense https://scigraph.springernature.com/explorer/license/
    34 schema:sdPublisher Nb1d2384486d3416e96cd17ccef7303fe
    35 schema:url https://link.springer.com/10.1186%2Fs13640-018-0380-9
    36 sgo:license sg:explorer/license/
    37 sgo:sdDataset articles
    38 rdf:type schema:ScholarlyArticle
    39 N007228775971467eab5bd152f2a73549 schema:name pubmed_id
    40 schema:value 30873210
    41 rdf:type schema:PropertyValue
    42 N0e90e87b5c7d495d879f9c7882d87000 schema:name doi
    43 schema:value 10.1186/s13640-018-0380-9
    44 rdf:type schema:PropertyValue
    45 N5fea9865878347f6acb8436397bc30b8 rdf:first sg:person.011676611345.85
    46 rdf:rest rdf:nil
    47 N760e38c3c592405bbe85f409cac0d1f6 schema:name readcube_id
    48 schema:value 79d68ed7675675dc92dffbd265344dae3dfdbed620c7f87624c8478d57458f28
    49 rdf:type schema:PropertyValue
    50 N801b54c2076946d1ab209771078bb489 schema:name nlm_unique_id
    51 schema:value 101304025
    52 rdf:type schema:PropertyValue
    53 N924e6805f4ef496f960ec6612a31cf5c rdf:first sg:person.01302047704.06
    54 rdf:rest Na2cd9595576344408c47b5849b0859e3
    55 Na2cd9595576344408c47b5849b0859e3 rdf:first sg:person.013063441715.59
    56 rdf:rest N5fea9865878347f6acb8436397bc30b8
    57 Nb1d2384486d3416e96cd17ccef7303fe schema:name Springer Nature - SN SciGraph project
    58 rdf:type schema:Organization
    59 Nc6cae6af1ceb4fdc847f955799645e46 schema:issueNumber 1
    60 rdf:type schema:PublicationIssue
    61 Nd3dce5d4726c49c5b4fcf158995870ea schema:volumeNumber 2019
    62 rdf:type schema:PublicationVolume
    63 Nfe70e4898382459e8422904d68789000 schema:name dimensions_id
    64 schema:value pub.1111105176
    65 rdf:type schema:PropertyValue
    66 anzsrc-for:08 schema:inDefinedTermSet anzsrc-for:
    67 schema:name Information and Computing Sciences
    68 rdf:type schema:DefinedTerm
    69 anzsrc-for:0801 schema:inDefinedTermSet anzsrc-for:
    70 schema:name Artificial Intelligence and Image Processing
    71 rdf:type schema:DefinedTerm
    72 sg:journal.1038561 schema:issn 1687-5176
    73 1687-5281
    74 schema:name EURASIP Journal on Image and Video Processing
    75 rdf:type schema:Periodical
    76 sg:person.011676611345.85 schema:affiliation https://www.grid.ac/institutes/grid.4858.1
    77 schema:familyName Schutte
    78 schema:givenName Klamer
    79 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.011676611345.85
    80 rdf:type schema:Person
    81 sg:person.01302047704.06 schema:affiliation https://www.grid.ac/institutes/grid.4858.1
    82 schema:familyName Nieuwenhuizen
    83 schema:givenName Robert
    84 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01302047704.06
    85 rdf:type schema:Person
    86 sg:person.013063441715.59 schema:affiliation https://www.grid.ac/institutes/grid.4858.1
    87 schema:familyName Dijk
    88 schema:givenName Judith
    89 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.013063441715.59
    90 rdf:type schema:Person
    91 sg:pub.10.1007/978-3-540-74936-3_22 schema:sameAs https://app.dimensions.ai/details/publication/pub.1016508230
    92 https://doi.org/10.1007/978-3-540-74936-3_22
    93 rdf:type schema:CreativeWork
    94 sg:pub.10.1007/978-3-540-88458-3_36 schema:sameAs https://app.dimensions.ai/details/publication/pub.1033569074
    95 https://doi.org/10.1007/978-3-540-88458-3_36
    96 rdf:type schema:CreativeWork
    97 sg:pub.10.1007/s11263-010-0390-2 schema:sameAs https://app.dimensions.ai/details/publication/pub.1034215603
    98 https://doi.org/10.1007/s11263-010-0390-2
    99 rdf:type schema:CreativeWork
    100 sg:pub.10.1007/s11554-007-0037-x schema:sameAs https://app.dimensions.ai/details/publication/pub.1046013605
    101 https://doi.org/10.1007/s11554-007-0037-x
    102 rdf:type schema:CreativeWork
    103 https://doi.org/10.1016/0004-3702(81)90024-2 schema:sameAs https://app.dimensions.ai/details/publication/pub.1039049535
    104 rdf:type schema:CreativeWork
    105 https://doi.org/10.1016/j.sigpro.2008.10.012 schema:sameAs https://app.dimensions.ai/details/publication/pub.1040519414
    106 rdf:type schema:CreativeWork
    107 https://doi.org/10.1109/tpami.2012.82 schema:sameAs https://app.dimensions.ai/details/publication/pub.1061744413
    108 rdf:type schema:CreativeWork
    109 https://doi.org/10.1109/tpami.2012.97 schema:sameAs https://app.dimensions.ai/details/publication/pub.1061744422
    110 rdf:type schema:CreativeWork
    111 https://doi.org/10.1364/ao.53.001181 schema:sameAs https://app.dimensions.ai/details/publication/pub.1065130243
    112 rdf:type schema:CreativeWork
    113 https://doi.org/10.1364/oe.23.005091 schema:sameAs https://app.dimensions.ai/details/publication/pub.1030246437
    114 rdf:type schema:CreativeWork
    115 https://www.grid.ac/institutes/grid.4858.1 schema:alternateName Netherlands Organisation for Applied Scientific Research
    116 schema:name TNO, PO Box 96864, 2509 JG, The Hague, The Netherlands
    117 rdf:type schema:Organization
     




    Preview window. Press ESC to close (or click here)


    ...