Swapped face detection using deep learning and subjective assessment View Full Text


Ontology type: schema:ScholarlyArticle      Open Access: True


Article Info

DATE

2020-05-19

AUTHORS

Xinyi Ding, Zohreh Raziei, Eric C. Larson, Eli V. Olinick, Paul Krueger, Michael Hahsler

ABSTRACT

The tremendous success of deep learning for imaging applications has resulted in numerous beneficial advances. Unfortunately, this success has also been a catalyst for malicious uses such as photo-realistic face swapping of parties without consent. In this study, we use deep transfer learning for face swapping detection, showing true positive rates greater than 96% with very few false alarms. Distinguished from existing methods that only provide detection accuracy, we also provide uncertainty for each prediction, which is critical for trust in the deployment of such detection systems. Moreover, we provide a comparison to human subjects. To capture human recognition performance, we build a website to collect pairwise comparisons of images from human subjects. Based on these comparisons, we infer a consensus ranking from the image perceived as most real to the image perceived as most fake. Overall, the results show the effectiveness of our method. As part of this study, we create a novel dataset that is, to the best of our knowledge, the largest swapped face dataset created using still images. This dataset will be available for academic research use per request. Our goal of this study is to inspire more research in the field of image forensics through the creation of a dataset and initial analysis. More... »

PAGES

6

References to SciGraph publications

  • 2006. SURF: Speeded Up Robust Features in COMPUTER VISION – ECCV 2006
  • Identifiers

    URI

    http://scigraph.springernature.com/pub.10.1186/s13635-020-00109-8

    DOI

    http://dx.doi.org/10.1186/s13635-020-00109-8

    DIMENSIONS

    https://app.dimensions.ai/details/publication/pub.1127728538


    Indexing Status Check whether this publication has been indexed by Scopus and Web Of Science using the SN Indexing Status Tool
    Incoming Citations Browse incoming citations for this publication using opencitations.net

    JSON-LD is the canonical representation for SciGraph data.

    TIP: You can open this SciGraph record using an external JSON-LD service: JSON-LD Playground Google SDTT

    [
      {
        "@context": "https://springernature.github.io/scigraph/jsonld/sgcontext.json", 
        "about": [
          {
            "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/08", 
            "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
            "name": "Information and Computing Sciences", 
            "type": "DefinedTerm"
          }, 
          {
            "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/0801", 
            "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
            "name": "Artificial Intelligence and Image Processing", 
            "type": "DefinedTerm"
          }
        ], 
        "author": [
          {
            "affiliation": {
              "alternateName": "Department of Computer Science, Southern Methodist University, Dallas, USA", 
              "id": "http://www.grid.ac/institutes/grid.263864.d", 
              "name": [
                "Department of Computer Science, Southern Methodist University, Dallas, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Ding", 
            "givenName": "Xinyi", 
            "id": "sg:person.015514754304.07", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.015514754304.07"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "Department of Engineering Management, Information and Systems, Southern Methodist University, Dallas, USA", 
              "id": "http://www.grid.ac/institutes/grid.263864.d", 
              "name": [
                "Department of Engineering Management, Information and Systems, Southern Methodist University, Dallas, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Raziei", 
            "givenName": "Zohreh", 
            "id": "sg:person.015101525556.59", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.015101525556.59"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "Department of Computer Science, Southern Methodist University, Dallas, USA", 
              "id": "http://www.grid.ac/institutes/grid.263864.d", 
              "name": [
                "Department of Computer Science, Southern Methodist University, Dallas, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Larson", 
            "givenName": "Eric C.", 
            "id": "sg:person.010654065201.24", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.010654065201.24"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "Department of Engineering Management, Information and Systems, Southern Methodist University, Dallas, USA", 
              "id": "http://www.grid.ac/institutes/grid.263864.d", 
              "name": [
                "Department of Engineering Management, Information and Systems, Southern Methodist University, Dallas, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Olinick", 
            "givenName": "Eli V.", 
            "id": "sg:person.015364427015.50", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.015364427015.50"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "Department of Mechanical Engineering, Southern Methodist University, Dallas, USA", 
              "id": "http://www.grid.ac/institutes/grid.263864.d", 
              "name": [
                "Department of Mechanical Engineering, Southern Methodist University, Dallas, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Krueger", 
            "givenName": "Paul", 
            "id": "sg:person.01352635635.73", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01352635635.73"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "Department of Engineering Management, Information and Systems, Southern Methodist University, Dallas, USA", 
              "id": "http://www.grid.ac/institutes/grid.263864.d", 
              "name": [
                "Department of Engineering Management, Information and Systems, Southern Methodist University, Dallas, USA"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Hahsler", 
            "givenName": "Michael", 
            "id": "sg:person.01343724027.36", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01343724027.36"
            ], 
            "type": "Person"
          }
        ], 
        "citation": [
          {
            "id": "sg:pub.10.1007/11744023_32", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1000158525", 
              "https://doi.org/10.1007/11744023_32"
            ], 
            "type": "CreativeWork"
          }
        ], 
        "datePublished": "2020-05-19", 
        "datePublishedReg": "2020-05-19", 
        "description": "The tremendous success of deep learning for imaging applications has resulted in numerous beneficial advances. Unfortunately, this success has also been a catalyst for malicious uses such as photo-realistic face swapping of parties without consent. In this study, we use deep transfer learning for face swapping detection, showing true positive rates greater than 96% with very few false alarms. Distinguished from existing methods that only provide detection accuracy, we also provide uncertainty for each prediction, which is critical for trust in the deployment of such detection systems. Moreover, we provide a comparison to human subjects. To capture human recognition performance, we build a website to collect pairwise comparisons of images from human subjects. Based on these comparisons, we infer a consensus ranking from the image perceived as most real to the image perceived as most fake. Overall, the results show the effectiveness of our method. As part of this study, we create a novel dataset that is, to the best of our knowledge, the largest swapped face dataset created using still images. This dataset will be available for academic research use per request. Our goal of this study is to inspire more research in the field of image forensics through the creation of a dataset and initial analysis.", 
        "genre": "article", 
        "id": "sg:pub.10.1186/s13635-020-00109-8", 
        "inLanguage": "en", 
        "isAccessibleForFree": true, 
        "isPartOf": [
          {
            "id": "sg:journal.1381262", 
            "issn": [
              "1687-4161", 
              "2510-523X"
            ], 
            "name": "EURASIP Journal on Information Security", 
            "publisher": "Springer Nature", 
            "type": "Periodical"
          }, 
          {
            "issueNumber": "1", 
            "type": "PublicationIssue"
          }, 
          {
            "type": "PublicationVolume", 
            "volumeNumber": "2020"
          }
        ], 
        "keywords": [
          "deep learning", 
          "human recognition performance", 
          "true positive rate", 
          "such detection systems", 
          "image forensics", 
          "deep transfer", 
          "academic research use", 
          "face datasets", 
          "detection accuracy", 
          "malicious uses", 
          "recognition performance", 
          "photo-realistic faces", 
          "detection system", 
          "tremendous success", 
          "false alarms", 
          "dataset", 
          "novel dataset", 
          "images", 
          "beneficial advances", 
          "learning", 
          "positive rate", 
          "imaging applications", 
          "forensics", 
          "deployment", 
          "requests", 
          "alarms", 
          "detection", 
          "websites", 
          "accuracy", 
          "trust", 
          "research use", 
          "subjective assessment", 
          "applications", 
          "creation", 
          "performance", 
          "method", 
          "effectiveness", 
          "initial analysis", 
          "system", 
          "goal", 
          "human subjects", 
          "face", 
          "success", 
          "pairwise comparisons", 
          "parties", 
          "knowledge", 
          "uncertainty", 
          "prediction", 
          "advances", 
          "research", 
          "comparison", 
          "use", 
          "field", 
          "part", 
          "results", 
          "consensus", 
          "analysis", 
          "more research", 
          "uses", 
          "assessment", 
          "transfer", 
          "study", 
          "rate", 
          "subjects", 
          "consent", 
          "catalyst", 
          "numerous beneficial advances"
        ], 
        "name": "Swapped face detection using deep learning and subjective assessment", 
        "pagination": "6", 
        "productId": [
          {
            "name": "dimensions_id", 
            "type": "PropertyValue", 
            "value": [
              "pub.1127728538"
            ]
          }, 
          {
            "name": "doi", 
            "type": "PropertyValue", 
            "value": [
              "10.1186/s13635-020-00109-8"
            ]
          }
        ], 
        "sameAs": [
          "https://doi.org/10.1186/s13635-020-00109-8", 
          "https://app.dimensions.ai/details/publication/pub.1127728538"
        ], 
        "sdDataset": "articles", 
        "sdDatePublished": "2022-01-01T18:57", 
        "sdLicense": "https://scigraph.springernature.com/explorer/license/", 
        "sdPublisher": {
          "name": "Springer Nature - SN SciGraph project", 
          "type": "Organization"
        }, 
        "sdSource": "s3://com-springernature-scigraph/baseset/20220101/entities/gbq_results/article/article_860.jsonl", 
        "type": "ScholarlyArticle", 
        "url": "https://doi.org/10.1186/s13635-020-00109-8"
      }
    ]
     

    Download the RDF metadata as:  json-ld nt turtle xml License info

    HOW TO GET THIS DATA PROGRAMMATICALLY:

    JSON-LD is a popular format for linked data which is fully compatible with JSON.

    curl -H 'Accept: application/ld+json' 'https://scigraph.springernature.com/pub.10.1186/s13635-020-00109-8'

    N-Triples is a line-based linked data format ideal for batch operations.

    curl -H 'Accept: application/n-triples' 'https://scigraph.springernature.com/pub.10.1186/s13635-020-00109-8'

    Turtle is a human-readable linked data format.

    curl -H 'Accept: text/turtle' 'https://scigraph.springernature.com/pub.10.1186/s13635-020-00109-8'

    RDF/XML is a standard XML format for linked data.

    curl -H 'Accept: application/rdf+xml' 'https://scigraph.springernature.com/pub.10.1186/s13635-020-00109-8'


     

    This table displays all metadata directly associated to this object as RDF triples.

    168 TRIPLES      22 PREDICATES      93 URIs      84 LITERALS      6 BLANK NODES

    Subject Predicate Object
    1 sg:pub.10.1186/s13635-020-00109-8 schema:about anzsrc-for:08
    2 anzsrc-for:0801
    3 schema:author Nd19cbdfa135a453e9ddbbde34180c3a4
    4 schema:citation sg:pub.10.1007/11744023_32
    5 schema:datePublished 2020-05-19
    6 schema:datePublishedReg 2020-05-19
    7 schema:description The tremendous success of deep learning for imaging applications has resulted in numerous beneficial advances. Unfortunately, this success has also been a catalyst for malicious uses such as photo-realistic face swapping of parties without consent. In this study, we use deep transfer learning for face swapping detection, showing true positive rates greater than 96% with very few false alarms. Distinguished from existing methods that only provide detection accuracy, we also provide uncertainty for each prediction, which is critical for trust in the deployment of such detection systems. Moreover, we provide a comparison to human subjects. To capture human recognition performance, we build a website to collect pairwise comparisons of images from human subjects. Based on these comparisons, we infer a consensus ranking from the image perceived as most real to the image perceived as most fake. Overall, the results show the effectiveness of our method. As part of this study, we create a novel dataset that is, to the best of our knowledge, the largest swapped face dataset created using still images. This dataset will be available for academic research use per request. Our goal of this study is to inspire more research in the field of image forensics through the creation of a dataset and initial analysis.
    8 schema:genre article
    9 schema:inLanguage en
    10 schema:isAccessibleForFree true
    11 schema:isPartOf N86fe84e1fa8e43da8049c6c7fb51e45b
    12 Nb89dbbd540d74d67a55bdc3ec448004e
    13 sg:journal.1381262
    14 schema:keywords academic research use
    15 accuracy
    16 advances
    17 alarms
    18 analysis
    19 applications
    20 assessment
    21 beneficial advances
    22 catalyst
    23 comparison
    24 consensus
    25 consent
    26 creation
    27 dataset
    28 deep learning
    29 deep transfer
    30 deployment
    31 detection
    32 detection accuracy
    33 detection system
    34 effectiveness
    35 face
    36 face datasets
    37 false alarms
    38 field
    39 forensics
    40 goal
    41 human recognition performance
    42 human subjects
    43 image forensics
    44 images
    45 imaging applications
    46 initial analysis
    47 knowledge
    48 learning
    49 malicious uses
    50 method
    51 more research
    52 novel dataset
    53 numerous beneficial advances
    54 pairwise comparisons
    55 part
    56 parties
    57 performance
    58 photo-realistic faces
    59 positive rate
    60 prediction
    61 rate
    62 recognition performance
    63 requests
    64 research
    65 research use
    66 results
    67 study
    68 subjective assessment
    69 subjects
    70 success
    71 such detection systems
    72 system
    73 transfer
    74 tremendous success
    75 true positive rate
    76 trust
    77 uncertainty
    78 use
    79 uses
    80 websites
    81 schema:name Swapped face detection using deep learning and subjective assessment
    82 schema:pagination 6
    83 schema:productId N11cbb8e957e045bd8034903199466a2b
    84 Ndc6139ee1cb74ee7a4ed27e4421300dc
    85 schema:sameAs https://app.dimensions.ai/details/publication/pub.1127728538
    86 https://doi.org/10.1186/s13635-020-00109-8
    87 schema:sdDatePublished 2022-01-01T18:57
    88 schema:sdLicense https://scigraph.springernature.com/explorer/license/
    89 schema:sdPublisher N07ed3364486f452f8d64a369d162193d
    90 schema:url https://doi.org/10.1186/s13635-020-00109-8
    91 sgo:license sg:explorer/license/
    92 sgo:sdDataset articles
    93 rdf:type schema:ScholarlyArticle
    94 N00520510c35e4672897cf89ce293b699 rdf:first sg:person.01343724027.36
    95 rdf:rest rdf:nil
    96 N05e9d9ae369c44b19570320168b7d43f rdf:first sg:person.015364427015.50
    97 rdf:rest Ndd5c20826380458c90c7e2c63ffe8a4b
    98 N07ed3364486f452f8d64a369d162193d schema:name Springer Nature - SN SciGraph project
    99 rdf:type schema:Organization
    100 N11cbb8e957e045bd8034903199466a2b schema:name dimensions_id
    101 schema:value pub.1127728538
    102 rdf:type schema:PropertyValue
    103 N70af841131214138a520ee7062eb9ee1 rdf:first sg:person.015101525556.59
    104 rdf:rest N8ce87c19846e45c9ac60f8344689700b
    105 N86fe84e1fa8e43da8049c6c7fb51e45b schema:issueNumber 1
    106 rdf:type schema:PublicationIssue
    107 N8ce87c19846e45c9ac60f8344689700b rdf:first sg:person.010654065201.24
    108 rdf:rest N05e9d9ae369c44b19570320168b7d43f
    109 Nb89dbbd540d74d67a55bdc3ec448004e schema:volumeNumber 2020
    110 rdf:type schema:PublicationVolume
    111 Nd19cbdfa135a453e9ddbbde34180c3a4 rdf:first sg:person.015514754304.07
    112 rdf:rest N70af841131214138a520ee7062eb9ee1
    113 Ndc6139ee1cb74ee7a4ed27e4421300dc schema:name doi
    114 schema:value 10.1186/s13635-020-00109-8
    115 rdf:type schema:PropertyValue
    116 Ndd5c20826380458c90c7e2c63ffe8a4b rdf:first sg:person.01352635635.73
    117 rdf:rest N00520510c35e4672897cf89ce293b699
    118 anzsrc-for:08 schema:inDefinedTermSet anzsrc-for:
    119 schema:name Information and Computing Sciences
    120 rdf:type schema:DefinedTerm
    121 anzsrc-for:0801 schema:inDefinedTermSet anzsrc-for:
    122 schema:name Artificial Intelligence and Image Processing
    123 rdf:type schema:DefinedTerm
    124 sg:journal.1381262 schema:issn 1687-4161
    125 2510-523X
    126 schema:name EURASIP Journal on Information Security
    127 schema:publisher Springer Nature
    128 rdf:type schema:Periodical
    129 sg:person.010654065201.24 schema:affiliation grid-institutes:grid.263864.d
    130 schema:familyName Larson
    131 schema:givenName Eric C.
    132 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.010654065201.24
    133 rdf:type schema:Person
    134 sg:person.01343724027.36 schema:affiliation grid-institutes:grid.263864.d
    135 schema:familyName Hahsler
    136 schema:givenName Michael
    137 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01343724027.36
    138 rdf:type schema:Person
    139 sg:person.01352635635.73 schema:affiliation grid-institutes:grid.263864.d
    140 schema:familyName Krueger
    141 schema:givenName Paul
    142 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01352635635.73
    143 rdf:type schema:Person
    144 sg:person.015101525556.59 schema:affiliation grid-institutes:grid.263864.d
    145 schema:familyName Raziei
    146 schema:givenName Zohreh
    147 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.015101525556.59
    148 rdf:type schema:Person
    149 sg:person.015364427015.50 schema:affiliation grid-institutes:grid.263864.d
    150 schema:familyName Olinick
    151 schema:givenName Eli V.
    152 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.015364427015.50
    153 rdf:type schema:Person
    154 sg:person.015514754304.07 schema:affiliation grid-institutes:grid.263864.d
    155 schema:familyName Ding
    156 schema:givenName Xinyi
    157 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.015514754304.07
    158 rdf:type schema:Person
    159 sg:pub.10.1007/11744023_32 schema:sameAs https://app.dimensions.ai/details/publication/pub.1000158525
    160 https://doi.org/10.1007/11744023_32
    161 rdf:type schema:CreativeWork
    162 grid-institutes:grid.263864.d schema:alternateName Department of Computer Science, Southern Methodist University, Dallas, USA
    163 Department of Engineering Management, Information and Systems, Southern Methodist University, Dallas, USA
    164 Department of Mechanical Engineering, Southern Methodist University, Dallas, USA
    165 schema:name Department of Computer Science, Southern Methodist University, Dallas, USA
    166 Department of Engineering Management, Information and Systems, Southern Methodist University, Dallas, USA
    167 Department of Mechanical Engineering, Southern Methodist University, Dallas, USA
    168 rdf:type schema:Organization
     




    Preview window. Press ESC to close (or click here)


    ...