Feature-Guided Black-Box Safety Testing of Deep Neural Networks View Full Text


Ontology type: schema:Chapter      Open Access: True


Chapter Info

DATE

2018-04-12

AUTHORS

Matthew Wicker , Xiaowei Huang , Marta Kwiatkowska

ABSTRACT

Despite the improved accuracy of deep neural networks, the discovery of adversarial examples has raised serious safety concerns. Most existing approaches for crafting adversarial examples necessitate some knowledge (architecture, parameters, etc) of the network at hand. In this paper, we focus on image classifiers and propose a feature-guided black-box approach to test the safety of deep neural networks that requires no such knowledge. Our algorithm employs object detection techniques such as SIFT (Scale Invariant Feature Transform) to extract features from an image. These features are converted into a mutable saliency distribution, where high probability is assigned to pixels that affect the composition of the image with respect to the human visual system. We formulate the crafting of adversarial examples as a two-player turn-based stochastic game, where the first player’s objective is to minimise the distance to an adversarial example by manipulating the features, and the second player can be cooperative, adversarial, or random. We show that, theoretically, the two-player game can converge to the optimal strategy, and that the optimal strategy represents a globally minimal adversarial image. For Lipschitz networks, we also identify conditions that provide safety guarantees that no adversarial examples exist. Using Monte Carlo tree search we gradually explore the game state space to search for adversarial examples. Our experiments show that, despite the black-box setting, manipulations guided by a perception-based saliency distribution are competitive with state-of-the-art methods that rely on white-box saliency matrices or sophisticated optimization procedures. Finally, we show how our method can be used to evaluate robustness of neural networks in safety-critical applications such as traffic sign recognition in self-driving cars. More... »

PAGES

408-426

Book

TITLE

Tools and Algorithms for the Construction and Analysis of Systems

ISBN

978-3-319-89959-6
978-3-319-89960-2

Identifiers

URI

http://scigraph.springernature.com/pub.10.1007/978-3-319-89960-2_22

DOI

http://dx.doi.org/10.1007/978-3-319-89960-2_22

DIMENSIONS

https://app.dimensions.ai/details/publication/pub.1103225695


Indexing Status Check whether this publication has been indexed by Scopus and Web Of Science using the SN Indexing Status Tool
Incoming Citations Browse incoming citations for this publication using opencitations.net

JSON-LD is the canonical representation for SciGraph data.

TIP: You can open this SciGraph record using an external JSON-LD service: JSON-LD Playground Google SDTT

[
  {
    "@context": "https://springernature.github.io/scigraph/jsonld/sgcontext.json", 
    "about": [
      {
        "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/08", 
        "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
        "name": "Information and Computing Sciences", 
        "type": "DefinedTerm"
      }, 
      {
        "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/0801", 
        "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
        "name": "Artificial Intelligence and Image Processing", 
        "type": "DefinedTerm"
      }
    ], 
    "author": [
      {
        "affiliation": {
          "alternateName": "University of Georgia, Athens, USA", 
          "id": "http://www.grid.ac/institutes/grid.213876.9", 
          "name": [
            "University of Georgia, Athens, USA"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Wicker", 
        "givenName": "Matthew", 
        "id": "sg:person.012500504516.80", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.012500504516.80"
        ], 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "University of Liverpool, Liverpool, UK", 
          "id": "http://www.grid.ac/institutes/grid.10025.36", 
          "name": [
            "University of Liverpool, Liverpool, UK"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Huang", 
        "givenName": "Xiaowei", 
        "id": "sg:person.013276065116.65", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.013276065116.65"
        ], 
        "type": "Person"
      }, 
      {
        "affiliation": {
          "alternateName": "University of Oxford, Oxford, UK", 
          "id": "http://www.grid.ac/institutes/grid.4991.5", 
          "name": [
            "University of Oxford, Oxford, UK"
          ], 
          "type": "Organization"
        }, 
        "familyName": "Kwiatkowska", 
        "givenName": "Marta", 
        "id": "sg:person.011375012273.39", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.011375012273.39"
        ], 
        "type": "Person"
      }
    ], 
    "datePublished": "2018-04-12", 
    "datePublishedReg": "2018-04-12", 
    "description": "Despite the improved accuracy of deep neural networks, the discovery of adversarial examples has raised serious safety concerns. Most existing approaches for crafting adversarial examples necessitate some knowledge (architecture, parameters, etc) of the network at hand. In this paper, we focus on image classifiers and propose a feature-guided black-box approach to test the safety of deep neural networks that requires no such knowledge. Our algorithm employs object detection techniques such as SIFT (Scale Invariant Feature Transform) to extract features from an image. These features are converted into a mutable saliency distribution, where high probability is assigned to pixels that affect the composition of the image with respect to the human visual system. We formulate the crafting of adversarial examples as a two-player turn-based stochastic game, where the first player\u2019s objective is to minimise the distance to an adversarial example by manipulating the features, and the second player can be cooperative, adversarial, or random. We show that, theoretically, the two-player game can converge to the optimal strategy, and that the optimal strategy represents a globally minimal adversarial image. For Lipschitz networks, we also identify conditions that provide safety guarantees that no adversarial examples exist. Using Monte Carlo tree search we gradually explore the game state space to search for adversarial examples. Our experiments show that, despite the black-box setting, manipulations guided by a perception-based saliency distribution are competitive with state-of-the-art methods that rely on white-box saliency matrices or sophisticated optimization procedures. Finally, we show how our method can be used to evaluate robustness of neural networks in safety-critical applications such as traffic sign recognition in self-driving cars.", 
    "editor": [
      {
        "familyName": "Beyer", 
        "givenName": "Dirk", 
        "type": "Person"
      }, 
      {
        "familyName": "Huisman", 
        "givenName": "Marieke", 
        "type": "Person"
      }
    ], 
    "genre": "chapter", 
    "id": "sg:pub.10.1007/978-3-319-89960-2_22", 
    "inLanguage": "en", 
    "isAccessibleForFree": true, 
    "isPartOf": {
      "isbn": [
        "978-3-319-89959-6", 
        "978-3-319-89960-2"
      ], 
      "name": "Tools and Algorithms for the Construction and Analysis of Systems", 
      "type": "Book"
    }, 
    "keywords": [
      "deep neural networks", 
      "adversarial examples", 
      "neural network", 
      "saliency distribution", 
      "safety-critical applications", 
      "self-driving cars", 
      "traffic sign recognition", 
      "Monte Carlo Tree Search", 
      "black-box approach", 
      "black-box setting", 
      "Carlo Tree Search", 
      "human visual system", 
      "adversarial images", 
      "sophisticated optimization procedures", 
      "image classifier", 
      "sign recognition", 
      "two-player games", 
      "art methods", 
      "tree search", 
      "safety guarantees", 
      "turn-based stochastic games", 
      "player's objective", 
      "detection techniques", 
      "network", 
      "stochastic games", 
      "state space", 
      "images", 
      "optimal strategy", 
      "improved accuracy", 
      "visual system", 
      "game", 
      "SIFT", 
      "optimization procedure", 
      "classifier", 
      "such knowledge", 
      "algorithm", 
      "pixels", 
      "guarantees", 
      "features", 
      "example", 
      "robustness", 
      "high probability", 
      "recognition", 
      "accuracy", 
      "search", 
      "knowledge", 
      "applications", 
      "car", 
      "method", 
      "second player", 
      "system", 
      "space", 
      "technique", 
      "strategies", 
      "objective", 
      "discovery", 
      "players", 
      "probability", 
      "experiments", 
      "manipulation", 
      "distance", 
      "hand", 
      "testing", 
      "crafting", 
      "concern", 
      "state", 
      "matrix", 
      "safety", 
      "respect", 
      "setting", 
      "procedure", 
      "safety testing", 
      "distribution", 
      "serious safety concerns", 
      "safety concerns", 
      "conditions", 
      "composition", 
      "approach", 
      "paper", 
      "feature-guided black-box approach", 
      "mutable saliency distribution", 
      "two-player turn-based stochastic game", 
      "first player\u2019s objective", 
      "minimal adversarial image", 
      "Lipschitz networks", 
      "game state space", 
      "perception-based saliency distribution", 
      "white-box saliency matrices", 
      "saliency matrices", 
      "Feature-Guided Black-Box Safety Testing", 
      "Black-Box Safety Testing"
    ], 
    "name": "Feature-Guided Black-Box Safety Testing of Deep Neural Networks", 
    "pagination": "408-426", 
    "productId": [
      {
        "name": "dimensions_id", 
        "type": "PropertyValue", 
        "value": [
          "pub.1103225695"
        ]
      }, 
      {
        "name": "doi", 
        "type": "PropertyValue", 
        "value": [
          "10.1007/978-3-319-89960-2_22"
        ]
      }
    ], 
    "publisher": {
      "name": "Springer Nature", 
      "type": "Organisation"
    }, 
    "sameAs": [
      "https://doi.org/10.1007/978-3-319-89960-2_22", 
      "https://app.dimensions.ai/details/publication/pub.1103225695"
    ], 
    "sdDataset": "chapters", 
    "sdDatePublished": "2021-11-01T18:46", 
    "sdLicense": "https://scigraph.springernature.com/explorer/license/", 
    "sdPublisher": {
      "name": "Springer Nature - SN SciGraph project", 
      "type": "Organization"
    }, 
    "sdSource": "s3://com-springernature-scigraph/baseset/20211101/entities/gbq_results/chapter/chapter_125.jsonl", 
    "type": "Chapter", 
    "url": "https://doi.org/10.1007/978-3-319-89960-2_22"
  }
]
 

Download the RDF metadata as:  json-ld nt turtle xml License info

HOW TO GET THIS DATA PROGRAMMATICALLY:

JSON-LD is a popular format for linked data which is fully compatible with JSON.

curl -H 'Accept: application/ld+json' 'https://scigraph.springernature.com/pub.10.1007/978-3-319-89960-2_22'

N-Triples is a line-based linked data format ideal for batch operations.

curl -H 'Accept: application/n-triples' 'https://scigraph.springernature.com/pub.10.1007/978-3-319-89960-2_22'

Turtle is a human-readable linked data format.

curl -H 'Accept: text/turtle' 'https://scigraph.springernature.com/pub.10.1007/978-3-319-89960-2_22'

RDF/XML is a standard XML format for linked data.

curl -H 'Accept: application/rdf+xml' 'https://scigraph.springernature.com/pub.10.1007/978-3-319-89960-2_22'


 

This table displays all metadata directly associated to this object as RDF triples.

176 TRIPLES      23 PREDICATES      116 URIs      109 LITERALS      7 BLANK NODES

Subject Predicate Object
1 sg:pub.10.1007/978-3-319-89960-2_22 schema:about anzsrc-for:08
2 anzsrc-for:0801
3 schema:author N56632d45da2445e693b8faa57845289b
4 schema:datePublished 2018-04-12
5 schema:datePublishedReg 2018-04-12
6 schema:description Despite the improved accuracy of deep neural networks, the discovery of adversarial examples has raised serious safety concerns. Most existing approaches for crafting adversarial examples necessitate some knowledge (architecture, parameters, etc) of the network at hand. In this paper, we focus on image classifiers and propose a feature-guided black-box approach to test the safety of deep neural networks that requires no such knowledge. Our algorithm employs object detection techniques such as SIFT (Scale Invariant Feature Transform) to extract features from an image. These features are converted into a mutable saliency distribution, where high probability is assigned to pixels that affect the composition of the image with respect to the human visual system. We formulate the crafting of adversarial examples as a two-player turn-based stochastic game, where the first player’s objective is to minimise the distance to an adversarial example by manipulating the features, and the second player can be cooperative, adversarial, or random. We show that, theoretically, the two-player game can converge to the optimal strategy, and that the optimal strategy represents a globally minimal adversarial image. For Lipschitz networks, we also identify conditions that provide safety guarantees that no adversarial examples exist. Using Monte Carlo tree search we gradually explore the game state space to search for adversarial examples. Our experiments show that, despite the black-box setting, manipulations guided by a perception-based saliency distribution are competitive with state-of-the-art methods that rely on white-box saliency matrices or sophisticated optimization procedures. Finally, we show how our method can be used to evaluate robustness of neural networks in safety-critical applications such as traffic sign recognition in self-driving cars.
7 schema:editor Nec1a9f74094743f09830b66054374b7f
8 schema:genre chapter
9 schema:inLanguage en
10 schema:isAccessibleForFree true
11 schema:isPartOf Ne173b3a51fd94cbca26e273169d09b0c
12 schema:keywords Black-Box Safety Testing
13 Carlo Tree Search
14 Feature-Guided Black-Box Safety Testing
15 Lipschitz networks
16 Monte Carlo Tree Search
17 SIFT
18 accuracy
19 adversarial examples
20 adversarial images
21 algorithm
22 applications
23 approach
24 art methods
25 black-box approach
26 black-box setting
27 car
28 classifier
29 composition
30 concern
31 conditions
32 crafting
33 deep neural networks
34 detection techniques
35 discovery
36 distance
37 distribution
38 example
39 experiments
40 feature-guided black-box approach
41 features
42 first player’s objective
43 game
44 game state space
45 guarantees
46 hand
47 high probability
48 human visual system
49 image classifier
50 images
51 improved accuracy
52 knowledge
53 manipulation
54 matrix
55 method
56 minimal adversarial image
57 mutable saliency distribution
58 network
59 neural network
60 objective
61 optimal strategy
62 optimization procedure
63 paper
64 perception-based saliency distribution
65 pixels
66 player's objective
67 players
68 probability
69 procedure
70 recognition
71 respect
72 robustness
73 safety
74 safety concerns
75 safety guarantees
76 safety testing
77 safety-critical applications
78 saliency distribution
79 saliency matrices
80 search
81 second player
82 self-driving cars
83 serious safety concerns
84 setting
85 sign recognition
86 sophisticated optimization procedures
87 space
88 state
89 state space
90 stochastic games
91 strategies
92 such knowledge
93 system
94 technique
95 testing
96 traffic sign recognition
97 tree search
98 turn-based stochastic games
99 two-player games
100 two-player turn-based stochastic game
101 visual system
102 white-box saliency matrices
103 schema:name Feature-Guided Black-Box Safety Testing of Deep Neural Networks
104 schema:pagination 408-426
105 schema:productId N588c5df03e1c4e369639d5bb5429da59
106 Na8872d545cae419285cc7e3ea5043b90
107 schema:publisher Nef370dec49b6415883db6ec487894663
108 schema:sameAs https://app.dimensions.ai/details/publication/pub.1103225695
109 https://doi.org/10.1007/978-3-319-89960-2_22
110 schema:sdDatePublished 2021-11-01T18:46
111 schema:sdLicense https://scigraph.springernature.com/explorer/license/
112 schema:sdPublisher N4f10e6c2162b4c949e561f0989c2fc94
113 schema:url https://doi.org/10.1007/978-3-319-89960-2_22
114 sgo:license sg:explorer/license/
115 sgo:sdDataset chapters
116 rdf:type schema:Chapter
117 N0639cf02a05c4231822097550a985038 schema:familyName Huisman
118 schema:givenName Marieke
119 rdf:type schema:Person
120 N4f10e6c2162b4c949e561f0989c2fc94 schema:name Springer Nature - SN SciGraph project
121 rdf:type schema:Organization
122 N56632d45da2445e693b8faa57845289b rdf:first sg:person.012500504516.80
123 rdf:rest N6f18211bd865479291608e7b52b998f2
124 N588c5df03e1c4e369639d5bb5429da59 schema:name dimensions_id
125 schema:value pub.1103225695
126 rdf:type schema:PropertyValue
127 N6f18211bd865479291608e7b52b998f2 rdf:first sg:person.013276065116.65
128 rdf:rest Nc9c53f136c4f463e90a4582e8223eddd
129 N8478ae4a4f5841e49714a8f0c07781db rdf:first N0639cf02a05c4231822097550a985038
130 rdf:rest rdf:nil
131 Na8872d545cae419285cc7e3ea5043b90 schema:name doi
132 schema:value 10.1007/978-3-319-89960-2_22
133 rdf:type schema:PropertyValue
134 Nc3a9c74d74a44e0b8aa28dd339fd3b31 schema:familyName Beyer
135 schema:givenName Dirk
136 rdf:type schema:Person
137 Nc9c53f136c4f463e90a4582e8223eddd rdf:first sg:person.011375012273.39
138 rdf:rest rdf:nil
139 Ne173b3a51fd94cbca26e273169d09b0c schema:isbn 978-3-319-89959-6
140 978-3-319-89960-2
141 schema:name Tools and Algorithms for the Construction and Analysis of Systems
142 rdf:type schema:Book
143 Nec1a9f74094743f09830b66054374b7f rdf:first Nc3a9c74d74a44e0b8aa28dd339fd3b31
144 rdf:rest N8478ae4a4f5841e49714a8f0c07781db
145 Nef370dec49b6415883db6ec487894663 schema:name Springer Nature
146 rdf:type schema:Organisation
147 anzsrc-for:08 schema:inDefinedTermSet anzsrc-for:
148 schema:name Information and Computing Sciences
149 rdf:type schema:DefinedTerm
150 anzsrc-for:0801 schema:inDefinedTermSet anzsrc-for:
151 schema:name Artificial Intelligence and Image Processing
152 rdf:type schema:DefinedTerm
153 sg:person.011375012273.39 schema:affiliation grid-institutes:grid.4991.5
154 schema:familyName Kwiatkowska
155 schema:givenName Marta
156 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.011375012273.39
157 rdf:type schema:Person
158 sg:person.012500504516.80 schema:affiliation grid-institutes:grid.213876.9
159 schema:familyName Wicker
160 schema:givenName Matthew
161 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.012500504516.80
162 rdf:type schema:Person
163 sg:person.013276065116.65 schema:affiliation grid-institutes:grid.10025.36
164 schema:familyName Huang
165 schema:givenName Xiaowei
166 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.013276065116.65
167 rdf:type schema:Person
168 grid-institutes:grid.10025.36 schema:alternateName University of Liverpool, Liverpool, UK
169 schema:name University of Liverpool, Liverpool, UK
170 rdf:type schema:Organization
171 grid-institutes:grid.213876.9 schema:alternateName University of Georgia, Athens, USA
172 schema:name University of Georgia, Athens, USA
173 rdf:type schema:Organization
174 grid-institutes:grid.4991.5 schema:alternateName University of Oxford, Oxford, UK
175 schema:name University of Oxford, Oxford, UK
176 rdf:type schema:Organization
 




Preview window. Press ESC to close (or click here)


...