Information-based evaluation criterion for classifier's performance View Full Text


Ontology type: schema:ScholarlyArticle      Open Access: True


Article Info

DATE

1991-01

AUTHORS

Igor Kononenko, Ivan Bratko

ABSTRACT

In the past few years many systems for learning decision rules from examples were developed. As different systems allow different types of answers when classifying new instances, it is difficult to appropriately evaluate the systems' classification power in comparison with other classification systems or in comparison with human experts. Classification accuracy is usually used as a measure of classification performance. This measure is, however, known to have several defects. A fair evaluation criterion should exclude the influence of the class probabilities which may enable a completely uninformed classifier to trivially achieve high classification accuracy. In this paper a method for evaluating the information score of a classifier's answers is proposed. It excludes the influence of prior probabilities, deals with various types of imperfect or probabilistic answers and can be used also for comparing the performance in different domains. More... »

PAGES

67-80

References to SciGraph publications

  • 1986-03. Induction of decision trees in MACHINE LEARNING
  • Identifiers

    URI

    http://scigraph.springernature.com/pub.10.1007/bf00153760

    DOI

    http://dx.doi.org/10.1007/bf00153760

    DIMENSIONS

    https://app.dimensions.ai/details/publication/pub.1036513123


    Indexing Status Check whether this publication has been indexed by Scopus and Web Of Science using the SN Indexing Status Tool
    Incoming Citations Browse incoming citations for this publication using opencitations.net

    JSON-LD is the canonical representation for SciGraph data.

    TIP: You can open this SciGraph record using an external JSON-LD service: JSON-LD Playground Google SDTT

    [
      {
        "@context": "https://springernature.github.io/scigraph/jsonld/sgcontext.json", 
        "about": [
          {
            "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/08", 
            "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
            "name": "Information and Computing Sciences", 
            "type": "DefinedTerm"
          }, 
          {
            "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/0801", 
            "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
            "name": "Artificial Intelligence and Image Processing", 
            "type": "DefinedTerm"
          }
        ], 
        "author": [
          {
            "affiliation": {
              "alternateName": "Faculty of Electrical and Computer Engineering, Trzaska 25, Ljubljana, Yugoslavia", 
              "id": "http://www.grid.ac/institutes/None", 
              "name": [
                "Faculty of Electrical and Computer Engineering, Trzaska 25, Ljubljana, Yugoslavia"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Kononenko", 
            "givenName": "Igor", 
            "id": "sg:person.015561447361.41", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.015561447361.41"
            ], 
            "type": "Person"
          }, 
          {
            "affiliation": {
              "alternateName": "Stefan Institute, Jamova 39, Ljubljana, Yugoslavia", 
              "id": "http://www.grid.ac/institutes/None", 
              "name": [
                "Faculty of Electrical and Computer Engineering, Trzaska 25, Ljubljana, Yugoslavia", 
                "Stefan Institute, Jamova 39, Ljubljana, Yugoslavia"
              ], 
              "type": "Organization"
            }, 
            "familyName": "Bratko", 
            "givenName": "Ivan", 
            "id": "sg:person.01113747300.33", 
            "sameAs": [
              "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01113747300.33"
            ], 
            "type": "Person"
          }
        ], 
        "citation": [
          {
            "id": "sg:pub.10.1007/bf00116251", 
            "sameAs": [
              "https://app.dimensions.ai/details/publication/pub.1019422208", 
              "https://doi.org/10.1007/bf00116251"
            ], 
            "type": "CreativeWork"
          }
        ], 
        "datePublished": "1991-01", 
        "datePublishedReg": "1991-01-01", 
        "description": "In the past few years many systems for learning decision rules from examples were developed. As different systems allow different types of answers when classifying new instances, it is difficult to appropriately evaluate the systems' classification power in comparison with other classification systems or in comparison with human experts. Classification accuracy is usually used as a measure of classification performance. This measure is, however, known to have several defects. A fair evaluation criterion should exclude the influence of the class probabilities which may enable a completely uninformed classifier to trivially achieve high classification accuracy. In this paper a method for evaluating the information score of a classifier's answers is proposed. It excludes the influence of prior probabilities, deals with various types of imperfect or probabilistic answers and can be used also for comparing the performance in different domains.", 
        "genre": "article", 
        "id": "sg:pub.10.1007/bf00153760", 
        "inLanguage": "en", 
        "isAccessibleForFree": true, 
        "isPartOf": [
          {
            "id": "sg:journal.1125588", 
            "issn": [
              "0885-6125", 
              "1573-0565"
            ], 
            "name": "Machine Learning", 
            "publisher": "Springer Nature", 
            "type": "Periodical"
          }, 
          {
            "issueNumber": "1", 
            "type": "PublicationIssue"
          }, 
          {
            "type": "PublicationVolume", 
            "volumeNumber": "6"
          }
        ], 
        "keywords": [
          "classification accuracy", 
          "classification power", 
          "high classification accuracy", 
          "human experts", 
          "classification performance", 
          "probabilistic answers", 
          "new instances", 
          "class probabilities", 
          "classifier performance", 
          "evaluation criteria", 
          "different domains", 
          "information score", 
          "decision rules", 
          "performance", 
          "prior probability", 
          "different systems", 
          "classification system", 
          "accuracy", 
          "classifier", 
          "system", 
          "influence", 
          "answers", 
          "rules", 
          "different types", 
          "power", 
          "experts", 
          "instances", 
          "probability", 
          "comparison", 
          "domain", 
          "example", 
          "types", 
          "defects", 
          "method", 
          "deal", 
          "measures", 
          "criteria", 
          "years", 
          "scores", 
          "paper", 
          "systems' classification power", 
          "fair evaluation criterion", 
          "uninformed classifier", 
          "classifier's answers", 
          "Information-based evaluation criterion"
        ], 
        "name": "Information-based evaluation criterion for classifier's performance", 
        "pagination": "67-80", 
        "productId": [
          {
            "name": "dimensions_id", 
            "type": "PropertyValue", 
            "value": [
              "pub.1036513123"
            ]
          }, 
          {
            "name": "doi", 
            "type": "PropertyValue", 
            "value": [
              "10.1007/bf00153760"
            ]
          }
        ], 
        "sameAs": [
          "https://doi.org/10.1007/bf00153760", 
          "https://app.dimensions.ai/details/publication/pub.1036513123"
        ], 
        "sdDataset": "articles", 
        "sdDatePublished": "2021-12-01T19:08", 
        "sdLicense": "https://scigraph.springernature.com/explorer/license/", 
        "sdPublisher": {
          "name": "Springer Nature - SN SciGraph project", 
          "type": "Organization"
        }, 
        "sdSource": "s3://com-springernature-scigraph/baseset/20211201/entities/gbq_results/article/article_249.jsonl", 
        "type": "ScholarlyArticle", 
        "url": "https://doi.org/10.1007/bf00153760"
      }
    ]
     

    Download the RDF metadata as:  json-ld nt turtle xml License info

    HOW TO GET THIS DATA PROGRAMMATICALLY:

    JSON-LD is a popular format for linked data which is fully compatible with JSON.

    curl -H 'Accept: application/ld+json' 'https://scigraph.springernature.com/pub.10.1007/bf00153760'

    N-Triples is a line-based linked data format ideal for batch operations.

    curl -H 'Accept: application/n-triples' 'https://scigraph.springernature.com/pub.10.1007/bf00153760'

    Turtle is a human-readable linked data format.

    curl -H 'Accept: text/turtle' 'https://scigraph.springernature.com/pub.10.1007/bf00153760'

    RDF/XML is a standard XML format for linked data.

    curl -H 'Accept: application/rdf+xml' 'https://scigraph.springernature.com/pub.10.1007/bf00153760'


     

    This table displays all metadata directly associated to this object as RDF triples.

    116 TRIPLES      22 PREDICATES      72 URIs      63 LITERALS      6 BLANK NODES

    Subject Predicate Object
    1 sg:pub.10.1007/bf00153760 schema:about anzsrc-for:08
    2 anzsrc-for:0801
    3 schema:author Nd2b239893c584f98acb2a057deb4406a
    4 schema:citation sg:pub.10.1007/bf00116251
    5 schema:datePublished 1991-01
    6 schema:datePublishedReg 1991-01-01
    7 schema:description In the past few years many systems for learning decision rules from examples were developed. As different systems allow different types of answers when classifying new instances, it is difficult to appropriately evaluate the systems' classification power in comparison with other classification systems or in comparison with human experts. Classification accuracy is usually used as a measure of classification performance. This measure is, however, known to have several defects. A fair evaluation criterion should exclude the influence of the class probabilities which may enable a completely uninformed classifier to trivially achieve high classification accuracy. In this paper a method for evaluating the information score of a classifier's answers is proposed. It excludes the influence of prior probabilities, deals with various types of imperfect or probabilistic answers and can be used also for comparing the performance in different domains.
    8 schema:genre article
    9 schema:inLanguage en
    10 schema:isAccessibleForFree true
    11 schema:isPartOf N61634aac8931496bbc9854244ed0d7d5
    12 Ndda10c6a94f54f378b71aad18f5438f5
    13 sg:journal.1125588
    14 schema:keywords Information-based evaluation criterion
    15 accuracy
    16 answers
    17 class probabilities
    18 classification accuracy
    19 classification performance
    20 classification power
    21 classification system
    22 classifier
    23 classifier performance
    24 classifier's answers
    25 comparison
    26 criteria
    27 deal
    28 decision rules
    29 defects
    30 different domains
    31 different systems
    32 different types
    33 domain
    34 evaluation criteria
    35 example
    36 experts
    37 fair evaluation criterion
    38 high classification accuracy
    39 human experts
    40 influence
    41 information score
    42 instances
    43 measures
    44 method
    45 new instances
    46 paper
    47 performance
    48 power
    49 prior probability
    50 probabilistic answers
    51 probability
    52 rules
    53 scores
    54 system
    55 systems' classification power
    56 types
    57 uninformed classifier
    58 years
    59 schema:name Information-based evaluation criterion for classifier's performance
    60 schema:pagination 67-80
    61 schema:productId N99d38eff70d34b4baa6f51da45b13adb
    62 Nf934609c3f404abbbc60c31d3cb34ed0
    63 schema:sameAs https://app.dimensions.ai/details/publication/pub.1036513123
    64 https://doi.org/10.1007/bf00153760
    65 schema:sdDatePublished 2021-12-01T19:08
    66 schema:sdLicense https://scigraph.springernature.com/explorer/license/
    67 schema:sdPublisher N895b471dfe3b4a85b214a813b875fc26
    68 schema:url https://doi.org/10.1007/bf00153760
    69 sgo:license sg:explorer/license/
    70 sgo:sdDataset articles
    71 rdf:type schema:ScholarlyArticle
    72 N61634aac8931496bbc9854244ed0d7d5 schema:issueNumber 1
    73 rdf:type schema:PublicationIssue
    74 N895b471dfe3b4a85b214a813b875fc26 schema:name Springer Nature - SN SciGraph project
    75 rdf:type schema:Organization
    76 N99d38eff70d34b4baa6f51da45b13adb schema:name doi
    77 schema:value 10.1007/bf00153760
    78 rdf:type schema:PropertyValue
    79 Nc97d3b53b1bb461c8a1d5c546db33fa7 rdf:first sg:person.01113747300.33
    80 rdf:rest rdf:nil
    81 Nd2b239893c584f98acb2a057deb4406a rdf:first sg:person.015561447361.41
    82 rdf:rest Nc97d3b53b1bb461c8a1d5c546db33fa7
    83 Ndda10c6a94f54f378b71aad18f5438f5 schema:volumeNumber 6
    84 rdf:type schema:PublicationVolume
    85 Nf934609c3f404abbbc60c31d3cb34ed0 schema:name dimensions_id
    86 schema:value pub.1036513123
    87 rdf:type schema:PropertyValue
    88 anzsrc-for:08 schema:inDefinedTermSet anzsrc-for:
    89 schema:name Information and Computing Sciences
    90 rdf:type schema:DefinedTerm
    91 anzsrc-for:0801 schema:inDefinedTermSet anzsrc-for:
    92 schema:name Artificial Intelligence and Image Processing
    93 rdf:type schema:DefinedTerm
    94 sg:journal.1125588 schema:issn 0885-6125
    95 1573-0565
    96 schema:name Machine Learning
    97 schema:publisher Springer Nature
    98 rdf:type schema:Periodical
    99 sg:person.01113747300.33 schema:affiliation grid-institutes:None
    100 schema:familyName Bratko
    101 schema:givenName Ivan
    102 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01113747300.33
    103 rdf:type schema:Person
    104 sg:person.015561447361.41 schema:affiliation grid-institutes:None
    105 schema:familyName Kononenko
    106 schema:givenName Igor
    107 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.015561447361.41
    108 rdf:type schema:Person
    109 sg:pub.10.1007/bf00116251 schema:sameAs https://app.dimensions.ai/details/publication/pub.1019422208
    110 https://doi.org/10.1007/bf00116251
    111 rdf:type schema:CreativeWork
    112 grid-institutes:None schema:alternateName Faculty of Electrical and Computer Engineering, Trzaska 25, Ljubljana, Yugoslavia
    113 Stefan Institute, Jamova 39, Ljubljana, Yugoslavia
    114 schema:name Faculty of Electrical and Computer Engineering, Trzaska 25, Ljubljana, Yugoslavia
    115 Stefan Institute, Jamova 39, Ljubljana, Yugoslavia
    116 rdf:type schema:Organization
     




    Preview window. Press ESC to close (or click here)


    ...