Bias in Estimating the Variance of K-Fold Cross-Validation View Full Text


Ontology type: schema:Chapter     


Chapter Info

DATE

2005-01-01

AUTHORS

Yoshua Bengio , Yves Grandvalet

ABSTRACT

Most machine learning researchers perform quantitative experiments to estimate generalization error and compare the perforniance of different algorithms (in particular, their proposed algorithmn). In order to be able to draw statistically convincing conclusions, it is important to estimate the uncertainty of such estimates. This paper studies the very commonly used K-fold cross-validation estimator of generalization performance. The main theorem shows that there exists no universal (valid under all distributions) unbiased estimator of the variance of K-fold cross-validation, based on a single computation of the K-fold cross-validation estimator. The analysis that accompanies this result is based on the eigen-decomposition of the covariance matrix of errors, which has only three different eigenvalues corresponding to three degrees of freedom of the matrix and three components of the total variance. This analysis helps to better understand the nature of the problem and how it can make naive estimators (that don't take into account the error correlations due to the overlap between training and test sets) grossly underestimate variance. This is confirmed by numerical experiments in which the three components of the variance are compared when the difficulty of the learning problem and the number of folds are varied. More... »

PAGES

75-95

Book

TITLE

Statistical Modeling and Analysis for Complex Data Problems

ISBN

978-0-387-24554-6
978-0-387-24555-3

Identifiers

URI

http://scigraph.springernature.com/pub.10.1007/0-387-24555-3_5

DOI

http://dx.doi.org/10.1007/0-387-24555-3_5

DIMENSIONS

https://app.dimensions.ai/details/publication/pub.1028718727


Indexing Status Check whether this publication has been indexed by Scopus and Web Of Science using the SN Indexing Status Tool
Incoming Citations Browse incoming citations for this publication using opencitations.net

JSON-LD is the canonical representation for SciGraph data.

TIP: You can open this SciGraph record using an external JSON-LD service: JSON-LD Playground Google SDTT

[
  {
    "@context": "https://springernature.github.io/scigraph/jsonld/sgcontext.json", 
    "about": [
      {
        "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/01", 
        "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
        "name": "Mathematical Sciences", 
        "type": "DefinedTerm"
      }, 
      {
        "id": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/0104", 
        "inDefinedTermSet": "http://purl.org/au-research/vocabulary/anzsrc-for/2008/", 
        "name": "Statistics", 
        "type": "DefinedTerm"
      }
    ], 
    "author": [
      {
        "familyName": "Bengio", 
        "givenName": "Yoshua", 
        "id": "sg:person.01303456133.13", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01303456133.13"
        ], 
        "type": "Person"
      }, 
      {
        "familyName": "Grandvalet", 
        "givenName": "Yves", 
        "id": "sg:person.015255215731.52", 
        "sameAs": [
          "https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.015255215731.52"
        ], 
        "type": "Person"
      }
    ], 
    "datePublished": "2005-01-01", 
    "datePublishedReg": "2005-01-01", 
    "description": "Most machine learning researchers perform quantitative experiments to estimate generalization error and compare the perforniance of different algorithms (in particular, their proposed algorithmn). In order to be able to draw statistically convincing conclusions, it is important to estimate the uncertainty of such estimates. This paper studies the very commonly used K-fold cross-validation estimator of generalization performance. The main theorem shows that there exists no universal (valid under all distributions) unbiased estimator of the variance of K-fold cross-validation, based on a single computation of the K-fold cross-validation estimator. The analysis that accompanies this result is based on the eigen-decomposition of the covariance matrix of errors, which has only three different eigenvalues corresponding to three degrees of freedom of the matrix and three components of the total variance. This analysis helps to better understand the nature of the problem and how it can make naive estimators (that don't take into account the error correlations due to the overlap between training and test sets) grossly underestimate variance. This is confirmed by numerical experiments in which the three components of the variance are compared when the difficulty of the learning problem and the number of folds are varied.", 
    "editor": [
      {
        "familyName": "Duchesne", 
        "givenName": "Pierre", 
        "type": "Person"
      }, 
      {
        "familyName": "R\u00c9Millard", 
        "givenName": "Bruno", 
        "type": "Person"
      }
    ], 
    "genre": "chapter", 
    "id": "sg:pub.10.1007/0-387-24555-3_5", 
    "inLanguage": "en", 
    "isAccessibleForFree": false, 
    "isPartOf": {
      "isbn": [
        "978-0-387-24554-6", 
        "978-0-387-24555-3"
      ], 
      "name": "Statistical Modeling and Analysis for Complex Data Problems", 
      "type": "Book"
    }, 
    "keywords": [
      "most machine", 
      "different algorithms", 
      "generalization performance", 
      "quantitative experiments", 
      "generalization error", 
      "cross-validation estimator", 
      "single computation", 
      "learning problem", 
      "Fold Cross-Validation", 
      "Cross-Validation", 
      "algorithm", 
      "covariance matrix", 
      "degrees of freedom", 
      "machine", 
      "error", 
      "estimator", 
      "computation", 
      "numerical experiments", 
      "number of folds", 
      "experiments", 
      "performance", 
      "researchers", 
      "order", 
      "uncertainty", 
      "such estimates", 
      "unbiased estimator", 
      "matrix", 
      "components", 
      "difficulties", 
      "number", 
      "main theorem", 
      "theorem", 
      "analysis", 
      "results", 
      "different eigenvalues", 
      "freedom", 
      "naive estimator", 
      "estimates", 
      "variance", 
      "eigenvalues", 
      "nature", 
      "convincing conclusions", 
      "degree", 
      "problem", 
      "bias", 
      "conclusion", 
      "total variance", 
      "folds", 
      "paper", 
      "perforniance", 
      "fold cross-validation estimator", 
      "universal (valid under all distributions) unbiased estimator", 
      "underestimate variance"
    ], 
    "name": "Bias in Estimating the Variance of K-Fold Cross-Validation", 
    "pagination": "75-95", 
    "productId": [
      {
        "name": "dimensions_id", 
        "type": "PropertyValue", 
        "value": [
          "pub.1028718727"
        ]
      }, 
      {
        "name": "doi", 
        "type": "PropertyValue", 
        "value": [
          "10.1007/0-387-24555-3_5"
        ]
      }
    ], 
    "publisher": {
      "name": "Springer Nature", 
      "type": "Organisation"
    }, 
    "sameAs": [
      "https://doi.org/10.1007/0-387-24555-3_5", 
      "https://app.dimensions.ai/details/publication/pub.1028718727"
    ], 
    "sdDataset": "chapters", 
    "sdDatePublished": "2022-01-01T19:20", 
    "sdLicense": "https://scigraph.springernature.com/explorer/license/", 
    "sdPublisher": {
      "name": "Springer Nature - SN SciGraph project", 
      "type": "Organization"
    }, 
    "sdSource": "s3://com-springernature-scigraph/baseset/20220101/entities/gbq_results/chapter/chapter_353.jsonl", 
    "type": "Chapter", 
    "url": "https://doi.org/10.1007/0-387-24555-3_5"
  }
]
 

Download the RDF metadata as:  json-ld nt turtle xml License info

HOW TO GET THIS DATA PROGRAMMATICALLY:

JSON-LD is a popular format for linked data which is fully compatible with JSON.

curl -H 'Accept: application/ld+json' 'https://scigraph.springernature.com/pub.10.1007/0-387-24555-3_5'

N-Triples is a line-based linked data format ideal for batch operations.

curl -H 'Accept: application/n-triples' 'https://scigraph.springernature.com/pub.10.1007/0-387-24555-3_5'

Turtle is a human-readable linked data format.

curl -H 'Accept: text/turtle' 'https://scigraph.springernature.com/pub.10.1007/0-387-24555-3_5'

RDF/XML is a standard XML format for linked data.

curl -H 'Accept: application/rdf+xml' 'https://scigraph.springernature.com/pub.10.1007/0-387-24555-3_5'


 

This table displays all metadata directly associated to this object as RDF triples.

120 TRIPLES      23 PREDICATES      78 URIs      71 LITERALS      7 BLANK NODES

Subject Predicate Object
1 sg:pub.10.1007/0-387-24555-3_5 schema:about anzsrc-for:01
2 anzsrc-for:0104
3 schema:author Ndae49c6e2771456fb53d9fc2fe35cf9e
4 schema:datePublished 2005-01-01
5 schema:datePublishedReg 2005-01-01
6 schema:description Most machine learning researchers perform quantitative experiments to estimate generalization error and compare the perforniance of different algorithms (in particular, their proposed algorithmn). In order to be able to draw statistically convincing conclusions, it is important to estimate the uncertainty of such estimates. This paper studies the very commonly used K-fold cross-validation estimator of generalization performance. The main theorem shows that there exists no universal (valid under all distributions) unbiased estimator of the variance of K-fold cross-validation, based on a single computation of the K-fold cross-validation estimator. The analysis that accompanies this result is based on the eigen-decomposition of the covariance matrix of errors, which has only three different eigenvalues corresponding to three degrees of freedom of the matrix and three components of the total variance. This analysis helps to better understand the nature of the problem and how it can make naive estimators (that don't take into account the error correlations due to the overlap between training and test sets) grossly underestimate variance. This is confirmed by numerical experiments in which the three components of the variance are compared when the difficulty of the learning problem and the number of folds are varied.
7 schema:editor N4423e6be00cc409cbb6b12f494aaab38
8 schema:genre chapter
9 schema:inLanguage en
10 schema:isAccessibleForFree false
11 schema:isPartOf Nb64edbc6fb0d4680b04341ef80e2d2c7
12 schema:keywords Cross-Validation
13 Fold Cross-Validation
14 algorithm
15 analysis
16 bias
17 components
18 computation
19 conclusion
20 convincing conclusions
21 covariance matrix
22 cross-validation estimator
23 degree
24 degrees of freedom
25 different algorithms
26 different eigenvalues
27 difficulties
28 eigenvalues
29 error
30 estimates
31 estimator
32 experiments
33 fold cross-validation estimator
34 folds
35 freedom
36 generalization error
37 generalization performance
38 learning problem
39 machine
40 main theorem
41 matrix
42 most machine
43 naive estimator
44 nature
45 number
46 number of folds
47 numerical experiments
48 order
49 paper
50 performance
51 perforniance
52 problem
53 quantitative experiments
54 researchers
55 results
56 single computation
57 such estimates
58 theorem
59 total variance
60 unbiased estimator
61 uncertainty
62 underestimate variance
63 universal (valid under all distributions) unbiased estimator
64 variance
65 schema:name Bias in Estimating the Variance of K-Fold Cross-Validation
66 schema:pagination 75-95
67 schema:productId Na63f3826e7834888bdcda69c836a9bfc
68 Nb52adde8f6214d9287236aa57af26e55
69 schema:publisher Nb3e84d3ef0464affa22ee4e6fea9cbcd
70 schema:sameAs https://app.dimensions.ai/details/publication/pub.1028718727
71 https://doi.org/10.1007/0-387-24555-3_5
72 schema:sdDatePublished 2022-01-01T19:20
73 schema:sdLicense https://scigraph.springernature.com/explorer/license/
74 schema:sdPublisher Naf80a5c580c04f81b469e27cf90b38b1
75 schema:url https://doi.org/10.1007/0-387-24555-3_5
76 sgo:license sg:explorer/license/
77 sgo:sdDataset chapters
78 rdf:type schema:Chapter
79 N078e2ec3626b462497aef891c92cb6b3 schema:familyName Duchesne
80 schema:givenName Pierre
81 rdf:type schema:Person
82 N4423e6be00cc409cbb6b12f494aaab38 rdf:first N078e2ec3626b462497aef891c92cb6b3
83 rdf:rest N67af6b10b88646d9a9e2f3601a1f261f
84 N45b7ee8e84884ddcb4988851e63c5983 rdf:first sg:person.015255215731.52
85 rdf:rest rdf:nil
86 N67af6b10b88646d9a9e2f3601a1f261f rdf:first Ncac7ae236f7448ee95ff7d8b3e9cc94b
87 rdf:rest rdf:nil
88 Na63f3826e7834888bdcda69c836a9bfc schema:name doi
89 schema:value 10.1007/0-387-24555-3_5
90 rdf:type schema:PropertyValue
91 Naf80a5c580c04f81b469e27cf90b38b1 schema:name Springer Nature - SN SciGraph project
92 rdf:type schema:Organization
93 Nb3e84d3ef0464affa22ee4e6fea9cbcd schema:name Springer Nature
94 rdf:type schema:Organisation
95 Nb52adde8f6214d9287236aa57af26e55 schema:name dimensions_id
96 schema:value pub.1028718727
97 rdf:type schema:PropertyValue
98 Nb64edbc6fb0d4680b04341ef80e2d2c7 schema:isbn 978-0-387-24554-6
99 978-0-387-24555-3
100 schema:name Statistical Modeling and Analysis for Complex Data Problems
101 rdf:type schema:Book
102 Ncac7ae236f7448ee95ff7d8b3e9cc94b schema:familyName RÉMillard
103 schema:givenName Bruno
104 rdf:type schema:Person
105 Ndae49c6e2771456fb53d9fc2fe35cf9e rdf:first sg:person.01303456133.13
106 rdf:rest N45b7ee8e84884ddcb4988851e63c5983
107 anzsrc-for:01 schema:inDefinedTermSet anzsrc-for:
108 schema:name Mathematical Sciences
109 rdf:type schema:DefinedTerm
110 anzsrc-for:0104 schema:inDefinedTermSet anzsrc-for:
111 schema:name Statistics
112 rdf:type schema:DefinedTerm
113 sg:person.01303456133.13 schema:familyName Bengio
114 schema:givenName Yoshua
115 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.01303456133.13
116 rdf:type schema:Person
117 sg:person.015255215731.52 schema:familyName Grandvalet
118 schema:givenName Yves
119 schema:sameAs https://app.dimensions.ai/discover/publication?and_facet_researcher=ur.015255215731.52
120 rdf:type schema:Person
 




Preview window. Press ESC to close (or click here)


...