Kdtyjb

Time-reversing a quantum computation: what does this mean for the future of physics and is it considered to be time travel?

Confusion with the nameplate of an induction motor

Why doesn't the EU now just force the UK to choose between referendum and no-deal?

How does airport security verify that you can carry a battery bank over 100 Wh?

Why don't MCU characters ever seem to have language issues?

BitNot does not flip bits in the way I expected

When were linguistics departments first established

Do f-stop and exposure time perfectly cancel?

Provisioning profile doesn't include the application-identifier and keychain-access-groups entitlements

Why would a jet engine that runs at temps excess of 2000°C burn when it crashes?

infinitive telling the purpose

"One can do his homework in the library"

Silly Sally's Movie

How to discourage/prevent PCs from using door choke-points?

Time dilation for a moving electronic clock

What wound would be of little consequence to a biped but terrible for a quadruped?

Is Adobe Illustrator too heavy for handling complex EPSs?

Prove that the total distance is minimised (when travelling across the longest path)

Best approach to update all entries in a list that is paginated?

Why do Australian milk farmers need to protest supermarkets' milk price?

Is having access to past exams cheating and, if yes, could it be proven just by a good grade?

Force user to remove USB token

What is the dot in “1.2.4."

Rejected in 4th interview round citing insufficient years of experience

validation vs test vs training accuracy, which one to compare for claiming overfit?

Which observation to use when doing k-fold validation or boostrap?why k-fold cross validation (CV) overfits? Or why discrepancy occurs between CV and test set?Consistently inconsistent cross-validation results that are wildly different from original model accuracyWhy use both validation set and test set?Reporting test result for cross-validation with Neural Networkvalidation/training accuracy and overfittingValidation accuracy for neural networkTraining score at parameter tuning lower than on hold out test set (RandomForestClassifier)Terminology - cross-validation, testing and validation set for classification taskValidation accuracy is always close to training accuracy

2

$begingroup$

I have read on the several answers here and on the internet that cross-validation helps to indicate that if the model will generalize well or not and about overfitting.

But I am confused that which two accuracies/errors amoung test/training/validation should I compare to be able to see if the model is overfitting or not?

For example:

I divide my data for 70% training and 30% test.

When I get to run 10 fold cross-validation, I get 10 accuracies that I can take the average/mean of. should I call this mean as validation accuracy?

Afterward, I test the model on 30% test data and get Test Accuracy.

In this case, what will be training accuracy? and which two accuracies I compare to see if the model is overfitting or not?

This is my first question on this platform so please ignore errors.

machine-learning cross-validation accuracy overfitting

share|improve this question

asked 3 hours ago

A.BA.B

1113

New contributor

A.B is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

add a comment | 

2

$begingroup$

I have read on the several answers here and on the internet that cross-validation helps to indicate that if the model will generalize well or not and about overfitting.

But I am confused that which two accuracies/errors amoung test/training/validation should I compare to be able to see if the model is overfitting or not?

For example:

I divide my data for 70% training and 30% test.

When I get to run 10 fold cross-validation, I get 10 accuracies that I can take the average/mean of. should I call this mean as validation accuracy?

Afterward, I test the model on 30% test data and get Test Accuracy.

In this case, what will be training accuracy? and which two accuracies I compare to see if the model is overfitting or not?

This is my first question on this platform so please ignore errors.

machine-learning cross-validation accuracy overfitting

share|improve this question

asked 3 hours ago

A.BA.B

1113

New contributor

A.B is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

add a comment | 

$begingroup$

I have read on the several answers here and on the internet that cross-validation helps to indicate that if the model will generalize well or not and about overfitting.

But I am confused that which two accuracies/errors amoung test/training/validation should I compare to be able to see if the model is overfitting or not?

For example:

I divide my data for 70% training and 30% test.

When I get to run 10 fold cross-validation, I get 10 accuracies that I can take the average/mean of. should I call this mean as validation accuracy?

Afterward, I test the model on 30% test data and get Test Accuracy.

In this case, what will be training accuracy? and which two accuracies I compare to see if the model is overfitting or not?

This is my first question on this platform so please ignore errors.

machine-learning cross-validation accuracy overfitting

share|improve this question

asked 3 hours ago

A.BA.B

1113

New contributor

A.B is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

share|improve this question

asked 3 hours ago

A.BA.B

1113

New contributor

A.B is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

share|improve this question

asked 3 hours ago

A.BA.B

1113

New contributor

A.B is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

asked 3 hours ago

A.BA.B

1113

New contributor

A.B is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

asked 3 hours ago

A.BA.B

1113

2 Answers
2

active

oldest

votes

3

$begingroup$

When I get to run 10 fold cross-validation, I get 10 accuracies that I
can take the average/mean of. should I call this mean as validation
accuracy?

No. It is a [estimate of] test accuracy.

The difference between validation and test sets (and their corresponding accuracies) is that validation set is used to build/select a better model (e.g. avoid over-fitting), meaning it affects the final model. However, in your case, 10-fold CV tests an already-built model on the 10% hold-out, thus the hold-out is a test set not a validation set.

Afterward, I test the model on 30% test data and get Test Accuracy.

If you don't use the K-fold to select/build a better model, this part is not needed, run K-fold on 100% of data to get the test accuracy. Otherwise, you should keep a final test set, since the result of K-fold would be a validation accuracy.

In this case, what will be training accuracy?

From each of 10 folds you can get a test accuracy on 10% of data, and a training accuracy on 90% of data. In python, method cross_val_score only returns the test accuracies. Here is how to get both:

from  sklearn import model_selection

from sklearn import datasets

from sklearn import svm



iris = datasets.load_iris()

clf = svm.SVC(kernel='linear', C=1)

scores = model_selection.cross_validate(clf, iris.data, iris.target, cv=5, return_train_score=True)

print('Train scores:')

print(scores['train_score'])

print('Test scores:')

print(scores['test_score'])

and which two accuracies I compare to see if the model is overfitting or not?

You should compare the training and test accuracies to identify over-fitting. A training accuracy subjectively far higher than test accuracy indicates over-fitting.

I suggest "Bias and Variance" and "Learning curves" parts of "Machine Learning Yearning - Andrew Ng". It presents plots and interpretations for all the cases with a clear narration.

share|improve this answer

edited 1 hour ago

answered 2 hours ago

EsmailianEsmailian

922110

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  I think I disagree with "30% test set not needed." If you are using CV to select a better model, then you are exposing the test folds (which I would call a validation set in this case) and risk overfitting there. The final test set should remain untouched (by both you and your algorithms) until the end, to estimate the final model performance (if that's something you need). But yes, while model-building, the (averaged) training fold score vs. the (averaged) validation fold score is what you're looking at for overfitting indication.
  $endgroup$
  – Ben Reiniger
  2 hours ago
  
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  @BenReiniger You are right I should clear this case.
  $endgroup$
  – Esmailian
  2 hours ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  @Esmailian train_score is also an average of 10 scores? Also, to do a similar kind of thing with GridSearchCV(in case hyper paramter tuning and cross-validation are required in one step) can we use return_train_score=true? is it same?
  $endgroup$
  – A.B
  4 mins ago
  

  
  
  
  

add a comment | 

2

$begingroup$

Cross validation splits your data into K folds. Each fold contains a set of training data and test data. You are correct that you get K different error rates that you then take the mean of. These error rates come from the test set of each of your K folds. If you want to get the training error rate, you would calculate the error rate on the training part of each of these K folds and then take the average.

share|improve this answer

answered 2 hours ago

astelastel

211

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Thank you for answer.
  $endgroup$
  – A.B
  6 mins ago
  

  
  
  
  

add a comment | 

Your Answer

StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "557"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});

A.B is a new contributor. Be nice, and check out our Code of Conduct.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47263%2fvalidation-vs-test-vs-training-accuracy-which-one-to-compare-for-claiming-overf%23new-answer', 'question_page');
}
);

Post as a guest

Name

Email

Required, but never shown

3

$begingroup$

When I get to run 10 fold cross-validation, I get 10 accuracies that I
can take the average/mean of. should I call this mean as validation
accuracy?

No. It is a [estimate of] test accuracy.

The difference between validation and test sets (and their corresponding accuracies) is that validation set is used to build/select a better model (e.g. avoid over-fitting), meaning it affects the final model. However, in your case, 10-fold CV tests an already-built model on the 10% hold-out, thus the hold-out is a test set not a validation set.

Afterward, I test the model on 30% test data and get Test Accuracy.

If you don't use the K-fold to select/build a better model, this part is not needed, run K-fold on 100% of data to get the test accuracy. Otherwise, you should keep a final test set, since the result of K-fold would be a validation accuracy.

In this case, what will be training accuracy?

From each of 10 folds you can get a test accuracy on 10% of data, and a training accuracy on 90% of data. In python, method cross_val_score only returns the test accuracies. Here is how to get both:

from  sklearn import model_selection

from sklearn import datasets

from sklearn import svm



iris = datasets.load_iris()

clf = svm.SVC(kernel='linear', C=1)

scores = model_selection.cross_validate(clf, iris.data, iris.target, cv=5, return_train_score=True)

print('Train scores:')

print(scores['train_score'])

print('Test scores:')

print(scores['test_score'])

and which two accuracies I compare to see if the model is overfitting or not?

You should compare the training and test accuracies to identify over-fitting. A training accuracy subjectively far higher than test accuracy indicates over-fitting.

I suggest "Bias and Variance" and "Learning curves" parts of "Machine Learning Yearning - Andrew Ng". It presents plots and interpretations for all the cases with a clear narration.

share|improve this answer

edited 1 hour ago

answered 2 hours ago

EsmailianEsmailian

922110

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  I think I disagree with "30% test set not needed." If you are using CV to select a better model, then you are exposing the test folds (which I would call a validation set in this case) and risk overfitting there. The final test set should remain untouched (by both you and your algorithms) until the end, to estimate the final model performance (if that's something you need). But yes, while model-building, the (averaged) training fold score vs. the (averaged) validation fold score is what you're looking at for overfitting indication.
  $endgroup$
  – Ben Reiniger
  2 hours ago
  
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  @BenReiniger You are right I should clear this case.
  $endgroup$
  – Esmailian
  2 hours ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  @Esmailian train_score is also an average of 10 scores? Also, to do a similar kind of thing with GridSearchCV(in case hyper paramter tuning and cross-validation are required in one step) can we use return_train_score=true? is it same?
  $endgroup$
  – A.B
  4 mins ago
  

  
  
  
  

add a comment | 

3

$begingroup$

When I get to run 10 fold cross-validation, I get 10 accuracies that I
can take the average/mean of. should I call this mean as validation
accuracy?

No. It is a [estimate of] test accuracy.

The difference between validation and test sets (and their corresponding accuracies) is that validation set is used to build/select a better model (e.g. avoid over-fitting), meaning it affects the final model. However, in your case, 10-fold CV tests an already-built model on the 10% hold-out, thus the hold-out is a test set not a validation set.

Afterward, I test the model on 30% test data and get Test Accuracy.

If you don't use the K-fold to select/build a better model, this part is not needed, run K-fold on 100% of data to get the test accuracy. Otherwise, you should keep a final test set, since the result of K-fold would be a validation accuracy.

In this case, what will be training accuracy?

From each of 10 folds you can get a test accuracy on 10% of data, and a training accuracy on 90% of data. In python, method cross_val_score only returns the test accuracies. Here is how to get both:

from  sklearn import model_selection

from sklearn import datasets

from sklearn import svm



iris = datasets.load_iris()

clf = svm.SVC(kernel='linear', C=1)

scores = model_selection.cross_validate(clf, iris.data, iris.target, cv=5, return_train_score=True)

print('Train scores:')

print(scores['train_score'])

print('Test scores:')

print(scores['test_score'])

and which two accuracies I compare to see if the model is overfitting or not?

You should compare the training and test accuracies to identify over-fitting. A training accuracy subjectively far higher than test accuracy indicates over-fitting.

I suggest "Bias and Variance" and "Learning curves" parts of "Machine Learning Yearning - Andrew Ng". It presents plots and interpretations for all the cases with a clear narration.

share|improve this answer

edited 1 hour ago

answered 2 hours ago

EsmailianEsmailian

922110

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  I think I disagree with "30% test set not needed." If you are using CV to select a better model, then you are exposing the test folds (which I would call a validation set in this case) and risk overfitting there. The final test set should remain untouched (by both you and your algorithms) until the end, to estimate the final model performance (if that's something you need). But yes, while model-building, the (averaged) training fold score vs. the (averaged) validation fold score is what you're looking at for overfitting indication.
  $endgroup$
  – Ben Reiniger
  2 hours ago
  
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  @BenReiniger You are right I should clear this case.
  $endgroup$
  – Esmailian
  2 hours ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  @Esmailian train_score is also an average of 10 scores? Also, to do a similar kind of thing with GridSearchCV(in case hyper paramter tuning and cross-validation are required in one step) can we use return_train_score=true? is it same?
  $endgroup$
  – A.B
  4 mins ago
  

  
  
  
  

add a comment | 

$begingroup$

When I get to run 10 fold cross-validation, I get 10 accuracies that I
can take the average/mean of. should I call this mean as validation
accuracy?

No. It is a [estimate of] test accuracy.

The difference between validation and test sets (and their corresponding accuracies) is that validation set is used to build/select a better model (e.g. avoid over-fitting), meaning it affects the final model. However, in your case, 10-fold CV tests an already-built model on the 10% hold-out, thus the hold-out is a test set not a validation set.

Afterward, I test the model on 30% test data and get Test Accuracy.

If you don't use the K-fold to select/build a better model, this part is not needed, run K-fold on 100% of data to get the test accuracy. Otherwise, you should keep a final test set, since the result of K-fold would be a validation accuracy.

In this case, what will be training accuracy?

From each of 10 folds you can get a test accuracy on 10% of data, and a training accuracy on 90% of data. In python, method cross_val_score only returns the test accuracies. Here is how to get both:

from  sklearn import model_selection

from sklearn import datasets

from sklearn import svm



iris = datasets.load_iris()

clf = svm.SVC(kernel='linear', C=1)

scores = model_selection.cross_validate(clf, iris.data, iris.target, cv=5, return_train_score=True)

print('Train scores:')

print(scores['train_score'])

print('Test scores:')

print(scores['test_score'])

and which two accuracies I compare to see if the model is overfitting or not?

You should compare the training and test accuracies to identify over-fitting. A training accuracy subjectively far higher than test accuracy indicates over-fitting.

I suggest "Bias and Variance" and "Learning curves" parts of "Machine Learning Yearning - Andrew Ng". It presents plots and interpretations for all the cases with a clear narration.

share|improve this answer

edited 1 hour ago

answered 2 hours ago

EsmailianEsmailian

922110

$endgroup$

from  sklearn import model_selection

from sklearn import datasets

from sklearn import svm



iris = datasets.load_iris()

clf = svm.SVC(kernel='linear', C=1)

scores = model_selection.cross_validate(clf, iris.data, iris.target, cv=5, return_train_score=True)

print('Train scores:')

print(scores['train_score'])

print('Test scores:')

print(scores['test_score'])

share|improve this answer

edited 1 hour ago

answered 2 hours ago

EsmailianEsmailian

922110

answered 2 hours ago

EsmailianEsmailian

922110

answered 2 hours ago

EsmailianEsmailian

922110

2

$begingroup$

Cross validation splits your data into K folds. Each fold contains a set of training data and test data. You are correct that you get K different error rates that you then take the mean of. These error rates come from the test set of each of your K folds. If you want to get the training error rate, you would calculate the error rate on the training part of each of these K folds and then take the average.

share|improve this answer

answered 2 hours ago

astelastel

211

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Thank you for answer.
  $endgroup$
  – A.B
  6 mins ago
  

  
  
  
  

add a comment | 

2

$begingroup$

Cross validation splits your data into K folds. Each fold contains a set of training data and test data. You are correct that you get K different error rates that you then take the mean of. These error rates come from the test set of each of your K folds. If you want to get the training error rate, you would calculate the error rate on the training part of each of these K folds and then take the average.

share|improve this answer

answered 2 hours ago

astelastel

211

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Thank you for answer.
  $endgroup$
  – A.B
  6 mins ago
  

  
  
  
  

add a comment | 

$begingroup$

Cross validation splits your data into K folds. Each fold contains a set of training data and test data. You are correct that you get K different error rates that you then take the mean of. These error rates come from the test set of each of your K folds. If you want to get the training error rate, you would calculate the error rate on the training part of each of these K folds and then take the average.

share|improve this answer

answered 2 hours ago

astelastel

211

$endgroup$

share|improve this answer

answered 2 hours ago

astelastel

211

answered 2 hours ago

astelastel

211

answered 2 hours ago

astelastel

211