Kdtyjb

Why complex landing gears are used instead of simple, reliable and light weight muscle wire or shape memory alloys?

Weaponising the Grasp-at-a-Distance spell

Centre cell contents vertically

Did pre-Columbian Americans know the spherical shape of the Earth?

Why not use the yoke to control yaw, as well as pitch and roll?

.bashrc alias for a command with fixed second parameter

How much damage would a cupful of neutron star matter do to the Earth?

How to resize main filesystem

My mentor says to set image to Fine instead of RAW — how is this different from JPG?

Determine whether an integer is a palindrome

Order between one to one functions and their inverses

Vertical ranges of Column Plots in 12

Statistical analysis applied to methods coming out of Machine Learning

malloc in main() or malloc in another function: allocating memory for a struct and its members

How to make triangles with rounded sides and corners? (squircle with 3 sides)

Can gravitational waves pass through a black hole?

Maximum duration for Canada's short term visas?

Did any compiler fully use 80-bit floating point?

One-one communication

Short story about astronauts fertilizing soil with their own bodies

Find general formula for the terms

Why can't fire hurt Daenerys but it did to Jon Snow in season 1?

newbie Q : How to read an output file in one command line

Is Normal(mean, variance) mod x still a normal distribution?

Is it OK to use the testing sample to compare algorithms?

1

$begingroup$

I'm working on a little project where my dataset have 6k lines and around 300 features, with a simple binary outcome.

Since I'm still learning ML, I want to try all the algorithms I can manage to find and compare the results.

As I've read in tutorials, I split my dataset into a training sample (80%) and a testing sample (20%), and then trained my algorithms on the training sample with cross-validation (5 folds).

My plan is to train all my models this way, and then measure their performance on the testing sample to chose the best algorithm.

Could this cause overfitting? If so, since I cannot compare several models inside model_selection.GridSearchCV, how can I prevent it to overfit?

machine-learning scikit-learn sampling

share|improve this question

asked 5 hours ago

Dan ChaltielDan Chaltiel

1385

$endgroup$

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  $begingroup$
  Possible duplicate of Can I use the test dataset to select a model?
  $endgroup$
  – Ben Reiniger
  3 hours ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  @BenReiniger You are right, this is quite the same question, but I like Simon's answer better.
  $endgroup$
  – Dan Chaltiel
  1 hour ago
  

  
  
  
  

add a comment | 

1

$begingroup$

I'm working on a little project where my dataset have 6k lines and around 300 features, with a simple binary outcome.

Since I'm still learning ML, I want to try all the algorithms I can manage to find and compare the results.

As I've read in tutorials, I split my dataset into a training sample (80%) and a testing sample (20%), and then trained my algorithms on the training sample with cross-validation (5 folds).

My plan is to train all my models this way, and then measure their performance on the testing sample to chose the best algorithm.

Could this cause overfitting? If so, since I cannot compare several models inside model_selection.GridSearchCV, how can I prevent it to overfit?

machine-learning scikit-learn sampling

share|improve this question

asked 5 hours ago

Dan ChaltielDan Chaltiel

1385

$endgroup$

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  $begingroup$
  Possible duplicate of Can I use the test dataset to select a model?
  $endgroup$
  – Ben Reiniger
  3 hours ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  @BenReiniger You are right, this is quite the same question, but I like Simon's answer better.
  $endgroup$
  – Dan Chaltiel
  1 hour ago
  

  
  
  
  

add a comment | 

$begingroup$

I'm working on a little project where my dataset have 6k lines and around 300 features, with a simple binary outcome.

Since I'm still learning ML, I want to try all the algorithms I can manage to find and compare the results.

As I've read in tutorials, I split my dataset into a training sample (80%) and a testing sample (20%), and then trained my algorithms on the training sample with cross-validation (5 folds).

My plan is to train all my models this way, and then measure their performance on the testing sample to chose the best algorithm.

Could this cause overfitting? If so, since I cannot compare several models inside model_selection.GridSearchCV, how can I prevent it to overfit?

machine-learning scikit-learn sampling

share|improve this question

asked 5 hours ago

Dan ChaltielDan Chaltiel

1385

$endgroup$

share|improve this question

asked 5 hours ago

Dan ChaltielDan Chaltiel

1385

share|improve this question

asked 5 hours ago

Dan ChaltielDan Chaltiel

1385

asked 5 hours ago

Dan ChaltielDan Chaltiel

1385

asked 5 hours ago

Dan ChaltielDan Chaltiel

1385

2 Answers
2

active

oldest

votes

1

$begingroup$

No, that is not the purpose of the test set. Test set is only for final evaluation when your model is done. The problem is that if you include the test set in your decisions your evaluation will no longer be reliable.

To compare algorithms you instead set aside another chunk of your data called the validation set.

Here is some info about good splits depending on data size:

Train / Dev / Test sets from Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization by Prof. Andrew Ng.

(Andrew uses the word dev set instead of validation set)

share|improve this answer

edited 5 hours ago

answered 5 hours ago

Simon LarssonSimon Larsson

1,015214

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  I thought so but I couldn't read anything about this. Could you provide some material on which I could learn? Like about what a common splitting would be (70/10/20) ?
  $endgroup$
  – Dan Chaltiel
  5 hours ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Depends on the size of your dataset. But I would say 70/15/15 would be good in your case.
  $endgroup$
  – Simon Larsson
  5 hours ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  I added a video on the subject.
  $endgroup$
  – Simon Larsson
  5 hours ago
  

  
  
  
  

add a comment | 

1

$begingroup$

Basically, every time you use the results of a train/test split to make decisions about a model- whether that's tuning the hyperparameters of a single model, or choosing the most effective of a number of different models, you cannot infer anything about the performance of the model after making those decisions until you have "frozen" your model and evaluated it on a portion of data that has not been touched.

The general concept addressing this issue is called nested cross validation. If you use a train/test split to choose the best parameters for a model, that's fine. But if you want to estimate the performance of that, you need to then evaluate on a second held out set.

If you then repeat process for multiple models and choose the best performing one, again, that's fine, but by choosing the best result the value of your performance metric is inherently biased, and you need to validate the entire procedure on yet another held out set to get an unbiased estimate of how your model will perform on unseen data.

share|improve this answer

answered 1 hour ago

Cameron KingCameron King

211

New contributor

Cameron King is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Great answer too! So since this seems like an order 2 cross-validation (a cross-validation of cross-validations), should I pool my samples (70+15 according to Simon's answer) before I evaluate my final algorithm on the test sample?
  $endgroup$
  – Dan Chaltiel
  1 hour ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  With so few samples and features (relatively speaking) personally I would use multiple rounds of stratified k-fold cross validation, rather than holding a fixed set or sets out. The theoretical guarantees of evaluating on held out data only hold in the limit of the number of samples. I'm working under the assumption that training and testing your model is not a huge deal in terms of time. I would do something like shuffling the samples within each class and setting aside each fold in turn, and for the inner loop, combine it all, split again, do your model selection, then yes, pool.
  $endgroup$
  – Cameron King
  43 mins ago
  
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  I forgot to mention here, this means that you are setting aside, say, 1/k points to test on, and dividing the 1-1/k remaining points into an inner cross validation run, but repeating this so that you are training, validating, and testing on every single data point at least once. With larger datasets and more time consuming models, deciding on splitting the data once into 3 partitions makes sense, but it will always be less robust than k-fold cross validation. develop with that method to save yourself time, that's a good idea, but when it comes to making decisions, don't cut corners.
  $endgroup$
  – Cameron King
  33 mins ago
  

  
  
  
  

add a comment | 

Your Answer

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "557"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f49683%2fis-it-ok-to-use-the-testing-sample-to-compare-algorithms%23new-answer', 'question_page');
}
);

Post as a guest

Name

Email

Required, but never shown

1

$begingroup$

No, that is not the purpose of the test set. Test set is only for final evaluation when your model is done. The problem is that if you include the test set in your decisions your evaluation will no longer be reliable.

To compare algorithms you instead set aside another chunk of your data called the validation set.

Here is some info about good splits depending on data size:

Train / Dev / Test sets from Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization by Prof. Andrew Ng.

(Andrew uses the word dev set instead of validation set)

share|improve this answer

edited 5 hours ago

answered 5 hours ago

Simon LarssonSimon Larsson

1,015214

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  I thought so but I couldn't read anything about this. Could you provide some material on which I could learn? Like about what a common splitting would be (70/10/20) ?
  $endgroup$
  – Dan Chaltiel
  5 hours ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Depends on the size of your dataset. But I would say 70/15/15 would be good in your case.
  $endgroup$
  – Simon Larsson
  5 hours ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  I added a video on the subject.
  $endgroup$
  – Simon Larsson
  5 hours ago
  

  
  
  
  

add a comment | 

1

$begingroup$

No, that is not the purpose of the test set. Test set is only for final evaluation when your model is done. The problem is that if you include the test set in your decisions your evaluation will no longer be reliable.

To compare algorithms you instead set aside another chunk of your data called the validation set.

Here is some info about good splits depending on data size:

Train / Dev / Test sets from Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization by Prof. Andrew Ng.

(Andrew uses the word dev set instead of validation set)

share|improve this answer

edited 5 hours ago

answered 5 hours ago

Simon LarssonSimon Larsson

1,015214

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  I thought so but I couldn't read anything about this. Could you provide some material on which I could learn? Like about what a common splitting would be (70/10/20) ?
  $endgroup$
  – Dan Chaltiel
  5 hours ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Depends on the size of your dataset. But I would say 70/15/15 would be good in your case.
  $endgroup$
  – Simon Larsson
  5 hours ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  I added a video on the subject.
  $endgroup$
  – Simon Larsson
  5 hours ago
  

  
  
  
  

add a comment | 

$begingroup$

No, that is not the purpose of the test set. Test set is only for final evaluation when your model is done. The problem is that if you include the test set in your decisions your evaluation will no longer be reliable.

To compare algorithms you instead set aside another chunk of your data called the validation set.

Here is some info about good splits depending on data size:

Train / Dev / Test sets from Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization by Prof. Andrew Ng.

(Andrew uses the word dev set instead of validation set)

share|improve this answer

edited 5 hours ago

answered 5 hours ago

Simon LarssonSimon Larsson

1,015214

$endgroup$

share|improve this answer

edited 5 hours ago

answered 5 hours ago

Simon LarssonSimon Larsson

1,015214

answered 5 hours ago

Simon LarssonSimon Larsson

1,015214

answered 5 hours ago

Simon LarssonSimon Larsson

1,015214

1

$begingroup$

Basically, every time you use the results of a train/test split to make decisions about a model- whether that's tuning the hyperparameters of a single model, or choosing the most effective of a number of different models, you cannot infer anything about the performance of the model after making those decisions until you have "frozen" your model and evaluated it on a portion of data that has not been touched.

The general concept addressing this issue is called nested cross validation. If you use a train/test split to choose the best parameters for a model, that's fine. But if you want to estimate the performance of that, you need to then evaluate on a second held out set.

If you then repeat process for multiple models and choose the best performing one, again, that's fine, but by choosing the best result the value of your performance metric is inherently biased, and you need to validate the entire procedure on yet another held out set to get an unbiased estimate of how your model will perform on unseen data.

share|improve this answer

answered 1 hour ago

Cameron KingCameron King

211

New contributor

Cameron King is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Great answer too! So since this seems like an order 2 cross-validation (a cross-validation of cross-validations), should I pool my samples (70+15 according to Simon's answer) before I evaluate my final algorithm on the test sample?
  $endgroup$
  – Dan Chaltiel
  1 hour ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  With so few samples and features (relatively speaking) personally I would use multiple rounds of stratified k-fold cross validation, rather than holding a fixed set or sets out. The theoretical guarantees of evaluating on held out data only hold in the limit of the number of samples. I'm working under the assumption that training and testing your model is not a huge deal in terms of time. I would do something like shuffling the samples within each class and setting aside each fold in turn, and for the inner loop, combine it all, split again, do your model selection, then yes, pool.
  $endgroup$
  – Cameron King
  43 mins ago
  
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  I forgot to mention here, this means that you are setting aside, say, 1/k points to test on, and dividing the 1-1/k remaining points into an inner cross validation run, but repeating this so that you are training, validating, and testing on every single data point at least once. With larger datasets and more time consuming models, deciding on splitting the data once into 3 partitions makes sense, but it will always be less robust than k-fold cross validation. develop with that method to save yourself time, that's a good idea, but when it comes to making decisions, don't cut corners.
  $endgroup$
  – Cameron King
  33 mins ago
  

  
  
  
  

add a comment | 

1

$begingroup$

Basically, every time you use the results of a train/test split to make decisions about a model- whether that's tuning the hyperparameters of a single model, or choosing the most effective of a number of different models, you cannot infer anything about the performance of the model after making those decisions until you have "frozen" your model and evaluated it on a portion of data that has not been touched.

The general concept addressing this issue is called nested cross validation. If you use a train/test split to choose the best parameters for a model, that's fine. But if you want to estimate the performance of that, you need to then evaluate on a second held out set.

If you then repeat process for multiple models and choose the best performing one, again, that's fine, but by choosing the best result the value of your performance metric is inherently biased, and you need to validate the entire procedure on yet another held out set to get an unbiased estimate of how your model will perform on unseen data.

share|improve this answer

answered 1 hour ago

Cameron KingCameron King

211

New contributor

Cameron King is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Great answer too! So since this seems like an order 2 cross-validation (a cross-validation of cross-validations), should I pool my samples (70+15 according to Simon's answer) before I evaluate my final algorithm on the test sample?
  $endgroup$
  – Dan Chaltiel
  1 hour ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  With so few samples and features (relatively speaking) personally I would use multiple rounds of stratified k-fold cross validation, rather than holding a fixed set or sets out. The theoretical guarantees of evaluating on held out data only hold in the limit of the number of samples. I'm working under the assumption that training and testing your model is not a huge deal in terms of time. I would do something like shuffling the samples within each class and setting aside each fold in turn, and for the inner loop, combine it all, split again, do your model selection, then yes, pool.
  $endgroup$
  – Cameron King
  43 mins ago
  
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  I forgot to mention here, this means that you are setting aside, say, 1/k points to test on, and dividing the 1-1/k remaining points into an inner cross validation run, but repeating this so that you are training, validating, and testing on every single data point at least once. With larger datasets and more time consuming models, deciding on splitting the data once into 3 partitions makes sense, but it will always be less robust than k-fold cross validation. develop with that method to save yourself time, that's a good idea, but when it comes to making decisions, don't cut corners.
  $endgroup$
  – Cameron King
  33 mins ago
  

  
  
  
  

add a comment | 

$begingroup$

Basically, every time you use the results of a train/test split to make decisions about a model- whether that's tuning the hyperparameters of a single model, or choosing the most effective of a number of different models, you cannot infer anything about the performance of the model after making those decisions until you have "frozen" your model and evaluated it on a portion of data that has not been touched.

The general concept addressing this issue is called nested cross validation. If you use a train/test split to choose the best parameters for a model, that's fine. But if you want to estimate the performance of that, you need to then evaluate on a second held out set.

If you then repeat process for multiple models and choose the best performing one, again, that's fine, but by choosing the best result the value of your performance metric is inherently biased, and you need to validate the entire procedure on yet another held out set to get an unbiased estimate of how your model will perform on unseen data.

share|improve this answer

answered 1 hour ago

Cameron KingCameron King

211

New contributor

Cameron King is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

share|improve this answer

answered 1 hour ago

Cameron KingCameron King

211

New contributor

Cameron King is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

answered 1 hour ago

Cameron KingCameron King

211

New contributor

Cameron King is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

answered 1 hour ago

Cameron KingCameron King

211