Kdtyjb

Where is the 1/8 CR apprentice in Volo's Guide to Monsters?

Could the Saturn V actually have launched astronauts around Venus?

How do I hide Chekhov's Gun?

Can elves maintain concentration in a trance?

Be in awe of my brilliance!

Does the statement `int val = (++i > ++j) ? ++i : ++j;` invoke undefined behavior?

How can I change step-down my variable input voltage? [Microcontroller]

Dot in front of file

Who is our nearest planetary neighbor, on average?

Informing my boss about remarks from a nasty colleague

The use of "touch" and "touch on" in context

Current sense amp + op-amp buffer + ADC: Measuring down to 0 with single supply

What is IP squat space

Can hydraulic brake levers get hot when brakes overheat?

Happy pi day, everyone!

What are the possible solutions of the given equation?

How to make healing in an exploration game interesting

Why does Deadpool say "You're welcome, Canada," after shooting Ryan Reynolds in the end credits?

Calculus II Professor will not accept my correct integral evaluation that uses a different method, should I bring this up further?

Does splitting a potentially monolithic application into several smaller ones help prevent bugs?

Employee lack of ownership

An Accountant Seeks the Help of a Mathematician

Life insurance that covers only simultaneous/dual deaths

Professor being mistaken for a grad student

Distribution of Maximum Likelihood Estimator

Maximum likelihood of function of the mean on a restricted parameter spaceMaximum Likelihood Estimator (MLE)Maximum Likelihood Estimator of the exponential function parameter based on Order StatisticsFind maximum likelihood estimateMaximum Likelihood Estimation in case of some specific uniform distributionsMaximum likelihood estimate for a univariate gaussianFind the Maximum Likelihood Estimator given two pdfsMaximum Likelihood Estimate for a likelihood defined by partsVariance of distribution for maximum likelihood estimatorHow is $P(D;theta) = P(D|theta)$?

1

$begingroup$

Why is the Maximum Likelihood Estimator Normally distributed? I can't figure out why it is true for large n in general. My attempt (for single parameter)

Let $L(theta)$ be the maximum likelihood function for the distribution $f(x;theta)$

Then after taking sample of size n

$$L(theta)=f(x_1;theta)cdot f(x_2theta)...f(x_n;theta)$$

And we want to find $theta_{max}$ such that $L(theta)$ is maximized and $theta_{max}$ is our estimate (once a sample has actually been selected)

Since $theta_{max}$ maximizes $L(theta)$ it also maximizes $ln(L(theta))$

where

$$ln(L(theta))=ln(f(x_1;theta))+ln(f(x_2;theta))...+ln(f(x_n;theta))$$

Taking the derivative with respect to $theta$

$$frac{f'(x_1;theta)}{f(x_1;theta)}+frac{f'(x_2;theta)}{f(x_2;theta)}...+frac{f'(x_n;theta)}{f(x_n;theta)}$$

$theta_{max}$ would be the solution of the above when set to 0 (after selecting values for all $x_1,x_2...x_n$) but why is it normally distributed and how do I show that it's true for large n?

probability distributions normal-distribution estimation sampling

share|cite|improve this question

asked 3 hours ago

Colin HicksColin Hicks

1353

New contributor

Colin Hicks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

add a comment | 

1

$begingroup$

Why is the Maximum Likelihood Estimator Normally distributed? I can't figure out why it is true for large n in general. My attempt (for single parameter)

Let $L(theta)$ be the maximum likelihood function for the distribution $f(x;theta)$

Then after taking sample of size n

$$L(theta)=f(x_1;theta)cdot f(x_2theta)...f(x_n;theta)$$

And we want to find $theta_{max}$ such that $L(theta)$ is maximized and $theta_{max}$ is our estimate (once a sample has actually been selected)

Since $theta_{max}$ maximizes $L(theta)$ it also maximizes $ln(L(theta))$

where

$$ln(L(theta))=ln(f(x_1;theta))+ln(f(x_2;theta))...+ln(f(x_n;theta))$$

Taking the derivative with respect to $theta$

$$frac{f'(x_1;theta)}{f(x_1;theta)}+frac{f'(x_2;theta)}{f(x_2;theta)}...+frac{f'(x_n;theta)}{f(x_n;theta)}$$

$theta_{max}$ would be the solution of the above when set to 0 (after selecting values for all $x_1,x_2...x_n$) but why is it normally distributed and how do I show that it's true for large n?

probability distributions normal-distribution estimation sampling

share|cite|improve this question

asked 3 hours ago

Colin HicksColin Hicks

1353

New contributor

Colin Hicks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

add a comment | 

$begingroup$

Why is the Maximum Likelihood Estimator Normally distributed? I can't figure out why it is true for large n in general. My attempt (for single parameter)

Let $L(theta)$ be the maximum likelihood function for the distribution $f(x;theta)$

Then after taking sample of size n

$$L(theta)=f(x_1;theta)cdot f(x_2theta)...f(x_n;theta)$$

And we want to find $theta_{max}$ such that $L(theta)$ is maximized and $theta_{max}$ is our estimate (once a sample has actually been selected)

Since $theta_{max}$ maximizes $L(theta)$ it also maximizes $ln(L(theta))$

where

$$ln(L(theta))=ln(f(x_1;theta))+ln(f(x_2;theta))...+ln(f(x_n;theta))$$

Taking the derivative with respect to $theta$

$$frac{f'(x_1;theta)}{f(x_1;theta)}+frac{f'(x_2;theta)}{f(x_2;theta)}...+frac{f'(x_n;theta)}{f(x_n;theta)}$$

$theta_{max}$ would be the solution of the above when set to 0 (after selecting values for all $x_1,x_2...x_n$) but why is it normally distributed and how do I show that it's true for large n?

probability distributions normal-distribution estimation sampling

share|cite|improve this question

asked 3 hours ago

Colin HicksColin Hicks

1353

New contributor

Colin Hicks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

share|cite|improve this question

asked 3 hours ago

Colin HicksColin Hicks

1353

New contributor

Colin Hicks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

share|cite|improve this question

asked 3 hours ago

Colin HicksColin Hicks

1353

New contributor

Colin Hicks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

asked 3 hours ago

Colin HicksColin Hicks

1353

New contributor

Colin Hicks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

asked 3 hours ago

Colin HicksColin Hicks

1353

1 Answer
1

active

oldest

votes

3

$begingroup$

MLE requires $$frac{partial ln L(theta)}{partial theta} = sum_{i=1}^n frac{ f'(x_i;theta)}{f(x_i;theta)},$$
where $f'(x_i;theta)$ could denote a gradient (allowing for the multivariate case, but still sticking to your notation). Define a new function $g(x;theta)=frac{ f'(x;theta)}{f(x;theta)}.$ Then ${g(x_i;theta)}_{i=1}^n$ is a new iid sequence of random variables, with $Eg(x_1;theta)=0$. If $Eg(x_1;theta)g(x_1;theta)'<infty$, CLT implies,
$$sqrt{n}(bar{g}_n(theta)-Eg(x_1;theta))=sqrt{n}bar{g}_n(theta) rightarrow_D N(0,E(g(x;theta)g(x;theta)'),$$
where $bar{g}_n(theta)=frac{1}{n} sum_{i=1}^n g(x_i;theta).$ The ML estimator solves the equation
$$bar{g}_n(theta)=0.$$
It follows that the ML estimator is given by
$$hat{theta}=bar{g}_n^{-1}(0).$$
So long as the set of discontinuity points of $bar{g}_n^{-1}(z)$, i.e. the set of all values of $z$ such that $bar{g}_n^{-1}(z)$ is not continuous, occur with probability zero, the continuous mapping theorem gives us asymptotic normality of $theta$.

share|cite|improve this answer

edited 2 hours ago

answered 2 hours ago

dlnBdlnB

3916

New contributor

dlnB is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  $frac{f'(x)}{f(x)}$ is just another function of x so central limit theorem applies thank you for that
  $endgroup$
  – Colin Hicks
  2 hours ago
  
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  You're welcome :)
  $endgroup$
  – dlnB
  2 hours ago
  

  
  
  
  

add a comment | 

Your Answer

StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "65"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});

Colin Hicks is a new contributor. Be nice, and check out our Code of Conduct.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f397619%2fdistribution-of-maximum-likelihood-estimator%23new-answer', 'question_page');
}
);

Post as a guest

Name

Email

Required, but never shown

3

$begingroup$

MLE requires $$frac{partial ln L(theta)}{partial theta} = sum_{i=1}^n frac{ f'(x_i;theta)}{f(x_i;theta)},$$
where $f'(x_i;theta)$ could denote a gradient (allowing for the multivariate case, but still sticking to your notation). Define a new function $g(x;theta)=frac{ f'(x;theta)}{f(x;theta)}.$ Then ${g(x_i;theta)}_{i=1}^n$ is a new iid sequence of random variables, with $Eg(x_1;theta)=0$. If $Eg(x_1;theta)g(x_1;theta)'<infty$, CLT implies,
$$sqrt{n}(bar{g}_n(theta)-Eg(x_1;theta))=sqrt{n}bar{g}_n(theta) rightarrow_D N(0,E(g(x;theta)g(x;theta)'),$$
where $bar{g}_n(theta)=frac{1}{n} sum_{i=1}^n g(x_i;theta).$ The ML estimator solves the equation
$$bar{g}_n(theta)=0.$$
It follows that the ML estimator is given by
$$hat{theta}=bar{g}_n^{-1}(0).$$
So long as the set of discontinuity points of $bar{g}_n^{-1}(z)$, i.e. the set of all values of $z$ such that $bar{g}_n^{-1}(z)$ is not continuous, occur with probability zero, the continuous mapping theorem gives us asymptotic normality of $theta$.

share|cite|improve this answer

edited 2 hours ago

answered 2 hours ago

dlnBdlnB

3916

New contributor

dlnB is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  $frac{f'(x)}{f(x)}$ is just another function of x so central limit theorem applies thank you for that
  $endgroup$
  – Colin Hicks
  2 hours ago
  
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  You're welcome :)
  $endgroup$
  – dlnB
  2 hours ago
  

  
  
  
  

add a comment | 

3

$begingroup$

MLE requires $$frac{partial ln L(theta)}{partial theta} = sum_{i=1}^n frac{ f'(x_i;theta)}{f(x_i;theta)},$$
where $f'(x_i;theta)$ could denote a gradient (allowing for the multivariate case, but still sticking to your notation). Define a new function $g(x;theta)=frac{ f'(x;theta)}{f(x;theta)}.$ Then ${g(x_i;theta)}_{i=1}^n$ is a new iid sequence of random variables, with $Eg(x_1;theta)=0$. If $Eg(x_1;theta)g(x_1;theta)'<infty$, CLT implies,
$$sqrt{n}(bar{g}_n(theta)-Eg(x_1;theta))=sqrt{n}bar{g}_n(theta) rightarrow_D N(0,E(g(x;theta)g(x;theta)'),$$
where $bar{g}_n(theta)=frac{1}{n} sum_{i=1}^n g(x_i;theta).$ The ML estimator solves the equation
$$bar{g}_n(theta)=0.$$
It follows that the ML estimator is given by
$$hat{theta}=bar{g}_n^{-1}(0).$$
So long as the set of discontinuity points of $bar{g}_n^{-1}(z)$, i.e. the set of all values of $z$ such that $bar{g}_n^{-1}(z)$ is not continuous, occur with probability zero, the continuous mapping theorem gives us asymptotic normality of $theta$.

share|cite|improve this answer

edited 2 hours ago

answered 2 hours ago

dlnBdlnB

3916

New contributor

dlnB is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  $frac{f'(x)}{f(x)}$ is just another function of x so central limit theorem applies thank you for that
  $endgroup$
  – Colin Hicks
  2 hours ago
  
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  You're welcome :)
  $endgroup$
  – dlnB
  2 hours ago
  

  
  
  
  

add a comment | 

$begingroup$

MLE requires $$frac{partial ln L(theta)}{partial theta} = sum_{i=1}^n frac{ f'(x_i;theta)}{f(x_i;theta)},$$
where $f'(x_i;theta)$ could denote a gradient (allowing for the multivariate case, but still sticking to your notation). Define a new function $g(x;theta)=frac{ f'(x;theta)}{f(x;theta)}.$ Then ${g(x_i;theta)}_{i=1}^n$ is a new iid sequence of random variables, with $Eg(x_1;theta)=0$. If $Eg(x_1;theta)g(x_1;theta)'<infty$, CLT implies,
$$sqrt{n}(bar{g}_n(theta)-Eg(x_1;theta))=sqrt{n}bar{g}_n(theta) rightarrow_D N(0,E(g(x;theta)g(x;theta)'),$$
where $bar{g}_n(theta)=frac{1}{n} sum_{i=1}^n g(x_i;theta).$ The ML estimator solves the equation
$$bar{g}_n(theta)=0.$$
It follows that the ML estimator is given by
$$hat{theta}=bar{g}_n^{-1}(0).$$
So long as the set of discontinuity points of $bar{g}_n^{-1}(z)$, i.e. the set of all values of $z$ such that $bar{g}_n^{-1}(z)$ is not continuous, occur with probability zero, the continuous mapping theorem gives us asymptotic normality of $theta$.

share|cite|improve this answer

edited 2 hours ago

answered 2 hours ago

dlnBdlnB

3916

New contributor

dlnB is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

share|cite|improve this answer

edited 2 hours ago

answered 2 hours ago

dlnBdlnB

3916

New contributor

dlnB is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

answered 2 hours ago

dlnBdlnB

3916

New contributor

dlnB is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

answered 2 hours ago

dlnBdlnB

3916