##### Differences

This shows you the differences between two versions of the page.

 — cs-677sp2010:mom-and-mle [2014/12/12 13:32] (current)ryancha created 2014/12/12 13:32 ryancha created 2014/12/12 13:32 ryancha created Line 1: Line 1: + = Still Proofreading,​ but this is essentially correct... = + = Expectations = + + == Expected value of the sum == + + If $X_1, ... ,X_n$ are n independent random variables such that the expectation $E[X_i]$ exists for all $i=1..n$ then prove: + + $E[X_1 + X_2 + ... + X_n] = E[X_1] + E[X_2] + ... + E[X_n]$ + + Now, If $X_1, ... ,X_n$ are n (not necessarily independent) random variables such that the expectation $E[X_i]$ exists for all $i=1..n$ then prove: + + $E[X_1 + X_2 + ... + X_n] = E[X_1] + E[X_2] + ... + E[X_n]$ + + Humm, the expectation of the sum is the sum of the expectations,​ even if they are NOT independent! + + == Expectations with independence == + + If $X$ and $Y$ are independent random variables show that: + + $E_{p(X,​Y)}[f(X) ]=E_{p(X)}[f(X) ]$ + + In this case I am using the subscript to indicate the distribution with respect to which the expectation is taken + + == Messy expectations == + + Prove that + + $exp \lbrace E_{p(x)}[ln(g(y)^{f(x)})] \rbrace =g(y)^{E_{p(x)}[f(x)]}$ + + This is only about four or five lines, so if you are going on much longer, you have made this harder than it was intended. + + == Markov'​s Inequality == + + Prove that + + $Pr(X \ge t) \le E_{p(x)}[X]/​t$ + + assuming that $Pr(X \ge 0) = 1$ and that $t > 0$. + + == Variance == + + If $X_1, ... ,X_n$ are n independent random variables such that the variance $Var[X_i]$ exists for all $i=1..n$ then prove: + + $Var[X_1 + X_2 + ... + X_n] = Var[X_1] + Var[X_2] + ... + Var[X_n]$ + + == Calculating Moments == + + $f(x; \alpha, \beta, \gamma) = \begin{cases} c & \mathrm{if}\,​ \alpha < x \le \beta, \\ 2c & \mathrm{if}\,​ \beta < x \le \gamma, \\ 0 & \mathrm{otherwise}. \end{cases}$ + + Calculate $c$ (the constant of integration),​ and then calculate the first, second, and third moments about zero, in terms of $\alpha$, $\beta$, and $\gamma$. + + = Method of Moments= + + == Negative Binomial == + + === Part 1 === + + The Negative Binomial distribution can be parameterized as follows: + + $f(x; \alpha, \beta) = {x + \alpha - 1 \choose \alpha - 1} \left(\frac{\beta}{\beta+1}\right)^\alpha \left(\frac{1}{\beta+1}\right)^x,​\,​ x=0, 1, 2, \ldots$ + + For common distributions,​ the moments are well known and appear in tables of distributions. ​ In this case, the mean is: + + $\mu = E[X] = \frac{\alpha}{\beta}$ + + and the variance is: + + $\sigma^2 = E[(X-E[X])^2] = \frac{\alpha}{\beta^2}(\beta + 1)$ + + Find a formula for $\alpha$ and $\beta$ using Method of Moments. + + === Part 2 === + + Suppose that you are given the first moment ​ and  the second central moment (moment about the mean): $\mu = 3$ and $\sigma^2 = 4$.  Use your formula from Part 1 to find $\alpha$ and $\beta$. + + === Part 3 === + + Now suppose that rather than the first moment and the second central moment, you are instead given both the first and second moments (about zero): $\mu = 6$ and $E[X^2] = 45$.  Find the variance and then use the mean and variance to find $\alpha$ and $\beta$. + + = Maximum Likelihood = + + The book talks about MLE's on page 719. + + == Bernoulli == + + Suppose that $x_1, x_2, \ldots x_n$ are independently and identically distributed as $\textrm{Bernoulli}(\theta)$ where $0 \le \theta \le 1$.  Thus: + + $f(x_i | \theta) = \theta ^ {x_i} (1 - \theta) ^ {1 - x_i},\, x_i \in \{0, 1\}.$ + + Find the Maximum Likelihood Estimator for $\theta$ by defining $L(\theta; x_1, x_2, \ldots x_n)$ and taking the derivative of $\log\, L(\theta)$. + + == Uniform == + + Suppose that $x_1, x_2, \ldots x_n$ are independently and identically distributed as $\textrm{Uniform}(\theta)$ where $0 \le \theta$. + + $f(x_i | \theta) = \begin{cases} \frac{1}{\theta} & \mathrm{if}\,​ 0 \le x_i \le \theta \\ 0 & \mathrm{otherwise}. \end{cases}$ + + === Part 1 === + + Find a formula for $L(\theta; x_1, \ldots x_n)$. + + === Part 2 === + + Find $L(4;\, x_1=3,\, x_2=7,\, x_3=5,\, x_4=6)$ and then sketch the function $L(\theta;​\,​ x_1=3,\, x_2=7,\, x_3=5,\, x_4=6)$. Identify where the maximum likelihood occurs. + + === Part 3 === + + Find the MLE of $\theta$ in general (for any data $x_1, x_2, \ldots x_n$). + + What is the derivative at this point? Is it 0? Look at your graph from Part 2...