If $X_1, \ldots ,X_n$ are n independent random variables such that the expectation $E[X_i]$ exists for all $i=1..n$ then prove:

$E[X_1 + X_2 + \ldots + X_n] = E[X_1] + E[X_2] + \ldots + E[X_n]$

Now, If $X_1, \ldots ,X_n$ are n (not necessarily independent) random variables such that the expectation $E[X_i]$ exists for all $i=1..n$ then prove:

$E[X_1 + X_2 + \ldots + X_n] = E[X_1] + E[X_2] + \ldots + E[X_n]$

Humm, the expectation of the sum is the sum of the expectations, even if they are NOT independent!

If $X$ and $Y$ are independent random variables show that:

$E_{p(X,Y)}[f(X) ]=E_{p(X)}[f(X) ]$

In this case I am using the subscript to indicate the distribution with respect to which the expectation is taken

Prove that

$exp \lbrace E_{p(x)}[ln(g(y)^{f(x)})] \rbrace =g(y)^{E_{p(x)}[f(x)]}$

This is only about four or five lines, so if you are going on much longer, you have made this harder than it was intended.

Prove that

$Pr(X \ge t) \le E_{p(x)}[X]/t$

assuming that $Pr(X \ge 0) = 1$ and that $t > 0$.

If $X_1, \ldots ,X_n$ are n independent random variables such that the variance $Var[X_i]$ exists for all $i=1..n$ then prove:

$Var[X_1 + X_2 + \ldots + X_n] = Var[X_1] + Var[X_2] + \ldots + Var[X_n]$

$f(x; \alpha, \beta, \gamma) = \begin{cases} c & \mathrm{if}\, \alpha < x \le \beta, \\ 2c & \mathrm{if}\, \beta < x \le \gamma, \\ 0 & \mathrm{otherwise}. \end{cases}$

Calculate $c$ (the constant of integration), and then calculate the first, second, and third moments about zero, in terms of $\alpha$, $\beta$, and $\gamma$.

The Negative Binomial distribution can be parameterized as follows:

$f(x; \alpha, \beta) = {x + \alpha - 1 \choose \alpha - 1} \left(\frac{\beta}{\beta+1}\right)^\alpha \left(\frac{1}{\beta+1}\right)^x,\, x=0, 1, 2, \ldots$

For common distributions, the moments are well known and appear in tables of distributions. In this case, the mean is:

$\mu = E[X] = \frac{\alpha}{\beta}$

and the variance is:

$\sigma^2 = E[(X-E[X])^2] = \frac{\alpha}{\beta^2}(\beta + 1)$

Find a formula for $\alpha$ and $\beta$ using Method of Moments.

Suppose that you are given the first moment and the second central moment (moment about the mean): $\mu = 3$ and $\sigma^2 = 4$. Use your formula from Part 1 to find $\alpha$ and $\beta$.

Now suppose that rather than the first moment and the second central moment, you are instead given both the first and second moments (about zero): $\mu = 6$ and $E[X^2] = 45$. Find the variance and then use the mean and variance to find $\alpha$ and $\beta$.

The book talks about MLE's on page 719.

Suppose that $x_1, x_2, \ldots x_n$ are independently and identically distributed as $\textrm{Bernoulli}(\theta)$ where $0 \le \theta \le 1$. Thus:

$f(x_i | \theta) = \theta ^ {x_i} (1 - \theta) ^ {1 - x_i},\, x_i \in \{0, 1\}.$

Find the Maximum Likelihood Estimator for $\theta$ by defining $L(\theta; x_1, x_2, \ldots x_n)$ and taking the derivative of $\log\, L(\theta)$.

Suppose that $x_1, x_2, \ldots x_n$ are independently and identically distributed as $\textrm{Uniform}(\theta)$ where $0 \le \theta$.

$f(x_i | \theta) = \begin{cases} \frac{1}{\theta} & \mathrm{if}\, 0 \le x_i \le \theta \\ 0 & \mathrm{otherwise}. \end{cases}$

Find a formula for $L(\theta; x_1, \ldots x_n)$.

Find $L(4;\, x_1=3,\, x_2=7,\, x_3=5,\, x_4=6)$ and then sketch the function $L(\theta;\, x_1=3,\, x_2=7,\, x_3=5,\, x_4=6)$. Identify where the maximum likelihood occurs.

Find the MLE of $\theta$ in general (for any data $x_1, x_2, \ldots x_n$).

What is the derivative at this point? Is it 0? Look at your graph from Part 2…