Kdtyjb

My question is not meant to be a criticism of Bayesian methods; I am simply trying to understand the Bayesian view. Why is it reasonable to believe we know the distribution of our parameters, but not our parameters given data?

If you have a belief about the distribution of your data after seeing data, then why would you be estimating its parameters with data? You already have the parameters.

Well, in Bayesian statistics, you don't just "make up" your priors. You should be building a prior that best captures your knowledge before seeing the data. Otherwise, why anyone should care about the output of your Bayesian analysis is very hard to justify.

So while it's true that the practitioner has some sense of freedom in creating a prior, it should be tied to something meaningful in order for an analysis to be useful. With that said, the prior isn't the only part of a Bayesian analysis that allows this freedom. A practitioner is offered the same freedom in constructing the likelihood function, which defines the relation between the data and the model. Just as using nonsense priors will lead to a nonsense posterior, using a nonsense likelihood will also lead to a nonsense posterior. So in practice, ideally one should chose a likelihood function such that it is flexible enough to handle one's uncertainty, yet constrained enough to make inference with limited data possible.

To demonstrate, consider two somewhat extreme examples. Suppose we are interested in determining the effect of a continuous-valued treatment on patients. In order to learn anything from the data, we must choice a model with that flexibility. If we were to simply leave out "treatment" for our set of regression parameters, no matter what our outcome was, we could report "given the data, our model estimates no effect of treatment". On the other extreme, suppose we have a model so flexible that we don't constrain the treatment effect to have a finite number of discontinuities. Then, (without strong priors, at least), we have almost no hope of having any sort of convergence of our estimated treatment effect no matter our sample size. Thus, our inference can be completely butchered by poor choices of likelihood functions, just as it could be by poor choices of priors.

Of course, in reality we wouldn't chose either of these extremes, but we still do make these types of choices. How flexible a treatment effect are we going to allow: linear, splines, interaction with other variables? There's always the tradeoff between "sufficiently flexible" and "estimatable given our sample size". If we're smart, our likelihood functions should include reasonable constraints (i.e., treatment continuous treatment effect probably relatively smooth, probably doesn't include very high order interaction effects). This is essentially the same art as picking a prior: you want to constrain your inference with prior knowledge, and allow flexibility where there is uncertainty. The whole point of using data is to help constrain some of that the flexibility that stems from our uncertainty.

In summary, a practitioner has freedom in selection of both the prior and the likelihood function. In order for an analysis to be in anyway meaningful, both choices should be a relatively good approximation of real phenomena.

In case of many problems in statistics you have some data, let's denote it as $X$, and want to learn about some "parameter" $theta$ of the distribution of the data, i.e. calculate the $theta|X$ kind of things (conditional distribution, conditional expectation etc.). There are several ways how can this be achieved, including maximum likelihood, and without getting into discussion if and which of them is better, you can consider using Bayes theorem as one of them. One of the advantages of using Bayes theorem, is that it let's you directly given that you know conditional distribution of the data given the parameter (likelihood) and the distribution of the parameter (prior), then you simply calculate

$$
overbrace{p(theta|X)}^text{posterior} = frac{overbrace{p(X|theta)}^text{likelihood};overbrace{p(theta)}^text{prior}}{p(X)}
$$

Likelihood is the conditional distribution of your data, so it is a matter of understanding your data and choosing some distribution that approximates it best, and it is rather uncontroversial concept. As about prior, notice that for the above formula to work you need some prior. In perfect world, you would know a priori the distribution of $theta$ and applied it to get the posterior. In real world, this is something that you assume, given your best knowledge, and plug-in to Bayes theorem. You could choose an "uninformative" prior $p(theta) propto 1$, but there are many arguments that such priors are neither "uninformative", nor reasonable. What I'm trying to say, is that there are many ways how you could come up with some distribution for a prior. Some consider priors as a blessing, since they make it possible to bring your out-of-data knowledge into the model, while others, for exactly the same reason, consider them as problematic.

Answering your question, sure you can assume that the distribution of the parameter given data is something. On day-to-day basis all the time we make our decisions based on some assumptions, that not always are rigorously validated. However the difference between prior and posterior is that the posterior is something that you learned from the data (and the prior). If it isn't, but your wild guess, then it's not a posterior any more. As about why we allow ourselves to "make up" priors, there are two answers depending on who you ask: either it is that (a) for the machinery to work we need some prior, or (b) we know something in advance that want to include it in our model, and thanks to priors this is possible. In either case, we usually expect the data to have "final word" rather then the priors.