I was going through my notes on the unitary irreducible representations of the Poincare group and the subsequent construction of one particle states and I stumbled across the following steps in the method of induced representations: ($\mathscr{H}$ is the Hilbert space and $\mathscr{H}_p$ is the subspace of one-particle states with momenta p. $\mathcal{S}_k$ is just the little group of the standard momentum $k$. $L(p)$ is the standard Lorentz boost; $L(p)k=p$)
(1) Let $|k,\sigma\rangle$ be a basis for $\mathscr{H}_k$ and let $\mathcal{D}$ be the (finite) unitary irrep of $\mathcal{S}_k$ induced by the unitary irrep $\mathcal{U}$. Then $\mathcal{D}$ acts on $\mathscr{H}_k$ by mixing the spin polarization states, \begin{align} \mathcal{U}(h)|k,\sigma\rangle=\sum_{\sigma'}\mathcal{D}_{\sigma'\sigma}(h)|k,\sigma'\rangle,~~~ \forall h\in\mathcal{S}_k,~~~\mathcal{U}(h)\equiv\mathcal{U}(h,0). \end{align} (2) Define the basis for $\mathscr{H}_p$ by \begin{align} |p,\sigma\rangle=\mathcal{U}(L(p))|k,\sigma\rangle. \end{align} (3) The action of Lorentz transformations on these states is then \begin{align} \mathcal{U}(\Lambda)|p,\sigma\rangle=\sum_{\sigma'}\mathcal{D}_{\sigma'\sigma}\big(h(\Lambda,p)\big)|\Lambda p,\sigma'\rangle \end{align} where $h(\Lambda,p)=L^{-1}(\Lambda p)\Lambda L(p)\in \mathcal{S}_k$ is known as the Wigner rotation.
(4) Finally, we extend this action to an arbitrary state in the Hilbert space $|\Psi\rangle=\sum_{p,\sigma}\Psi_{\sigma}(p)|p,\sigma\rangle\in\mathscr{H}$ for some coefficients $\Psi_{\sigma}(p)$ by, \begin{align} \mathcal{U}(\Lambda)|\Psi\rangle&=\sum_{p,\sigma}\big[\mathcal{U}(\Lambda)\cdot\Psi\big]_{\sigma}\big( p\big)| p,\sigma\rangle,\notag\\ \text{ where } \big[\mathcal{U}(\Lambda)\cdot\Psi\big]_{\sigma}\big(p\big)&=\sum_{\sigma'}\mathcal{D}_{\sigma\sigma'}\big(h(\Lambda,\Lambda^{-1}p)\big)\Psi_{\sigma'}(\Lambda^{-1}p). \end{align}
These are my own notes which are largely influenced by Weinbergs construction, so they may be incorrect. I understand all of the steps except for one part of step 4; The part where I say $|\Psi\rangle=\sum_{p,\sigma}\Psi_{\sigma}(p)|p,\sigma\rangle\in\mathscr{H}$ for some coefficients $\Psi_{\sigma}(p)$. What are these coefficients? Are they the wavefunctions? Do they depend continuously on $p$, if so should the sum over $p$ be replaced by an integral? Should these coefficients be square-integrable; $\Psi_{\sigma}(p)\in L^2(\Sigma_m,\mu,\mathbb{C})$ where $\Sigma_m$ is the mass shell and $\mu$ a Lorentz invariant measure? Is this expansion of an arbitrary state even correct?
By construction, the $\lvert p,\sigma\rangle$ are a basis of your vector space. So all your states are linear combinations of these basis vectors. However, you are right to inquire about the nature of the coefficients since it is not that simple in this case:
There are three different notions of "basis" for a quantum mechanical Hilbert space, which unfortunately few physics texts bother to properly distinguish:
A finite-dimensional basis. In case your vector space is finite-dimensional, you have a finite number of basis vectors $\lvert v_i\rangle, i = 1,\dots,n$ and every vector is a linear combination $$ \lvert v\rangle = \sum_{i = 1}^n c_i \lvert v_i\rangle, c_i\in\mathbb{C}.$$ This definition has no further issues.
A Hilbert basis. In case your vector space is infinite-dimensional and a separable Hilbert space you can take a countable number of basis vectors $\lvert v_i\rangle,i\in\mathbb{N}$ and every vector is an infinite linear combination $$ \lvert v\rangle = \sum_{i=1}^\infty c_i\lvert v_i\rangle,c\in\ell^2(\mathbb{C}),$$ where $c\in \ell^2(\mathbb{C})$ means that the sequence $c = (c_i)_{i\in\mathbb{N}}$ is square-summable as $\sum_{i=1}^\infty \lvert c_i \rvert^2 < \infty$.
A rigged basis1. Again, in the case where the vector space is infinite-dimensional, we have a number of "basis" "vectors" $\lvert v(x)\rangle, x\in\mathbb{R}$ such that every vector is a "uncountably infinite" "linear combination" $$ "\lvert v\rangle = \int \psi(x)\lvert v(x)\rangle\mathrm{d}x",$$ where $\psi(x)\in L^2(\mathbb{R})$ is a square-integrable complex function, i.e. a "wavefunction". The reason for all the scare quotes here is that this notion is "easy" for the physicist to write down, but very hard for the mathematician to make sense of. Since $\langle v(x')\vert v(x)\rangle = \delta(x' -x)$, their inner product is ill-defined and these things are not vectors in the Hilbert space proper, but lie outside of it in the larger part of a rigged Hilbert space, see also this question and answer. The integral is also difficult to make rigorous, so you might as well replace it by a sum as Weinberg does, it is certainly neither an integral nor a sum as we would naively understand them, but it is clearly a generalization of the sum like the integral. Nevertheless, you should be familiar with this from basic quantum mechanics where we usually have no qualms about writing $1= \int \lvert x\rangle\langle x\rvert \mathrm{d}x$, and also $\psi(x) = \langle x \vert \psi \rangle$ and $\lvert \psi\rangle = \int \langle x\vert \psi\rangle \lvert x\rangle \mathrm{d}x$.
In this case, your $\lvert p,\sigma\rangle$ are a mixture of a finite basis and a rigged basis - there are only finitely many options for one-particle spins $\sigma$, but there are uncountably many choices of momentum. So, for fixed $p$ we have $\Phi(p)\in \mathbb{C}^n$ for $n = 2s + 1$ for $s$ the total spin of the particle. For fixed $\sigma$, we have $\Phi_\sigma \in L^2(\Sigma_m)$. So, altogether, $\Phi_\sigma(p)$ are the values of a $\mathbb{C}^n$-valued function that is square-integrable in each component, and to which you can only feed momenta $p$ with the correct $p^2 = -m^2$ relation.
1This terminology is my personal invention. If there's a proper name for these things, I'd be happy to hear it.
