mathematical physics - Subtlety in the proof of 2-to-1 homomorphism between $SU(2)$ and $SO(3)$

In physics, it's common to use the relations

$$\textbf{r}^\prime=\mathscr{R}\textbf{r};~~\text{and}~~\textbf{r}^\prime\cdot\boldsymbol{\sigma} =\mathscr{U}(\textbf{r}\cdot\boldsymbol{\sigma}) \mathscr{U}^{\dagger}\tag{1}$$ to establish a two-to-one homomorphism between ${\rm SU(2)}$ and ${\rm SO(3)}$ where $\textbf{r}\in \mathbb{R}^3$, $\mathscr{R}\in {\rm SO(3)}$, $\mathscr{U}\in {\rm SU(2)}$ and $\boldsymbol{\sigma}=(\sigma_1,\sigma_2,\sigma_3)$ are three Pauli matrices. Both the relations of Eq.(1) represent rotation of coordinates in real three-dimensioanl space because both of them satisfy $|\textbf{r}^\prime|^2=|\textbf{r}|^2$. It's easy to see from (1) that corresponding to every $3\times 3$ matrix $\mathscr{R}\in {\rm SO(3)}$ there exist two $2\times 2$ matrices $\pm \mathscr{U}\in {\rm SU(2)}$ that represent the same rotation.

Question Note that the above proof of 2-to-1 homomorphism is based on fundamental representations of $SO(3)$ and $SU(2)$. But for any odd-dimensional representation of $SU(2)$, if $\mathscr{U}$ has determinant $+1$, $-\mathscr{U}$ is not a representation of $SU(2)$ since it has determinant $-1$. Hence, if $\mathscr{U}$ is a member of an odd-dimensional representation of $SU(2)$. $\mathscr{U}$ is not. Does it mean that 2-to-1 homomorphism between $SU(2)$ and $SO(3)$ is not true in general?