operators - Quantum non-unitary transformation?

Let us say that I apply a non-unitary transformation $\def\ket#1{| #1\rangle} \def\braket#1#2{\langle #1|#2\rangle} \hat A$ to the ket's:

$$\ket{\psi} \rightarrow \hat A \ket{\psi}$$ $$\ket{\phi}\rightarrow \hat A \ket{\phi}$$ Clearly in this case the probability: $$P=|\braket{\phi}{\psi}|^2$$ Will change. What physically is going on here? i.e. why for unitary operators we can perform such a transformation but for non-unitary operators we can't?