quantum mechanics - Shankar's Active/Passive Change of Basis

I'm working my way through Shankar's Quantum Mechanics (7th printing, and I'm doing it alone, so I apologize if I have core concepts completely wrong).

He has a section on Active and Passive Transformations (which seems to be slightly different from what I know as Active & Passive Transformations, but that would be a different question :) ), and it reads like this:

Suppose we subject all the vectors $|V\rangle$ in a space to a unitary transformation $$ |V\rangle \rightarrow U|V\rangle $$ Under this transformation, the matrix elements of any operator $\Omega$ are modified as follows: $$ \langle V'|\Omega|V\rangle \rightarrow \langle UV'|\Omega|UV\rangle = \langle V'|U^\dagger\Omega U|V\rangle $$ It is clear that the same change would be effected if we left the vectors alone and subjected all operators to the change $$ \Omega\rightarrow U^\dagger\Omega U$$

I was trying to wrap my head around this, so I was working through some examples I made up and realized that the formula $\Omega\rightarrow U^\dagger\Omega U$ only works when $U$ is unitary. In my own examples I worked it out for an arbitrary (viz. non-unitary) change of basis T to be $$\Omega\rightarrow T^{-1}\Omega T$$ Like I said, I worked this out myself, so it may be wrong, but it checks out for the unitary case because when $U$ is unitary, $U^{-1} = U^\dagger$.

My question is: what part of Shankar's logic above makes use of the fact that U is unitary? It seems like his logic holds fine even without that constraint, but then it gives us a formula that isn't true for non-unitary U.

Answer

By definition the matrix elements of an operator $\Omega$ are the expressions $\langle V|\Omega|V'\rangle$, and they transform as DanielSank described. However, their interpretation as actual coefficients of the matrix of $\Omega$ in the basis $V,V',\ldots$ only is correct when the $V,V',\ldots$ form an orthonormal basis. If $A$ is the matrix of $\Omega$ in any basis, and $T$ is the matrix mapping it to a different basis, the matrix of $\Omega$ in the new basis becomes $T^{-1}AT$.

In other words, the inner products $\langle TV|\Omega|TV'\rangle$, called matrix elements, are equal to $\langle V|T^\dagger\Omega T|V'\rangle$, but the elements of the matrix of $\Omega$ in the basis consisting of the columns of $T$ are equal to $\langle V|T^{-1}\Omega T|V'\rangle$, if the $V,V',\ldots$ form an orthonormal basis. These are equal only if the $TV, TV',\ldots$ still form an orthonormal basis, which is equivalent to $T$ being unitary.