Impossibility of time travel due to energy conservation?

I just watched the movie Terminator Genisys and started pondering time travel.

While pontificating and stretching my limited remembrance of AP Physics, science documentaries, and YouTube videos I remembered the laws of Conservation of Energy and Mass.

Would that not make time travel impossible? You can't send something elsewhere in time as you would be destroying energy and/or mass, by removing it from the current time. And it would be impossible to arrive elsewhere in time, as you would be creating energy and/or mass.

I know this is theoretical and a real answer can't be found till someone invents a time machine. I'm just curious as to what people think of that.