singularities - When are infinities okay?

I'm a little confused about how to handle infinities in physics. On the one hand, I always here things like infinities are not physically tenable answers. GR can't answer anything about the origin of the universe because of singularities. Throw away the radial solution that gives an infinite potential when solving the Laplace equation in spherical coordinates. But then there are other times that the infinities are okay. For instance, it is usually okay to set the reference point for your potential at infinity. There are certain times in David Tong's quantum field theory lecture notes where he seems to almost to hand wave away infinities. So, philosophically, physically, and mathematically when are infinities okay, and when are they not okay?