What kind of statements aren't considered grounds to have a judge removed from a case?

Here's what one judge got away with saying in his courtroom at the start of a hearing in which a father was aiming for the visitation rights to his child. The bid to have him removed from the case was denied:

Before we get started on this, I want everybody to understand I don’t
set visitation. If I do, I specifically make it so absolutely ridiculous
that nobody can adhere to it. And I hold people in contempt and put
them in jail for it.

I’ve been doing this for 22 years and I’ve set visitations five times.
All five times both parents have ended up in jail for contempt. If I’m
forced to set it, I will set it. But it will be so absolutely ridiculous that
nobody can adhere to it and people will go to jail. That’s a promise.

So, if you all would like to get together to try to settle this visitation
issue, it might be advantageous.

Source: Judge Andrew Jackson, as cited in the State of Tennessee Court Ruling

See also a news article covering this story.