Switching servers while observing (“server hopping”) is not considered antag rolling if you have either not joined the round that you are leaving or if you are observing the round.
How you get to observe a round might be an issue. For example, round start suicide may be considered abandoning your role. Pressing the observe button from the lobby then quitting or server hopping is not considered antag rolling.
Please keep in mind that being connected to multiple servers at the same time is different than server hopping.