if no one is on the tee or green, then SWING AWAY!
however if there is, then just wait. i once played in a 4 some on a very crowded day and everyone in the 4 some had to pee. there is only 1 person allowed in the bathroom at a time in the course that i played at that day. while we were releiving ourselves at the bathroom by the 5th tee, not one but TWO groups decided to jump us(one of the groups was a 6 some)and they couldn`t go anywhere because the group we were originally following was waiting to tee off, and so we had to wait for two more groups, and the 6 some did not keep up with the groups in front of them, and they weren`t playing ready golf, so what the two groups gained from jumping us was NOTHING, plus we just had to wait even longer. at about the 10th hole, after another 4+ hour front 9, they FINALLY let us play through. we didn`t have to wait again intil hole 14, but still it was about a 6 hour round. some people are just plain rude!