Duel server, I'm playing in warm-up with another guy, XYZ joins the server. I clicked on "Join match" after adjusting some settings ==> I went to spec and XYZ joins the game.

"Join match" should be the same as "Return to match".