That's because public FFA/CA/CTF is ridiculously easy to "do very well" at. There is far more involved in duel than in all of those modes. You're getting humped at duel because you are bad at it. Play more, you will improve.
On your profile it says you've played 39 duels and won 22 - seems pretty fair to me. Normally when people have "issues" with the ranking system it's because they want to scan servers and find the highest chance of a win. Luckily you're going to have a hard time doing that in duel because even the "low elo" players will be better than players who sit around on FFA/CA all day.
Got two hands, a brain and a computer? Then we're skill matched.