You are doing it wrong.
You have to whitelist the ip in fail2ban otherwise fail2ban will insert a rule with higher priority than your whitelist in iptables.