A ship can't be blacklisted and whitelisted at the same time. If you add a blacklisted player to the whitelist, it will remove him from the blacklist and the opposite as well.
Also yes, it does make the base hostile. The blacklist is meant to be for people you do NOT want around your base.
Defensemode 0 = can dock = whitelist, unknown = hostility based on IFF, blacklist = you die
Defensemode 2 = anyone can dock, hostile IFFs can't dock and will be shot at, blacklist = you die
Defensemode 1 = everybody dies except whitelisted people.