I agree it's not the best logic. However, slavers have a liner, viable routes etc.
There's no viable route that I can see that involves Refugees and a ship designed for human transport. On those grounds, I make an exception. Albeit one I agree doesn't make the best of sense when viewed outside the context of a workable traderoute.
Edit: That said, having upgraded from a Mammoth to a Mastadon, I am no longer entirely comfortable carrying refugees for the reason you mentioned and I'm looking at reworking my routes.
I may well end up financing a separate ship and/or a refugee transportation initiative.
Further edit: I haven't yet carried refugees since my ship change and now likely won't, in that ship anyway. I WOULD like to do something RP wise regarding refugees though.