Damage from ramming asteroids ? No...seriously. Examples have been named. Of course it's no problem for, lets say, a Liberty Dreadnought, since those don't ever fly through asteroid fields, just on very rare occasions.
Of course it's "realistic" and "inRP", but it will just piss players off, and by god we've enough things that do just that already.