Frustration comes from the way they interact with the universe, jamming up the gates and lanes, and shooting the tradelane rings. They make you wait, or they bump into you and make you spend time getting your ship back on course so you can wait for the next batch to jam you again.
They are also completely predictable in their spawn patterns, popping out when you enter and exit a lane, or undock from a base.
I like having them around as atmosphere, I just wish they wouldn't interact with stuff.