Why would you live in the woods and piss off the bears? The Phantoms are the bears, and The Brotherhood has to live in the woods.
We don't have city walls to keep out the wild animals, so we must try to live in harmony with as many of them as possible.
Scratch that, harmony isn't the right word. But no reason to make unnecessary enemies. Most people draw the line at decimating planets, and we don't have enough planets to afford something like that.
Too sum it up, as it's late and I'm tired:
Don't poke the bear and the bear wont rip out your intestines and feed you to it's cubs.