What about bases like ours?
Melbourne station is a Military Installation built within an area restricted to Military only traffic.
Would you expect to be able to just walk into a Military base without them trying to stop you?
As for your "restaurant" example, not all buildings are restaurants. You can't expect to be able to just walk into a company office, or someones house, and expect to be welcomed.