I agree with Xoria that this is still a not defined area in the PvP. If I was to advocate on it I'd say that anytime you're within weapons range (say 2k) and allows yourself to be successfully attacked, the engagement has started.
I did this myself. I needed to know what I was up against and did a flyby at some 4-5k distance to assess the situation before they even noticed me and decide what to do (if I should use a fighter or a bomber in that occasion).
If instead I went within the guy's weapon's range and let him shoot me just to figure out I should have been smarter and properly dressed for the occasion, if I did I would be fleeing from battle.
If you're doing recon and/or don't want to be so easily surprised get a adv. cargo scanner. That will keep you outside of the engagement zone and still allow you to figure out what you're up against.
One thing though, if you want to do this, keep all your vessels based in the very same spot.
As for the ruling on it. Since there's no specific rule rule for "when an engagement starts", it's up for the admins. I think it's likely they'll go for the 50% shield ruling.
try the engagement rules for retreating then. - you are allowed to scout and retreat to join reinforcements ( even in the very same system ) if you have not been shooting. so you can fly up to an enemy battleship evade its shot - return to your buddies and then go for it in a pack.
when you went up to the enemy battleship but shot at it - then there would have been an engagement and even if you managed to retreat, the retreat rules would apply and you would have to leave the system.
one can also take the rules for self defense - that state that an enegaement happens when ones shields are down by 50% or more. at that time, a player can, regardless of any other rule for re-engagement fully engage in self defense.
for such an occasion like switching ships - i d apply the retreat rule and say, as long as there has not been weaponfire from both sides - there has not been an engagement and both would be free to switch ships ( if thats so important )