Just, so strong dependency of agility and speed from size and armor that in really just arcade assumption. If we stay with it, we to be honestly no need so big gap between fighters as class, they can be balanced inside one big fighters class. At least for HF and VHF, because faster cruise for smallest fighters make difference.
If going from more simulator space dofight side, size will affect only on ship inertia (response in freel balance), while in same size, armor and powerplant, engineer will choose between two variants - concentrate most thrust to forward vector and turning systems, or split it on sides and allow more strafe. That system exists in some games, and work okey enough.
There already exists fighters which more good for strafe and which more good for out-turning. So it possible make bigger accent on this characteristics, instead of playing with armor and core for VHF, while other classes have small sense.