Educational theorists (and health policy theorists) may be the problem, however they seem to drive policy. It just seems that the unions have become so resistant to change, any change, with the desire to go back to very little responsibility for outcomes, however they should be measured.
I'm not convinced it is not possible to measure data intelligently and control for a lot of the variables you mention. The league tables of A level grades would seem to be a ridiculously crude measure, but going back to the bad days of the 80s should not be an option. The problem with the cruder measures is often lack of peer review.
That is particularly the case with the medical profession, and yes, surgeons should be rewarded for above average performance and the bar should be high for firing the bad ones. Medical data is one area where careful data collecting is making a huge improvement in quality of care in some NHS Trusts. The reduction in hospital borne infections is thanks to the 6-sigma people working in the NHS, not the doctors. Even simple data analysis would have stopped the Bristol baby killing scandal earlier.
I accept teaching is more complex as you are not just counting MSRA infections or dead bodies. A-level success is tempting, but way too crude, and what about the heroic teachers that work with special needs? There are internationally used methods, in which Britain does not come top of the class.
I just think saying we should not try to measure effectiveness because its difficult is not going to work anymore. There also has to be a method of identifying weak or burnt out teachers and helping them improve or retrain as Ofsted inspectors or something. At the same time it would seem logical to try a reward the outstandingly teachers, and doing that through the pay packet would seem better than having a "teacher of the month" parking spot next to the headteacher's.