...
- Trying to emulate keras BatchNormalization 'modes', but actually, I mixed up mode 1 and 3 (keras mode 1 is what I called mode 3)
- The problem is that in test mode, the running mean and std still get updated. Here is tensorflow documentation on control dependencies
- note the use of tf.control_dependencies(boolVar, callableA, callableB) in the UseBatchAndUpdateAvg class - it is tieing this 'assign' ops for the running mean/stddev to an op used during training
Adding Ops to Training
Another way to do this is
...
If you look at stackoverflow posts (links below), you see there is a better, more automatic way to do some of this, there appears to be a training group of ops, and you should be able to add the running mean/std updates into this group by accessing the default computational graph - then you can just run the normal training op of minimizing the optimizer. The computational graph is a sort of global variable that is in scope - this appears to be the real tensorflow way to do this, not there yet! 
.
Stack overflow activity:
...