...

• Trying to emulate keras BatchNormalization 'modes', but actually, I mixed up mode 1 and 3 (keras mode 1 is what I called mode 3)
• The problem is that in test mode, the running mean and std still get updated. Here is tensorflow documentation on control dependencies
• note the use of tf.control_dependencies(boolVar, callableA, callableB) in the UseBatchAndUpdateAvg class - it is tieing this 'assign' ops for the running mean/stddev to an op used during training

Adding Ops to Training

Another way to do this is

...

If you look at stackoverflow posts (links below), you see there is a better, more automatic way to do some of this, there appears to be a training group of ops, and you should be able to add the running mean/std updates into this group by accessing the default computational graph - then you can just run the normal training op of minimizing the optimizer. The computational graph is a sort of global variable that is in scope - this appears to be the real tensorflow way to do this, not there yet! .

Stack overflow activity:

• how-could-i-use-batch-normalization-in-tensorflow
• implementing-batch-normalization-with-tensorflow

...

• create the session
• creating the op to initializing all the variables
• create the tf.Saver() object, we'll call it saver
• run the init op
  • saver automatically ties ops to computational graph to save variables
• call saver.save()
• For restoring, between creating the
  • create initialize variables op
  and running that op,
  • create saver
  • run init op
  • call saver.restore()

Code

• A BatchNormalization class that keeps track of additional training ops: BatchNormalization.py
• Driver program: ex06_tf_batchnorm.py. Note:
  
  • use of a new boolean flag placeholder
  • model class keeps list of instances of the batch normalization classes
  • model returns trainOps by querying all the batch normalization instances
  • trainOps are added to ops used during training