Train/Test behavior

Typically want the following behavior during train/test

• train:
  • compute and use batch moments
  • update exponential moving averages
• validation:
  • use saved exponential moving average's (ema's) (no updates or computation of batch stastics)
• Final/Test:
  • Ideally, once model trained, do one more pass through data - replace exponential moving averages with averages over whole data (I think most people are lazy here, they just use the last ema's)

Computational Graph

Have to understand the tensorflow computational graph to some degree. When you do:

Stack overflow activity: