Tips:

1. Look at the data
   1. Human Baseline: Can a human can predict the Y label, if human can't it will be tough for a machine to do so.
   2. Do exploratory data analysis
      1. Missing data
      2. Outliers
      3. Imbalanced data
      4. Data features correlation
2. Start Simple
   1. Start with a simple baseline model
   2. Try adding features and layers unless it overfits on a small sample of data (manually hand written easy-to-learn data)
      1. Train and validate on same data
      2. Assumption is if the model can't learn a small amount of data, how can it learn a lot of data
      3. When overfitted add the extra data
      4. And add more features or layers to increase performance on the whole daya
      5. Add Regularization when model starts Overfitting
3. Test each part part by part, test dataloader, test each of the model part with some custom data and finally run through the whole model
4. Use model.summary() or model.plot() to check layer output shapes, network connection.
5. Check model layer initialization
6. Look for exploding or vanishing gradient
7. Deep Learning fails silently

References

1. Keras Debugging Tips
2. Google Developer Course
3. Andrej Karpathy