Abstract

The performance of machine learning algorithms is growing quickly due to the development of hardware and training model and method. In some areas, machine learning even has a better performance than natural humans. The neural network is a powerful model for machine learning. To build a practical neural network, there are different choices that must be made at different stages. For deeper and more complex architectures, these minor differences could leads to an enormous model converge speed difference. This talk is a discussion about the general neural network. The content includes different choices and trade-offs on loss function, activation function, parameter update method and forward/backward propagation by both maths model and practical training evidence.