Navigating beyond backpropagation: on alternative training methods for deep neural networks

Navigating beyond backpropagation: on alternative training methods for deep neural networks

연구 분야: Artificial Intelligence

학회: Knowledge and Information Systems

Backpropagation has long been the de facto algorithm for training deep neural networks due to its effectiveness in optimising network parameters. However, the algorithm is not without its limitations, such as high computational requirement, sensitivity to initialisation, weight transport problem, vanishing and exploding gradient, and convergence issues. Many such issues have been highlighted, underscoring the significance of finding an alternative to train deep neural networks. In this survey paper, we aim to critically assess the commended methodologies that either mitigate the constraints inherent in the backpropagation algorithm or provide alternative strategies that obviate the requirement of utilising backpropagation for training neural models. By categorising these methods, we provide a comprehensive understanding of the advancements made in the respective spheres, which can be a valuable resource for researchers and practitioners seeking to explore alternatives to backpropagation in deep neural network training.