Sparsity Aware Learning in Feedback-Driven Differential Recurrent Neural Networks

Sparsity Aware Learning in Feedback-Driven Differential Recurrent Neural Networks

연구 분야: Artificial Intelligence

학회: International Conference on Artificial Neural Networks

The avenue of training differential recurrent neural networks (d-RNNs) emerges as a promising route to address spatio-temporally evolving systems. The effective learning of variable information gain makes training d-RNNs important for their inherent derivative of states property. In addition to training readout weights, the optimization of the intrinsic recurrent connection of the d-RNNs prove significant for performance enhancement. We introduce sparsity aware learning for feedback-driven differential RNNs, tailored to adapt neuron-specific learnable thresholds. This enables neurons with lower sparsity thresholds to play a more significant role in decision-making processes, while simultaneously dampening the influence of neurons with higher thresholds. This learning paradigm is in addition to optimizing the recurrent connectivity matrix of the d-RNN for mastering tasks demanding complex spatio-temporal input-output mappings. Our learning approach yields networks capable of accomplishing classification and sequential learning tasks with fewer neurons while exhibiting heightened performance compared to existing differential recurrent network training least-squares methods. Sparse d-RNN improves on spatio-temporal learning tasks by a cumulative error rate reduction of 20 % in mean squared error for dynamic system mimicking tasks and 2.97 % increase in test accuracy for classification tasks compared to existing target based learning of recurrence in feedback driven d-RNNs.