In the paper, a novel gradient-based feature importance method for neural networks is described. This method is compared to the existing feature importance method using a trained neural network, which predicts the optimal gains in real time, for a steering controller on a mobile robot. The neural network is trained using the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) algorithm, in order to minimize an objective function. From an analysis using the feature importance methods, key inputs are determined, and their contribution to the neural network’s prediction are observed. Furthermore, using a first-order Taylor approximation of the neural network, an improved control law is determined and tested based on the results of the gradient-based feature importance method. This analysis is then applied to an existing neural network using real-world experiments, in order to determine the behavior of the gains with respect to each input, and allows for a glimpse into the neural network’s inner workings in order to improve its explainability.