Hi Pavan, Im studying for the ML Certification, so not an expert on the subject but at least familiar with it.
For the dying neurons problem during the learning phase we have methods like Leaky RELU, which allows those previously dead neurons on other methods (RELU, TanH, Sigmoid...) to "leak" part of its weights to the backpropagation calculation. It basically allows you to define a negative slope for your activator so that these nodes contribute also to the calculation. Another method is the Maxout, which is supposed to be superior to Leaky RELY but it doubles the amount of parameters used in the calculation of the gradients.
As a personal recomendation, one of the best books Ive found on ML is from one of our IBM colleagues, Charu Aggarwal : "Neural Networks and Deep Learning" , if you really want to understand the most complex topics, you can find them here.
------------------------------
Daniel Lopez Sainz
------------------------------
Original Message:
Sent: Wed October 12, 2022 01:46 PM
From: Pavan Saish Naru
Subject: Hyperparameter Tuning in Deep learning
Is ReLu the only option as a better optimizer for a neural network to give better performance? And what could be the best possible way to deal with Gradient Vanishing problems in Neural networks?
------------------------------
Pavan Saish Naru
------------------------------
#GlobalAIandDataScience
#GlobalDataScience