# Optimizers in tensorflow with formulas

To understand optimizers in tensorflow, we firstly denote a cost function where represents parameters or weights in the model.

The rule for updating parameters follow:

,

where is is learning_rate.

The python code for gradient descent optimizer is:

optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)


Parameters are updated by the following rule:

The tensorflow code for this optimizer is

optimizer = tf.train.AdagradOptimizer(learning_rate=0.001, initial_accumulator_value=0.1).minimize(cost)


3) RMSprop optimizer

RMSprop is a improvement from Rprop. Following this link, if you want to see the Rprop algorithm. Because RMSprop is designed on mini batch training, so we replace notation with . The scheme for RMSprop is:

Tensorflow supports RMSprop optimization by this following code:

optimizer = tf.train.RMSPropOptimizer(learning_rate=0.001, decay =0.9, momentum=0.0, epsilon=1e-10).minimize(cost)


Adadelta opitmization is similar to RMSprop. It is beacuse the both optimizations solve the fast learning-rate decay problem of Adagrad. The rule for updating weights is:

optimizer = tf.train.AdadeltaOptimizer(learning_rate=0.001, rho=0.95, epsilon = 1e-08).minimize(cost)


where rho, learning_rate, and epsilon represent , and , respectively.

The usage code for Adam optimizer is the following:

optimizer = tf.train.AdamOptimizer(learning_rate=0.001,beta1=0.9,beta2=0.999,epsilon=1e-08).minimize(cost)


6) Momentum optimizer and Nesterov algorithm

Weights with momentum optimizer are updated by the following rule:

.

With Nesterow accelerated gradient technique, the scheme for momentum optimizer is:

One can use the code python for momentum optimization as following:

optimizer = tf.train.MomentumOptimizer.(learning_rate=0.001, momentum=0.9,use_nesterov=False).minimize(cost)