Incremental (Stochastic) Gradient Descent

Batch mode Gradient Descent:

Do until satisfied

1.

Compute the gradient $\nabla E_{D}[\vec{w}]$

2.

$\vec{w} \leftarrow \vec{w} -\eta \nabla E_{D}[\vec{w}]$

Incremental mode Gradient Descent:

Do until satisfied

• For each training example d in D

  1.

  Compute the gradient $\nabla E_{d}[\vec{w}]$

  2.

  $\vec{w} \leftarrow \vec{w} -\eta \nabla E_{d}[\vec{w}]$

$$E_{D}[\vec{w}] \equiv \frac{1}{2}\sum_{d \in D}(t_{d} - o_{d})^{2}$$

$$E_{d}[\vec{w}] \equiv \frac{1}{2}(t_{d} - o_{d})^{2}$$

Incremental Gradient Descent can approximate Batch Gradient Descent arbitrarily closely if $\eta$ made small enough