Gradient Descent

GRADIENT-DESCENT( $training\_examples, \eta$)

Each training example is a pair of the form $\langle \vec{x}, t \rangle$, where $\vec{x}$ is the vector of input values, and t is the target output value. $\eta$ is the learning rate (e.g., .05).

• Initialize each w_i to some small random value
• Until the termination condition is met, Do
  • Initialize each $\Delta w_{i}$ to zero.
  • For each $\langle \vec{x}, t \rangle$ in $training\_examples$, Do
    • Input the instance $\vec{x}$ to the unit and compute the output o
    • For each linear unit weight w_i, Do
      
      $$\Delta w_{i} \leftarrow\Delta w_{i} + \eta (t - o) x_{i}$$
      
      

  • For each linear unit weight w_i, Do
    
    $$w_i \leftarrow w_i + \Delta w_i$$