Chapter 1: Artificial Neural Network Fundamentals

Feedforward

Activations

Linear

Python

Copy

return x

Sigmoid

Python

Copy

return 1 / (1 + np.exp(-x)) 

Tanh

Python

Copy

return (np.exp(x) - np.exp(-x)) / (np.exp(x) + np.exp(-x))

ReLU (stand for rectified linear unit)

Python

Copy

return np.where(x > 0, x, 0)

Softmax: s(x)=eix∑iexis(x)=\frac{e^x_i}{\sum_i e^{x_i}}s(x)=∑i​exi​eix​​﻿, applied on a entire array of values

Python

Copy

return np.exp(x) / np.sum(np.exp(x))

Loss

Continuous: MSE

L(p,y)=1m∑im(pi−yi)2L(p,y)=\frac{1}{m}\sum_i^m(p_i-y_i)^2L(p,y)=m1​∑im​(pi​−yi​)2﻿​

Python

Copy

return np.mean(np.square(p - y))

Continuous: MAE

Python

Copy

return np.mean(np.abs(p - y))

Categorical: Binary Cross-Entropy

L(p,y)=−1m∑imyilog(pi)+(1−yi)log(1−pi)L(p, y)=-\frac{1}{m}\sum_i^my_ilog(p_i)+(1-y_i)log(1-p_i)L(p,y)=−m1​∑im​yi​log(pi​)+(1−yi​)log(1−pi​)﻿​

Python

Copy

return -np.mean(y * log(p) + (1 - y) * log(1 - p))

Categorical: Category Cross-Entropy

L(p,y)=−1m∑jC∑imyilog(pi)L(p,y)=-\frac{1}{m}\sum^C_j\sum^m_iy_ilog(p_i)L(p,y)=−m1​∑jC​∑im​yi​log(pi​)﻿​

Python

Copy

return np.mean(np.sum(y * log(p), axis=0), axis=1)

Backpropagation

Batch size: the incremental contribution of a greater number of data points while calculating the loss value would follow the law of diminishing returns

batch size is between 32 and 1024, much smaller compared to the total number of data points.

we will apply gradient descent (after feedforward propagation) using one batch at a time until we exhaust all data points within one epoch of training.

Chain rule

MSE  Loss(C)=(y−y^)2MSE\;Loss(C)=(y-\hat{y})^2MSELoss(C)=(y−y^​)2﻿​

y^=a11∗w31+a12∗w32+a13∗w33\hat{y}=a_{11}*w_{31} + a_{12}*w_{32}+a_{13}*w_{33}y^​=a11​∗w31​+a12​∗w32​+a13​∗w33​﻿​

a11=11+e−h11a_{11}=\frac{1}{1+e^{-h_{11}}}a11​=1+e−h11​1​﻿​

h11=x1∗w11+x2∗w21h_{11}=x_1*w_{11}+x_2*w_{21}h11​=x1​∗w11​+x2​∗w21​﻿​

∂L∂w11=∂L∂y^∂y^∂a11∂a11∂h11∂h11∂w11\frac{\partial L}{\partial w_{11}}=\frac{\partial L}{\partial \hat{y}}\frac{\partial \hat{y}}{\partial a_{11}} \frac{\partial a_{11}}{\partial h_{11}}\frac{\partial h_{11}}{\partial w_{11}}∂w11​∂L​=∂y^​∂L​∂a11​∂y^​​∂h11​∂a11​​∂w11​∂h11​​﻿​

so

∂L∂y^=−2(y−y^)\frac{\partial L}{\partial \hat{y}}=-2(y-\hat{y})∂y^​∂L​=−2(y−y^​)﻿​

∂y^∂a11=w31\frac{\partial \hat{y}}{\partial a_{11}}=w_{31}∂a11​∂y^​​=w31​﻿​

∂a11∂h11=e−h111+e−h11=a11∗(1−a11)\frac{\partial a_{11}}{\partial h_{11}}=\frac{e^{-h_{11}}}{1+e^{-h_{11}}}=a_{11}*(1-a_{11})∂h11​∂a11​​=1+e−h11​e−h11​​=a11​∗(1−a11​)﻿​

∂h11∂w11=x1\frac{\partial h_{11}}{\partial w_{11}}=x_1∂w11​∂h11​​=x1​﻿​

then

∂L∂w11=−2(y−y^)∗w31∗a11∗(1−a11)∗x1\frac{\partial L}{\partial w_{11}}=-2(y-\hat{y})*w_{31}*a_{11}*(1-a_{11})*x_1∂w11​∂L​=−2(y−y^​)∗w31​∗a11​∗(1−a11​)∗x1​﻿​

finally

w11=w11−lr∂L∂w11w_{11}=w_{11}-lr\frac{\partial L}{\partial w_{11}}w11​=w11​−lr∂w11​∂L​﻿​

As we update parameters across all layers, the whole process of updating parameters can be parallelized, enabling core GPU