matrix calculus

https://en.wikipedia.org/wiki/Matrix_calculus

 

numerator layout : $$\frac{\partial y}{\partial \textbf{x}} = \begin{bmatrix} \frac{\partial y}{\partial x_1} & \frac{\partial y}{\partial x_2} & \cdots  & \frac{\partial y}{\partial x_n} \\ \end{bmatrix}$$

 

쟈코비안 형태로 

$$\frac{\partial \textbf{y}}{\partial \textbf{x}}=\begin{bmatrix} \frac{\partial y_1}{\partial x_1}& \frac{\partial y_1}{\partial x_2} & \cdots & \frac{\partial y_1}{\partial x_n} \\  \frac{\partial y_2}{\partial x_1}& \frac{\partial y_2}{\partial x_2} & \cdots  & \frac{\partial y_2}{\partial x_n} \\ \vdots & \vdots & \ddots & \vdots\\ \frac{\partial y_n}{\partial x_1} & \frac{\partial y_n}{\partial x_2} & \cdots  & \frac{\partial y_n}{\partial x_n} \\ \end{bmatrix}$$

 

 

denominator layout : $$\frac{\partial y}{\partial \textbf{x}} = \begin{bmatrix}  \frac{\partial y}{\partial x_1}\\ \frac{\partial y}{\partial x_2} \\ \vdots   \\ \frac{\partial y}{\partial x_n} \\ \end{bmatrix}     = \begin{bmatrix} \frac{\partial y}{\partial x_1} & \frac{\partial y}{\partial x_2} & \cdots  & \frac{\partial y}{\partial x_n} \\ \end{bmatrix}^T$$

 

$$\frac{\partial \textbf{y}}{\partial \textbf{x}}=\begin{bmatrix} \frac{\partial y_1}{\partial x_1}& \frac{\partial y_2}{\partial x_1} & \cdots & \frac{\partial y_n}{\partial x_1} \\  \frac{\partial y_1}{\partial x_2}& \frac{\partial y_2}{\partial x_2} & \cdots  & \frac{\partial y_n}{\partial x_2} \\ \vdots & \vdots & \ddots & \vdots\\ \frac{\partial y_1}{\partial x_n} & \frac{\partial y_2}{\partial x_n} & \cdots  & \frac{\partial y_n}{\partial x_n} \\ \end{bmatrix}$$

 

자 그러면, 

x는 열벡터

$$\textbf{x}=\begin{bmatrix} x_1 \\ x_2 \end{bmatrix}$$

A는 행렬

$$\textbf{A}=\begin{bmatrix} 2 & 1\\ 3& 4 \\ \end{bmatrix}$$

일 때,

denominator layout 관점에서

$$\frac{\partial \textbf{Ax}}{\partial \textbf{x}} = ?$$

$$\textbf{Ax} =  \begin{bmatrix} 2x_1+x_2 \\ 3x_1+4x_2 \end{bmatrix}$$

$$\frac{\partial \textbf{Ax}}{\partial \textbf{x}} =  \begin{bmatrix} \frac{\partial (2x_1+x_2)}{\partial x_1} & \frac{\partial (3x_1+4x_2)}{\partial x_1} \\  \frac{\partial (2x_1+x_2)}{\partial x_2}&  \frac{\partial (3x_1+4x_2)}{\partial x_2}\\ \end{bmatrix} = \begin{bmatrix} 2 & 3 \\ 1& 4 \\ \end{bmatrix} = A^T \cdots \left( 1 \right )$$

 

또한

$$\frac{\partial \textbf{x}^T\textbf{A}\textbf{x}}{\partial \textbf{x}} = ?$$

$\textbf{x}^T\textbf{A}\textbf{x} = \begin{bmatrix} x_1 & x_2 \\ \end{bmatrix} \begin{bmatrix} 2 & 1 \\ 3 & 4 \\ \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = \begin{bmatrix} 2x_1+3x_2 & x_1+4x_2 \\ \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = 2x_1^2+4x_1x_2+4x_2^2, \;\; scalar$

$$\frac{\partial \textbf{x}^T\textbf{A}\textbf{x}}{\partial \textbf{x}} =  \begin{bmatrix} \frac{\partial (2x_1^2+4x_1x_2+4x_2^2)}{\partial x_1} \\ \frac{\partial (2x_1^2+4x_1x_2+4x_2^2)}{\partial x_2}\end{bmatrix} = \begin{bmatrix} 4x_1+4x_2 \\ 4x_1+8x_2\end{bmatrix}$$

이 때

$\left ( \textbf{A} + \textbf{A}^T \right )\textbf{x} = \left( \begin{bmatrix} 2 & 1 \\ 3 & 4 \end{bmatrix} + \begin{bmatrix} 2 & 3 \\ 1 & 4 \end{bmatrix} \right )\begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = \begin{bmatrix} 4 & 4 \\ 4 & 8\end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = \begin{bmatrix} 4x_1+4x_2 \\ 4x_1+8x_2\end{bmatrix} = \frac{\partial \textbf{x}^T\textbf{A}\textbf{x}}{\partial \textbf{x}} \cdots \left( 2 \right )$

 

또한

$$\textbf{u}=\textbf{u(x)} = \begin{bmatrix} u_1 \\ u_2 \end{bmatrix} 에 대해$$

$\textbf{Au} = \begin{bmatrix} 2 & 1 \\ 3 & 4 \end{bmatrix} \begin{bmatrix} u_1 \\ u_2\end{bmatrix} = \begin{bmatrix} 2u_1+u_2 \\ 3u_1 + 4u_2\end{bmatrix}$

 $\frac{\partial \textbf{Au}}{\partial \textbf{u}} = \begin{bmatrix} \frac{\partial (2u_1+u_2)}{\partial x_1} & \frac{\partial (3u_1+4u_2)}{\partial x_1} \\ \frac{\partial (2u_1+u_2)}{\partial x_2} & \frac{\partial (3u_1+4u_2)}{\partial x_2}\end{bmatrix}$

이 때 

$\frac{\partial \textbf{u}}{\partial \textbf{x}}\textbf{A}^T = \begin{bmatrix}\frac{\partial u_1}{\partial x_1} & \frac{\partial u_2}{\partial x_1} \\ \frac{\partial u_1}{\partial x_2} & \frac{\partial u_2}{\partial x_2}\end{bmatrix} \begin{bmatrix}2 & 3 \\ 1 & 4\end{bmatrix} = \frac{\partial \textbf{Au}}{\partial \textbf{u}} \cdots \left( 3 \right )$