matrix calculus

https://en.wikipedia.org/wiki/Matrix_calculus

 

numerator layout : $$  \frac{\partial y}{\partial \textbf{x}} = \begin{bmatrix} \frac{\partial y}{\partial x_1} & \frac{\partial y}{\partial x_2} & \cdots  & \frac{\partial y}{\partial x_n} \\
\end{bmatrix} $$

 

쟈코비안 형태로 

$$ \frac{\partial \textbf{y}}{\partial \textbf{x}}=\begin{bmatrix} \frac{\partial y_1}{\partial x_1}& \frac{\partial y_1}{\partial x_2} & \cdots & \frac{\partial y_1}{\partial x_n} \\
 \frac{\partial y_2}{\partial x_1}& \frac{\partial y_2}{\partial x_2} & \cdots  & \frac{\partial y_2}{\partial x_n} \\ \vdots & \vdots & \ddots & \vdots\\ \frac{\partial y_n}{\partial x_1} & \frac{\partial y_n}{\partial x_2} & \cdots  & \frac{\partial y_n}{\partial x_n} \\ \end{bmatrix} $$

 

 

denominator layout : $$ \frac{\partial y}{\partial \textbf{x}} = \begin{bmatrix}
 \frac{\partial y}{\partial x_1}\\ \frac{\partial y}{\partial x_2} \\ \vdots 
 \\ \frac{\partial y}{\partial x_n} \\ \end{bmatrix}    
= \begin{bmatrix}
\frac{\partial y}{\partial x_1} & \frac{\partial y}{\partial x_2} & \cdots  & \frac{\partial y}{\partial x_n} \\
\end{bmatrix}^T $$

 

$$  \frac{\partial \textbf{y}}{\partial \textbf{x}}=\begin{bmatrix} \frac{\partial y_1}{\partial x_1}& \frac{\partial y_2}{\partial x_1} & \cdots & \frac{\partial y_n}{\partial x_1} \\
 \frac{\partial y_1}{\partial x_2}& \frac{\partial y_2}{\partial x_2} & \cdots  & \frac{\partial y_n}{\partial x_2} \\ \vdots & \vdots & \ddots & \vdots\\ \frac{\partial y_1}{\partial x_n} & \frac{\partial y_2}{\partial x_n} & \cdots  & \frac{\partial y_n}{\partial x_n} \\ \end{bmatrix} $$

 

자 그러면, 

x는 열벡터

$$ \textbf{x}=\begin{bmatrix} x_1 \\ x_2 \end{bmatrix} $$

A는 행렬

$$ \textbf{A}=\begin{bmatrix} 2 & 1\\ 3& 4 \\ \end{bmatrix}  $$

일 때,

denominator layout 관점에서

$$  \frac{\partial \textbf{Ax}}{\partial \textbf{x}} = ? $$

$$ \textbf{Ax} =  \begin{bmatrix} 2x_1+x_2 \\ 3x_1+4x_2 \end{bmatrix} $$

$$ \frac{\partial \textbf{Ax}}{\partial \textbf{x}} =  \begin{bmatrix}
\frac{\partial (2x_1+x_2)}{\partial x_1} & \frac{\partial (3x_1+4x_2)}{\partial x_1} \\
 \frac{\partial (2x_1+x_2)}{\partial x_2}&  \frac{\partial (3x_1+4x_2)}{\partial x_2}\\
\end{bmatrix} = \begin{bmatrix} 2 & 3 \\ 1& 4 \\ \end{bmatrix} = A^T \cdots \left( 1 \right ) $$

 

또한

$$ \frac{\partial \textbf{x}^T\textbf{A}\textbf{x}}{\partial \textbf{x}} = ?$$

$  \textbf{x}^T\textbf{A}\textbf{x} = \begin{bmatrix} x_1 & x_2 \\ \end{bmatrix} \begin{bmatrix} 2 & 1 \\ 3 & 4 \\ \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = \begin{bmatrix} 2x_1+3x_2 & x_1+4x_2 \\ \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = 2x_1^2+4x_1x_2+4x_2^2, \;\; scalar $

$$  \frac{\partial \textbf{x}^T\textbf{A}\textbf{x}}{\partial \textbf{x}} =  \begin{bmatrix} \frac{\partial (2x_1^2+4x_1x_2+4x_2^2)}{\partial x_1} \\ \frac{\partial (2x_1^2+4x_1x_2+4x_2^2)}{\partial x_2}\end{bmatrix} = \begin{bmatrix} 4x_1+4x_2 \\ 4x_1+8x_2\end{bmatrix} $$

이 때

$ \left ( \textbf{A} + \textbf{A}^T \right )\textbf{x} = \left( \begin{bmatrix} 2 & 1 \\ 3 & 4 \end{bmatrix} + \begin{bmatrix} 2 & 3 \\ 1 & 4 \end{bmatrix} \right )\begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = \begin{bmatrix} 4 & 4 \\ 4 & 8\end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = \begin{bmatrix} 4x_1+4x_2 \\ 4x_1+8x_2\end{bmatrix} = \frac{\partial \textbf{x}^T\textbf{A}\textbf{x}}{\partial \textbf{x}} \cdots \left( 2 \right ) $

 

또한

$$ \textbf{u}=\textbf{u(x)} = \begin{bmatrix} u_1 \\ u_2 \end{bmatrix} 에 대해$$

$ \textbf{Au} = \begin{bmatrix} 2 & 1 \\ 3 & 4 \end{bmatrix} \begin{bmatrix} u_1 \\ u_2\end{bmatrix} = \begin{bmatrix} 2u_1+u_2 \\ 3u_1 + 4u_2\end{bmatrix} $

 $ \frac{\partial \textbf{Au}}{\partial \textbf{u}} = \begin{bmatrix} \frac{\partial (2u_1+u_2)}{\partial x_1} & \frac{\partial (3u_1+4u_2)}{\partial x_1} \\ \frac{\partial (2u_1+u_2)}{\partial x_2} & \frac{\partial (3u_1+4u_2)}{\partial x_2}\end{bmatrix} $

이 때 

$ \frac{\partial \textbf{u}}{\partial \textbf{x}}\textbf{A}^T = \begin{bmatrix}\frac{\partial u_1}{\partial x_1} & \frac{\partial u_2}{\partial x_1} \\ \frac{\partial u_1}{\partial x_2} & \frac{\partial u_2}{\partial x_2}\end{bmatrix} \begin{bmatrix}2 & 3 \\ 1 & 4\end{bmatrix} = \frac{\partial \textbf{Au}}{\partial \textbf{u}} \cdots \left( 3 \right )$