|
|
| Line 1: |
Line 1: |
| {{Calculus |Multivariable}}
| | == that man gratefully == |
|
| |
|
| In [[mathematics]], '''matrix calculus''' is a specialized notation for doing [[multivariable calculus]], especially over spaces of [[matrix (mathematics)|matrices]]. It collects the various [[partial derivative]]s of a single function with respect to many variables, and/or of a multivariate function with respect to a single variable, into vectors and matrices that can be treated as single entities. This greatly simplifies operations such as finding the maximum or minimum of a multivariate function and solving systems of [[differential equation]]s. The notation used here is commonly used in [[statistics]] and [[engineering]], while the [[tensor index notation]] is preferred in [[physics]].
| | When standing in the first row position of [http://www.dmwai.com/webalizer/kate-spade-4.html ケイトスペード 時計 取り扱い店] poor households, Wang Bin cattle led to his hands, his face excited, but also knock heads, Wang Bin unexpectedly slapped straight on the head, kick to kick away, that man gratefully, hug Tau, ouch, but also intimate than the gossips<br><br>It is the county several units Bafu funding, in addition to a poverty alleviation projects, but also to raise funds to purchase thirty high-quality [http://www.dmwai.com/webalizer/kate-spade-12.html kate spade 長財布] cattle, transport police came back and saw joy of the villagers, I [http://www.dmwai.com/webalizer/kate-spade-4.html ケイトスペード ショルダーバッグ] sin laughed, though still so cheap cheap place, but a little more comforting colors, this cheerful, successful outcome, although belated, and for many days, but after all, to the Well.<br><br>he sat in the car, [http://www.dmwai.com/webalizer/kate-spade-15.html ケイトスペード マザーズバッグ] what wistfully thinking, looking at the mountains, watching the jungle, watching the excitement, emotions rising masses, what he seems to think he has [http://www.dmwai.com/webalizer/kate-spade-14.html ケイトスペード クラッチバッグ] enough ⊥ exciting thing, quietly underground car, the crowd booing Li Yifeng being grabbed, and ran to the police station to the wall of the ............<br><br>in rapturous distributed into the climax of the ceremony, when a v police are maneuvering to go to catch Sheepshead Cliff Township, driving |
| | 相关的主题文章： |
| | <ul> |
| | |
| | <li>[http://home.ejiangtang.com.cn/space.php?uid=275624&do=blog&id=3373503 http://home.ejiangtang.com.cn/space.php?uid=275624&do=blog&id=3373503]</li> |
| | |
| | <li>[http://happy-marron.com/cgi/diary/nicky/nicky.cgi http://happy-marron.com/cgi/diary/nicky/nicky.cgi]</li> |
| | |
| | <li>[http://bzzlyw.dns143.53nic.com/forum.php?mod=viewthread&tid=191491 http://bzzlyw.dns143.53nic.com/forum.php?mod=viewthread&tid=191491]</li> |
| | |
| | </ul> |
|
| |
|
| Two competing notational conventions split the field of '''matrix calculus''' into two separate groups. The two groups can be distinguished by whether they write the derivative of a scalar with respect to a vector as a [[column vector]] or a row vector. Both of these conventions are possible even when the common assumption is made that vectors should be treated as column vectors when combined with matrices (rather than [[row vector]]s). A single convention can be somewhat standard throughout a single field that commonly use matrix calculus (e.g. [[econometrics]], [[statistics]], [[estimation theory]] and [[machine learning]]). However, even within a given field different authors can be found using competing conventions. Authors of both groups often write as though their specific convention is standard. Serious mistakes can result when combining results from different authors without carefully verifying that compatible notations are used. Therefore great care should be taken to ensure notational consistency. Definitions of these two conventions and comparisons between them are collected in the [[#Layout conventions|layout conventions]] section.
| | == 'signal at three.' == |
|
| |
|
| == Scope == | | Section ...... Yes, [http://www.dmwai.com/webalizer/kate-spade-7.html ケイトスペードのバッグ] that two newspaper jing phone ...... check [http://www.dmwai.com/webalizer/kate-spade-9.html kate spade 財布 ゴールド] should he deliberately threw out, the purpose is to divert attention, indirectly blocked jing mobilize our forces ...... Tut, may have been late, maybe now exit the. 'Xu Pingqiu repentant authentic.<br><br>Lee Cheuk also figured out, drove, then step forward in the notification field.<br><br>evil, and soon come back, and that the Department reported jing phone, actually pass forward.<br><br>'grasping' Lee Cheuk apart from anything else, follow the directions, jing [http://www.dmwai.com/webalizer/kate-spade-2.html マザーズバッグ ケイトスペード] whistle sounded, the rampage in the streets, straight to the direction of the signal source.<br><br>Overseas Chinese Hospital, there is no sign of approaching several cars, a group of plainclothes criminal off jing follow directions, separately ran into the hospital.<br><br>follow-up came a dozen cars jing car, front door, back door, fence, in a very short period of time, to [http://www.dmwai.com/webalizer/kate-spade-6.html kate spade マザーズバッグ] a packed around here.<br><br>'signal at three.'<br><br>'on [http://www.dmwai.com/webalizer/kate-spade-3.html 財布 kate spade] you, you keep the floor ...... mouth.'<br><br>'find |
| | 相关的主题文章： |
| | <ul> |
| | |
| | <li>[http://ykusuda.s87.xrea.com/cgi/zatudan/yybbs.cgi http://ykusuda.s87.xrea.com/cgi/zatudan/yybbs.cgi]</li> |
| | |
| | <li>[http://www.tomikain.com/index.php?title=User:Ityykrvtzv#mine_was_a_brigade_vice_squad http://www.tomikain.com/index.php?title=User:Ityykrvtzv#mine_was_a_brigade_vice_squad]</li> |
| | |
| | <li>[http://www.midwayramblers.com/cgi-bin/guestbook.cgi http://www.midwayramblers.com/cgi-bin/guestbook.cgi]</li> |
| | |
| | </ul> |
|
| |
|
| Matrix calculus refers to a number of different notations that use matrices and vectors to collect the derivative of each component of the dependent variable with respect to each component of the independent variable. In general, the independent variable can be a scalar, a vector, or a matrix while the dependent variable can be any of these as well. Each different situation will lead to a different set of rules, or a separate [[calculus]], using the broader sense of the term. Matrix notation serves as a convenient way to collect the many derivatives in an organized way.
| | == or else the two hole play was so badly == |
|
| |
|
| As a first example, consider the [[gradient]] from [[vector calculus]]. For a scalar function of three independent variables, <math>f(x_1, x_2, x_3)</math>, the gradient is given by the vector equation
| | Temperature of minus nine degrees, visibility twenty meters, raids quite difficult .........<br><br>This is a less than a big net, the team's reaction heavy case is nothing less rapidly diffuse color of snow seen within two hours has informed the five neighboring city, a radius [http://www.dmwai.com/webalizer/kate-spade-1.html ケイトスペード リボン バッグ] of two [http://www.dmwai.com/webalizer/kate-spade-0.html ケイトスペード 人気 財布] hundred kilometers have been stationed on the investigation of police, considering the weather, the suspect fled into the city is likely to turn to when the incident, so the investigation and search the city, almost carpet paved, address, work, social relationships, possible hidden place, [http://www.dmwai.com/webalizer/kate-spade-10.html ケイトスペード財布評判] soon to be a one plane out of Interpol.<br><br>'Gebao Long? New Year back home, right? [http://www.dmwai.com/webalizer/kate-spade-4.html kate spade バッグ] out what matter?' neighbor, was the first to ask Interpol up.<br><br>'that the couple often in battle, usually hit than New Year is also warming up.' neighbors, gloating<br><br>'Yes, comrades, I heard ...... I heard ah, Ge Baolong babysitter wife said yes, in fact, is to give [http://www.dmwai.com/webalizer/kate-spade-3.html ケイトスペード バッグ 通販] people as concubine, a good friend outside a lot, really, I did not talk nonsense ah, or else the two hole play was so badly? 'neighbor, a bitch type of wretched men |
| :<math>\nabla f = \frac{\partial f}{\partial x_1} \hat{x_1} + \frac{\partial f}{\partial x_2} \hat{x_2} + \frac{\partial f}{\partial x_3} \hat{x_3}</math>, | | 相关的主题文章： |
| where <math>\hat{x_i}</math> represents a unit vector in the <math>x_i</math> direction for <math>1\le i \le 3</math>. This type of generalized derivative can be seen as the derivative of a scalar, ''f'', with respect to a vector, <math>\mathbf{x}</math> and its result can be easily collected in vector form.
| | <ul> |
| :<math>\nabla f = \frac{\partial f}{\partial \mathbf{x}} = | | |
| \begin{bmatrix}
| | <li>[http://www.sdxinxi.net/plus/feedback.php?aid=20 http://www.sdxinxi.net/plus/feedback.php?aid=20]</li> |
| \frac{\partial f}{\partial x_1} &
| | |
| \frac{\partial f}{\partial x_2} &
| | <li>[http://blog.ingridnewkirk.com/movabletype/mt-search.cgi http://blog.ingridnewkirk.com/movabletype/mt-search.cgi]</li> |
| \frac{\partial f}{\partial x_3} \\
| | |
| \end{bmatrix}.
| | <li>[http://kangdavalve.com/plus/view.php?aid=1140942 http://kangdavalve.com/plus/view.php?aid=1140942]</li> |
| </math>{{citation needed|reason=This is not the usual notation of a vector; a vector is normally written as a column matrix|date=March 2013}} | | |
| More complicated examples include the derivative of a scalar function with respect to a matrix, known as the [[#Derivatives with matrices|gradient matrix]], which collects the derivative with respect to each matrix element in the corresponding position in the resulting matrix. In that case the scalar must be a function of each of the independent variables in the matrix. As another example, if we have an ''n''-vector of dependent variables, or functions, of ''m'' independent variables we might consider the derivative of the dependent vector with respect to the independent vector. The result could be collected in an ''m×n'' matrix consisting of all of the possible derivative combinations. There are, of course, a total of nine possibilities using scalars, vectors, and matrices. Notice that as we consider higher numbers of components in each of the independent and dependent variables we can be left with a very large number of possibilities.
| | </ul> |
|
| |
|
| The six kinds of derivatives that can be most neatly organized in matrix form are collected in the following table.<ref name="minka"/>
| | == Ji Tiger camp got a piece of land a year ago == |
| {| class="wikitable" style="text-align:center; width:50%;"
| |
| |+ Types of Matrix Derivatives
| |
| ! scope="col" | Types
| |
| ! scope="col" | Scalar
| |
| ! scope="col" | Vector
| |
| ! scope="col" | Matrix
| |
| |-
| |
| ! scope="row" | Scalar
| |
| | <math>\frac{\partial y}{\partial x} </math> || <math>\frac{\partial \mathbf{y}}{\partial x} </math> || <math>\frac{\partial \mathbf{Y}}{\partial x} </math>
| |
| |-
| |
| ! scope="row" | Vector
| |
| | <math>\frac{\partial y}{\partial \mathbf{x}} </math> ||<math>\frac{\partial \mathbf{y}}{\partial \mathbf{x}} </math> ||
| |
| |-
| |
| ! scope="row" | Matrix
| |
| | <math>\frac{\partial y}{\partial \mathbf{X}} </math> || ||
| |
| |}
| |
|
| |
|
| Here, we have used the term "matrix" in its most general sense, recognizing that vectors and scalars are simply matrices with one column and then one row respectively. Moreover, we have used bold letters to indicate vectors and bold capital letters for matrices. This notation is used throughout.
| | , Wei Jincheng is a mysterious smile, and I looked very mysterious crime, obscure authentic with: 'I advise you one, do not know you listen.'<br><br>'Nonsense, I [http://www.dmwai.com/webalizer/kate-spade-14.html kate spade マザーズバッグ] came to ask you to come.' I sin road.<br><br>'. then farther away from Xinghai investment,' Wei [http://www.dmwai.com/webalizer/kate-spade-6.html ケイトスペード リボン バッグ] Jincheng cautiously authentic look silly daze over the crime, he added with: [http://www.dmwai.com/webalizer/kate-spade-8.html ハンドバッグ ケイトスペード] 'Xinghai investment real estate are subordinate Xinghai Xinghai Group, Ji Tiger camp got a piece of land a year ago, when they can not even office locations are not, on this approval directly sold to Jinda coal, sold a four-point hundreds of millions ...... real boss, I do not know who he is, but the state-owned enterprises do business, it is not [http://www.dmwai.com/webalizer/kate-spade-13.html kate spade ハンドバッグ] generally people ...... Now you see, is suddenly set of funds, they may also have a big move ...... at least I know that strong sales of [http://www.dmwai.com/webalizer/kate-spade-6.html ケイトスペード リボン バッグ] coal a few years, they forcibly shares, and can be more than a coal mine, but they do not do real business, and short-term speculation, today shares, sale of shares ......... tomorrow you say how much they earn? then stake in the mine, but of millions of units. '<br><br>hissing, I |
| | 相关的主题文章： |
| | <ul> |
| | |
| | <li>[http://www.runqingcika.com/plus/view.php?aid=20148 http://www.runqingcika.com/plus/view.php?aid=20148]</li> |
| | |
| | <li>[http://www.showcase-usa.com/cgi-bin/classifieds/classifieds.cgi http://www.showcase-usa.com/cgi-bin/classifieds/classifieds.cgi]</li> |
| | |
| | <li>[http://www7a.biglobe.ne.jp/~rokka/cgi-bin/honey/honey.cgi http://www7a.biglobe.ne.jp/~rokka/cgi-bin/honey/honey.cgi]</li> |
| | |
| | </ul> |
|
| |
|
| Notice that we could also talk about the derivative of a vector with respect to a matrix, or any of the other unfilled cells in our table. However, these derivatives are most naturally organized in a [[tensor]] of rank higher than 2, so that they do not fit neatly into a matrix. In the following three sections we will define each one of these derivatives and relate them to other branches of mathematics. See the [[#Layout conventions|layout conventions]] section for a more detailed table.
| | == There is the feeling of fear == |
|
| |
|
| === Relation to other derivatives ===
| | There is the feeling of fear, although it is people feel fear. He quietly watching, like thinking about something, as in somewhere looking for something.<br>When<br>Zhou Juan to speak, was stopped 肖梦琪 the gently back [http://www.dmwai.com/webalizer/kate-spade-11.html ケイトスペードニューヨーク 財布] a few steps, she knew a lot of Ming Wu always appear inadvertently, when, for example, this time.<br><br>abruptly, I stood up wildly crime shock, he [http://www.dmwai.com/webalizer/kate-spade-10.html ケイトスペード財布評判] muttered, not knowing the language, ran on the upstairs, they did not understand when he ran down, ran into the [http://www.dmwai.com/webalizer/kate-spade-2.html 時計 ケイトスペード] main room, seems to do a few violent action ...... and soon they rushed out, went straight to the gate, made a stinging attack behind the action ...... all of a sudden as if he was committing the crime, as in the big mouth panting, shortness saying:<br><br>'...... play it several times, just three minutes ...... he was drunk in a state of extreme excitement completed ...... Last night neighbors heard the loud wailing call ...... [http://www.dmwai.com/webalizer/kate-spade-0.html ケイトスペード中古バッグ] killed a man [http://www.dmwai.com/webalizer/kate-spade-14.html ケイトスペード ハンドバッグ] after the first time it should be ... ... right, it is crazy, but is happy again regret ...... hated Lao Zhangren one, kill the old couple very happy, Lian Tong dozen knives; no children of his own, |
| | 相关的主题文章： |
| | <ul> |
| | |
| | <li>[http://www.yemaowo.com/thread-19701-1-1.html http://www.yemaowo.com/thread-19701-1-1.html]</li> |
| | |
| | <li>[http://www.vibrantsantafe.com/cgi-bin/vibrant/user.cgi http://www.vibrantsantafe.com/cgi-bin/vibrant/user.cgi]</li> |
| | |
| | <li>[http://www.letmebreathe.net/cgi-bin/gb.cgi http://www.letmebreathe.net/cgi-bin/gb.cgi]</li> |
| | |
| | </ul> |
|
| |
|
| The matrix derivative is a convenient notation for keeping track of partial derivatives for doing calculations. The [[Fréchet derivative]] is the standard way in the setting of [[functional analysis]] to take derivatives with respect to vectors. In the case that a matrix function of a matrix is Fréchet differentiable, the two derivatives will agree up to translation of notations. As is the case in general for [[partial derivative]]s, some formulae may extend under weaker analytic conditions than the existence of the derivative as approximating linear mapping.
| | == the history of the Qing Huai stopping == |
|
| |
|
| === Usages ===
| | Come. He could [http://www.dmwai.com/webalizer/kate-spade-11.html ケイトスペードニューヨーク 財布] not explain directly. 'I stood kinky crime two light, proud authentic.<br><br>'You can not pick persimmons net soft pinch ah, I sympathize Qinju Chang's experience.' Li Mei dumbfounding authentic with, whoever I met sin be unlucky, coax fraud, estimated to be underwear they have been cheated . Speaking here, she and Xiaomeng Qi, Yu Feng three people smile overjoyed, especially the I sin induced [http://www.dmwai.com/webalizer/kate-spade-1.html kate spade トートバッグ] Li Mei Qin meritorious deeds recording a place that seek private secret whispering, I do not know, absolutely not suspect is a black police bribes.<br><br>sins are embarrassed that I hear, he jumped out of the car, and the history [http://www.dmwai.com/webalizer/kate-spade-15.html ケイトスペード アウトレット バッグ] of the Qing Huai greeted, asked to report the matter to the middle, then go [http://www.dmwai.com/webalizer/kate-spade-14.html kate spade マザーズバッグ] out 'bribes' the mouse has returned the message came: five hundred thousand , a lot of points<br><br>marked brother sigh, this really Nima money na, a black plastic bag containing five bundles, thrown into my car and ran away, just like [http://www.dmwai.com/webalizer/kate-spade-14.html ケイトスペード ハンドバッグ] throwing a roll of toilet paper.<br><br>'Oh ...... Qin Bureau so much money, it seems a little less open.' I reported the crime to things down, going back to the years, the history of the Qing Huai stopping |
| | | 相关的主题文章： |
| Matrix calculus is used for deriving optimal stochastic estimators, often involving the use of [[Lagrange multipliers]]. This includes the derivation of:
| | <ul> |
| * [[Kalman filter]]
| | |
| * [[Wiener filter]]
| | <li>[http://www.theneitherworld.com/cgi-bin/bjguestbook/guestbook.cgi http://www.theneitherworld.com/cgi-bin/bjguestbook/guestbook.cgi]</li> |
| * [[Expectation-maximization algorithm#Gaussian mixture|Expectation-maximization algorithm for Gaussian mixture]]
| | |
| | | <li>[http://www.commonapp.com.cn/plus/feedback.php?aid=30 http://www.commonapp.com.cn/plus/feedback.php?aid=30]</li> |
| == Notation ==
| | |
| | | <li>[http://www.baiweidrill.com.cn/plus/feedback.php?aid=163 http://www.baiweidrill.com.cn/plus/feedback.php?aid=163]</li> |
| The vector and matrix derivatives presented in the sections to follow take full advantage of [[matrix notation]], using a single variable to represent a large number of variables. In what follows we will distinguish scalars, vectors and matrices by their typeface. We will let ''M''(''n'',''m'') denote the space of [[real number|real]] ''n×m'' matrices with ''n'' rows and ''m'' columns. Such matrices will be denoted using bold capital letters: '''A''', '''X''', '''Y''', etc. An element of ''M''(''n'',1), that is, a [[column vector]], is denoted with a boldface lowercase letter: '''a''', '''x''', '''y''', etc. An element of ''M''(1,1) is a scalar, denoted with lowercase italic typeface: ''a'', ''t'', ''x'', etc. '''X'''<sup>T</sup> denotes matrix [[transpose]], tr('''X''') is the [[Trace (linear algebra)|trace]], and det('''X''') is the [[determinant]]. All functions are assumed to be of [[differentiability class]] ''C''<sup>1</sup> unless otherwise noted. Generally letters from first half of the alphabet (a, b, c, …) will be used to denote constants, and from the second half (t, x, y, …) to denote variables.
| | |
| | | </ul> |
| '''NOTE''': As mentioned above, there are competing notations for laying out systems of [[partial derivative]]s in vectors and matrices, and no standard appears to be emerging yet. The next two introductory sections use the [[#Layout conventions|numerator layout convention]] simply for the purposes of convenience, to avoid overly complicating the discussion. The section after them discusses [[#Layout conventions|layout conventions]] in more detail. It is important to realize the following:
| |
| #Despite the use of the terms "numerator layout" and "denominator layout", there are actually more than two possible notational choices involved. The reason is that the choice of numerator vs. denominator (or in some situations, numerator vs. mixed) can be made independently for scalar-by-vector, vector-by-scalar, vector-by-vector, and scalar-by-matrix derivatives, and a number of authors mix and match their layout choices in various ways.
| |
| #The choice of numerator layout in the introductory sections below does not imply that this is the "correct" or "superior" choice. There are advantages and disadvantages to the various layout types. Serious mistakes can result from carelessly combining formulas written in different layouts, and converting from one layout to another requires care to avoid errors. As a result, when working with existing formulas the best policy is probably to identify whichever layout is used and maintain consistency with it, rather than attempting to use the same layout in all situations.
| |
| | |
| === Alternatives ===
| |
| | |
| The [[tensor index notation]] with its [[Einstein summation]] convention is very similar to the matrix calculus, except one writes only a single component at a time. It has the advantage that one can easily manipulate arbitrarily high rank tensors, whereas tensors of rank higher than two are quite unwieldy with matrix notation. All of the work here can be done in this notation without use of the single-variable matrix notation. However, many problems in estimation theory and other areas of applied mathematics would result in too many indices to properly keep track of, pointing in favor of matrix calculus in those areas. Also, Einstein notation can be very useful in proving the identities presented here, as an alternative to typical element notation, which can become cumbersome when the explicit sums are carried around. Note that a matrix can be considered a tensor of rank two.
| |
| | |
| == Derivatives with vectors ==
| |
| | |
| {{Main|Vector calculus}}
| |
| Because vectors are matrices with only one column, the simplest matrix derivatives are vector derivatives.
| |
| | |
| The notations developed here can accommodate the usual operations of [[vector calculus]] by identifying the space ''M''(''n'',1) of ''n''-vectors with the [[Euclidean space]] '''R'''<sup>''n''</sup>, and the scalar ''M''(1,1) is identified with '''R'''. The corresponding concept from vector calculus is indicated at the end of each subsection.
| |
| | |
| '''NOTE''': The discussion in this section assumes the [[#Layout conventions|numerator layout convention]] for pedagogical purposes. Some authors use different conventions. The section on [[#Layout conventions|layout conventions]] discusses this issue in greater detail. The identities given further down are presented in forms that can be used in conjunction with all common layout conventions.
| |
| | |
| === Vector-by-scalar ===
| |
| | |
| The [[derivative]] of a [[Euclidean vector|vector]]
| |
| <math>\mathbf{y} =
| |
| \begin{bmatrix}
| |
| y_1 \\
| |
| y_2 \\
| |
| \vdots \\
| |
| y_m \\
| |
| \end{bmatrix}
| |
| </math>,
| |
| by a [[Scalar (mathematics)|scalar]] ''x'' is written (in [[#Layout conventions|numerator layout notation]]) as
| |
| | |
| :<math>
| |
| \frac{\partial \mathbf{y}}{\partial x} =
| |
| \begin{bmatrix}
| |
| \frac{\partial y_1}{\partial x}\\
| |
| \frac{\partial y_2}{\partial x}\\
| |
| \vdots\\
| |
| \frac{\partial y_m}{\partial x}\\
| |
| \end{bmatrix}.
| |
| </math>
| |
| | |
| In [[vector calculus]] the derivative of a vector '''y''' with respect to a scalar ''x'' is known as the '''[[tangent vector]]''' of the vector '''y''', <math>\frac{\partial \mathbf{y}}{\partial x}</math>. Notice here that '''y''':'''R''' <math>\rightarrow</math> '''R'''<sup>''m''</sup>.
| |
| | |
| '''Example''' Simple examples of this include the [[velocity]] vector in [[Euclidean space]], which is the [[tangent vector]] of the [[Position (vector)|position]] vector (considered as a function of time). Also, the [[acceleration]] is the tangent vector of the velocity.
| |
| | |
| === Scalar-by-vector ===
| |
| | |
| The [[derivative]] of a [[scalar (mathematics)|scalar]] ''y'' by a vector
| |
| <math>\mathbf{x} =
| |
| \begin{bmatrix}
| |
| x_1 \\
| |
| x_2 \\
| |
| \vdots \\
| |
| x_n \\
| |
| \end{bmatrix}
| |
| </math>,
| |
| is written (in [[#Layout conventions|numerator layout notation]]) as
| |
| | |
| :<math>
| |
| \frac{\partial y}{\partial \mathbf{x}} =
| |
| \left[
| |
| \frac{\partial y}{\partial x_1}
| |
| \ \ \frac{\partial y}{\partial x_2}
| |
| \ \ \cdots
| |
| \ \ \frac{\partial y}{\partial x_n}
| |
| \right].
| |
| </math>
| |
| | |
| In [[vector calculus]] the [[gradient]] of a scalar field ''y'', in the space '''R'''<sup>''n''</sup> whose independent coordinates are the components of '''x''' is the derivative of a scalar by a vector. In physics, the [[electric field]] is the vector [[gradient]] of the [[electric potential]].
| |
| | |
| The [[directional derivative]] of a scalar function ''f''('''x''') of the space vector '''x''' in the direction of the unit vector '''u''' is defined using the gradient as follows.
| |
| :<math>\nabla_{\bold{u}}{f}(\bold{x}) = \nabla f(\bold{x}) \cdot \bold{u}</math>
| |
| Using the notation just defined for the derivative of a scalar with respect to a vector we can re-write the directional derivative as
| |
| <math>\nabla_\mathbf{u} f = \frac{\partial f}{\partial \mathbf{x}}\mathbf{u}.</math>
| |
| This type of notation will be nice when proving product rules and chain rules that come out looking similar to what we are familiar with for the scalar [[derivative]].
| |
| | |
| === Vector-by-vector ===
| |
| | |
| Each of the previous two cases can be considered as an application of the derivative of a vector with respect to a vector, using a vector of size one appropriately. Similarly we will find that the derivatives involving matrices will reduce to derivatives involving vectors in a corresponding way.
| |
| | |
| The derivative of a [[vector function]] (a vector whose components are functions)
| |
| <math>\mathbf{y} =
| |
| \begin{bmatrix}
| |
| y_1 \\
| |
| y_2 \\
| |
| \vdots \\
| |
| y_m \\
| |
| \end{bmatrix}
| |
| </math>,
| |
| of an independent vector
| |
| <math>\mathbf{x} =
| |
| \begin{bmatrix}
| |
| x_1 \\
| |
| x_2 \\
| |
| \vdots \\
| |
| x_n \\
| |
| \end{bmatrix}
| |
| </math>,
| |
| is written (in [[#Layout conventions|numerator layout notation]]) as
| |
| | |
| :<math>
| |
| \frac{\partial \mathbf{y}}{\partial \mathbf{x}} =
| |
| \begin{bmatrix}
| |
| \frac{\partial y_1}{\partial x_1} & \frac{\partial y_1}{\partial x_2} & \cdots & \frac{\partial y_1}{\partial x_n}\\
| |
| \frac{\partial y_2}{\partial x_1} & \frac{\partial y_2}{\partial x_2} & \cdots & \frac{\partial y_2}{\partial x_n}\\
| |
| \vdots & \vdots & \ddots & \vdots\\
| |
| \frac{\partial y_m}{\partial x_1} & \frac{\partial y_m}{\partial x_2} & \cdots & \frac{\partial y_m}{\partial x_n}\\
| |
| \end{bmatrix}.
| |
| </math>
| |
| | |
| In [[vector calculus]], the derivative of a vector function '''y''' with respect to a vector '''x''' whose components represent a space is known as the '''[[pushforward (differential)|pushforward or differential]]''', or the '''[[Jacobian matrix]]'''.
| |
| | |
| The pushforward along a vector function '''f''' with respect to vector '''v''' in '''R'''<sup>''m''</sup> is given by
| |
| <math>d\,\mathbf{f}(\mathbf{v}) = \frac{\partial \mathbf{f}}{\partial \mathbf{x}} \mathbf{v}.</math>
| |
| | |
| == Derivatives with matrices ==
| |
| | |
| There are two types of derivatives with matrices that can be organized into a matrix of the same size. These are the derivative of a matrix by a scalar and the derivative of a scalar by a matrix respectively. These can be useful in minimization problems found many areas of applied mathematics and have adopted the names '''tangent matrix''' and '''gradient matrix''' respectively after their analogs for vectors.
| |
| | |
| '''NOTE''': The discussion in this section assumes the [[#Layout conventions|numerator layout convention]] for pedagogical purposes. Some authors use different conventions. The section on [[#Layout conventions|layout conventions]] discusses this issue in greater detail. The identities given further down are presented in forms that can be used in conjunction with all common layout conventions.
| |
| | |
| === Matrix-by-scalar ===
| |
| | |
| The derivative of a matrix function '''Y''' by a scalar ''x'' is known as the '''tangent matrix''' and is given (in [[#Layout conventions|numerator layout notation]]) by
| |
| | |
| :<math>
| |
| \frac{\partial \mathbf{Y}}{\partial x} =
| |
| \begin{bmatrix}
| |
| \frac{\partial y_{11}}{\partial x} & \frac{\partial y_{12}}{\partial x} & \cdots & \frac{\partial y_{1n}}{\partial x}\\
| |
| \frac{\partial y_{21}}{\partial x} & \frac{\partial y_{22}}{\partial x} & \cdots & \frac{\partial y_{2n}}{\partial x}\\
| |
| \vdots & \vdots & \ddots & \vdots\\
| |
| \frac{\partial y_{m1}}{\partial x} & \frac{\partial y_{m2}}{\partial x} & \cdots & \frac{\partial y_{mn}}{\partial x}\\
| |
| \end{bmatrix}.
| |
| </math>
| |
| | |
| === Scalar-by-matrix ===
| |
| | |
| The derivative of a scalar ''y'' function of a matrix '''X''' of independent variables, with respect to the matrix '''X''', is given (in [[#Layout conventions|numerator layout notation]]) by
| |
| | |
| :<math>
| |
| \frac{\partial y}{\partial \mathbf{X}} =
| |
| \begin{bmatrix}
| |
| \frac{\partial y}{\partial x_{11}} & \frac{\partial y}{\partial x_{21}} & \cdots & \frac{\partial y}{\partial x_{p1}}\\
| |
| \frac{\partial y}{\partial x_{12}} & \frac{\partial y}{\partial x_{22}} & \cdots & \frac{\partial y}{\partial x_{p2}}\\
| |
| \vdots & \vdots & \ddots & \vdots\\
| |
| \frac{\partial y}{\partial x_{1q}} & \frac{\partial y}{\partial x_{2q}} & \cdots & \frac{\partial y}{\partial x_{pq}}\\
| |
| \end{bmatrix}.
| |
| </math>
| |
| | |
| Notice that the indexing of the gradient with respect to '''X''' is transposed as compared with the indexing of '''X'''. Important examples of scalar functions of matrices include the [[trace (linear algebra)|trace]] of a matrix and the [[determinant]].
| |
| | |
| In analog with [[vector calculus]] this derivative is often written as the following.
| |
| :<math>
| |
| \nabla_\mathbf{X} y(\mathbf{X}) = \frac{\partial y(\mathbf{X})}{\partial \mathbf{X}} </math>
| |
| Also in analog with [[vector calculus]], the '''directional derivative''' of a scalar ''f''('''X''') of a matrix '''X''' in the direction of matrix '''Y''' is given by
| |
| :<math>\nabla_\mathbf{Y} f = \operatorname{tr} \left(\frac{\partial f}{\partial \mathbf{X}} \mathbf{Y}\right).</math>
| |
| | |
| It is the gradient matrix, in particular, that finds many uses in minimization problems in [[estimation theory]], particularly in the [[Kalman filter#Derivations|derivation]] of the [[Kalman filter]] algorithm, which is of great importance in the field.
| |
| | |
| === Other matrix derivatives ===
| |
| | |
| {{Disputed-section|Disputed information: Matrix derivative|date=July 2009}}
| |
| | |
| The three types of derivatives that have not been considered are those involving vectors-by-matrices, matrices-by-vectors, and matrices-by-matrices. These are not as widely considered and a notation is not widely agreed upon. As for vectors, the other two types of higher matrix derivatives can be seen as applications of the derivative of a matrix by a matrix by using a matrix with one column in the correct place. For this reason, in this subsection we consider only how one can write the derivative of a matrix by another matrix.
| |
| | |
| The differential or the matrix derivative of a matrix function '''F'''('''X''') that maps from ''n''×''m'' matrices to ''p''×''q'' matrices, '''F''' : ''M''(''n'',''m'') <math>\to</math> ''M''(''p'',''q''), is an element of ''M''(''p'',''q'') ? ''M''(''m'',''n''), a [[Tensor order|fourth-rank]] [[tensor]] (the reversal of ''m'' and ''n'' here indicates the [[dual space]] of ''M''(''n'',''m'')). In short it is an ''m''×''n'' matrix each of whose entries is a ''p''×''q'' matrix.{{Citation needed|date=July 2009}}
| |
| :<math>\frac{\partial\mathbf{F}} {\partial\mathbf{X}}=
| |
| \begin{bmatrix}
| |
| \frac{\partial\mathbf{F}}{\partial X_{1,1}} & \cdots & \frac{\partial \mathbf{F}}{\partial X_{n,1}}\\
| |
| \vdots & \ddots & \vdots\\
| |
| \frac{\partial\mathbf{F}}{\partial X_{1,m}} & \cdots & \frac{\partial \mathbf{F}}{\partial X_{n,m}}\\
| |
| \end{bmatrix},
| |
| </math>
| |
| and note that each <math>\frac{\partial \mathbf{F}}{\partial \mathbf{X}_{ij}}</math> is a ''p''×''q'' matrix defined as above. Note also that this matrix has its indexing transposed; ''m'' rows and ''n'' columns. The pushforward along '''F''' of an ''n''×''m'' matrix '''Y''' in ''M''(''n'',''m'') is then
| |
| :<math>d\mathbf{F}(\mathbf{Y}) = \operatorname{tr}\left(\frac{\partial\mathbf{F}} {\partial\mathbf{X}}\mathbf{Y}\right),</math> as formal block matrices.
| |
| Note that this definition encompasses all of the preceding definitions as special cases.
| |
| | |
| According to Jan R. Magnus and Heinz Neudecker, the following notations are both unsuitable, as the determinant of the second resulting matrix would have "no interpretation" and "a useful chain rule does not exist" if these notations are being used:<ref name="Magnus and Neudecker 1999">
| |
| {{cite book | last1=Magnus | first1=Jan R. | last2=Neudecker | first2=Heinz | title=Matrix Differential Calculus with Applications in Statistics and Econometrics | publisher=Wiley | series=Wiley Series in Probability and Statistics | edition=2nd | year=1999 | pages=171–173}}</ref>
| |
| <!-- I looked up Magnus and Neudecker at http://www.amazon.com/dp/047198633X using "Search inside this book", and found the relevant pages by searching "bad notation". Also, without the two "Given" sentences below, the <math> mark-up doesn't ahve not enough context to make much sense. But does it exceed fair use this way? -->
| |
| :Given <math>\phi</math>, a differentiable function of an <math>n \times m</math> matrix <math>\mathbf X = (x_{i, j})</math>,
| |
| ::<math>\frac{\partial \mathbf\phi(\mathbf X)} {\partial \mathbf{X}}=
| |
| \begin{bmatrix}
| |
| \frac{\partial \mathbf\phi}{\partial x_{1,1}} & \cdots & \frac{\partial \mathbf\phi}{\partial x_{1,q}}\\
| |
| \vdots & \ddots & \vdots\\
| |
| \frac{\partial \mathbf\phi}{\partial x_{n,1}} & \cdots & \frac{\partial \mathbf\phi}{\partial x_{n,q}}\\
| |
| \end{bmatrix}
| |
| </math>
| |
| :Given <math>\mathbf F = (f_{s,t})</math>, a differentiable <math>m \times n</math> function of an <math>n \times m</math> matrix <math>\mathbf X</math>,
| |
| ::<math>\frac{\partial \mathbf F (\mathbf X)} {\partial \mathbf{X}}=
| |
| \begin{bmatrix}
| |
| \frac{\partial f_{1,1}}{\partial \mathbf X} & \cdots & \frac{\partial f_{1,p}}{\partial \mathbf X}\\
| |
| \vdots & \ddots & \vdots\\
| |
| \frac{\partial f_{m,1}}{\partial \mathbf X} & \cdots & \frac{\partial f_{m,p}}{\partial \mathbf X}\\
| |
| \end{bmatrix}
| |
| </math>
| |
| <!-- The following appears to leave out some essential context, but I don't understand this part of Magnus and Neudecker well enough to know how much additional text is necessary, or how much is fair use to include. -->
| |
| The [[Jacobian matrix]], according to Magnus and Neudecker,<ref name="Magnus and Neudecker 1999"/> is
| |
| :<math>\mathrm D\, \mathbf F\left(\mathbf X\right) = \frac{\partial\, \mathrm{vec}\ \mathbf F\left(\mathbf X\right)}{\partial\left(\mathrm{vec}\ \mathbf X\right)^{\prime}}.
| |
| </math>
| |
| | |
| == Layout conventions ==
| |
| | |
| This section discusses the similarities and differences between notational conventions that are used in the various fields that take advantage of matrix calculus. Although there are largely two consistent conventions, some authors find it convenient to mix the two conventions in forms that are discussed below. After this section equations will be listed in both competing forms separately.
| |
| | |
| The fundamental issue is that the derivative of a vector with respect to a vector, i.e. <math>\frac{\partial \mathbf{y}}{\partial \mathbf{x}}</math>, is often written in two competing ways. If the numerator '''y''' is of size ''m'' and the denominator '''x''' of size ''n'', then the result can be laid out as either an ''m×n'' matrix or ''n×m'' matrix, i.e. the elements of '''y''' laid out in columns and the elements of '''x''' laid out in rows, or vice-versa. This leads to the following possibilities:
| |
| #''Numerator layout'', i.e. lay out according to '''y''' and '''x'''<sup>T</sup> (i.e. contrarily to '''x'''). This is sometimes known as the ''Jacobian formulation''.
| |
| #''Denominator layout'', i.e. lay out according to '''y'''<sup>T</sup> and '''x''' (i.e. contrarily to '''y'''). This is sometimes known as the ''Hessian formulation''. Some authors term this layout the ''gradient'', in distinction to the ''Jacobian'' (numerator layout), which is its transpose. (However, "[[gradient]]" more commonly means the derivative <math>\frac{\partial y}{\partial \mathbf{x}},</math> regardless of layout.)
| |
| #A third possibility sometimes seen is to insist on writing the derivative as <math>\frac{\partial \mathbf{y}}{\partial \mathbf{x}'},</math> (i.e. the derivative is taken with respect to the transpose of '''x''') and follow the numerator layout. This makes it possible to claim that the matrix is laid out according to both numerator and denominator. In practice this produces results the same as the numerator layout.
| |
| | |
| When handling the [[gradient]] <math>\frac{\partial y}{\partial \mathbf{x}}</math> and the opposite case <math>\frac{\partial \mathbf{y}}{\partial x},</math> we have the same issues. To be consistent, we should do one of the following:
| |
| #If we choose numerator layout for <math>\frac{\partial \mathbf{y}}{\partial \mathbf{x}},</math> we should lay out the [[gradient]] <math>\frac{\partial y}{\partial \mathbf{x}}</math> as a row vector, and <math>\frac{\partial \mathbf{y}}{\partial x}</math> as a column vector.
| |
| #If we choose denominator layout for <math>\frac{\partial \mathbf{y}}{\partial \mathbf{x}},</math> we should lay out the [[gradient]] <math>\frac{\partial y}{\partial \mathbf{x}}</math> as a column vector, and <math>\frac{\partial \mathbf{y}}{\partial x}</math> as a row vector.
| |
| #In the third possibility above, we write <math>\frac{\partial y}{\partial \mathbf{x}'}</math> and <math>\frac{\partial \mathbf{y}}{\partial x},</math> and use numerator layout.
| |
| | |
| Not all math textbooks and papers are consistent in this respect throughout the entire paper. That is, sometimes different conventions are used in different contexts within the same paper. For example, some choose denominator layout for gradients (laying them out as column vectors), but numerator layout for the vector-by-vector derivative <math>\frac{\partial \mathbf{y}}{\partial \mathbf{x}}.</math>
| |
| | |
| Similarly, when it comes to scalar-by-matrix derivatives <math>\frac{\partial y}{\partial \mathbf{X}}</math> and matrix-by-scalar derivatives <math>\frac{\partial \mathbf{Y}}{\partial x},</math> then consistent numerator layout lays out according to '''Y''' and '''X'''<sup>T</sup>, while consistent denominator layout lays out according to '''Y'''<sup>T</sup> and '''X'''. In practice, however, following a denominator layout for <math>\frac{\partial \mathbf{Y}}{\partial x},</math> and laying the result out according to '''Y'''<sup>T</sup>, is rarely seen because it makes for ugly formulas that do not correspond to the scalar formulas. As a result, the following layouts can often be found:
| |
| #''Consistent numerator layout'', which lays out <math>\frac{\partial \mathbf{Y}}{\partial x}</math> according to '''Y''' and <math>\frac{\partial y}{\partial \mathbf{X}}</math> according to '''X'''<sup>T</sup>.
| |
| #''Mixed layout'', which lays out <math>\frac{\partial \mathbf{Y}}{\partial x}</math> according to '''Y''' and <math>\frac{\partial y}{\partial \mathbf{X}}</math> according to '''X'''.
| |
| #Use the notation <math>\frac{\partial y}{\partial \mathbf{X}^{\rm T}},</math> with results the same as consistent numerator layout.
| |
| | |
| In the following formulas, we handle the five possible combinations <math>\frac{\partial y}{\partial \mathbf{x}}, \frac{\partial \mathbf{y}}{\partial x}, \frac{\partial \mathbf{y}}{\partial \mathbf{x}}, \frac{\partial y}{\partial \mathbf{X}}</math> and <math>\frac{\partial \mathbf{Y}}{\partial x}</math> separately. We also handle cases of scalar-by-scalar derivatives that involve an intermediate vector or matrix. (This can arise, for example, if a multi-dimensional [[parametric curve]] is defined in terms of a scalar variable, and then a derivative of a scalar function of the curve is taken with respect to the scalar that parameterizes the curve.) For each of the various combinations, we give numerator-layout and denominator-layout results, except in the cases above where denominator layout rarely occurs. In cases involving matrices where it makes sense, we give numerator-layout and mixed-layout results. As noted above, cases where vector and matrix denominators are written in transpose notation are equivalent to numerator layout with the denominators written without the transpose.
| |
| | |
| Keep in mind that various authors use different combinations of numerator and denominator layouts for different types of derivatives, and there is no guarantee that an author will consistently use either numerator or denominator layout for all types. Match up the formulas below with those quoted in the source to determine the layout used for that particular type of derivative, but be careful not to assume that derivatives of other types necessarily follow the same kind of layout.
| |
| | |
| When taking derivatives with an aggregate (vector or matrix) denominator in order to find a maximum or minimum of the aggregate, it should be kept in mind that using numerator layout will produce results that are transposed with respect to the aggregate. For example, in attempting to find the [[maximum likelihood]] estimate of a [[multivariate normal distribution]] using matrix calculus, if the domain is a ''k''x1 column vector, then the result using the numerator layout will be in the form of a 1x''k'' row vector. Thus, either the results should be transposed at the end or the denominator layout (or mixed layout) should be used.
| |
| | |
| :{|class="wikitable"
| |
| |+ Result of differentiating various kinds of aggregates with other kinds of aggregates
| |
| !rowspan=2| !!colspan=2|Scalar ''y'' !! colspan=2|Vector '''y''' (size ''m'') !! colspan=2|Matrix '''Y''' (size ''m''×''n'')
| |
| |-
| |
| ! Notation !! Type !! Notation !! Type !! Notation !! Type
| |
| |-
| |
| ! Scalar ''x''
| |
| | <math>\frac{\partial y}{\partial x}</math>
| |
| | scalar
| |
| | <math>\frac{\partial \mathbf{y}}{\partial x}</math>
| |
| | (numerator layout) size-''m'' [[column vector]] <br/>
| |
| (denominator layout) size-''m'' [[row vector]] <br/>
| |
| | <math>\frac{\partial \mathbf{Y}}{\partial x}</math>
| |
| | (numerator layout) ''m''×''n'' matrix
| |
| |-
| |
| ! Vector '''x''' (size ''n'')
| |
| | <math>\frac{\partial y}{\partial \mathbf{x}}</math>
| |
| | (numerator layout) size-''n'' [[row vector]] <br/>
| |
| (denominator layout) size-''n'' [[column vector]] <br/>
| |
| | <math>\frac{\partial \mathbf{y}}{\partial \mathbf{x}}</math>
| |
| | (numerator layout) ''m''×''n'' matrix <br/>
| |
| (denominator layout) ''n''×''m'' matrix <br/>
| |
| | <math>\frac{\partial \mathbf{Y}}{\partial \mathbf{x}}</math>
| |
| | ?
| |
| |-
| |
| ! Matrix '''X''' (size ''p''×''q'')
| |
| | <math>\frac{\partial y}{\partial \mathbf{X}}</math>
| |
| | (numerator layout) ''q''×''p'' matrix <br/>
| |
| (denominator layout) ''p''×''q'' matrix <br/>
| |
| | <math>\frac{\partial \mathbf{y}}{\partial \mathbf{X}}</math>
| |
| | ?
| |
| | <math>\frac{\partial \mathbf{Y}}{\partial \mathbf{X}}</math>
| |
| | ?
| |
| |}
| |
| | |
| The results of operations will be transposed when switching between numerator-layout and denominator-layout notation.
| |
| | |
| === Numerator-layout notation ===
| |
| | |
| Using numerator-layout notation, we have:<ref name="minka">Minka, Thomas P. "Old and New Matrix Algebra Useful for Statistics." December 28, 2000. [http://research.microsoft.com/en-us/um/people/minka/papers/matrix/]</ref>
| |
| | |
| :<math>
| |
| \frac{\partial y}{\partial \mathbf{x}} =
| |
| \left[
| |
| \frac{\partial y}{\partial x_1}
| |
| \frac{\partial y}{\partial x_2}
| |
| \cdots
| |
| \frac{\partial y}{\partial x_n}
| |
| \right].
| |
| </math>
| |
| | |
| :<math>
| |
| \frac{\partial \mathbf{y}}{\partial x} =
| |
| \begin{bmatrix}
| |
| \frac{\partial y_1}{\partial x}\\
| |
| \frac{\partial y_2}{\partial x}\\
| |
| \vdots\\
| |
| \frac{\partial y_m}{\partial x}\\
| |
| \end{bmatrix}.
| |
| </math>
| |
| | |
| :<math>
| |
| \frac{\partial \mathbf{y}}{\partial \mathbf{x}} =
| |
| \begin{bmatrix}
| |
| \frac{\partial y_1}{\partial x_1} & \frac{\partial y_1}{\partial x_2} & \cdots & \frac{\partial y_1}{\partial x_n}\\
| |
| \frac{\partial y_2}{\partial x_1} & \frac{\partial y_2}{\partial x_2} & \cdots & \frac{\partial y_2}{\partial x_n}\\
| |
| \vdots & \vdots & \ddots & \vdots\\
| |
| \frac{\partial y_m}{\partial x_1} & \frac{\partial y_m}{\partial x_2} & \cdots & \frac{\partial y_m}{\partial x_n}\\
| |
| \end{bmatrix}.
| |
| </math>
| |
| | |
| :<math>
| |
| \frac{\partial y}{\partial \mathbf{X}} =
| |
| \begin{bmatrix}
| |
| \frac{\partial y}{\partial x_{11}} & \frac{\partial y}{\partial x_{21}} & \cdots & \frac{\partial y}{\partial x_{p1}}\\
| |
| \frac{\partial y}{\partial x_{12}} & \frac{\partial y}{\partial x_{22}} & \cdots & \frac{\partial y}{\partial x_{p2}}\\
| |
| \vdots & \vdots & \ddots & \vdots\\
| |
| \frac{\partial y}{\partial x_{1q}} & \frac{\partial y}{\partial x_{2q}} & \cdots & \frac{\partial y}{\partial x_{pq}}\\
| |
| \end{bmatrix}.
| |
| </math>
| |
| | |
| The following definitions are only provided in numerator-layout notation:
| |
| | |
| :<math>
| |
| \frac{\partial \mathbf{Y}}{\partial x} =
| |
| \begin{bmatrix}
| |
| \frac{\partial y_{11}}{\partial x} & \frac{\partial y_{12}}{\partial x} & \cdots & \frac{\partial y_{1n}}{\partial x}\\
| |
| \frac{\partial y_{21}}{\partial x} & \frac{\partial y_{22}}{\partial x} & \cdots & \frac{\partial y_{2n}}{\partial x}\\
| |
| \vdots & \vdots & \ddots & \vdots\\
| |
| \frac{\partial y_{m1}}{\partial x} & \frac{\partial y_{m2}}{\partial x} & \cdots & \frac{\partial y_{mn}}{\partial x}\\
| |
| \end{bmatrix}.
| |
| </math>
| |
| | |
| :<math>
| |
| d\mathbf{X} =
| |
| \begin{bmatrix}
| |
| dx_{11} & dx_{12} & \cdots & dx_{1n}\\
| |
| dx_{21} & dx_{22} & \cdots & dx_{2n}\\
| |
| \vdots & \vdots & \ddots & \vdots\\
| |
| dx_{m1} & dx_{m2} & \cdots & dx_{mn}\\
| |
| \end{bmatrix}.
| |
| </math>
| |
| | |
| ===Denominator-layout notation===
| |
| Using denominator-layout notation, we have:<ref>[http://www.colorado.edu/engineering/CAS/courses.d/IFEM.d/IFEM.AppD.pdf]</ref>
| |
| :<math>
| |
| \frac{\partial y}{\partial \mathbf{x}} =
| |
| \begin{bmatrix}
| |
| \frac{\partial y}{\partial x_1}\\
| |
| \frac{\partial y}{\partial x_2}\\
| |
| \vdots\\
| |
| \frac{\partial y}{\partial x_n}\\
| |
| \end{bmatrix}.
| |
| </math>
| |
| | |
| :<math>
| |
| \frac{\partial \mathbf{y}}{\partial x} = \left[
| |
| \frac{\partial y_1}{\partial x}
| |
| \frac{\partial y_2}{\partial x}
| |
| \cdots
| |
| \frac{\partial y_m}{\partial x}
| |
| \right].
| |
| </math>
| |
| | |
| :<math>
| |
| \frac{\partial \mathbf{y}}{\partial \mathbf{x}} =
| |
| \begin{bmatrix}
| |
| \frac{\partial y_1}{\partial x_1} & \frac{\partial y_2}{\partial x_1} & \cdots & \frac{\partial y_m}{\partial x_1}\\
| |
| \frac{\partial y_1}{\partial x_2} & \frac{\partial y_2}{\partial x_2} & \cdots & \frac{\partial y_m}{\partial x_2}\\
| |
| \vdots & \vdots & \ddots & \vdots\\
| |
| \frac{\partial y_1}{\partial x_n} & \frac{\partial y_2}{\partial x_n} & \cdots & \frac{\partial y_m}{\partial x_n}\\
| |
| \end{bmatrix}.
| |
| </math>
| |
| | |
| :<math>
| |
| \frac{\partial y}{\partial \mathbf{X}} =
| |
| \begin{bmatrix}
| |
| \frac{\partial y}{\partial x_{11}} & \frac{\partial y}{\partial x_{12}} & \cdots & \frac{\partial y}{\partial x_{1q}}\\
| |
| \frac{\partial y}{\partial x_{21}} & \frac{\partial y}{\partial x_{22}} & \cdots & \frac{\partial y}{\partial x_{2q}}\\
| |
| \vdots & \vdots & \ddots & \vdots\\
| |
| \frac{\partial y}{\partial x_{p1}} & \frac{\partial y}{\partial x_{p2}} & \cdots & \frac{\partial y}{\partial x_{pq}}\\
| |
| \end{bmatrix}.
| |
| </math>
| |
| | |
| == Identities ==
| |
| | |
| As noted above, in general, the results of operations will be transposed when switching between numerator-layout and denominator-layout notation.
| |
| | |
| To help make sense of all the identities below, keep in mind the most important rules: the [[chain rule]], [[product rule]] and [[sum rule in differentiation|sum rule]]. The sum rule applies universally, and the product rule applies in most of the cases below, provided that the order of matrix products is maintained, since matrix products are not commutative. The chain rule applies in some of the cases, but unfortunately does ''not'' apply in matrix-by-scalar derivatives or scalar-by-matrix derivatives (in the latter case, mostly involving the [[trace (linear algebra)|trace]] operator applied to matrices). In the latter case, the product rule can't quite be applied directly, either, but the equivalent can be done with a bit more work using the differential identities.
| |
| | |
| === Vector-by-vector identities ===
| |
| This is presented first because all of the operations that apply to vector-by-vector differentiation apply directly to vector-by-scalar or scalar-by-vector differentiation simply by reducing the appropriate vector in the numerator or denominator to a scalar.
| |
| | |
| :{|class="wikitable" style="text-align: center;"
| |
| |+ Identities: vector-by-vector <math>\frac{\partial \mathbf{y}}{\partial \mathbf{x}}</math>
| |
| ! scope="col" width="150" | Condition
| |
| ! scope="col" width="10" | Expression
| |
| ! scope="col" width="100" | Numerator layout, i.e. by '''y''' and '''x'''<sup>T</sup>
| |
| ! scope="col" width="100" | Denominator layout, i.e. by '''y'''<sup>T</sup> and '''x'''
| |
| |-
| |
| | '''a''' is not a function of '''x''' || <math>\frac{\partial \mathbf{a}}{\partial \mathbf{x}} =</math> ||colspan=2| <math>\mathbf{0}</math>
| |
| |-
| |
| | || <math>\frac{\partial \mathbf{x}}{\partial \mathbf{x}} =</math> || colspan=2|<math>\mathbf{I}</math>
| |
| |-
| |
| | '''A''' is not a function of '''x''' || <math>\frac{\partial \mathbf{A}\mathbf{x}}{\partial \mathbf{x}} =</math> || <math>\mathbf{A}</math> || <math>\mathbf{A}^{\rm T}</math>
| |
| |-
| |
| | '''A''' is not a function of '''x''' || <math>\frac{\partial \mathbf{x}^{\rm T}\mathbf{A}}{\partial \mathbf{x}} =</math> || <math>\mathbf{A}^{\rm T}</math> || <math>\mathbf{A}</math>
| |
| |-
| |
| | ''a'' is not a function of '''x''',<br />'''u''' = '''u'''('''x''') || <math>\frac{\partial a\mathbf{u}}{\partial\, \mathbf{x}} =</math>
| |
| | colspan=2|<math>a\frac{\partial \mathbf{u}}{\partial \mathbf{x}}</math>
| |
| |-
| |
| | ''a'' = ''a''('''x'''), '''u''' = '''u'''('''x''') || <math>\frac{\partial a\mathbf{u}}{\partial \mathbf{x}} =</math> || <math>a \frac{\partial \mathbf{u}}{\partial \mathbf{x}} + \mathbf{u}\frac{\partial a}{\partial \mathbf{x}} </math> || <math>a\frac{\partial \mathbf{u}}{\partial \mathbf{x}} + \frac{\partial a}{\partial \mathbf{x}} \mathbf{u}^{\rm T} </math>
| |
| |-
| |
| | '''A''' is not a function of '''x''',<br />'''u''' = '''u'''('''x''') || <math>\frac{\partial \mathbf{A}\mathbf{u}}{\partial \mathbf{x}} =</math> || <math>\mathbf{A}\frac{\partial \mathbf{u}}{\partial \mathbf{x}}</math> || <math>\frac{\partial \mathbf{u}}{\partial \mathbf{x}}\mathbf{A}^{\rm T}</math>
| |
| |-
| |
| | '''u''' = '''u'''('''x'''), '''v''' = '''v'''('''x''') || <math>\frac{\partial (\mathbf{u} + \mathbf{v})}{\partial \mathbf{x}} =</math>
| |
| | colspan=2|<math>\frac{\partial \mathbf{u}}{\partial \mathbf{x}} + \frac{\partial \mathbf{v}}{\partial \mathbf{x}}</math>
| |
| |-
| |
| | '''u''' = '''u'''('''x''') || <math>\frac{\partial \mathbf{g(u)}}{\partial \mathbf{x}} =</math> || <math>\frac{\partial \mathbf{g(u)}}{\partial \mathbf{u}} \frac{\partial \mathbf{u}}{\partial \mathbf{x}}</math> || <math>\frac{\partial \mathbf{u}}{\partial \mathbf{x}} \frac{\partial \mathbf{g(u)}}{\partial \mathbf{u}}</math>
| |
| |-
| |
| | '''u''' = '''u'''('''x''') || <math>\frac{\partial \mathbf{f(g(u))}}{\partial \mathbf{x}} =</math> || <math>\frac{\partial \mathbf{f(g)}}{\partial \mathbf{g}} \frac{\partial \mathbf{g(u)}}{\partial \mathbf{u}} \frac{\partial \mathbf{u}}{\partial \mathbf{x}}</math> || <math>\frac{\partial \mathbf{u}}{\partial \mathbf{x}} \frac{\partial \mathbf{g(u)}}{\partial \mathbf{u}} \frac{\partial \mathbf{f(g)}}{\partial \mathbf{g}}</math>
| |
| |}
| |
| | |
| === Scalar-by-vector identities ===
| |
| | |
| The fundamental identities are placed above the thick black line.
| |
| | |
| :{|class="wikitable" style="text-align: center;"
| |
| |+ Identities: scalar-by-vector <math>\frac{\partial y}{\partial \mathbf{x}} = \nabla_\mathbf{x} y</math>
| |
| ! scope="col" width="150" | Condition
| |
| ! scope="col" width="200" | Expression
| |
| ! scope="col" width="200" | Numerator layout,<br />i.e. by '''x'''<sup>T</sup>; result is row vector
| |
| ! scope="col" width="200" | Denominator layout,<br />i.e. by '''x'''; result is column vector
| |
| |-
| |
| | ''a'' is not a function of '''x''' || <math>\frac{\partial a}{\partial \mathbf{x}} =</math>
| |
| | <math>\mathbf{0}^{\rm T}</math> <ref name="zerovec">Here, <math>\mathbf{0}</math> refers to a [[column vector]] of all 0's, of size ''n'', where ''n'' is the length of '''x'''.</ref> || <math>\mathbf{0}</math> <ref name="zerovec"/>
| |
| |-
| |
| | ''a'' is not a function of '''x''', <br />''u'' = ''u''('''x''') || <math>\frac{\partial au}{\partial \mathbf{x}} =</math>
| |
| | colspan=2|<math>a\frac{\partial u}{\partial \mathbf{x}}</math>
| |
| |-
| |
| | ''u'' = ''u''('''x'''), ''v'' = ''v''('''x''') || <math>\frac{\partial (u+v)}{\partial \mathbf{x}} =</math>
| |
| | colspan=2|<math>\frac{\partial u}{\partial \mathbf{x}} + \frac{\partial v}{\partial \mathbf{x}} </math>
| |
| |-
| |
| | ''u'' = ''u''('''x'''), ''v'' = ''v''('''x''') || <math>\frac{\partial uv}{\partial \mathbf{x}} =</math>
| |
| | colspan=2|<math>u\frac{\partial v}{\partial \mathbf{x}} + v\frac{\partial u}{\partial \mathbf{x}} </math>
| |
| |-
| |
| | ''u'' = ''u''('''x''') || <math>\frac{\partial g(u)}{\partial \mathbf{x}} =</math>
| |
| | colspan=2|<math>\frac{\partial g(u)}{\partial u} \frac{\partial u}{\partial \mathbf{x}} </math>
| |
| |-
| |
| | ''u'' = ''u''('''x''') || <math>\frac{\partial f(g(u))}{\partial \mathbf{x}} =</math>
| |
| | colspan=2|<math>\frac{\partial f(g)}{\partial g} \frac{\partial g(u)}{\partial u} \frac{\partial u}{\partial \mathbf{x}} </math>
| |
| |-
| |
| | '''u''' = '''u'''('''x'''), '''v''' = '''v'''('''x''')
| |
| | <math>\frac{\partial (\mathbf{u} \cdot \mathbf{v})}{\partial \mathbf{x}} = \frac{\partial \mathbf{u}^{\rm T}\mathbf{v}}{\partial \mathbf{x}} =</math>
| |
| | <math>\mathbf{u}^{\rm T}\frac{\partial \mathbf{v}}{\partial \mathbf{x}} + \mathbf{v}^{\rm T}\frac{\partial \mathbf{u}}{\partial \mathbf{x}} </math>
| |
| *assumes numerator layout of <math>\frac{\partial \mathbf{u}}{\partial \mathbf{x}}, \frac{\partial \mathbf{v}}{\partial \mathbf{x}}</math>
| |
| | <math>\frac{\partial \mathbf{u}}{\partial \mathbf{x}}\mathbf{v} + \frac{\partial \mathbf{v}}{\partial \mathbf{x}}\mathbf{u}</math>
| |
| *assumes denominator layout of <math>\frac{\partial \mathbf{u}}{\partial \mathbf{x}}, \frac{\partial \mathbf{v}}{\partial \mathbf{x}}</math>
| |
| |-
| |
| | '''u''' = '''u'''('''x'''), '''v''' = '''v'''('''x'''),<br /> '''A''' is not a function of '''x'''
| |
| | <math>\frac{\partial (\mathbf{u} \cdot \mathbf{A}\mathbf{v})}{\partial \mathbf{x}} = \frac{\partial \mathbf{u}^{\rm T}\mathbf{A}\mathbf{v}}{\partial \mathbf{x}} =</math>
| |
| | <math>\mathbf{u}^{\rm T}\mathbf{A}\frac{\partial \mathbf{v}}{\partial \mathbf{x}} + \mathbf{v}^{\rm T}\mathbf{A}^{\rm T}\frac{\partial \mathbf{u}}{\partial \mathbf{x}} </math>
| |
| *assumes numerator layout of <math>\frac{\partial \mathbf{u}}{\partial \mathbf{x}}, \frac{\partial \mathbf{v}}{\partial \mathbf{x}}</math>
| |
| | <math>\frac{\partial \mathbf{u}}{\partial \mathbf{x}}\mathbf{A}\mathbf{v} + \frac{\partial \mathbf{v}}{\partial \mathbf{x}}\mathbf{A}^{\rm T}\mathbf{u}</math>
| |
| *assumes denominator layout of <math>\frac{\partial \mathbf{u}}{\partial \mathbf{x}}, \frac{\partial \mathbf{v}}{\partial \mathbf{x}}</math>
| |
| |- style="border-top: 3px solid;"
| |
| | '''a''' is not a function of '''x''' || <math>\frac{\partial (\mathbf{a}\cdot\mathbf{x})}{\partial \mathbf{x}} = \frac{\partial (\mathbf{x}\cdot\mathbf{a})}{\partial \mathbf{x}} =</math> <br /> <br /> <math>\frac{\partial \mathbf{a}^{\rm T}\mathbf{x}}{\partial \mathbf{x}} = \frac{\partial \mathbf{x}^{\rm T}\mathbf{a}}{\partial \mathbf{x}} =</math> || <math>\mathbf{a}^{\rm T}</math> || <math>\mathbf{a}</math>
| |
| |-
| |
| | '''A''' is not a function of '''x''' <br />'''b''' is not a function of '''x''' || <math>\frac{\partial \mathbf{b}^{\rm T}\mathbf{A}\mathbf{x}}{\partial \mathbf{x}} =</math> || <math>\mathbf{b}^{\rm T}\mathbf{A}</math> || <math>\mathbf{A}^{\rm T}\mathbf{b}</math>
| |
| |-
| |
| | '''A''' is not a function of '''x''' || <math>\frac{\partial \mathbf{x}^{\rm T}\mathbf{A}\mathbf{x}}{\partial \mathbf{x}} =</math> || <math>\mathbf{x}^{\rm T}(\mathbf{A} + \mathbf{A}^{\rm T})</math> || <math>(\mathbf{A} + \mathbf{A}^{\rm T})\mathbf{x}</math>
| |
| |-
| |
| | '''A''' is not a function of '''x''' <br />'''A''' is [[symmetric matrix|symmetric]] || <math>\frac{\partial \mathbf{x}^{\rm T}\mathbf{A}\mathbf{x}}{\partial \mathbf{x}} =</math> || <math>2\mathbf{x}^{\rm T}\mathbf{A}</math> || <math>2\mathbf{A}\mathbf{x}</math>
| |
| |-
| |
| | '''A''' is not a function of '''x''' || <math>\frac{\partial^2 \mathbf{x}^{\rm T}\mathbf{A}\mathbf{x}}{\partial \mathbf{x}^2} =</math> || colspan=2|<math>\mathbf{A} + \mathbf{A}^{\rm T}</math>
| |
| |-
| |
| | '''A''' is not a function of '''x''' <br />'''A''' is [[symmetric matrix|symmetric]] || <math>\frac{\partial^2 \mathbf{x}^{\rm T}\mathbf{A}\mathbf{x}}{\partial \mathbf{x}^2} =</math> || colspan=2|<math>2\mathbf{A}</math>
| |
| |-
| |
| | || <math>\frac{\partial (\mathbf{x} \cdot \mathbf{x})}{\partial \mathbf{x}} = \frac{\partial \mathbf{x}^{\rm T}\mathbf{x}}{\partial \mathbf{x}} =</math> || <math>2\mathbf{x}^{\rm T} </math> || <math>2\mathbf{x}</math>
| |
| |-
| |
| | '''a''' is not a function of '''x''', <br />'''u''' = '''u'''('''x''')
| |
| | <math>\frac{\partial (\mathbf{a} \cdot \mathbf{u})}{\partial \mathbf{x}} = \frac{\partial \mathbf{a}^{\rm T}\mathbf{u}}{\partial \mathbf{x}} =</math>
| |
| | <math>\mathbf{a}^{\rm T}\frac{\partial \mathbf{u}}{\partial \mathbf{x}} </math>
| |
| *assumes numerator layout of <math>\frac{\partial \mathbf{u}}{\partial \mathbf{x}}</math>
| |
| | <math>\frac{\partial \mathbf{u}}{\partial \mathbf{x}}\mathbf{a}</math>
| |
| *assumes denominator layout of <math>\frac{\partial \mathbf{u}}{\partial \mathbf{x}}</math>
| |
| |-
| |
| | '''a''', '''b''' are not functions of '''x''' || <math>\frac{\partial \; \textbf{a}^{\rm T}\textbf{x}\textbf{x}^{\rm T}\textbf{b}}{\partial \; \textbf{x}} = </math> || <math>\textbf{x}^{\rm T}(\textbf{a}\textbf{b}^{\rm T} + \textbf{b}\textbf{a}^{\rm T})</math> || <math>(\textbf{a}\textbf{b}^{\rm T} + \textbf{b}\textbf{a}^{\rm T})\textbf{x}</math>
| |
| |-
| |
| | '''A''', '''b''', '''C''', '''D''', '''e''' are not functions of '''x''' || <math>\frac{\partial \; (\textbf{A}\textbf{x} + \textbf{b})^{\rm T} \textbf{C} (\textbf{D}\textbf{x} + \textbf{e}) }{\partial \; \textbf{x}} = </math> || <math>(\textbf{D}\textbf{x} + \textbf{e})^{\rm T} \textbf{C}^{\rm T} \textbf{A} + (\textbf{A}\textbf{x} + \textbf{b})^{\rm T} \textbf{C} \textbf{D}</math> || <math> \textbf{D}^{\rm T} \textbf{C}^{\rm T} (\textbf{A}\textbf{x} + \textbf{b}) + \textbf{A}^{\rm T}\textbf{C}(\textbf{D}\textbf{x} + \textbf{e})</math>
| |
| |-
| |
| | '''a''' is not a function of '''x''' || <math>\frac{\partial \; \|\mathbf{x}-\mathbf{a}\|}{\partial \; \mathbf{x}} = </math> || <math>\frac{(\mathbf{x}-\mathbf{a})^{\rm T}}{\|\mathbf{x}-\mathbf{a}\|}</math> || <math>\frac{\mathbf{x}-\mathbf{a}}{\|\mathbf{x}-\mathbf{a}\|}</math>
| |
| |}
| |
| | |
| === Vector-by-scalar identities ===
| |
| | |
| :{|class="wikitable" style="text-align: center;"
| |
| |+ Identities: vector-by-scalar <math>\frac{\partial \mathbf{y}}{\partial x}</math>
| |
| ! scope="col" width="150" | Condition
| |
| ! scope="col" width="100" | Expression
| |
| ! scope="col" width="100" | Numerator layout, i.e. by '''y''',<br />result is column vector
| |
| ! scope="col" width="100" | Denominator layout, i.e. by '''y'''<sup>T</sup>,<br />result is row vector
| |
| |-
| |
| | '''a''' is not a function of ''x'' || <math>\frac{\partial \mathbf{a}}{\partial x} =</math> ||colspan=2| <math>\mathbf{0}</math><ref name="zerovec">Here, <math>\mathbf{0}</math> refers to a [[column vector]] of all 0's, of size ''n'', where ''n'' is the length of '''a'''.</ref>
| |
| |-
| |
| | ''a'' is not a function of ''x'',<br />'''u''' = '''u'''(''x'') || <math>\frac{\partial a\mathbf{u}}{\partial x} =</math>
| |
| | colspan=2|<math>a\frac{\partial \mathbf{u}}{\partial x}</math>
| |
| |-
| |
| | '''A''' is not a function of ''x'',<br />'''u''' = '''u'''(''x'') || <math>\frac{\partial \mathbf{A}\mathbf{u}}{\partial x} =</math> || <math>\mathbf{A}\frac{\partial \mathbf{u}}{\partial x}</math> || <math>\frac{\partial \mathbf{u}}{\partial x}\mathbf{A}^{\rm T}</math>
| |
| |-
| |
| | '''u''' = '''u'''(''x'') || <math>\frac{\partial \mathbf{u}^{\rm T}}{\partial x} =</math>
| |
| | colspan=2|<math>\left(\frac{\partial \mathbf{u}}{\partial x}\right)^{\rm T}</math>
| |
| |-
| |
| | '''u''' = '''u'''(''x''), '''v''' = '''v'''(''x'') || <math>\frac{\partial (\mathbf{u} + \mathbf{v})}{\partial x} =</math>
| |
| | colspan=2|<math>\frac{\partial \mathbf{u}}{\partial x} + \frac{\partial \mathbf{v}}{\partial x}</math>
| |
| |-
| |
| | '''u''' = '''u'''(''x''), '''v''' = '''v'''(''x'') || <math>\frac{\partial (\mathbf{u} \times \mathbf{v})}{\partial x} =</math>
| |
| | colspan=2|<math>\mathbf{u} \times \frac{\partial \mathbf{v}}{\partial x} + \frac{\partial \mathbf{u}}{\partial x} \times \mathbf{v} </math>
| |
| |-
| |
| | rowspan=2|'''u''' = '''u'''(''x'') || rowspan=2|<math>\frac{\partial \mathbf{g(u)}}{\partial x} =</math> || <math>\frac{\partial \mathbf{g(u)}}{\partial \mathbf{u}} \frac{\partial \mathbf{u}}{\partial x}</math> || <math>\frac{\partial \mathbf{u}}{\partial x} \frac{\partial \mathbf{g(u)}}{\partial \mathbf{u}}</math>
| |
| |-
| |
| |colspan=2|Assumes consistent matrix layout; see below.
| |
| |-
| |
| | rowspan=2|'''u''' = '''u'''(''x'') || rowspan=2|<math>\frac{\partial \mathbf{f(g(u))}}{\partial x} =</math> || <math>\frac{\partial \mathbf{f(g)}}{\partial \mathbf{g}} \frac{\partial \mathbf{g(u)}}{\partial \mathbf{u}} \frac{\partial \mathbf{u}}{\partial x}</math> || <math>\frac{\partial \mathbf{u}}{\partial x} \frac{\partial \mathbf{g(u)}}{\partial \mathbf{u}} \frac{\partial \mathbf{f(g)}}{\partial \mathbf{g}}</math>
| |
| |-
| |
| |colspan=2|Assumes consistent matrix layout; see below.
| |
| |}
| |
| | |
| '''NOTE''': The formulas involving the vector-by-vector derivatives <math>\frac{\partial \mathbf{g(u)}}{\partial \mathbf{u}}</math> and <math>\frac{\partial \mathbf{f(g)}}{\partial \mathbf{g}}</math> (whose outputs are matrices) assume the matrices are laid out consistent with the vector layout, i.e. numerator-layout matrix when numerator-layout vector and vice-versa; otherwise, transpose the vector-by-vector derivatives.
| |
| | |
| === Scalar-by-matrix identities ===
| |
| | |
| Note that exact equivalents of the scalar [[product rule]] and [[chain rule]] do not exist when applied to matrix-valued functions of matrices. However, the product rule of this sort does apply to the differential form (see below), and this is the way to derive many of the identities below involving the [[trace (linear algebra)|trace]] function, combined with the fact that the trace function allows transposing and cyclic permutation, i.e.:
| |
| :<math>{\rm tr}(\mathbf{A}) = {\rm tr}(\mathbf{A^{\rm T}})</math>
| |
| :<math>{\rm tr}(\mathbf{ABCD}) = {\rm tr}(\mathbf{BCDA}) = {\rm tr}(\mathbf{CDAB}) = {\rm tr}(\mathbf{DABC})</math>
| |
| | |
| For example, to compute <math>\frac{\partial {\rm tr}(\mathbf{AXBX^{\rm T}C})}{\partial \mathbf{X}}:</math>
| |
| | |
| :<math>
| |
| \begin{align}
| |
| d\, {\rm tr}(\mathbf{AXBX^{\rm T}C}) &= d\, {\rm tr}(\mathbf{CAXBX^{\rm T}}) = {\rm tr}(d(\mathbf{CAXBX^{\rm T}})) \\
| |
| &= {\rm tr}(\mathbf{CAX} d(\mathbf{BX^{\rm T}}) + d(\mathbf{CAX})\mathbf{BX^{\rm T}}) \\
| |
| &= {\rm tr}(\mathbf{CAX} d(\mathbf{BX^{\rm T}})) + {\rm tr}(d(\mathbf{CAX})\mathbf{BX^{\rm T}}) \\
| |
| &= {\rm tr}(\mathbf{CAXB} d(\mathbf{X^{\rm T}})) + {\rm tr}(\mathbf{CA}(d\mathbf{X})\mathbf{BX^{\rm T}}) \\
| |
| &= {\rm tr}(\mathbf{CAXB} (d\mathbf{X})^{\rm T}) + {\rm tr}(\mathbf{CA}(d\mathbf{X})\mathbf{BX^{\rm T}}) \\
| |
| &= {\rm tr}\left((\mathbf{CAXB} (d\mathbf{X})^{\rm T})^{\rm T}\right) + {\rm tr}(\mathbf{CA}(d\mathbf{X})\mathbf{BX^{\rm T}}) \\
| |
| &= {\rm tr}((d\mathbf{X})\mathbf{B^{\rm T}X^{\rm T}A^{\rm T}C^{\rm T}}) + {\rm tr}(\mathbf{CA}(d\mathbf{X})\mathbf{BX^{\rm T}}) \\
| |
| &= {\rm tr}(\mathbf{B^{\rm T}X^{\rm T}A^{\rm T}C^{\rm T}}(d\mathbf{X})) + {\rm tr}(\mathbf{BX^{\rm T}}\mathbf{CA}(d\mathbf{X})) \\
| |
| &= {\rm tr}\left((\mathbf{B^{\rm T}X^{\rm T}A^{\rm T}C^{\rm T}} + \mathbf{BX^{\rm T}}\mathbf{CA})d\mathbf{X}\right)
| |
| \end{align}
| |
| </math>
| |
| | |
| Therefore,
| |
| | |
| :<math>\frac{\partial {\rm tr}(\mathbf{AXBX^{\rm T}C})}{\partial \mathbf{X}} = \mathbf{B^{\rm T}X^{\rm T}A^{\rm T}C^{\rm T}} + \mathbf{BX^{\rm T}}\mathbf{CA} .</math>
| |
| | |
| :{|class="wikitable" style="text-align: center;"
| |
| |+ Identities: scalar-by-matrix <math>\frac{\partial y}{\partial \mathbf{X}}</math>
| |
| ! scope="col" width="175" | Condition
| |
| ! scope="col" width="10" | Expression
| |
| ! scope="col" width="100" | Numerator layout, i.e. by '''X'''<sup>T</sup>
| |
| ! scope="col" width="100" | Denominator layout, i.e. by '''X'''
| |
| |-
| |
| | ''a'' is not a function of '''X''' || <math>\frac{\partial a}{\partial \mathbf{X}} =</math>
| |
| | <math>\mathbf{0}^{\rm T}</math> <ref name="zeromatrix">Here, <math>\mathbf{0}</math> refers to a matrix of all 0's, of the same shape as '''X'''.</ref> || <math>\mathbf{0}</math> <ref name="zeromatrix"/>
| |
| |-
| |
| | ''a'' is not a function of '''X''', ''u'' = ''u''('''X''') || <math>\frac{\partial au}{\partial \mathbf{X}} =</math>
| |
| | colspan=2|<math>a\frac{\partial u}{\partial \mathbf{X}}</math>
| |
| |-
| |
| | ''u'' = ''u''('''X'''), ''v'' = ''v''('''X''') || <math>\frac{\partial (u+v)}{\partial \mathbf{X}} =</math>
| |
| | colspan=2|<math>\frac{\partial u}{\partial \mathbf{X}} + \frac{\partial v}{\partial \mathbf{X}} </math>
| |
| |-
| |
| | ''u'' = ''u''('''X'''), ''v'' = ''v''('''X''') || <math>\frac{\partial uv}{\partial \mathbf{X}} =</math>
| |
| | colspan=2|<math>u\frac{\partial v}{\partial \mathbf{X}} + v\frac{\partial u}{\partial \mathbf{X}} </math>
| |
| |-
| |
| | ''u'' = ''u''('''X''') || <math>\frac{\partial g(u)}{\partial \mathbf{X}} =</math>
| |
| | colspan=2|<math>\frac{\partial g(u)}{\partial u} \frac{\partial u}{\partial \mathbf{X}} </math>
| |
| |-
| |
| | ''u'' = ''u''('''X''') || <math>\frac{\partial f(g(u))}{\partial \mathbf{X}} =</math>
| |
| | colspan=2|<math>\frac{\partial f(g)}{\partial g} \frac{\partial g(u)}{\partial u} \frac{\partial u}{\partial \mathbf{X}} </math>
| |
| |-
| |
| | rowspan=2|'''U''' = '''U'''('''X''') || rowspan=2|<ref name="matrix-cookbook"/> <math>\frac{\partial g(\mathbf{U})}{\partial X_{ij}} =</math>
| |
| || <math>{\rm tr}\left( \frac{\partial g(\mathbf{U})}{\partial \mathbf{U}} \frac{\partial \mathbf{U}}{\partial X_{ij}}\right)</math>
| |
| || <math>{\rm tr}\left( \left(\frac{\partial g(\mathbf{U})}{\partial \mathbf{U}}\right)^{\rm T} \frac{\partial \mathbf{U}}{\partial X_{ij}}\right)</math>
| |
| |-
| |
| | colspan=2|Both forms assume ''numerator'' layout for <math>\frac{\partial \mathbf{U}}{\partial X_{ij}},</math> <br />
| |
| i.e. mixed layout if denominator layout for '''X''' is being used.
| |
| |- style="border-top: 3px solid;"
| |
| | || <math>\frac{\partial {\rm tr}(\mathbf{X})}{\partial \mathbf{X}} =</math> || colspan=2|<math>\mathbf{I}</math>
| |
| |-
| |
| | '''U''' = '''U'''('''X'''), '''V''' = '''V'''('''X''') || <math>\frac{\partial {\rm tr}(\mathbf{U}+\mathbf{V})}{\partial \mathbf{X}} =</math> || colspan=2|<math>\frac{\partial {\rm tr}(\mathbf{U})}{\partial \mathbf{X}} + \frac{\partial {\rm tr}(\mathbf{V})}{\partial \mathbf{X}}</math>
| |
| |-
| |
| | ''a'' is not a function of '''X''',<br /> '''U''' = '''U'''('''X''') || <math>\frac{\partial {\rm tr}(a\mathbf{U})}{\partial \mathbf{X}} =</math> || colspan=2|<math>a\frac{\partial {\rm tr}(\mathbf{U})}{\partial \mathbf{X}}</math>
| |
| |-
| |
| | '''g'''('''X''') is any [[polynomial]] with scalar coefficients, or any matrix function defined by an infinite polynomial series (e.g. e<sup>'''X'''</sup>, sin('''X'''), cos('''X'''), ln('''X'''), etc. using a [[Taylor series]]); ''g''(''x'') is the equivalent scalar function, ''g''<big>′</big>(''x'') is its derivative, and '''g'''<big>′</big>('''X''') is the corresponding matrix function || <math>\frac{\partial {\rm tr}(\mathbf{g(X)})}{\partial \mathbf{X}} =</math> || <math>\mathbf{g}'(\mathbf{X})</math> || <math>(\mathbf{g}'(\mathbf{X}))^{\rm T}</math>
| |
| |-
| |
| | '''A''' is not a function of '''X''' || <ref>{{cite web|last=Duchi|first=John C|title=Properties of the Trace and Matrix Derivatives|url=http://www.cs.berkeley.edu/~jduchi/projects/matrix_prop.pdf|publisher=University of California at Berkeley|accessdate=19 July 2011}}</ref> <math>\frac{\partial {\rm tr}(\mathbf{AX})}{\partial \mathbf{X}} = \frac{\partial {\rm tr}(\mathbf{XA})}{\partial \mathbf{X}} =</math> || <math>\mathbf{A}</math> || <math>\mathbf{A}^{\rm T}</math>
| |
| |-
| |
| | '''A''' is not a function of '''X''' || <ref name="matrix-cookbook"/> <math>\frac{\partial {\rm tr}(\mathbf{AX^{\rm T}})}{\partial \mathbf{X}} = \frac{\partial {\rm tr}(\mathbf{X^{\rm T}A})}{\partial \mathbf{X}} =</math> || <math>\mathbf{A}^{\rm T}</math> || <math>\mathbf{A}</math>
| |
| |-
| |
| | '''A''' is not a function of '''X''' || <ref name="matrix-cookbook"/> <math>\frac{\partial {\rm tr}(\mathbf{X^{\rm T}AX})}{\partial \mathbf{X}} =</math> || <math>\mathbf{X}^{\rm T}(\mathbf{A}+\mathbf{A}^{\rm T})</math> || <math>(\mathbf{A}+\mathbf{A}^{\rm T})\mathbf{X}</math>
| |
| |-
| |
| | '''A''' is not a function of '''X''' || <ref name="matrix-cookbook"/> <math>\frac{\partial {\rm tr}(\mathbf{X^{-1}A})}{\partial \mathbf{X}} =</math> || <math>-(\mathbf{X}^{-1})^{\rm T}\mathbf{A}(\mathbf{X}^{-1})^{\rm T}</math> || <math>-\mathbf{X}^{-1}\mathbf{A}^{\rm T}\mathbf{X}^{-1}</math>
| |
| |-
| |
| | '''A''', '''B''' are not functions of '''X''' || <math>\frac{\partial {\rm tr}(\mathbf{AXB})}{\partial \mathbf{X}} = \frac{\partial {\rm tr}(\mathbf{BAX})}{\partial \mathbf{X}} =</math> || <math>\mathbf{BA}</math> || <math>\mathbf{A^{\rm T}B^{\rm T}}</math>
| |
| |-
| |
| | '''A''', '''B''', '''C''' are not functions of '''X''' || <math>\frac{\partial {\rm tr}(\mathbf{AXBX^{\rm T}C})}{\partial \mathbf{X}} = </math> || <math>\mathbf{BX^{\rm T}CA} + \mathbf{B^{\rm T}X^{\rm T}A^{\rm T}C^{\rm T}}</math> || <math>\mathbf{A^{\rm T}C^{\rm T}XB^{\rm T}} + \mathbf{CAXB}</math>
| |
| |-
| |
| | ''n'' is a positive integer || <ref name="matrix-cookbook"/> <math>\frac{\partial {\rm tr}(\mathbf{X}^n)}{\partial \mathbf{X}} =</math> || <math>n\mathbf{X}^{n-1}</math> || <math>n(\mathbf{X}^{n-1})^{\rm T}</math>
| |
| |-
| |
| | '''A''' is not a function of '''X''',<br /> ''n'' is a positive integer || <ref name="matrix-cookbook"/> <math>\frac{\partial {\rm tr}(\mathbf{A}\mathbf{X}^n)}{\partial \mathbf{X}} =</math> || <math>\sum_{i=0}^{n-1} \mathbf{X}^i\mathbf{A}\mathbf{X}^{n-i-1}</math> || <math>\sum_{i=0}^{n-1} (\mathbf{X}^i\mathbf{A}\mathbf{X}^{n-i-1})^{\rm T}</math>
| |
| |-
| |
| | || <ref name="matrix-cookbook"/> <math>\frac{\partial {\rm tr}(e^\mathbf{X})}{\partial \mathbf{X}} =</math> || <math>e^\mathbf{X}</math> || <math>(e^\mathbf{X})^{\rm T}</math>
| |
| |-
| |
| | || <ref name="matrix-cookbook"/> <math>\frac{\partial {\rm tr}(\sin(\mathbf{X}))}{\partial \mathbf{X}} =</math> || <math>\cos(\mathbf{X})</math> || <math>(\cos(\mathbf{X}))^{\rm T}</math>
| |
| |- style="border-top: 3px solid;"
| |
| | || <ref>See [[Determinant#Derivative]] for the derivation.</ref> <math>\frac{\partial |\mathbf{X}|}{\partial \mathbf{X}} =</math> || <math>\operatorname{cofactor}(X)^{\rm T} = |\mathbf{X}|\mathbf{X}^{-1}</math> ||<math>\operatorname{cofactor}(X) = |\mathbf{X}|(\mathbf{X}^{-1})^{\rm T}</math>
| |
| |-
| |
| | ''a'' is not a function of '''X''' || <ref name="matrix-cookbook"/><math>\frac{\partial \ln |a\mathbf{X}|}{\partial \mathbf{X}} =</math><ref>The constant ''a'' disappears in the result. This is intentional. In general,
| |
| :<math>\frac{d\,\ln au}{dx} = \frac{1}{au}\frac{d(au)}{dx} = \frac{1}{au}a\frac{du}{dx} = \frac{1}{u}\frac{du}{dx} = \frac{d\,\ln u}{dx}.</math></ref>
| |
| || <math>\mathbf{X}^{-1}</math> ||<math>(\mathbf{X}^{-1})^{\rm T}</math>
| |
| |-
| |
| | '''A''', '''B''' are not functions of '''X''' || <ref name="matrix-cookbook"/> <math>\frac{\partial |\mathbf{AXB}|}{\partial \mathbf{X}} =</math> || <math>|\mathbf{AXB}|\mathbf{X}^{-1}</math> ||<math>|\mathbf{AXB}|(\mathbf{X}^{-1})^{\rm T}</math>
| |
| |-
| |
| | ''n'' is a positive integer || <ref name="matrix-cookbook"/> <math>\frac{\partial |\mathbf{X}^n|}{\partial \mathbf{X}} =</math> || <math>n|\mathbf{X}^n|\mathbf{X}^{-1}</math> ||<math>n|\mathbf{X}^n|(\mathbf{X}^{-1})^{\rm T}</math>
| |
| |-
| |
| | (see [[pseudo-inverse]]) || <ref name="matrix-cookbook"/> <math>\frac{\partial \ln |\mathbf{X}^{\rm T}\mathbf{X}|}{\partial \mathbf{X}} =</math> || <math>2\mathbf{X}^{+}</math> ||<math>2(\mathbf{X}^{+})^{\rm T}</math>
| |
| |-
| |
| | (see [[pseudo-inverse]]) || <ref name="matrix-cookbook"/> <math>\frac{\partial \ln |\mathbf{X}^{\rm T}\mathbf{X}|}{\partial \mathbf{X}^{+}} =</math> || <math>-2\mathbf{X}</math> ||<math>-2\mathbf{X}^{\rm T}</math>
| |
| |-
| |
| | '''A''' is not a function of '''X''',<br />'''X''' is square and invertible || <math>\frac{\partial |\mathbf{X^{\rm T}}\mathbf{A}\mathbf{X}|}{\partial \mathbf{X}} =</math> || <math>2|\mathbf{X^{\rm T}}\mathbf{A}\mathbf{X}|\mathbf{X}^{-1}</math> ||<math>2|\mathbf{X^{\rm T}}\mathbf{A}\mathbf{X}|(\mathbf{X}^{-1})^{\rm T}</math>
| |
| |-
| |
| | '''A''' is not a function of '''X''',<br />'''X''' is non-square,<br />'''A''' is symmetric || <math>\frac{\partial |\mathbf{X^{\rm T}}\mathbf{A}\mathbf{X}|}{\partial \mathbf{X}} =</math> ||<math>2|\mathbf{X^{\rm T}}\mathbf{A}\mathbf{X}|(\mathbf{X^{\rm T}A^{\rm T}X})^{-1}\mathbf{X^{\rm T}A^{\rm T}}</math> ||<math>2|\mathbf{X^{\rm T}}\mathbf{A}\mathbf{X}|\mathbf{AX}(\mathbf{X^{\rm T}AX})^{-1}</math>
| |
| |-
| |
| | '''A''' is not a function of '''X''',<br />'''X''' is non-square,<br />'''A''' is non-symmetric || <math>\frac{\partial |\mathbf{X^{\rm T}}\mathbf{A}\mathbf{X}|}{\partial \mathbf{X}} =</math>
| |
| ||<math>|\mathbf{X^{\rm T}}\mathbf{A}\mathbf{X}|((\mathbf{X^{\rm T}AX})^{-1}\mathbf{X^{\rm T}A}</math>
| |
| ::<math>+ (\mathbf{X^{\rm T}A^{\rm T}X})^{-1}\mathbf{X^{\rm T}A^{\rm T}})</math>
| |
| ||<math>|\mathbf{X^{\rm T}}\mathbf{A}\mathbf{X}|(\mathbf{AX}(\mathbf{X^{\rm T}AX})^{-1}</math>
| |
| ::<math> + \mathbf{A^{\rm T}X}(\mathbf{X^{\rm T}A^{\rm T}X})^{-1})</math>
| |
| |}
| |
| | |
| === Matrix-by-scalar identities ===
| |
| | |
| :{|class="wikitable" style="text-align: center;"
| |
| |+ Identities: matrix-by-scalar <math>\frac{\partial \mathbf{Y}}{\partial x}</math>
| |
| ! scope="col" width="175" | Condition
| |
| ! scope="col" width="100" | Expression
| |
| ! scope="col" width="100" | Numerator layout, i.e. by '''Y'''
| |
| |-
| |
| | '''U''' = '''U'''(''x'') || <math>\frac{\partial a\mathbf{U}}{\partial x} =</math> || <math>a\frac{\partial \mathbf{U}}{\partial x}</math>
| |
| |-
| |
| | '''A''', '''B''' are not functions of ''x'',<br /> '''U''' = '''U'''(''x'') || <math>\frac{\partial \mathbf{AUB}}{\partial x} =</math> || <math>\mathbf{A}\frac{\partial \mathbf{U}}{\partial x}\mathbf{B}</math>
| |
| |-
| |
| | '''U''' = '''U'''(''x''), '''V''' = '''V'''(''x'') || <math>\frac{\partial (\mathbf{U}+\mathbf{V})}{\partial x} =</math> || <math>\frac{\partial \mathbf{U}}{\partial x} + \frac{\partial \mathbf{V}}{\partial x}</math>
| |
| |-
| |
| | '''U''' = '''U'''(''x''), '''V''' = '''V'''(''x'') || <math>\frac{\partial (\mathbf{U}\mathbf{V})}{\partial x} =</math> || <math>\mathbf{U}\frac{\partial \mathbf{V}}{\partial x} + \frac{\partial \mathbf{U}}{\partial x}\mathbf{V}</math>
| |
| |-
| |
| | '''U''' = '''U'''(''x''), '''V''' = '''V'''(''x'') || <math>\frac{\partial (\mathbf{U} \otimes \mathbf{V})}{\partial x} =</math> || <math>\mathbf{U} \otimes \frac{\partial \mathbf{V}}{\partial x} + \frac{\partial \mathbf{U}}{\partial x} \otimes \mathbf{V}</math>
| |
| |-
| |
| | '''U''' = '''U'''(''x''), '''V''' = '''V'''(''x'') || <math>\frac{\partial (\mathbf{U} \circ \mathbf{V})}{\partial x} =</math> || <math>\mathbf{U} \circ \frac{\partial \mathbf{V}}{\partial x} + \frac{\partial \mathbf{U}}{\partial x} \circ \mathbf{V}</math>
| |
| |-
| |
| | '''U''' = '''U'''(''x'') || <math>\frac{\partial \mathbf{U}^{-1}}{\partial x} =</math> || <math>-\mathbf{U}^{-1} \frac{\partial \mathbf{U}}{\partial x}\mathbf{U}^{-1}</math>
| |
| |-
| |
| | '''U''' = '''U'''(''x,y'') || <math>\frac{\partial^2 \mathbf{U}^{-1}}{\partial x \partial y} =</math> || <math>\mathbf{U}^{-1}\left(\frac{\partial \mathbf{U}}{\partial x}\mathbf{U}^{-1}\frac{\partial \mathbf{U}}{\partial y} - \frac{\partial^2 \mathbf{U}}{\partial x \partial y} + \frac{\partial \mathbf{U}}{\partial y}\mathbf{U}^{-1}\frac{\partial \mathbf{U}}{\partial x}\right)\mathbf{U}^{-1}</math>
| |
| |-
| |
| | '''A''' is not a function of ''x'', '''g'''('''X''') is any polynomial with scalar coefficients, or any matrix function defined by an infinite polynomial series (e.g. e<sup>'''X'''</sup>, sin('''X'''), cos('''X'''), ln('''X'''), etc.); ''g''(''x'') is the equivalent scalar function, ''g''<big>′</big>(''x'') is its derivative, and '''g'''<big>′</big>('''X''') is the corresponding matrix function || <math>\frac{\partial \, \mathbf{g}(x\mathbf{A})}{\partial x} =</math> || colspan=2|<math>\mathbf{A}\mathbf{g}'(x\mathbf{A}) = \mathbf{g}'(x\mathbf{A})\mathbf{A}</math>
| |
| |-
| |
| | '''A''' is not a function of ''x'' || <math>\frac{\partial e^{x\mathbf{A}}}{\partial x} =</math> || <math>\mathbf{A}e^{x\mathbf{A}} = e^{x\mathbf{A}}\mathbf{A}</math>
| |
| |}
| |
| | |
| === Scalar-by-scalar identities ===
| |
| | |
| ==== With vectors involved ====
| |
| | |
| :{|class="wikitable" style="text-align: center;"
| |
| |+ Identities: scalar-by-scalar, with vectors involved
| |
| ! scope="col" width="150" | Condition
| |
| ! scope="col" width="10" | Expression
| |
| ! scope="col" width="150" | Any layout (assumes dot product ignores row vs. column layout)
| |
| |-
| |
| | '''u''' = '''u'''(''x'') || <math>\frac{\partial g(\mathbf{u})}{\partial x} =</math> || <math>\frac{\partial g(\mathbf{u})}{\partial \mathbf{u}} \cdot \frac{\partial \mathbf{u}}{\partial x}</math>
| |
| |-
| |
| | '''u''' = '''u'''(''x''), '''v''' = '''v'''(''x'') || <math>\frac{\partial (\mathbf{u} \cdot \mathbf{v})}{\partial x} =</math>
| |
| | colspan=2|<math>\mathbf{u} \cdot \frac{\partial \mathbf{v}}{\partial x} + \frac{\partial \mathbf{u}}{\partial x} \cdot \mathbf{v} </math>
| |
| |}
| |
| | |
| ==== With matrices involved ====
| |
| | |
| :{|class="wikitable" style="text-align: center;"
| |
| |+ Identities: scalar-by-scalar, with matrices involved<ref name="matrix-cookbook">Petersen, Kaare Brandt and Michael Syskind Pedersen. ''The Matrix Cookbook.'' November 14, 2008. http://matrixcookbook.com. [http://matrixcookbook.com] This book uses a mixed layout, i.e. by '''Y''' in <math>\frac{\partial \mathbf{Y}}{\partial x},</math> by '''X''' in <math>\frac{\partial y}{\partial \mathbf{X}}.</math></ref>
| |
| ! scope="col" width="175" | Condition
| |
| ! scope="col" width="100" | Expression
| |
| ! scope="col" width="100" | Consistent numerator layout,<br />i.e. by '''Y''' and '''X'''<sup>T</sup>
| |
| ! scope="col" width="100" | Mixed layout,<br />i.e. by '''Y''' and '''X'''
| |
| |-
| |
| | '''U''' = '''U'''(''x'') || <math>\frac{\partial |\mathbf{U}|}{\partial x} =</math> || colspan=2|<math>|\mathbf{U}|{\rm tr}\left(\mathbf{U}^{-1}\frac{\partial \mathbf{U}}{\partial x}\right)</math>
| |
| |-
| |
| | '''U''' = '''U'''(''x'') || <math>\frac{\partial \ln|\mathbf{U}|}{\partial x} =</math> || colspan=2|<math>{\rm tr}\left(\mathbf{U}^{-1}\frac{\partial \mathbf{U}}{\partial x}\right)</math>
| |
| |-
| |
| | '''U''' = '''U'''(''x'') || <math>\frac{\partial^2 |\mathbf{U}|}{\partial x^2} =</math> || colspan=2| <math>|\mathbf{U}|\left[{\rm tr}\left(\mathbf{U}^{-1}\frac{\partial^2 \mathbf{U}}{\partial x^2}\right) + \left({\rm tr}\left(\mathbf{U}^{-1}\frac{\partial \mathbf{U}}{\partial x}\right)\right)^2-{\rm tr}\left(\left(\mathbf{U}^{-1}\frac{\partial \mathbf{U}}{\partial x}\right)\left(\mathbf{U}^{-1}\frac{\partial \mathbf{U}}{\partial x}\right)\right)\right]</math>
| |
| |-
| |
| | '''U''' = '''U'''(''x'') || <math>\frac{\partial g(\mathbf{U})}{\partial x} =</math> || <math>{\rm tr}\left( \frac{\partial g(\mathbf{U})}{\partial \mathbf{U}} \frac{\partial \mathbf{U}}{\partial x}\right)</math>
| |
| || <math>{\rm tr}\left( \left(\frac{\partial g(\mathbf{U})}{\partial \mathbf{U}}\right)^{\rm T} \frac{\partial \mathbf{U}}{\partial x}\right)</math>
| |
| |-
| |
| | '''A''' is not a function of ''x'', '''g'''('''X''') is any polynomial with scalar coefficients, or any matrix function defined by an infinite polynomial series (e.g. e<sup>'''X'''</sup>, sin('''X'''), cos('''X'''), ln('''X'''), etc.); ''g''(''x'') is the equivalent scalar function, ''g''<big>′</big>(''x'') is its derivative, and '''g'''<big>′</big>('''X''') is the corresponding matrix function. || <math>\frac{\partial \, {\rm tr}(\mathbf{g}(x\mathbf{A}))}{\partial x} =</math> || colspan=2|<math>{\rm tr}(\mathbf{A}\mathbf{g}'(x\mathbf{A}))</math>
| |
| |-
| |
| | '''A''' is not a function of ''x'' || <math>\frac{\partial \, {\rm tr}(e^{x\mathbf{A}})}{\partial x} =</math> || colspan=2|<math>{\rm tr}(\mathbf{A}e^{x\mathbf{A}})</math>
| |
| |}
| |
| | |
| === Identities in differential form ===
| |
| | |
| It is often easier to work in differential form and then convert back to normal derivatives. This only works well using the numerator layout.
| |
| | |
| :{|class="wikitable" style="text-align: center;"
| |
| |+ Differential identities: scalar involving matrix <ref name="minka"/><ref name="matrix-cookbook"/>
| |
| ! Condition !! Expression !! Result (numerator layout)
| |
| |-
| |
| | || <math>d({\rm tr}(\mathbf{X})) =</math> || <math>{\rm tr}(d\mathbf{X})</math>
| |
| |-
| |
| | || <math>d(|\mathbf{X}|) =</math> || <math>|\mathbf{X}|{\rm tr}(\mathbf{X}^{-1}d\mathbf{X})</math>
| |
| |-
| |
| | || <math>d(\ln|\mathbf{X}|) =</math> || <math>{\rm tr}(\mathbf{X}^{-1}d\mathbf{X})</math>
| |
| |}
| |
| | |
| :{|class="wikitable" style="text-align: center;"
| |
| |+ Differential identities: matrix <ref name="minka"/><ref name="matrix-cookbook"/>
| |
| ! Condition !! Expression !! Result (numerator layout)
| |
| |-
| |
| |'''A''' is not a function of '''X''' || <math>d(\mathbf{A}) =</math> || <math>0</math>
| |
| |-
| |
| |''a'' is not a function of '''X''' || <math>d(a\mathbf{X}) =</math> || <math>a\,d\mathbf{X}</math>
| |
| |-
| |
| | || <math>d(\mathbf{X}+\mathbf{Y}) =</math> || <math>d\mathbf{X}+d\mathbf{Y}</math>
| |
| |-
| |
| | || <math>d(\mathbf{X}\mathbf{Y}) =</math> || <math>(d\mathbf{X})\mathbf{Y}+\mathbf{X}(d\mathbf{Y})</math>
| |
| |-
| |
| | ([[Kronecker product]]) || <math>d(\mathbf{X} \otimes \mathbf{Y}) =</math> || <math>(d\mathbf{X})\otimes\mathbf{Y}+\mathbf{X}\otimes(d\mathbf{Y})</math>
| |
| |-
| |
| | ([[Hadamard product (matrices)|Hadamard product]]) || <math>d(\mathbf{X} \circ \mathbf{Y}) =</math> || <math>(d\mathbf{X})\circ\mathbf{Y}+\mathbf{X}\circ(d\mathbf{Y})</math>
| |
| |-
| |
| | || <math>d(\mathbf{X}^{\rm T}) =</math> || <math>(d\mathbf{X})^{\rm T}</math>
| |
| |-
| |
| | ([[conjugate transpose]]) || <math>d(\mathbf{X}^{\rm H}) =</math> || <math>(d\mathbf{X})^{\rm H}</math>
| |
| |}
| |
| | |
| To convert to normal derivative form, first convert it to one of the following canonical forms, and then use these identities:
| |
| | |
| :{|class="wikitable" style="text-align: center;"
| |
| |+ Conversion from differential to derivative form <ref name="minka"/>
| |
| ! Canonical differential form !! Equivalent derivative form
| |
| |-
| |
| | <math>dy = a\,dx</math> || <math>\frac{dy}{dx} = a</math>
| |
| |-
| |
| | <math>dy = \mathbf{a}\,d\mathbf{x}</math> || <math>\frac{dy}{d\mathbf{x}} = \mathbf{a}</math>
| |
| |-
| |
| | <math>dy = {\rm tr}(\mathbf{A}\,d\mathbf{X})</math> || <math>\frac{dy}{d\mathbf{X}} = \mathbf{A}</math>
| |
| |-
| |
| | <math>d\mathbf{y} = \mathbf{a}\,dx</math> || <math>\frac{d\mathbf{y}}{dx} = \mathbf{a}</math>
| |
| |-
| |
| | <math>d\mathbf{y} = \mathbf{A}\,d\mathbf{x}</math> || <math>\frac{d\mathbf{y}}{d\mathbf{x}} = \mathbf{A}</math>
| |
| |-
| |
| | <math>d\mathbf{Y} = \mathbf{A}\,dx</math> || <math>\frac{d\mathbf{Y}}{dx} = \mathbf{A}</math>
| |
| |}
| |
| | |
| == See also ==
| |
| | |
| {{Portal|Mathematics}}
| |
| * [[Derivative (generalizations)]]
| |
| * [[Product integral]]
| |
| | |
| == Notes ==
| |
| | |
| <references/>
| |
| | |
| == External links ==
| |
| * [http://www.colorado.edu/engineering/cas/courses.d/IFEM.d/IFEM.AppD.d/IFEM.AppD.pdf Linear Algebra: Determinants, Inverses, Rank] appendix D from ''Introduction to Finite Element Methods'' book on [[University of Colorado]] at [[Boulder, Colorado|Boulder]]. Uses the [[Hessian matrix|Hessian]] ([[transpose]] to [[Jacobian matrix and determinant|Jacobian]]) definition of vector and matrix derivatives.
| |
| * [http://www.psi.toronto.edu/matrix/calculus.html Matrix Reference Manual], Mike Brookes, [[Imperial College London]].
| |
| * [http://www.mit.edu/~wingated/stuff_i_use/matrix_cookbook.pdf The Matrix Cookbook], with a ''derivatives'' chapter. Uses the Hessian definition.
| |
| * [http://www.wiley.com/WileyCDA/WileyTitle/productCd-0471751561,descCd-authorInfo.html Linear Algebra and its Applications] (author information page; see Chapter 9 of book), [[Peter Lax]], [[Courant Institute]].
| |
| * [http://www.atmos.washington.edu/~dennis/MatrixCalculus.pdf Matrix Differentiation (and some other stuff)], Randal J. Barnes, Department of Civil Engineering, University of Minnesota.
| |
| * [http://www4.ncsu.edu/~pfackler/MatCalc.pdf Notes on Matrix Calculus], Paul L. Fackler, [[North Carolina State University]].
| |
| * [https://wiki.inf.ed.ac.uk/twiki/pub/CSTR/ListenSemester1_2006_7/slide.pdf Matrix Differential Calculus] (slide presentation), Zhang Le, [[University of Edinburgh]].
| |
| * [http://www.econ.ku.dk/metrics/Econometrics2_05_II/LectureNotes/matrixdiff.pdf Introduction to Vector and Matrix Differentiation] (notes on matrix differentiation, in the context of [[Econometrics]]), Heino Bohn Nielsen.
| |
| * [http://mpra.ub.uni-muenchen.de/1239/1/MPRA_paper_1239.pdf A note on differentiating matrices] (notes on matrix differentiation), Pawel Koval, from Munich Personal RePEc Archive.
| |
| * [http://www.personal.rdg.ac.uk/~sis01xh/teaching/CY4C9/ANN3.pdf Vector/Matrix Calculus] More notes on matrix differentiation.
| |
| * [http://itee.uq.edu.au/~comp4702/material/matrixid.pdf Matrix Identities] (notes on matrix differentiation), Sam Roweis.
| |
| | |
| [[Category:Matrix theory]]
| |
| [[Category:Linear algebra]]
| |
| [[Category:Multivariable calculus]]
| |