Intersection theory - Revision history

en>Pythasis: Not to be confused with...

2014-12-16T03:05:28Z

Not to be confused with...

← Older revision		Revision as of 05:05, 16 December 2014
Line 1:		Line 1:
	~~The~~ '''Frank–Wolfe algorithm''' is a simple [[iterative method\|iterative]] [[First-order approximation\|first-order]] [[Mathematical optimization\|optimization]] [[algorithm]] for [[constrained optimization\|constrained]] [[convex optimization]]. Also known as the '''conditional gradient method''',<ref>{{Cite doi\|10.1016/0041-5553(66)90114-5\|noedit}}</ref> '''reduced gradient algorithm''' and the '''convex combination algorithm''', the method was originally proposed by [[Marguerite Frank]] and [[Philip Wolfe (mathematician)\|Philip Wolfe]] in 1956.<ref>{{cite doi\|10.1002/nav.3800030109\|noedit}}</ref> In each iteration, the Frank–Wolfe algorithm considers a [[linear approximation]] of the objective function, and moves slightly towards a minimizer of this linear function (taken over the same domain).		Irwin Butts is what my wife enjoys to call me although I don't really like becoming known as like that. Body building is 1 of the things I adore most. For many years he's been operating as a receptionist. For years he's been residing in North Dakota and his family loves it.<br><br>my website: [http://www.streaming.iwarrior.net/user/WMcneil at home std test]

	~~==Problem statement==~~

	~~:Minimize <math> f(\mathbf{x})</math>~~
	~~:subject to <math> \mathbf{x} \in \mathcal{D}</math>~~.
	~~Where the function <math> f</math>~~ is ~~[[Convex function\|convex]] and [[differentiable function\|differentiable]], and the domain / feasible set <math>\mathcal{D}</math> is a [[Convex set\|convex]] and bounded set in some [[vector space]].~~

	~~==Algorithm==~~
	~~[[File:Frank-Wolfe-Algorithm.png\|thumbnail\|right\|A step of the Frank-Wolfe algorithm]]~~

	~~:''Initialization:'' Let <math>k \leftarrow 0</math>, and let <math>\mathbf{x}_0 \!</math> be any point in <math>\mathcal{D}</math>.~~

	~~:'''Step~~ 1~~.''' ''Direction-finding subproblem:'' Find <math>\mathbf{s}_k</math> solving~~
	~~::Minimize <math> \mathbf{s}^T \nabla f(\mathbf{x}_k)</math>~~
	~~::Subject to <math>\mathbf{s} \in \mathcal{D}</math>~~
	~~:''(Interpretation: Minimize the linear approximation~~ of the ~~problem given by the first-order [[Taylor series\|Taylor approximation]] of <math>f</math> around <math>\mathbf{x}_k \!</math>.)''~~

	~~:'''Step 2~~.'~~'' ''Step size determination:'' Set <math>\gamma \leftarrow \frac{2}{k+2}</math>, or alternatively find <math>\gamma</math> that minimizes <math> f(\mathbf{x}_k+\gamma(\mathbf{~~s~~}_k -\mathbf{x}_k))</math> subject to <math>0 \le \gamma \le 1</math> .~~

	~~:'''Step 3.''' ''Update:'' Let <math>\mathbf{x}_{k+1}\leftarrow \mathbf{x}_k+\gamma(\mathbf{s}_k-\mathbf{x}_k)</math>, let <math>k \leftarrow k+1</math> and go to Step 1.~~

	~~==Properties==~~
	While competing methods such as [[gradient descent]] for constrained optimization require a [[Projection (mathematics)\|projection step]] back to the feasible set in each iteration, the Frank–Wolfe algorithm only needs the solution of a linear problem over the same set in each iteration, and automatically stays in the feasible set.

	The convergence of the Frank–Wolfe algorithm is sublinear in general: the error to the optimum is <math>O(1/k)</math> after k iterations. The same convergence rate can also be shown if the sub-problems are only solved approximately.<ref>{{cite doi\|10.1016/0022-247X(78)90137-3\|noedit}}</ref>

	~~The iterates of the algorithm can always be represented~~ as a sparse convex combination of the extreme points of the feasible set, which has helped to the popularity of the algorithm for sparse greedy optimization in [[machine learning]] and [[signal processing]] problems,<ref>{{cite doi\|10.~~1145/1824777.1824783\|noedit}}</ref> as well as for example the optimization of [[flow network\|minimum–cost flow]]~~s in ~~[[transportation network]]s.<ref>{{cite doi\|10.1016/0191-2615(84)90029-8\|noedit}}</ref>~~

	~~If the feasible set is given by a set of linear constraints, then the subproblem to be solved in each iteration becomes a [[linear programming\|linear program]].~~

	While the worst-case convergence rate with <math>O(1/k)</math> can not be improved in general, faster convergence can be obtained for special problem classes, such as some strongly convex problems.<ref>{{Cite book\|title=Nonlinear Programming\|first= Dimitri \|last=Bertsekas\|year= 2003\|page= 222\|publisher= Athena Scientific\| isbn =1-886529-00-0}}</ref>

	~~==Lower bounds on the solution value, and primal-dual analysis==~~

	~~Since <math>f</math> is convex, <math>f(\mathbf{y})</math> is always above the [[Tangent\|tangent plane]] of <math>f</math> at any point <math>\mathbf{x} \in \mathcal{D}</math>:~~

	~~:<math>~~
	~~f(\mathbf{y}) \geq f(\mathbf{x}) + (\mathbf{y} - \mathbf{x})^T \nabla f(\mathbf{x})~~
	~~</math>~~

	~~This holds in particular for the (unknown) optimal solution <math>\mathbf{x}^*</math>. The best lower bound with respect to a given point <math>\mathbf{x}</math> is given by~~

	~~:<math>~~
	f(\mathbf{x}^*) \geq \min_{\mathbf{y} \in D} f(\mathbf{x}) + (\mathbf{y} - \mathbf{x})^T \nabla f(\mathbf{x}) = f(\mathbf{x}) - \mathbf{x}^T \nabla f(\mathbf{x}) + \min_{\mathbf{y} \in D} \mathbf{y}^T \nabla f(\mathbf{x})
	~~</math>~~

	The latter optimization problem is solved in every iteration of the Frank-Wolfe algorithm, therefore the solution <math>\mathbf{s}_k</math> of the direction-finding subproblem of the <math>k</math>-th iteration can be used to determine increasing lower bounds <math>l_k</math> during each iteration by setting <math>l_0 = - \infty</math> and

	~~:<math>~~
	~~l_k := \max (l_{k - 1}, f(\mathbf{x}_k) + (\mathbf{s}_k - \mathbf{x})^T \nabla f(\mathbf{x}_k))~~
	~~</math>~~
	Such lower bounds on the unknown optimal value are important in practice because they can be used as a stopping criterion, and give an efficient certificate of the approximation quality in every iteration, since always <math>l_k \leq f(\mathbf{x}^*) \leq f(\mathbf{x}_k)</math>.

	~~It has been shown that this corresponding [[duality gap]], that is the difference between <math>f(\mathbf{x}_k)</math>~~ and ~~the lower bound <math>l_k</math>, decreases with the same convergence rate, i~~.e.
	<~~math~~>
	~~f(\mathbf{x}_k) - l_k = O(1/k) .~~
	<~~/math~~>

	~~==Notes==~~
	~~{{Reflist}}~~

	~~==Bibliography==~~
	*{{cite journal\|last=Jaggi\|first=Martin\|title=Revisiting Frank-Wolfe: Projection-Free Sparse Convex Optimization\|journal=Journal of Machine Learning Research: ~~Workshop and Conference Proceedings \|volume=28\|issue=1\|pages=427–435\|year= 2013 \|url=http://jmlr.csail.mit.edu/proceedings/papers/v28/jaggi13.html}} (Overview paper)~~
	*[http://www.~~math~~.~~chalmers~~.se/~~Math~~/~~Grundutb/CTH/tma946/0203/fw_eng.pdf The Frank-Wolfe algorithm] description~~

	~~== See also ==~~
	* [[Proximal Gradient Methods]]

	~~{{Optimization algorithms\|convex}}~~

	~~{{DEFAULTSORT:Frank-Wolfe algorithm}}~~
	~~[[Category:Optimization algorithms and methods]]~~
	~~[[Category:Iterative methods]]~~
	~~[[Category:First order methods]]~~
	~~[[Category:Gradient methods]~~]

en>ElNuevoEinstein: /* Moving cycles */

2014-01-09T21:49:18Z

Moving cycles

← Older revision		Revision as of 23:49, 9 January 2014
Line 1:		Line 1:
	~~Hello~~, ~~my name is Andrew~~ and ~~my spouse doesn~~'~~t like it at all~~. ~~To climb~~ is some ~~thing she would never give up~~. ~~Office supervising~~ is ~~where her main earnings comes from but she'~~s ~~currently utilized~~ for ~~another~~ 1. ~~Mississippi~~ is ~~exactly where her home~~ is ~~but her spouse wants them~~ to ~~transfer~~.<br><br>~~Here is my blog~~: [http://~~chungmuroresidence~~.~~com~~/xe/~~reservation_branch2~~/~~152663 real psychic~~]		The '''Frank–Wolfe algorithm''' is a simple [[iterative method\|iterative]] [[First-order approximation\|first-order]] [[Mathematical optimization\|optimization]] [[algorithm]] for [[constrained optimization\|constrained]] [[convex optimization]]. Also known as the '''conditional gradient method''',<ref>{{Cite doi\|10.1016/0041-5553(66)90114-5\|noedit}}</ref> '''reduced gradient algorithm''' and the '''convex combination algorithm''', the method was originally proposed by [[Marguerite Frank]] and [[Philip Wolfe (mathematician)\|Philip Wolfe]] in 1956.<ref>{{cite doi\|10.1002/nav.3800030109\|noedit}}</ref> In each iteration, the Frank–Wolfe algorithm considers a [[linear approximation]] of the objective function, and moves slightly towards a minimizer of this linear function (taken over the same domain).

			==Problem statement==

			:Minimize <math> f(\mathbf{x})</math>
			:subject to <math> \mathbf{x} \in \mathcal{D}</math>.
			Where the function <math> f</math> is [[Convex function\|convex]] and [[differentiable function\|differentiable]], and the domain / feasible set <math>\mathcal{D}</math> is a [[Convex set\|convex]] and bounded set in some [[vector space]].

			==Algorithm==
			[[File:Frank-Wolfe-Algorithm.png\|thumbnail\|right\|A step of the Frank-Wolfe algorithm]]

			:''Initialization:'' Let <math>k \leftarrow 0</math>, and let <math>\mathbf{x}_0 \!</math> be any point in <math>\mathcal{D}</math>.

			:'''Step 1.''' ''Direction-finding subproblem:'' Find <math>\mathbf{s}_k</math> solving
			::Minimize <math> \mathbf{s}^T \nabla f(\mathbf{x}_k)</math>
			::Subject to <math>\mathbf{s} \in \mathcal{D}</math>
			:''(Interpretation: Minimize the linear approximation of the problem given by the first-order [[Taylor series\|Taylor approximation]] of <math>f</math> around <math>\mathbf{x}_k \!</math>.)''

			:'''Step 2.''' ''Step size determination:'' Set <math>\gamma \leftarrow \frac{2}{k+2}</math>, or alternatively find <math>\gamma</math> that minimizes <math> f(\mathbf{x}_k+\gamma(\mathbf{s}_k -\mathbf{x}_k))</math> subject to <math>0 \le \gamma \le 1</math> .

			:'''Step 3.''' ''Update:'' Let <math>\mathbf{x}_{k+1}\leftarrow \mathbf{x}_k+\gamma(\mathbf{s}_k-\mathbf{x}_k)</math>, let <math>k \leftarrow k+1</math> and go to Step 1.

			==Properties==
			While competing methods such as [[gradient descent]] for constrained optimization require a [[Projection (mathematics)\|projection step]] back to the feasible set in each iteration, the Frank–Wolfe algorithm only needs the solution of a linear problem over the same set in each iteration, and automatically stays in the feasible set.

			The convergence of the Frank–Wolfe algorithm is sublinear in general: the error to the optimum is <math>O(1/k)</math> after k iterations. The same convergence rate can also be shown if the sub-problems are only solved approximately.<ref>{{cite doi\|10.1016/0022-247X(78)90137-3\|noedit}}</ref>

			The iterates of the algorithm can always be represented as a sparse convex combination of the extreme points of the feasible set, which has helped to the popularity of the algorithm for sparse greedy optimization in [[machine learning]] and [[signal processing]] problems,<ref>{{cite doi\|10.1145/1824777.1824783\|noedit}}</ref> as well as for example the optimization of [[flow network\|minimum–cost flow]]s in [[transportation network]]s.<ref>{{cite doi\|10.1016/0191-2615(84)90029-8\|noedit}}</ref>

			If the feasible set is given by a set of linear constraints, then the subproblem to be solved in each iteration becomes a [[linear programming\|linear program]].

			While the worst-case convergence rate with <math>O(1/k)</math> can not be improved in general, faster convergence can be obtained for special problem classes, such as some strongly convex problems.<ref>{{Cite book\|title=Nonlinear Programming\|first= Dimitri \|last=Bertsekas\|year= 2003\|page= 222\|publisher= Athena Scientific\| isbn =1-886529-00-0}}</ref>

			==Lower bounds on the solution value, and primal-dual analysis==

			Since <math>f</math> is convex, <math>f(\mathbf{y})</math> is always above the [[Tangent\|tangent plane]] of <math>f</math> at any point <math>\mathbf{x} \in \mathcal{D}</math>:

			:<math>
			f(\mathbf{y}) \geq f(\mathbf{x}) + (\mathbf{y} - \mathbf{x})^T \nabla f(\mathbf{x})
			</math>

			This holds in particular for the (unknown) optimal solution <math>\mathbf{x}^*</math>. The best lower bound with respect to a given point <math>\mathbf{x}</math> is given by

			:<math>
			f(\mathbf{x}^*) \geq \min_{\mathbf{y} \in D} f(\mathbf{x}) + (\mathbf{y} - \mathbf{x})^T \nabla f(\mathbf{x}) = f(\mathbf{x}) - \mathbf{x}^T \nabla f(\mathbf{x}) + \min_{\mathbf{y} \in D} \mathbf{y}^T \nabla f(\mathbf{x})
			</math>

			The latter optimization problem is solved in every iteration of the Frank-Wolfe algorithm, therefore the solution <math>\mathbf{s}_k</math> of the direction-finding subproblem of the <math>k</math>-th iteration can be used to determine increasing lower bounds <math>l_k</math> during each iteration by setting <math>l_0 = - \infty</math> and

			:<math>
			l_k := \max (l_{k - 1}, f(\mathbf{x}_k) + (\mathbf{s}_k - \mathbf{x})^T \nabla f(\mathbf{x}_k))
			</math>
			Such lower bounds on the unknown optimal value are important in practice because they can be used as a stopping criterion, and give an efficient certificate of the approximation quality in every iteration, since always <math>l_k \leq f(\mathbf{x}^*) \leq f(\mathbf{x}_k)</math>.

			It has been shown that this corresponding [[duality gap]], that is the difference between <math>f(\mathbf{x}_k)</math> and the lower bound <math>l_k</math>, decreases with the same convergence rate, i.e.
			<math>
			f(\mathbf{x}_k) - l_k = O(1/k) .
			</math>

			==Notes==
			{{Reflist}}

			==Bibliography==
			*{{cite journal\|last=Jaggi\|first=Martin\|title=Revisiting Frank-Wolfe: Projection-Free Sparse Convex Optimization\|journal=Journal of Machine Learning Research: Workshop and Conference Proceedings \|volume=28\|issue=1\|pages=427–435\|year= 2013 \|url=http://jmlr.csail.mit.edu/proceedings/papers/v28/jaggi13.html}} (Overview paper)
			*[http://www.math.chalmers.se/Math/Grundutb/CTH/tma946/0203/fw_eng.pdf The Frank-Wolfe algorithm] description

			== See also ==
			* [[Proximal Gradient Methods]]

			{{Optimization algorithms\|convex}}

			{{DEFAULTSORT:Frank-Wolfe algorithm}}
			[[Category:Optimization algorithms and methods]]
			[[Category:Iterative methods]]
			[[Category:First order methods]]
			[[Category:Gradient methods]]

en>Myasuda: /* References */ dab

2012-02-25T02:01:33Z

References: dab

New page

Hello, my name is Andrew and my spouse doesn't like it at all. To climb is some thing she would never give up. Office supervising is where her main earnings comes from but she's currently utilized for another 1. Mississippi is exactly where her home is but her spouse wants them to transfer.<br><br>Here is my blog: [http://chungmuroresidence.com/xe/reservation_branch2/152663 real psychic]