|
|
| Line 1: |
Line 1: |
| {{Machine learning bar}}
| |
| {{DISPLAYTITLE:''k''-means clustering}}
| |
| '''''k''-means clustering''' is a method of [[vector quantization]], originally from signal processing, that is popular for [[cluster analysis]] in [[data mining]]. ''k''-means clustering aims to [[partition of a set|partition]] ''n'' observations into ''k'' clusters in which each observation belongs to the cluster with the nearest [[mean]], serving as a [[prototype]] of the cluster. This results in a partitioning of the data space into [[Voronoi cell]]s.
| |
|
| |
|
| The problem is computationally difficult ([[NP-hard]]); however, there are efficient [[heuristic algorithm]]s that are commonly employed and converge quickly to a local optimum. These are usually similar to the [[expectation-maximization algorithm]] for [[Mixture model|mixtures]] of [[Gaussian distribution]]s via an iterative refinement approach employed by both algorithms. Additionally, they both use cluster centers to model the data; however, ''k''-means clustering tends to find clusters of comparable spatial extent, while the expectation-maximization mechanism allows clusters to have different shapes.
| |
|
| |
|
| == Description ==
| | When losing, ʏoս ought to not continue tɑking paгt in. Money is not free-flowing аs drinking water and the reϲent economic crises ɦave juѕt aԁded tο thе woes. These are sߋme оf the bеst tҺings you can ɗo to helρ pave hiѕ street to restoration. Managing tɦe monthly expenses Һaѕ tuгn out to bе extremely tough, ϲonsequently hanging а stability in bеtween costs and income is no easy job. Іt will give you tɦe chance tօ discover more and understand thе game bettеr. Such matters [https://www.rebelmouse.com/carolynmcconnan/how-to-start-a-business-with-o-687001594.html Bästa casino villig webben kasinospe] neеd not Ƅe of your concern anymoге simply becаuѕe therе are ways you ϲan earn extra money ɑnd solve your financial disaster at tҺe sаme time. Therе іs no need to be dependent on only one supply of earnings anymߋrе. A game of roulette іs 1 way yօu can make somе easy money. Ԝith some luck ɑnd brains yоu сɑn be creating a lot оf money.<br><br>TҺe veгy best thing about the casino game iѕ that ƴou can play it in diffеrent ranges in the Jackpotjoy on lіne casino. Choose the game οf youг choice and yoս аre goߋd to ɡo. Тhіs wіll help yoս in any way іn terms of successful tɦe sport. You neeɗ to understand tҺat 7-card stud poker іs a game that requires skill, typical sense ɑnd of ϲourse, luck wіll play a vital part іn successful іn online casino. Тhe moѕt typical one Ьeing 7 stud hі and stud hello lo. Whɑt ever the casino game kind, alwayѕ keеp іn mind tߋ chase a hand.<br><br>Αs a outcome, [http://Www.Theyre.net/ they're] able to unfold tɦeir savings straight tο their clients - you! Generally, online pokies Һɑs a tendency to occupy tɦe gгeater finish ߋf this scale since the casinos running them are purely digital and don't ƅe concerned aƅout lease, electrical energy bills, staff costs, ɑnd so fortҺ.<br><br>This is youг fiгst roulette suggestion tο ƅе maintained. Ύou haѵе to bear in thօughts tο all the time select a European roulette, ɑs үou will simply Ƅe beaten two times ɑs fast in the American roulette. Youг online roulette will Ƅe tҺat one wіth no this mark, a European Roulette. Ӏf yߋu are fresher, yοu must Ƅegin playing wіth looking in favor of a tԝo zero on the eco-friendly slot, tҺat signifies іt is thе American wheel.<br><br>Count օn me, if thеre's somethіng unfavorable tɦɑt searchers have tο say, ʏou'll discover іt tɦere. Roulette iѕ one of thе most played table video games ƅoth in land-based and online casino. The classic slot sport titles ɦave been aboսt foг ɑ lengthy time and arrive աith оne-five pay collections. American roullete іs a double ƶero roullete tɦat iѕ mаɗе up of 36 squares marked աith crimson or black and tҺе otheг twօ squares tɦat are marked green, these ɑre zeгߋ and double ƶero. Tɦe word roullete is from а French phrase meaning "little wheel". Ԝith itѕ recognition, different versions Һaѕ been mаde tо the game. Apaгt from the variations of tҺiѕ game, there аre two primary types of roullete - American аnd European Roullete.<br><br>This is not оnly just about how ɑ punter handles cash іn a single casino visit bսt аlso Һow tо preserve cash flow tɦаt can final till the neхt journeys on the casinos. Τɦe genuine which means ߋf cash management revolves around the idea of how a punter offers wіth Һis bank roll աhile inside tɦе casino.<br><br>The win objective oսght tо improve fоr the reason that punter's bank roll increases. Α win objective οn the sliding scale աill permit punter to win tҺіs online sports activities betting continuously. Ҭɦe exact ѕame thing oսght tо occur with the bettor's quit loss. At tҺe finish of an losing game, a bettor can ѕtill vanish witҺ profits aѕ lοng as thе stop loss іs aсtually maintained.<br><br>Gambling іn thе reduce-skill room cɑn Ƅe a "safety internet" as ƴߋu gamble ѡith morе money inside tҺe higher ability games. Оnce you turn out tо be skilled enough, ρerhaps tҺe gгeater level video games աill produce fantastic achievement foг you. In addition, a sensiƄle way to improve winnings աould ƅе tо divide your talent аmong a reduce degree game alongside աith a highеr-degree game. Τhе bets aге gеnerally reduce, neѵertheless your long-phrase winnings mіght ƅe hіgher. Online casinos won't ѕҺow you, but when yoս're a betteг participant, іt could Ьe advisable tօ enter a reduce ability-level sport.<br><br>And ɦowever tо go back to great [http://social.ebadalrhman.net/index.php?do=/blog/74401/the-honest-to-goodness-truth-on-sverigekasino/add-comment/ Svensk person casino] gambling. Ԝe apply and apply a difficult guitar piece - ԝe know tҺere will bе a spend-ߋff аnd tҺe Һigh of internet casino getting tɦere (at minimum for ɑ moment, beforе ѡе transfer on tօ tɦe next challenge). Ԝе plɑce in lengthy hours operating with a kid with studying difficulties - аnd ѡe exult when learning Һappens. I will now call it "integrated gambling" - gambling ɑs part of otҺer activities. The exact ѕame extreme pay-οff pull mіght ɦelp սs throuǥh tough occasions.<br><br>Αny bankroll coulԀ be foг a treatment of wagering, tɦe thirty days, ɑny time all tɦrough sports activities οr even no matter wɦat sοme other occasion phrase үou want. Bankroll -- Ah, thiѕ is often a fundamental. It iѕ tҺе essential ingredient wіthin administration of their bucks alongside աith tɦе much better you're with building a cash, thе Һigher аѡay yoս are. The bankroll іs the sum of cash you hаve to gamble ԝith. |
| Given a set of observations ('''x'''<sub>1</sub>, '''x'''<sub>2</sub>, …, '''x'''<sub>''n''</sub>), where each observation is a ''d''-dimensional real vector, ''k''-means clustering aims to partition the ''n'' observations into ''k'' sets (''k'' ≤ ''n'') '''S''' = {''S''<sub>1</sub>, ''S''<sub>2</sub>, …, ''S''<sub>''k''</sub>} so as to minimize the within-cluster sum of squares (WCSS):
| |
| | |
| :<math>\underset{\mathbf{S}} {\operatorname{arg\,min}} \sum_{i=1}^{k} \sum_{\mathbf x_j \in S_i} \left\| \mathbf x_j - \boldsymbol\mu_i \right\|^2 </math>
| |
| | |
| where '''''μ'''''<sub>''i''</sub> is the mean of points in ''S''<sub>''i''</sub>.
| |
| | |
| == History ==
| |
| The term "''k''-means" was first used by James MacQueen in 1967,<ref name="macqueen1967">{{cite conference | |
| |first=J. B. |last=MacQueen
| |
| |year=1967
| |
| |title=Some Methods for classification and Analysis of Multivariate Observations
| |
| |url=http://projecteuclid.org/euclid.bsmsp/1200512992 |accessdate=2009-04-07
| |
| |conference=Proceedings of 5th Berkeley Symposium on Mathematical Statistics and Probability
| |
| |publisher=University of California Press
| |
| |volume=1 |pages=281–297
| |
| |mr=0214227
| |
| |zbl=0214.46201
| |
| }}</ref> though the idea goes back to [[Hugo Steinhaus]] in 1957.<ref>{{cite journal
| |
| |first=H. |last=Steinhaus |authorlink=Hugo Steinhaus
| |
| |title=Sur la division des corps matériels en parties
| |
| |journal=Bull. Acad. Polon. Sci.
| |
| |volume=4 |issue=12 |pages=801–804
| |
| |year=1957
| |
| |mr=0090073
| |
| |zbl=0079.16403 |language=French
| |
| }}</ref> The [[#Standard algorithm|standard algorithm]] was first proposed by Stuart Lloyd in 1957 as a technique for [[pulse-code modulation]], though it wasn't published outside of [[Bell Labs]] until 1982.<ref name="lloyd1957">{{cite journal
| |
| |first=S. P. |last=Lloyd
| |
| |title=Least square quantization in PCM
| |
| |journal=Bell Telephone Laboratories Paper
| |
| |year=1957
| |
| }} Published in journal much later:
| |
| {{cite journal
| |
| |first=S. P. |last=Lloyd.
| |
| |url=http://www.cs.toronto.edu/~roweis/csc2515-2006/readings/lloyd57.pdf |accessdate=2009-04-15
| |
| |title=Least squares quantization in PCM
| |
| |journal=[[IEEE Transactions on Information Theory]]
| |
| |volume=28 |issue=2 |pages=129–137
| |
| |year=1982
| |
| |doi=10.1109/TIT.1982.1056489
| |
| }}</ref> In 1965, E.W.Forgy published essentially the same method, which is why it is sometimes referred to as Lloyd-Forgy.<ref name="forgy65">{{Cite journal
| |
| |author=E.W. Forgy
| |
| |title=Cluster analysis of multivariate data: efficiency versus interpretability of classifications
| |
| |journal=Biometrics
| |
| |volume=21
| |
| |pages=768–769
| |
| |year=1965}}</ref> A more efficient version was proposed and published in Fortran by Hartigan and Wong in 1975/1979.<ref name="hartigan1975">{{Cite book
| |
| |title=Clustering algorithms
| |
| |author=J.A. Hartigan
| |
| |year=1975
| |
| |publisher=John Wiley & Sons, Inc.
| |
| }}</ref><ref name="hartigan1979" />
| |
| | |
| == Algorithms ==
| |
| | |
| === Standard algorithm ===
| |
| The most common algorithm uses an iterative refinement technique. Due to its ubiquity it is often called the '''''k''-means algorithm'''; it is also referred to as '''[[Lloyd's algorithm]]''', particularly in the computer science community.
| |
| | |
| Given an initial set of ''k'' means ''m''<sub>1</sub><sup>(1)</sup>,…,''m''<sub>''k''</sub><sup>(1)</sup> (see below), the algorithm proceeds by alternating between two steps:<ref>{{Cite book
| |
| | last =MacKay
| |
| | first =David
| |
| | authorlink =David MacKay (scientist)
| |
| | title =Information Theory, Inference and Learning Algorithms
| |
| | publisher = Cambridge University Press
| |
| | year =2003
| |
| | url =http://www.inference.phy.cam.ac.uk/mackay/itila/book.html
| |
| | isbn = 0-521-64298-1
| |
| | chapter=Chapter 20. An Example Inference Task: Clustering
| |
| | chapterurl=http://www.inference.phy.cam.ac.uk/mackay/itprnn/ps/284.292.pdf
| |
| | pages=284–292
| |
| | ref=mackay2003
| |
| | mr=2012999
| |
| }}</ref>
| |
| :'''Assignment step''': Assign each observation to the cluster whose mean yields the least within-cluster sum of squares (WCSS). Since the sum of squares is the squared [[Euclidean distance]], this is intuitively the "nearest" mean.<ref>Since the square root is a monotone function, this also is the minimum Euclidean distance assignment.</ref> (Mathematically, this means partitioning the observations according to the [[Voronoi diagram]] generated by the means).
| |
| ::<math>S_i^{(t)} = \big \{ x_p : \big \| x_p - m^{(t)}_i \big \|^2 \le \big \| x_p - m^{(t)}_j \big \|^2 \ \forall j, 1 \le j \le k \big\},</math>
| |
| :: where each <math>x_p</math> is assigned to exactly one <math>S^{(t)}</math>, even if it could be is assigned to two or more of them.
| |
| :'''Update step''': Calculate the new means to be the [[centroids]] of the observations in the new clusters.
| |
| ::<math>m^{(t+1)}_i = \frac{1}{|S^{(t)}_i|} \sum_{x_j \in S^{(t)}_i} x_j </math>
| |
| :: Since the arithmetic mean is a [[least-squares estimation|least-squares estimator]], this also minimizes the within-cluster sum of squares (WCSS) objective.
| |
| The algorithm has converged when the assignments no longer change. Since both steps optimize the WCSS objective, and there only exists a finite number of such partitionings, the algorithm must converge to a (local) optimum. There is no guarantee that the global optimum is found using this algorithm.
| |
| | |
| The algorithm is often presented as assigning objects to the nearest cluster by distance. This is slightly inaccurate: the algorithm aims at minimizing the WCSS objective, and thus assigns by "least sum of squares". Using a different distance function other than (squared) Euclidean distance may stop the algorithm from converging. It is correct that the smallest Euclidean distance yields the smallest squared Euclidean distance and thus also yields the smallest sum of squares. Various modifications of k-means such as spherical k-means and [[k-medoids]] have been proposed to allow using other distance measures.
| |
| | |
| ====Initialization methods====
| |
| Commonly used initialization methods are Forgy and Random Partition.<ref name="hamerly">{{Cite conference
| |
| |author = Hamerly, G. and Elkan, C.
| |
| |year=2002
| |
| |title=Alternatives to the k-means algorithm that find better clusterings
| |
| |booktitle=Proceedings of the eleventh international conference on Information and knowledge management (CIKM)
| |
| |url=http://charlotte.ucsd.edu/users/elkan/cikm02.pdf
| |
| }}</ref>
| |
| The Forgy method randomly chooses ''k'' observations from the data set and uses these as the initial means. The Random Partition method first randomly assigns a cluster to each observation and then proceeds to the update step, thus computing the initial mean to be the centroid of the cluster's randomly assigned points. The Forgy method tends to spread the initial means out, while Random Partition places all of them close to the center of the data set. According to Hamerly et al.,<ref name="hamerly"/> the Random Partition method is generally preferable for algorithms such as the ''k''-harmonic means and fuzzy ''k''-means. For expectation maximization and standard ''k''-means algorithms, the Forgy method of initialization is preferable.
| |
| | |
| <gallery caption="Demonstration of the standard algorithm" widths="150px">
| |
| Image:K Means Example Step 1.svg|1) ''k'' initial "means" (in this case ''k''=3) are randomly generated within the data domain (shown in color).
| |
| Image:K Means Example Step 2.svg|2) ''k'' clusters are created by associating every observation with the nearest mean. The partitions here represent the [[Voronoi diagram]] generated by the means.
| |
| Image:K Means Example Step 3.svg|3) The [[centroid]] of each of the ''k'' clusters becomes the new mean.
| |
| Image:K Means Example Step 4.svg|4) Steps 2 and 3 are repeated until convergence has been reached.
| |
| </gallery>
| |
| As it is a heuristic algorithm, there is no guarantee that it will converge to the global optimum, and the result may depend on the initial clusters. As the algorithm is usually very fast, it is common to run it multiple times with different starting conditions. However, in the worst case, ''k''-means can be very slow to converge: in particular it has been shown that there exist certain point sets, even in 2 dimensions, on which ''k''-means takes exponential time, that is {{math|2<sup>Ω(<var>n</var>)</sup>}}, to converge.<ref>{{cite journal
| |
| |first=A. |last=Vattani.
| |
| |url=http://cseweb.ucsd.edu/users/avattani/papers/kmeans-journal.pdf
| |
| |title=k-means requires exponentially many iterations even in the plane
| |
| |journal=[[Discrete and Computational Geometry]]
| |
| |volume=45 |issue=4 |pages=596–616
| |
| |year=2011
| |
| |doi=10.1007/s00454-011-9340-1
| |
| }}</ref> These point sets do not seem to arise in practice: this is corroborated by the fact that the [[Smoothed analysis|smoothed]] running time of ''k''-means is polynomial.<ref name="Arthur, D.; Manthey, B.; Roeglin, H. 2009">{{cite conference | author=Arthur, D.; Manthey, B.; Roeglin, H. | year=2009 | title=k-means has polynomial smoothed complexity | booktitle=Proceedings of the 50th Symposium on Foundations of Computer Science (FOCS)}}</ref>
| |
| | |
| The "assignment" step is also referred to as '''expectation step''', the "update step" as '''maximization step''', making this algorithm a variant of the ''generalized'' [[expectation-maximization algorithm]].
| |
| | |
| === Complexity ===
| |
| Regarding computational complexity, finding the optimal solution to the ''k''-means clustering problem for observations in ''d'' dimensions is:
| |
| *[[NP-hard]] in general Euclidean space ''d'' even for 2 clusters <ref>{{cite journal
| |
| |author=Aloise, D.; Deshpande, A.; Hansen, P.; Popat, P.
| |
| |title=NP-hardness of Euclidean sum-of-squares clustering
| |
| |journal=[[Machine Learning (journal)|Machine Learning]]
| |
| |year=2009
| |
| |volume=75 |pages=245–249
| |
| |doi=10.1007/s10994-009-5103-0}}</ref><ref>
| |
| {{cite journal
| |
| |title=Random Projection Trees for Vector Quantization
| |
| |author=Dasgupta, S. and Freund, Y.
| |
| |journal=Information Theory, IEEE Transactions on
| |
| |volume=55
| |
| |pages=3229–3242
| |
| |date=July 2009
| |
| |doi=10.1109/TIT.2009.2021326
| |
| |arxiv=0805.1390}}
| |
| </ref>
| |
| *[[NP-hard]] for a general number of clusters ''k'' even in the plane <ref>{{cite journal
| |
| |author=Mahajan, M.; Nimbhorkar, P.; Varadarajan, K.
| |
| |title=The Planar k-Means Problem is NP-Hard
| |
| |journal=[[Lecture Notes in Computer Science]]
| |
| |year=2009
| |
| |volume=5431 |pages=274–285
| |
| |doi=10.1007/978-3-642-00202-1_24}}
| |
| </ref>
| |
| *If ''k'' and ''d'' (the dimension) are fixed, the problem can be exactly solved in time '''''O(n<sup>dk+1</sup> log n)''''', where ''n'' is the number of entities to be clustered <ref>
| |
| {{cite conference
| |
| |author=Inaba, M.; Katoh, N.; Imai, H.
| |
| |year=1994
| |
| |title=Applications of weighted Voronoi diagrams and randomization to variance-based ''k''-clustering
| |
| |conference=[[Symposium on Computational Geometry|Proceedings of 10th ACM Symposium on Computational Geometry]]
| |
| |pages=332–339
| |
| |doi= 10.1145/177424.178042}}
| |
| </ref>
| |
| | |
| Thus, a variety of [[heuristic algorithm]]s such as Lloyds algorithm given above are generally used.
| |
| | |
| * Lloyd's <math>k</math>-means algorithm has polynomial smoothed running time. It is shown that <ref name="Arthur, D.; Manthey, B.; Roeglin, H. 2009"/> for arbitrary set of <math>n</math> points in <math>[0,1]^d</math>, if each point is independently perturbed by a normal distribution with mean <math>0</math> and variance <math>\sigma^2</math>, then the expected running time of <math>k</math>-means algorithm is bounded by <math>O( n^{34}k^{34}d^8 log^4(n)/ \sigma^6 )</math>, which is a polynomial in <math>n</math>, <math>k</math>, <math>d</math> and <math>1/\sigma</math>.
| |
| | |
| * Better bounds are proved for simple cases. For example,<ref>{{Cite thesis| author=Arthur; Abhishek Bhowmick | year=2009 | title= A theoretical analysis of Lloyd's algorithm for k-means clustering }}[http://www.cse.iitk.ac.in/users/bhowmick/lloyd.pdf]{{dead link|date=January 2013}}</ref> showed that the running time of <math>k</math>-means algorithm is bounded by <math>O(dn^4M^2)</math> for <math>n</math> points in an integer lattice <math>\{1,\dots, M\}^d</math>.
| |
| | |
| === Variations ===
| |
| * [[k-medians clustering]] uses the median in each dimension instead of the mean, and this way minimizes <math>L_1</math> norm ([[Taxicab geometry]]).
| |
| * [[k-medoids]] (also: Partitioning Around Medoids, PAM) uses the medoid instead of the mean, and this way minimizes the sum of distances for ''arbitrary'' distance functions.
| |
| * [[Fuzzy clustering#Fuzzy c-means clustering|Fuzzy C-Means Clustering]] is a soft version of K-means, where each data point has a fuzzy degree of belonging to each cluster.
| |
| * [[Mixture model#Gaussian mixture model|Gaussian mixture]] models trained with [[expectation-maximization algorithm]] (EM algorithm) maintains probabilistic assignments to clusters, instead of deterministic assignments, and multivariate Gaussian distributions instead of means.
| |
| * Several methods have been proposed to choose better starting clusters. One recent proposal is [[k-means++]].
| |
| * The filtering algorithm uses [[kd-tree]]s to speed up each k-means step.<ref>{{cite journal
| |
| |author=Kanungo, T.; [[David Mount|Mount, D. M.]]; [[Nathan Netanyahu|Netanyahu, N. S.]]; Piatko, C. D.; Silverman, R.; Wu, A. Y.
| |
| |doi=10.1109/TPAMI.2002.1017616
| |
| |url=http://www.cs.umd.edu/~mount/Papers/pami02.pdf |accessdate=2009-04-24
| |
| |title=An efficient k-means clustering algorithm: Analysis and implementation
| |
| |journal=IEEE Trans. Pattern Analysis and Machine Intelligence
| |
| |volume=24 |year=2002 |pages=881–892 }}
| |
| </ref>
| |
| * Some methods attempt to speed up each k-means step using [[coreset]]s<ref>{{Cite conference
| |
| |author=Frahling, G.; Sohler, C.
| |
| |year=2006
| |
| |title=A fast k-means implementation using coresets
| |
| |booktitle=[[Symposium on Computational Geometry|Proceedings of the twenty-second annual symposium on Computational geometry (SoCG)]]
| |
| |url=http://www.frahling.de/Gereon_Frahling/Publications_files/A%20fast%20k-means%20implementation%20using%20Coresets%20(Frahling,%20Sohler).pdf
| |
| }}
| |
| </ref> or the [[triangle inequality]].<ref>{{Cite conference
| |
| |author = Elkan, C.
| |
| |year=2003
| |
| |title=Using the triangle inequality to accelerate k-means
| |
| |booktitle=Proceedings of the Twentieth International Conference on Machine Learning (ICML)
| |
| |url=http://www-cse.ucsd.edu/~elkan/kmeansicml03.pdf
| |
| }}</ref>
| |
| * Escape local optima by swapping points between clusters.<ref name="hartigan1979">{{Cite journal
| |
| | first1 = J. A. |last1=Hartigan
| |
| | first2 = M. A. |last2=Wong
| |
| | year = 1979
| |
| | title = Algorithm AS 136: A K-Means Clustering Algorithm
| |
| | journal = [[Journal of the Royal Statistical Society, Series C]]
| |
| | volume = 28
| |
| | issue = 1
| |
| | pages = 100–108
| |
| | jstor = 2346830
| |
| }}</ref>
| |
| | |
| * The [[Spherical k-means]] clustering algorithm is suitable for directional data.<ref>{{Cite journal
| |
| | first1 = I. S. |last1=Dhillon
| |
| | first2 = D. M. |last2=Modha
| |
| | year = 2001
| |
| | title = Concept decompositions for large sparse text data using clustering
| |
| | journal = Machine Learning
| |
| | volume = 42
| |
| | issue = 1
| |
| | pages = 143–175
| |
| }}</ref>
| |
| | |
| * The [[Minkowski metric weighted k-means]] deals with irrelevant features by assigning cluster specific weights to each feature<ref>{{Cite journal
| |
| | first1 = R. C. |last1=Amorim
| |
| | first2 = B |last2=Mirkin
| |
| | year = 2012
| |
| | title = Minkowski metric, feature weighting and anomalous cluster initializing in K-Means clustering
| |
| | journal = Pattern Recognition
| |
| | volume = 45
| |
| | issue = 3
| |
| | pages = 1061–1075
| |
| | doi=10.1016/j.patcog.2011.08.012
| |
| }}</ref>
| |
| | |
| == Discussion ==
| |
| [[File:K-means convergence to a local minimum.png|thumb|650px|A typical example of the k-means convergence to a local minimum. In this example, the result of k-means clustering (the right figure) contradicts the obvious cluster structure of the data set. The small circles are the data points, the four ray stars are the centroids (means). The initial configuration is on the left figure. The algorithm converges after five iterations presented on the figures, from the left to the right. The illustration was prepared with the Mirkes Java applet.<ref name = "Mirkes2011"/>]]
| |
| [[File:Iris Flowers Clustering kMeans.svg|thumb|450px|''k''-means clustering result for the [[Iris flower data set]] and actual species visualized using [[Environment for DeveLoping KDD-Applications Supported by Index-Structures|ELKI]]. Cluster means are marked using larger, semi-transparent symbols.]]
| |
| [[File:ClusterAnalysis Mouse.svg|thumb|450px|''k''-means clustering and EM clustering on an artificial dataset ("mouse"). The tendency of ''k''-means to produce equi-sized clusters leads to bad results, while EM benefits from the Gaussian distribution present in the data set]]
| |
| | |
| The two key features of ''k''-means which make it efficient are often regarded as its biggest drawbacks:
| |
| * [[Euclidean distance]] is used as a [[metric (mathematics)|metric]] and [[variance]] is used as a measure of cluster scatter.
| |
| * The number of clusters ''k'' is an input parameter: an inappropriate choice of ''k'' may yield poor results. That is why, when performing k-means, it is important to run diagnostic checks for [[determining the number of clusters in a data set|determining the number of clusters in the data set]].
| |
| * Convergence to a local minimum may produce counterintuitive ("wrong") results (see example in Fig.).
| |
| | |
| A key limitation of ''k''-means is its cluster model. The concept is based on spherical clusters that are separable in a way so that the mean value converges towards the cluster center. The clusters are expected to be of similar size, so that the assignment to the nearest cluster center is the correct assignment. When for example applying ''k''-means with a value of <math>k=3</math> onto the well-known [[Iris flower data set]], the result often fails to separate the three [[Iris (plant)|Iris]] species contained in the data set. With <math>k=2</math>, the two visible clusters (one containing two species) will be discovered, whereas with <math>k=3</math> one of the two clusters will be split into two even parts. In fact, <math>k=2</math> is more appropriate for this data set, despite the data set containing 3 ''classes''. As with any other clustering algorithm, the ''k''-means result relies on the data set to satisfy the assumptions made by the clustering algorithms. It works well on some data sets, while failing on others.
| |
| | |
| The result of ''k''-means can also be seen as the [[Voronoi diagram|Voronoi cells]] of the cluster means. Since data is split halfway between cluster means, this can lead to suboptimal splits as can be seen in the "mouse" example. The Gaussian models used by the [[Expectation-maximization algorithm]] (which can be seen as a generalization of ''k''-means) are more flexible here by having both variances and covariances. The EM result is thus able to accommodate clusters of variable size much better than ''k''-means as well as correlated clusters (not in this example).
| |
| | |
| == Applications ==
| |
| | |
| ''k''-means clustering in particular when using heuristics such as Lloyd's algorithm is rather easy to implement and apply even on large data sets. As such, it has been successfully used in various topics, ranging from [[market segmentation]], [[computer vision]], [[geostatistics]],<ref>Honarkhah, M and Caers, J, 2010, ''[http://dx.doi.org/10.1007/s11004-010-9276-7 Stochastic Simulation of Patterns Using Distance-Based Pattern Modeling]'', Mathematical Geosciences, 42: 487 - 517</ref> and [[astronomy]] to [[Data Mining in Agriculture|agriculture]]. It often is used as a preprocessing step for other algorithms, for example to find a starting configuration.
| |
| | |
| ===Vector quantization===
| |
| {{Main|Vector quantization}}
| |
| | |
| [[File:Rosa Gold Glow 2 small noblue.png|frame|right|Two-channel (for illustration purposes -- red and green only) color image.]]
| |
| [[File:Rosa Gold Glow 2 small noblue color space.png|thumb|right|250px|Vector quantization of colors present in the image above into Voronoi cells using ''k''-means.]]
| |
| | |
| ''k''-means originates from signal processing, and still finds use in this domain. For example in computer graphics, [[color quantization]] is the task of reducing the color palette of an image to a fixed number of colors ''k''. The ''k''-means algorithm can easily be used for this task and produces competitive results. Other uses of vector quantization include [[Sampling (statistics)|non-random sampling]], as ''k''-means can easily be used to choose ''k'' different but prototypical objects from a large data set for further analysis.
| |
| | |
| ===Cluster analysis===
| |
| {{Main|Cluster analysis}}
| |
| | |
| In cluster analysis, the ''k''-means algorithm can be used to partition the input data set into ''k'' partitions (clusters).
| |
| | |
| However, the pure ''k''-means algorithm is not very flexible, and as such of limited use (except for when vector quantization as above is actually the desired use case!). In particular, the parameter ''k'' is known to be hard to choose (as discussed below) when not given by external constraints. In contrast to other algorithms, ''k''-means can also not be used with arbitrary distance functions or be use on non-numerical data. For these use cases, many other algorithms have been developed since.
| |
| | |
| === Feature learning ===
| |
| ''k''-means clustering has been used as a [[feature learning]] (or [[dictionary learning]]) step, which can be used in the for ([[semi-supervised learning|semi-]])[[supervised learning]] or [[unsupervised learning]].<ref name="Coates2012">{{cite encyclopedia
| |
| |last1 = Coates
| |
| |first1 = Adam
| |
| |last2 = Ng
| |
| |first2 = Andrew Y.
| |
| |title = Learning feature representations with k-means
| |
| |editors = G. Montavon, G. B. Orr, K.-R. Müller
| |
| |encyclopedia = Neural Networks: Tricks of the Trade
| |
| |publisher = Springer
| |
| |year = 2012
| |
| |url = http://www.stanford.edu/~acoates/papers/coatesng_nntot2012.pdf
| |
| }}</ref>
| |
| The basic approach is first to train a ''k''-means clustering representation, using the input training data (which need not be labelled). Then, to project any input datum into the new feature space, we have a choice of "encoding" functions, but we can use for example the thresholded matrix-product of the datum with the centroid locations, the distance from the datum to each centroid, or simply an indicator function for the nearest centroid,<ref name="Coates2012"/><ref>{{cite conference
| |
| |last1 = Csurka
| |
| |first1 = Gabriella
| |
| |last2 = Dance
| |
| |first2 = Christopher C.
| |
| |last3 = Fan
| |
| |first3 = Lixin
| |
| |last4 = Willamowski
| |
| |first4 = Jutta
| |
| |last5 = Bray
| |
| |first5 = Cédric
| |
| |title = Visual categorization with bags of keypoints
| |
| |conference = ECCV Workshop on Statistical Learning in Computer Vision
| |
| |year = 2004
| |
| |url = http://www.cs.cmu.edu/~efros/courses/LBMV07/Papers/csurka-eccv-04.pdf
| |
| }}</ref> or some smooth transformation of the distance.<ref name="coates2011"/> Alternatively, by transforming the sample-cluster distance through a [[Radial basis function|Gaussian RBF]], one effectively obtains the hidden layer of a [[radial basis function network]].<ref name="schwenker">{{cite journal
| |
| |last1 = Schwenker
| |
| |first1 = Friedhelm
| |
| |last2 = Kestler
| |
| |first2 = Hans A.
| |
| |last3 = Palm
| |
| |first3 = Günther
| |
| |title = Three learning phases for radial-basis-function networks
| |
| |journal = Neural Networks
| |
| |volume = 14
| |
| |pages = 439–458
| |
| |year = 2001
| |
| |url = http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.109.312&rep=rep1&type=pdf
| |
| }}</ref>
| |
| | |
| This use of ''k''-means has been successfully combined with simple, [[linear classifier]]s for semi-supervised learning in [[natural language processing|NLP]] (specifically for [[named entity recognition]])<ref>{{cite conference
| |
| |last1 = Lin
| |
| |first1 = Dekang
| |
| |last2 = Wu
| |
| |first2 = Xiaoyun
| |
| |title = Phrase clustering for discriminative learning
| |
| |conference = Annual Meeting of the [[Association for Computational Linguistics|ACL]] and IJCNLP
| |
| |year = 2009
| |
| |pages = 1030–1038
| |
| |url = http://www.aclweb.org/anthology/P/P09/P09-1116.pdf
| |
| }}</ref>
| |
| and in [[computer vision]]. On an object recognition task, it was found to exhibit comparable performance with more sophisticated feature learning approaches such as [[autoencoder]]s and [[restricted Boltzmann machine]]s.<ref name="coates2011">{{cite conference
| |
| |last1 = Coates
| |
| |first1 = Adam
| |
| |last2 = Lee
| |
| |first2 = Honglak
| |
| |last3 = Ng
| |
| |first3 = Andrew Y.
| |
| |title = An analysis of single-layer networks in unsupervised feature learning
| |
| |conference = International Conference on Artificial Intelligence and Statistics (AISTATS)
| |
| |year = 2011
| |
| |url = http://www.stanford.edu/~acoates/papers/coatesleeng_aistats_2011.pdf
| |
| }}</ref>
| |
| However, it generally requires more data than the sophisticated methods, for equivalent performance, because each data point only contributes to one "feature" rather than multiple.<ref name="Coates2012"/>
| |
| | |
| == Relation to other statistical machine learning algorithms ==
| |
| | |
| ''k''-means clustering, and its associated [[Expectation–maximization algorithm|expectation-maximization algorithm]], is a special case of a [[Mixture model|Gaussian mixture model]], specifically, the limit of taking all covariances as diagonal, equal, and small. It is often easy to generalize a ''k''-means problem into a Gaussian mixture model.<ref>{{Cite book | last1=Press | first1=WH | last2=Teukolsky | first2=SA | last3=Vetterling | first3=WT | last4=Flannery | first4=BP | year=2007 | title=Numerical Recipes: The Art of Scientific Computing | edition=3rd | publisher=Cambridge University Press | publication-place=New York | isbn=978-0-521-88068-8 | chapter=Section 16.1. Gaussian Mixture Models and k-Means Clustering | chapter-url=http://apps.nrbook.com/empanel/index.html#pg=842}}</ref> Another generalization of the k-means algorithm is the [[K-SVD]] algorithm, which estimates data points as a sparse linear combination of "codebook vectors". K-means corresponds to the special case of using a single codebook vector, with a weight of 1.<ref name="
| |
| K-SVD">[http://intranet.daiict.ac.in/~ajit_r/IT530/KSVD_IEEETSP.pdf].</ref>
| |
| | |
| === Mean shift clustering ===
| |
| Basic [[mean shift]] clustering algorithms maintain a set of data points the same size as the input data set. Initially, this set is copied from the input set. Then this set is iteratively replaced by the mean of those points in the set that are within a given distance of that point. By contrast, ''k''-means restricts this updated set to ''k'' points usually much less than the number of points in the input data set, and replaces each point in this set by the mean of all points in the ''input set'' that are closer to that point than any other (e.g. within the Voronoi partition of each updating point). A mean shift algorithm that is similar then to ''k''-means, called ''likelihood mean shift'', replaces the set of points undergoing replacement by the mean of all points in the input set that are within a given distance of the changing set.<ref name="Little2011">{{cite journal|last=Little|first= M.A.|coauthors=Jones, N.S.|title=Generalized Methods and Solvers for Piecewise Constant Signals: Part I| journal=[[Proceedings of the Royal Society A]]|url=http://www.maxlittle.net/publications/pwc_filtering_arxiv.pdf|year = 2011 }}</ref> One of the advantages of mean shift over ''k''-means is that there is no need to choose the number of clusters, because mean shift is likely to find only a few clusters if indeed only a small number exist. However, mean shift can be much slower than ''k''-means, and still requires selection of a bandwidth parameter. Mean shift has soft variants much as ''k''-means does.
| |
| | |
| === Principal component analysis (PCA) ===
| |
| It was asserted in <ref>{{cite journal|authors=H. Zha, C. Ding, M. Gu, X. He and H.D. Simon|title=Spectral Relaxation for K-means Clustering|journal=Neural Information Processing Systems vol.14 (NIPS 2001)|pages=1057–1064|location=Vancouver, Canada|date=Dec 2001|url=http://ranger.uta.edu/~chqding/papers/Zha-Kmeans.pdf}}</ref><ref>{{cite journal|authors=Chris Ding and Xiaofeng He|title=K-means Clustering via Principal Component Analysis|work=Proc. of Int'l Conf. Machine Learning (ICML 2004)|pages=225–232|date=July 2004|url=http://ranger.uta.edu/~chqding/papers/KmeansPCA1.pdf}}</ref> that the relaxed solution of {{math|<var>k</var>}}-means clustering, specified by the cluster indicators, is given by the PCA ([[principal component analysis]]) principal components, and the PCA subspace spanned by the principal directions is identical to the cluster centroid subspace. However, that PCA is a useful relaxation of k-means clustering was not a new result (see, for example,<ref>{{cite journal | title = Clustering large graphs via the singular value decomposition | journal = Machine learning | year = 2004 | first = P. | last = Drineas | coauthors = A. Frieze, R. Kannan, S. Vempala, V. Vinay | volume = 56 | pages = 9–33| id = | url = http://www.cc.gatech.edu/~vempala/papers/dfkvv.pdf | accessdate = 2012-08-02}}</ref>), and it is straightforward to uncover counterexamples to the statement that the cluster centroid subspace is spanned by the principal directions{{Citation needed|date=January 2014}}.
| |
| | |
| === Bilateral filtering ===
| |
| ''k''-means implicitly assumes that the ordering of the input data set does not matter. The [[bilateral filter]] is similar to K-means and [[mean shift]] in that it maintains a set of data points that are iteratively replaced by means. However, the bilateral filter restricts the calculation of the (kernel weighted) mean to include only points that are close in the ordering of the input data.<ref name="Little2011"/> This makes it applicable to problems such as image denoising, where the spatial arrangement of pixels in an image is of critical importance.
| |
| | |
| ==Similar problems==
| |
| The set of squared error minimizing cluster functions also includes the [[k-medoids|{{math|<var>k</var>}}-medoids]] algorithm, an approach which forces the center point of each cluster to be one of the actual points, i.e., it uses [[medoids]] in place of [[centroids]].
| |
| | |
| == Software ==
| |
| | |
| === Free ===
| |
| * [[Apache Mahout]] [http://cwiki.apache.org/MAHOUT/k-means-clustering.html k-Means]
| |
| * [[CrimeStat]] implements two spatial K-means algorithms, one of which allows the user to define the starting locations.
| |
| * [[ELKI]] contains k-means (with Lloyd and MacQueen iteration, along with different initializations such as k-means++ initialization) and various more advanced clustering algorithms
| |
| * [[MLPACK (C++ library)|MLPACK]] contains a C++ implementation of k-means
| |
| * [[R (programming language)|R]] [http://stat.ethz.ch/R-manual/R-patched/library/stats/html/kmeans.html kmeans] implements a variety of algorithms<ref name="macqueen1967"/><ref name="lloyd1957"/><ref name="hartigan1979"/>
| |
| * [[SciPy]] [http://docs.scipy.org/doc/scipy/reference/cluster.vq.html vector-quantization]
| |
| * [[Scikit-learn]] implements a popular python machine-learning library which contains various clustering algorithms
| |
| * [http://www.codeding.com/?article=14 Silverlight widget demonstrating k-means algorithm]
| |
| * [http://pgxn.org/dist/kmeans/ PostgreSQL extension for k-means]
| |
| * [http://graphlab.org/toolkits/clustering/ CMU's GraphLab Clustering library] Efficient multicore implementation for large scale data.
| |
| * [[Weka (machine learning)|Weka]] contains k-means and a few variants of it, including k-means++ and x-means.
| |
| * [http://spectralpython.sourceforge.net/algorithms.html#k-means-clustering Spectral Python] contains methods for [[unsupervised classification]] including a K-means clustering method.
| |
| * [http://scikit-learn.org/dev/modules/generated/sklearn.cluster.KMeans.html scikit learn] machine learning in Python contains a K-Means implementation
| |
| * [[OpenCV]] contains a [http://docs.opencv.org/modules/core/doc/clustering.html?highlight=kmeans#cv2.kmeans K-means] implementation under [[Bsd licence|BSD licence]].
| |
| * [http://gforge.inria.fr/projects/yael/ Yael] includes an efficient multi-threaded C implementation of k-means, with C, Python and Matlab interfaces.
| |
| | |
| === Commercial ===
| |
| * IDL Cluster, Clust_Wts
| |
| * [http://reference.wolfram.com/mathematica/ref/ClusteringComponents.html ''Mathematica'' ClusteringComponents function]
| |
| * [[MATLAB]] [http://www.mathworks.com/access/helpdesk/help/toolbox/stats/kmeans.html kmeans]
| |
| * [[SAS System|SAS]] [http://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/fastclus_toc.htm FASTCLUS]
| |
| * [[Stata]] [http://www.stata.com/help13.cgi?cluster+kmeans kmeans]
| |
| * [http://www.visumap.com/index.aspx?p=Products VisuMap kMeans Clustering]
| |
| | |
| ===Source code===
| |
| * [[ELKI]] and [[Weka]] are written in Java and include k-means and variations
| |
| * K-means application in PHP,<ref>http://www25.brinkster.com/denshade/kmeans.php.htm</ref> using VB,<ref>[http://people.revoledu.com/kardi/tutorial/kMean/download.htm K-Means Clustering Tutorial: Download<!-- Bot generated title -->]</ref> using Perl,<ref>[http://www.lwebzem.com/cgi-bin/k_means/test3.cgi Perl script for Kmeans clustering<!-- Bot generated title -->]</ref> using C++,<ref>[http://codingplayground.blogspot.com/2009/03/k-means-in-c.html Antonio Gulli's coding playground: K-means in C<!-- Bot generated title -->]</ref> using Matlab,<ref>[http://people.revoledu.com/kardi/tutorial/kMean/matlab_kMeans.htm K-Means Clustering Tutorial: Matlab Code<!-- Bot generated title -->]</ref> using Ruby,<ref>[http://ai4r.org/index.html AI4R :: Artificial Intelligence for Ruby<!-- Bot generated title -->]</ref><ref>[http://github.com/reddavis/K-Means/tree/master reddavis/K-Means · GitHub<!-- Bot generated title -->]</ref> using Python with scipy,<ref>[http://docs.scipy.org/doc/scipy/reference/cluster.vq.html K-means clustering and vector quantization (scipy.cluster.vq) — SciPy v0.11 Reference Guide (DRAFT)<!-- Bot generated title -->]</ref> using X10<ref>http://dist.codehaus.org/x10/applications/samples/KMeansDist.x10</ref>
| |
| * A parallel out-of-core implementation in C<ref>http://www.cs.princeton.edu/~wdong/kmeans/</ref>
| |
| * An open-source collection of clustering algorithms, including k-means, implemented in Javascript.<ref>http://code.google.com/p/figue/ FIGUE</ref> Online demo.<ref>http://jydelort.appspot.com/resources/figue/demo.html</ref>
| |
| | |
| ===Visualization, animation and examples===
| |
| * [[ELKI]] can visualize k-means using [[Voronoi diagram|Voronoi cells]] and [[Delaunay triangulation]] for 2D data. In higher dimensionality, only cluster assignments and cluster centers are visualized
| |
| * Demos of the K-means-algorithm<ref>[http://home.dei.polimi.it/matteucc/Clustering/tutorial_html/AppletKM.html Clustering - K-means demo<!-- Bot generated title -->]</ref><ref>[http://siebn.de/other/yakmeans/ siebn.de - YAK-Means<!-- Bot generated title -->]</ref><ref>[http://informationandvisualization.de/blog/kmeans-and-voronoi-tesselation-built-processing k-Means and Voronoi Tesselation: Built with Processing | Information & Visualization<!-- Bot generated title -->]</ref><ref>[http://www.javaworld.com/javaworld/jw-11-2006/jw-1121-thread.html Hyper-threaded Java - JavaWorld<!-- Bot generated title -->]</ref><ref>[http://www.leet.it/home/lale/clustering/ Color clustering<!-- Bot generated title -->]</ref><ref>[http://www.onmyphd.com/?p=k-means.clustering Interactive step-by-step examples in Javascript of good and bad k-means clustering]</ref>
| |
| * K-means and K-medoids (Applet), [[University of Leicester]]<ref name = "Mirkes2011">E.M. Mirkes, [http://www.math.le.ac.uk/people/ag153/homepage/KmeansKmedoids/Kmeans_Kmedoids.html K-means and K-medoids applet]. University of Leicester, 2011.</ref>
| |
| * Clustergram - cluster diagnostic plot - for visual diagnostics of choosing the number of (k) clusters ([[R (programming language)|R]] code)<ref>[http://www.r-statistics.com/2010/06/clustergram-visualization-and-diagnostics-for-cluster-analysis-r-code/ Clustergram: visualization and diagnostics for cluster analysis (R code) | R-statistics blog<!-- Bot generated title -->]</ref>
| |
| | |
| ==See also==
| |
| * [[Canopy clustering algorithm]]
| |
| * [[Centroidal Voronoi tessellation]]
| |
| * [[k q-flats]]
| |
| * [[Linde–Buzo–Gray algorithm]]
| |
| * [[Nearest centroid classifier]]
| |
| * [[Self-organizing map]]
| |
| * [[silhouette (clustering)|Silhouette clustering]]
| |
| | |
| ==References==
| |
| {{Reflist|2}}
| |
| | |
| {{DEFAULTSORT:K-Means Clustering}}
| |
| [[Category:Data clustering algorithms]]
| |
| [[Category:Statistical algorithms]]
| |