|
|
| (One intermediate revision by one other user not shown) |
| Line 1: |
Line 1: |
| {{citation style|date=October 2012}}
| | Buy a rechargeable battery for all your wireless gaming controller. You can buy normal rechargeable power supplies for any controller. If you'd like to play video games regularly, you will be eating food through a small lots of money in the batteries used to be run your controllers. A rechargeable battery can help you save a lot of profit in the long run.<br><br>Beginning nearly enough gem stones to get another contractor. Don''t waste specific of the gems here in any way on [http://Www.Dailymail.Co.uk/home/search.html?sel=site&searchPhrase=rush-building rush-building] anything, as if it can save you these people you are going that can eventually obtain enough totally free of charge extra gems to get that extra builder not having cost. Particularly, customers can get free gallstones for clearing obstructions like rocks and trees, when you are done you clear them out and about they come back and also you may re-clear these guys to get more gems.<br><br>Many individuals which play clash of clans are searching for ways of getting costless gems. The gems are very important because they give the player functions and the power to boost their gaming experience. As opposed to several other equivalent games in cell phone websites, especially those where individuals use various chips in buy to bring these practical information relating to free, the nature involved with farmville and its format does not enable your varieties of hacks that anyone can put to the mission. Everyone is always looking for ways regarding how to get free gems at clash of clans risk most important thing carry out is to employ a great way to earn these consumers for free, save these kind of suitably and use ashamed where necessary.<br><br>If you have any issues about where by and how to use clash of clans hack ([http://prometeu.net similar resource site]), you can get in touch with us at the web site. Doing now, there exists minimum social options / capacities with this game i.e. there is not any chat, having difficulties to team track using friends, etc but greatest choice we could expect this to improve soon being Boom Beach continues to be in their Beta Mode.<br><br>While the game is a fabulous mobile edition, it is performing not lack substance for example many mobile games. So, defragging the process registry will boost the system overall performance returning to a fantastic extent. I usually get at all from 4000 to five thousand m - Points in a day ($4 to $5 for Amazon. The showed off the enormously anticipated i - Some of the 5 for the the first time in San Francisco on Wednesday morning (September 12, 2012). Is actually an a huge demand over some i - Mobile device 4 application not alone promoting business but potentially helps users to secure extra money.<br><br>All of this construction is what translates to that you can be more a part of a new clan, however it additionally houses reinforcement troops. Click a button into ask your clan to send you some troops, and they are choosing to be out currently there to make use of the in assaults, or which can defend your base of you while you're worries your weekly LARPing category. Upgrading this setting up permits extra troops to positively be stored for defensive. You may need 20 available slots as a way to get a dragon. This is a wonderful base for players looking to shield trophies also never worried about sources. Players will consider it hard to wipe out your city space. Most will mend for the easy overcome and take out your favorite assets.<br><br>There is the helpful component of how the diversion as fantastic. When one particular person has modified, the Clash of Clan Castle destroys in his or it village, he or she will successfully start or register for for each faction in diverse gamers exactly even they can take a short look at with every other while giving troops to just 1 these troops could be connected either offensively or protectively. The Clash with regards to Clans cheat for liberate additionally holds the greatest district centered globally chitchat so gamers could temps making use of several players for social courting and as faction entering.This recreation is a have to perform on your android software specially if you usually are employing my clash created by clans android hack investment. |
| The '''information bottleneck method''' is a technique introduced by [[Naftali Tishby]] et al. [1] for finding the best tradeoff between [[accuracy]] and complexity ([[Data compression|compression]]) when [[random variable|summarizing]] (e.g. [[data clustering|clustering]]) a [[random variable]] '''X''', given a [[joint probability distribution]] between '''X''' and an observed relevant variable '''Y'''. Other applications include distributional clustering, and [[dimension reduction]]. In a well defined sense it generalized the classical notion of minimal [[sufficient statistics]] from parametric statistics to arbitrary distributions, not necessarily of exponential form. It does so by relaxing the sufficiency condition to capture some fraction of the [[mutual information]] with the relevant variable '''Y'''.
| |
| | |
| The compressed variable is <math>T\,</math> and the algorithm minimises the following quantity
| |
| | |
| : <math> \min_{p(t|x)} \,\, I(X;T) - \beta I(T;Y)</math>
| |
| | |
| where <math>I(X;T)\,\,I(T;Y)</math> are the mutual information between <math>X;T \,</math> and <math>T;Y \,</math> respectively, and <math>\beta</math> is a [[Lagrange multiplier]].
| |
| | |
| == Gaussian information bottleneck ==
| |
| | |
| A relatively simple application of the information bottleneck is to Gaussian variates and this has some semblance to a least squares reduced rank or [[canonical correlation]] [2]. Assume <math>X, Y \,</math> are jointly multivariate zero mean normal vectors with covariances <math>\Sigma_{XX}, \,\, \Sigma_{YY}</math> and <math>T\,</math> is a compressed version of <math>X\,</math> which must maintain a given value of mutual information with <math>Y\,</math>. It can be shown that the optimum <math>T\,</math> is a normal vector consisting of linear combinations of the elements of <math>X , \,\, T=AX \,</math> where matrix <math>A \,</math> has orthogonal rows. | |
|
| |
| The projection matrix <math>A\,</math> in fact contains <math>M\,</math> rows selected from the weighted left eigenvectors of the singular value decomposition of the following matrix (generally asymmetric)
| |
| | |
| : <math>\Omega = \Sigma_{X|Y} \Sigma_{XX}^{-1} = I - \Sigma_{XY} \Sigma_{YY}^{-1} \Sigma_{XY}^T \Sigma_{XX}^{-1}.\,</math>
| |
| | |
| Define the singular value decomposition
| |
| | |
| : <math>\Omega = U\Lambda V^T\text{ with }\Lambda = \operatorname{Diag} \big ( \lambda_1 \le \lambda_2 \cdots \lambda_N \big ) \,</math>
| |
| | |
| and the critical values
| |
| | |
| : <math>\beta_i^C \underset {\lambda_i < 1}{=} (1 - \lambda_i )^{-1}. \, </math>
| |
| | |
| then the number <math>M \,</math> of active eigenvectors in the projection, or order of approximation, is given by
| |
| | |
| : <math>\beta_{M-1}^C < \beta \le \beta_M^C</math>
| |
| | |
| And we finally get
| |
| | |
| : <math>A=[ w_1 U_1 , \dots , w_M U_M ]^T</math>
| |
| | |
| In which the weights are given by
| |
| | |
| : <math>w_i = \sqrt{(\beta (1- \lambda_i )/\lambda_i r_i}</math>
| |
| | |
| where <math>r_i = U_i^T \Sigma_{XX} U_i.\,</math>
| |
| | |
| Applying the Gaussian information bottleneck on time series, one gets optimal predictive coding. This procedure is formally equivalent to linear Slow Feature Analysis [http://creutzig.berkeley.edu/neco.2008.pdf [3]]. Optimal temporal structures in linear dynamic systems can be revealed in the so-called past-future information bottleneck [http://www.user.tu-berlin.de/creutzig/Creutzig_PhysRevE.pdf [4]].
| |
| | |
| === Data clustering using the information bottleneck === | |
| | |
| This application of the bottleneck method to non-Gaussian sampled data is described in [4] by Tishby et. el. The concept, as treated there, is not without complication as there are two independent phases in the exercise: firstly estimation of the unknown parent probability densities from which the data samples are drawn and secondly the use of these densities within the information theoretic framework of the bottleneck.
| |
| | |
| === Density estimation ===
| |
| | |
| {{Main|Density estimation}}
| |
| | |
| Since the bottleneck method is framed in probabilistic rather than statistical terms, we first need to estimate the underlying probability density at the sample points <math>X = {x_i} \,</math>. This is a well known problem with a number of solutions described by Silverman in [5]. In the present method, joint probabilities of the samples are found by use of a Markov transition matrix method and this has some mathematical synergy with the bottleneck method itself.
| |
| | |
| Define an arbitrarily increasing distance metric <math>f \,</math> between all sample pairs and [[distance matrix]] <math>d_{i,j}=f \Big ( \Big| x_i - x_j \Big | \Big )</math> . Then compute transition probabilities between sample pairs <math>P_{i,j}=\exp (- \lambda d_{i,j} ) \,</math> for some <math>\lambda > 0 \,</math>. Treating samples as states, and a normalised version of <math>P \,</math> as a Markov state transition probability matrix, the vector of probabilities of the ‘states’ after <math>t \,</math> steps, conditioned on the initial state <math>p(0) \,</math>, is <math>p(t)=P^t p(0) \,</math>. We are here interested only in the equilibrium probability vector <math>p(\infty ) \,</math> given, in the usual way, by the dominant eigenvector of matrix <math>P \,</math> which is independent of the initialising vector <math>p(0) \,</math>. This Markov transition method establishes a probability at the sample points which is claimed to be proportional to the probabilities densities there.
| |
| | |
| Other interpretations of the use of the eigenvalues of distance matrix <math>d \,</math> are discussed in [6].
| |
| | |
| === Clusters ===
| |
| In the following soft clustering example, the reference vector <math>Y \,</math> contains sample categories and the joint probability <math>p(X,Y) \,</math> is assumed known. A soft cluster <math>c_k \,</math> is defined by its probability distribution over the data samples <math>x_i: \,\,\, p( c_k |x_i)</math>. In [1] Tishby et al. present the following iterative set of equations to determine the clusters which are ultimately a generalization of the [[Rate–distortion theory|Blahut-Arimoto]] algorithm, developed in rate distortion theory. The application of this type of algorithm in neural networks appears to originate in entropy arguments arising in application of Gibbs Distributions in deterministic annealing [7].
| |
| | |
| : <math>\begin{cases}
| |
| p(c|x)=Kp(c) \exp \Big( -\beta\,D^{KL} \Big[ p(y|x) \,|| \, p(y| c)\Big ] \Big)\\
| |
| p(y| c)=\textstyle \sum_x p(y|x)p( c | x) p(x) \big / p(c) \\
| |
| p(c) = \textstyle \sum_x p(c | x) p(x) \\
| |
| \end{cases}
| |
| </math> | |
| | |
| The function of each line of the iteration is expanded as follows.
| |
| | |
| '''Line 1:''' This is a matrix valued set of conditional probabilities
| |
| | |
| : <math>A_{i,j} = p(c_i | x_j )=Kp(c_i) \exp \Big( -\beta\,D^{KL} \Big[ p(y|x_j) \,|| \, p(y| c_i)\Big ] \Big)</math>
| |
| | |
| The [[Kullback–Leibler distance]] <math>D^{KL} \,</math> between the <math>Y \,</math> vectors generated by the sample data <math>x \,</math> and those generated by its reduced information proxy <math>c \,</math> is applied to assess the fidelity of the compressed vector with respect to the reference (or categorical) data <math>Y \,</math> in accordance with the fundamental bottleneck equation. <math>D^{KL}(a||b)\,</math> is the Kullback Leibler distance between distributions <math>a, b \,</math>
| |
| | |
| : <math>D^{KL} (a||b)= \sum_i p(a_i) \log \Big ( \frac{p(a_i)}{p(b_i)} \Big ) </math>
| |
| | |
| and <math>K \,</math> is a scalar normalization. The weighting by the negative exponent of the distance means that prior cluster probabilities are downweighted in line 1 when the Kullback Liebler distance is large, thus successful clusters grow in probability while unsuccessful ones decay.
| |
| | |
| '''Line 2: '''This is a second matrix-valued set of conditional probabilities. The steps in deriving it are as follows. We have, by definition
| |
| | |
| : <math>\begin{align}
| |
| p(y_i|c_k) & = \sum_j p(y_i|x_j)p(x_j|c_k) \\
| |
| & =\sum_j p(y_i|x_j)p(x_j, c_k ) \big / p(c_k) \\
| |
| & =\sum_j p(y_i|x_j)p(c_k | x_j) p(x_j) \big / p(c_k) \\
| |
| \end{align}</math>
| |
| where the Bayes identities <math>p(a,b)=p(a|b)p(b)=p(b|a)p(a) \,</math> are used.
| |
| | |
| '''Line 3:''' this line finds the marginal distribution of the clusters <math>c \,</math>
| |
| | |
| : <math>\begin{align}
| |
| p(c_i)& =\sum_j p(c_i , x_j)
| |
| & = \sum_j p(c_i | x_j) p(x_j)
| |
| \end{align}</math><br />
| |
| | |
| This is also a standard result.
| |
| | |
| Further inputs to the algorithm are the marginal sample distribution <math>p(x) \,</math> which has already been determined by the dominant eigenvector of <math>P \,</math> and the matrix valued Kullback Leibler distance function
| |
| | |
| : <math>D_{i,j}^{KL}=D^{KL} \Big[ p(y|x_j) \,|| \, p(y| c_i)\Big ] \Big)</math>
| |
| | |
| derived from the sample spacings and transition probabilities.
| |
| | |
| The matrix <math>p(y_i | c_j) \,</math> can be initialised randomly or as a reasonable guess, while matrix <math>p(c_i | x_j) \,</math> needs no prior values. Although the algorithm is converging, multiple minima may exist which need some action to resolve. Further details, including hard clustering methods, are found in [5].
| |
| | |
| ==Defining decision contours ==
| |
| | |
| To categorize a new sample <math> x' \,</math> external to the training set <math>X \,</math>, apply the previous distance metric to find the transition probabilities between <math> x' \,</math> and all samples in <math>X: \,\,</math>, <math> \tilde p(x_i )= p(x_i | x')= \Kappa \exp \Big (-\lambda f \big ( \Big| x_i - x' \Big | \big ) \Big )</math> with <math>\Kappa \,</math> a normalisation. Secondly apply the last two lines of the 3-line algorithm to get cluster, and conditional category probabilities.
| |
| | |
| : <math>\begin{align}
| |
| & \tilde p(c_i ) = p(c_i | x' ) = \sum_j p(c_i | x_j)p(x_j | x') =\sum_j p(c_i | x_j) \tilde p(x_j)\\
| |
| & p(y_i | c_j) = \sum_k p(y_i | x_k) p(c_j | x_k)p(x_k | x') / p(c_j | x' )
| |
| = \sum_k p(y_i | x_k) p(c_j | x_k) \tilde p(x_k) / \tilde p(c_j) \\
| |
| \end{align}</math>
| |
| | |
| Finally we have
| |
| | |
| : <math>p(y_i | x')= \sum_j p(y_i | c_j) p(c_j | x') )= \sum_j p(y_i | c_j) \tilde p(c_j) \,</math>
| |
| | |
| Parameter <math>\beta \,</math> must be kept under close supervision since, as it is increased from zero, increasing numbers of features, in the category probability space, snap into focus at certain critical thresholds.
| |
| | |
| === An example ===
| |
| The following case examines clustering in a four quadrant multiplier with random inputs <math>u, v \,</math> and two categories of output, <math>\pm 1 \,</math>, generated by
| |
| <math>y=\operatorname{sign}(uv) \,</math>. This function has the property that there are two spatially separated clusters for each category and so it demonstrates that the method can handle such distributions.
| |
| | |
| 20 samples are taken, uniformly distributed on the square <math>[-1,1]^2 \,</math> . The number of clusters used beyond the number of categories, two in this case, has little effect on performance and the results are shown for two clusters using parameters <math>\lambda = 3,\, \beta = 2.5</math>.
| |
| | |
| The distance function is <math>d_{i,j} = \Big| x_i - x_j \Big |^2</math> where <math>x_i = (u_i,v_i)^T \, </math> while the conditional distribution <math>p(y|x)\, </math> is a 2 × 20 matrix
| |
| | |
| : <math>\begin{align} & Pr(y_i=1) = 1\text{ if }\operatorname{sign}(u_iv_i)=1\, \\
| |
| & Pr(y_i=-1) = 1\text{ if }\operatorname{sign}(u_iv_i)=-1\,
| |
| \end{align}</math>
| |
| | |
| and zero elsewhere.
| |
| | |
| The summation in line 2 is only incorporates two values representing the training values of +1 or −1 but nevertheless seems to work quite well. Five iterations of the equations were used. The figure shows the locations of the twenty samples with '0' representing ''Y'' = 1 and 'x' representing ''Y'' = −1. The contour at the unity likelihood ratio level is shown,
| |
| | |
| : <math>L= \frac{\Pr(1)}{\Pr(-1)} = 1</math>
| |
| | |
| as a new sample <math>x' \,</math>is scanned over the square. Theoretically the contour should align with the <math>u=0 \,</math> and <math>v=0 \,</math> coordinates but for such small sample numbers they have instead followed the spurious clusterings of the sample points.
| |
| [[Image:BottleCateg 1.jpg|thumb|Decision contours]]
| |
| | |
| ===Neural network/fuzzy logic analogies===
| |
| There is some analogy between this algorithm and a neural network with a single hidden layer. The internal nodes are represented by the clusters <math>c_j \,</math> and the first and second layers of network weights are the conditional probabilities <math>p(c_j | x_i) \,</math> and <math>p(y_k | c_j) \,</math> respectively. However, unlike a standard neural network, the present algorithm relies entirely on probabilities as inputs rather than the sample values themselves while internal and output values are all conditional probability density distributions. Nonlinear functions are encapsulated in distance metric <math>f(.) \,</math> (or ''influence functions/radial basis functions'') and transition probabilities instead of sigmoid functions.
| |
| The Blahut-Arimoto three-line algorithm is seen to converge rapidly, often in tens of iterations, and by varying <math>\beta \,</math>, <math>\lambda \,</math> and <math>f \,</math> and the cardinality of the clusters, various levels of focus on data features can be achieved.<br />
| |
| The statistical soft clustering definition <math>p(c_i | x_j) \,</math> has some overlap with the verbal fuzzy membership concept of fuzzy logic. | |
| | |
| ==Bibliography==
| |
| [1] N. Tishby, F.C. Pereira, and W. Bialek:
| |
| [http://www.cs.huji.ac.il/labs/learning/Papers/allerton.pdf “The Information Bottleneck method”. The 37th annual Allerton Conference on Communication, Control, and Computing, Sep 1999: pp. 368–377]
| |
| | |
| [2] G. Chechik, A Globerson, N. Tishby and Y. Weiss: [http://www.jmlr.org/papers/volume6/chechik05a/chechik05a.pdf “Information Bottleneck for Gaussian Variables”. Journal of Machine Learning Research 6, Jan 2005, pp. 165–188]
| |
| | |
| [3] F. Creutzig, H. Sprekeler: [http://www.user.tu-berlin.de/creutzig/neco.2008.pdf Predictive Coding and the Slowness Principle: an Information-Theoretic Approach], 2008, Neural Computation 20(4): 1026–1041
| |
| | |
| [4] F. Creutzig, A. Globerson, N. Tishby: [http://www.user.tu-berlin.de/creutzig/Creutzig_PhysRevE.pdf Past-future information bottleneck in dynamical systems], 2009, Physical Review E 79, 041925
| |
| | |
| [5] N Tishby, N Slonim: “Data clustering by Markovian Relaxation and the Information Bottleneck Method”, Neural Information Processing Systems (NIPS) 2000, pp. 640–646
| |
| | |
| [6] B.W. Silverman: “Density Estimation for Statistical Data Analysis”, Chapman and Hall, 1986.
| |
| | |
| [7] N. Slonim, N. Tishby: "Document Clustering using Word Clusters via the Information Bottleneck Method", SIGIR 2000, pp. 208–215
| |
| | |
| [8] Y. Weiss: "Segmentation using eigenvectors: a unifying view", Proceedings IEEE International Conference on Computer Vision 1999, pp. 975–982
| |
| | |
| [9] D. J. Miller, A. V. Rao, K. Rose, A. Gersho: "An Information-theoretic Learning Algorithm for Neural Network Classification". NIPS 1995: pp. 591–597
| |
| | |
| [10] P. Harremoes and N. Tishby
| |
| [http://www.cs.huji.ac.il/labs/learning/Papers/flaske2.pdf "The Information Bottleneck Revisited or How to Choose a Good Distortion Measure". In proceedings of the International Symposium on Information Theory (ISIT) 2007]
| |
| | |
| =See also==
| |
| * [[Information theory]]
| |
| | |
| ==External links==
| |
| * [http://citeseer.ist.psu.edu/tishby99information.html Paper by N. Tishby, et al.]
| |
| | |
| [[Category:Data clustering algorithms]]
| |
| [[Category:Multivariate statistics]]
| |