|
|
| (One intermediate revision by one other user not shown) |
| Line 1: |
Line 1: |
| {{merge from|Online NMF|discuss=Talk:Non-negative matrix factorization#Proposed merge with Online NMF|date=October 2013}}
| | Hello. Allow me introduce the writer. Her title is Refugia Shryock. Her family lives in Minnesota. Supervising is my profession. It's not a typical thing but what she likes performing is foundation jumping and now she is trying to make money with it.<br><br>Take a look at my web blog ... std testing at home ([http://www.asseryshit.com/groups/valuable-guidance-for-successfully-treating-yeast-infections/ check out here]) |
| [[File:NMF.png|thumb|400px|Illustration of approximate non-negative matrix factorization: the matrix ''V'' is represented by the two smaller matrices ''W'' and ''H'', which, when multiplied, approximately reconstruct ''V''.]]
| |
| : ''NMF redirects here. For the [[contract bridge|bridge]] convention, see [[new minor forcing]].''
| |
| | |
| '''Non-negative matrix factorization (NMF)''', also '''non-negative matrix approximation'''<ref name="dhillon"/><ref>{{cite journal|last=Tandon|first=Rashish|coauthors=Suvrit Sra|title=Sparse nonnegative matrix approximation: new formulations and algorithms|year=2010|series=TR|url=ftp://ftp.kyb.tuebingen.mpg.de/pub/mpi-memos/pdf/nmftr.pdf}}</ref> is a group of [[algorithm]]s in [[multivariate analysis]] and [[linear algebra]] where a [[matrix (mathematics)|matrix]] '''V''' is factorized into (usually) two matrices '''W''' and '''H''', with the property that all three matrices have no negative elements. This non-negativity makes the resulting matrices easier to inspect. Since the problem is not exactly solvable in general, it is commonly approximated numerically.
| |
| | |
| NMF finds applications in such fields as [[computer vision]], document [[Cluster analysis|clustering]],<ref name="dhillon"/> [[chemometrics]] and [[recommender system]]s.<ref name="gemulla">{{cite conference |authors=Rainer Gemulla, Erik Nijkamp, Peter J Haas, Yannis Sismanis |title=Large-scale matrix factorization with distributed stochastic gradient descent |conference=Proc. ACM SIGKDD Int'l Conf. on Knowledge discovery and data mining |url=http://www.mpi-inf.mpg.de/~rgemulla/publications/rj10481rev.pdf |year=2011 |pages=69-77}}</ref>
| |
| | |
| == History ==
| |
| In [[chemometrics]] non-negative matrix factorization has a long history under the name "self modeling curve resolution".<ref>{{Cite journal
| |
| | author1 = William H. Lawton
| |
| | author2 = Edward A. Sylvestre
| |
| | title= Self modeling curve resolution
| |
| | journal = [[Technometrics]]
| |
| | volume = 13
| |
| | issue = 3
| |
| | year = 1971
| |
| | page = 617+
| |
| | authorlink1 = William H. Lawton
| |
| | authorlink2 = Edward A. Sylvestre
| |
| }}</ref>
| |
| In this framework the vectors in the right matrix are continuous curves rather than discrete vectors.
| |
| Also early work on non-negative matrix factorizations was performed by a Finnish group of researchers in the middle of the 1990s under the name ''positive matrix factorization''.<ref>{{Cite journal
| |
| | author = P. Paatero, U. Tapper
| |
| | title = Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values
| |
| | journal = [[Environmetrics]]
| |
| | volume = 5
| |
| | pages = 111–126
| |
| | year = 1994
| |
| | doi = 10.1002/env.3170050203
| |
| | issue = 2
| |
| }}</ref><ref>{{Cite journal
| |
| | author = [[Pia Anttila]], [[Pentti Paatero]], Unto Tapper, Olli Järvinen
| |
| | title = Source identification of bulk wet deposition in Finland by positive matrix factorization
| |
| | journal = [[Atmospheric Environment (journal)|Atmospheric Environment]]
| |
| | volume = 29
| |
| | issue = 14
| |
| | pages = 1705–1718
| |
| | year = 1995
| |
| | doi = 10.1016/1352-2310(94)00367-T
| |
| }}</ref>
| |
| It became more widely known as ''non-negative matrix factorization'' after Lee and Seung investigated | |
| the properties of the algorithm and published some simple and useful
| |
| algorithms for two types of factorizations.<ref name="lee-seung">{{Cite journal
| |
| | author = Daniel D. Lee and [[Sebastian Seung|H. Sebastian Seung]]
| |
| | year = 1999
| |
| | title = Learning the parts of objects by non-negative matrix factorization
| |
| | journal = [[Nature (journal)|Nature]]
| |
| | volume = 401
| |
| | issue = 6755
| |
| | pages = 788–791
| |
| | doi = 10.1038/44565
| |
| | pmid = 10548103
| |
| }}</ref><ref name="lee2001algorithms">{{Cite conference
| |
| | author = Daniel D. Lee and H. Sebastian Seung
| |
| | year = 2001
| |
| | url = http://www.nips.cc/Web/Groups/NIPS/NIPS2000/00papers-pub-on-web/LeeSeung.ps.gz
| |
| | title = Algorithms for Non-negative Matrix Factorization
| |
| | booktitle = [[NIPS|Advances in Neural Information Processing Systems 13: Proceedings of the 2000 Conference]]
| |
| | pages = 556–562
| |
| | publisher = [[MIT Press]]
| |
| }}</ref>
| |
| | |
| == Background ==
| |
| | |
| Let matrix <math>\mathbf{V}</math> be the product of the matrices <math>\mathbf{W}</math> and <math>\mathbf{H}</math> such that:
| |
| | |
| <math>\mathbf{W}\mathbf{H} = \mathbf{V}</math>
| |
| | |
| Matrix multiplication can be implemented as linear combinations of column vectors in <math>\mathbf{W}</math> with coefficients supplied by cell values in <math>\mathbf{H}</math>. Each column in <math>\mathbf{V}</math> can be computed as follows:
| |
| | |
| <math>
| |
| \mathbf{v}_i = \sum_{j=1}^N \mathbf{H}_{ji}\mathbf{w}_j
| |
| </math>
| |
| | |
| where:
| |
| <math>N</math> is the number of columns in <math>\mathbf{W}</math>
| |
| | |
| <math>\mathbf{v}_i</math> is the <math>i^{th}</math> column vector of the product matrix <math>\mathbf{V}</math>
| |
| | |
| <math>\mathbf{H}_{ji}</math> is the cell value in the <math>j^{th}</math> row and <math>i^{th}</math> column of the matrix <math>\mathbf{H}</math>
| |
| | |
| <math>\mathbf{w}_j</math> is the <math>j^{th}</math> column of the matrix <math>\mathbf{W}</math>
| |
| | |
| When multiplying matrices, the dimensions of the factor matrices may be significantly lower than those of the product matrix and it's this property that forms the basis of NMF. NMF generates factors with significantly reduced dimensions compared to the original matrix. For example, if <math>\mathbf{V}</math> is an <math>m \times n</math> matrix, then <math>\mathbf{W}</math> is an <math>m \times p</math> matrix, and <math>\mathbf{H}</math> is a <math>p \times n</math> matrix. Here <math>p</math> can be significantly less than both <math>m</math> and <math>n</math>.
| |
| | |
| Here's an example based on a text-mining application:
| |
| * Let the input matrix (the matrix to be factored) be <math>\mathbf{V}</math> with 10000 rows and 500 columns where words are in rows and documents are in columns. In other words, we have 500 documents indexed by 10000 words. It follows that a column vector <math>\mathbf{v}</math> in <math>\mathbf{V}</math> represents a document.
| |
| * Assume we ask the algorithm to find 10 features in order to generate a features matrix <math>\mathbf{W}</math> with 10000 rows and 10 columns and a coefficients matrix <math>\mathbf{H}</math> with 10 rows and 500 columns.
| |
| * The product of <math>\mathbf{W}</math> and <math>\mathbf{H}</math> is a matrix with 10000 rows and 500 columns, the same shape as the input matrix <math>\mathbf{V}</math> and, if the factorization worked, also a reasonable approximation to the input matrix <math>\mathbf{V}</math>.
| |
| * From the treatment of matrix multiplication above it follows that each column in the product matrix <math>\mathbf{WH}</math> is a linear combination of the 10 column vectors in the features matrix <math>\mathbf{W}</math> with coefficients supplied by the coefficients matrix <math>\mathbf{H}</math>.
| |
| | |
| This last point is the basis of NMF because we can consider each original document in our example as being built from a small set of hidden features. NMF generates these features.
| |
| | |
| It's useful to think of each feature (column vector) in the features matrix <math>\mathbf{W}</math> as a document archetype comprising a set of words where each word's cell value defines the word's rank in the feature: The higher a word's cell value the higher the word's rank in the feature. A column in the coefficients matrix <math>\mathbf{H}</math> represents an original document with a cell value defining the document's rank for a feature. This follows because each row in <math>\mathbf{H}</math> represents a feature. We can now reconstruct a document (column vector) from our input matrix by a linear combination of our features (column vectors in <math>\mathbf{W}</math>) where each feature is weighted by the feature's cell value from the document's column in <math>\mathbf{H}</math>.
| |
| | |
| == Types ==
| |
| | |
| === Approximate non-negative matrix factorization ===
| |
| Usually the number of columns of '''W''' and the number of rows of '''H''' in NMF are selected so the product '''WH''' will become an approximation to '''V'''. The full decomposition of '''V''' then amounts to the two non-negative matrices '''W''' and '''H''' as well as a residual '''U''', such that: '''V''' = '''WH''' + '''U'''. The elements of the residual matrix can either be negative or positive.
| |
| | |
| When '''W''' and '''H''' are smaller than '''V''' they become easier to store and manipulate. Another reason for factorizing '''V''' into smaller matrices '''W''' and '''H''', is that if one is able to approximately represent the elements of '''V''' by significantly less data, then one has to infer some latent structure in the data.
| |
| | |
| === Different cost functions and regularizations ===
| |
| | |
| There are different types of non-negative matrix factorizations.
| |
| The different types arise from using different [[Loss function|cost function]]s for measuring the divergence between '''V''' and '''WH''' and possibly by [[regularization (mathematics)|regularization]] of the '''W''' and/or '''H''' matrices.<ref name="dhillon">{{Cite conference | author = [[Inderjit S. Dhillon]], [[Suvrit Sra]] | url = http://books.nips.cc/papers/files/nips18/NIPS2005_0203.pdf |format=PDF|title = Generalized Nonnegative Matrix Approximations with Bregman Divergences | booktitle = [[NIPS]] | year = 2005}}</ref>
| |
| | |
| Two simple divergence functions studied by Lee and Seung are the squared error (or [[Frobenius norm]]) and an extension of the Kullback-Leibler divergence to positive matrices (the original [[Kullback-Leibler divergence]] is defined on probability distributions).
| |
| Each divergence leads to a different NMF algorithm, usually minimizing the divergence using iterative update rules.
| |
| | |
| The factorization problem in the squared error version of NMF may be stated as:
| |
| Given a matrix <math>\mathbf{V}</math> find nonnegative matrices W and H that minimize the function
| |
| : <math>F(\mathbf{W},\mathbf{H}) = \|\mathbf{V} - \mathbf{WH}\|^2_F</math>
| |
| | |
| Another type of NMF for images is based on the [[total variation norm]].<ref>{{cite doi|10.1016/j.neucom.2008.01.022}}</ref>
| |
| | |
| When [[Tikhnov regularization|L1 regularization]] (akin to [[Least squares#Lasso_method|Lasso]]) is added to NMF with the mean squared error cost function, the resulting problem may be called '''non-negative sparse coding''' due to the similarity to the [[sparse coding]] problem,<ref name="hoyer02">{{cite conference |last=Hoyer |first=Patrik O. |title=Non-negative sparse coding |booktitle=Proc. IEEE Workshop on Neural Networks for Signal Processing |year=2002 |url=http://arxiv.org/pdf/cs/0202009}}</ref>
| |
| although it may also still be referred to as NMF.<ref>{{cite doi|10.1145/2020408.2020577}}</ref>
| |
| | |
| == Algorithms ==
| |
| There are several ways in which the '''W''' and '''H''' may be found: Lee and Seung's multiplicative update rule <ref name="lee2001algorithms"/> has been a popular method due to the simplicity of implementation. Since then, a few other algorithmic approaches have been developed.
| |
| | |
| Some successful algorithms are based on alternating [[non-negative least squares]]: in each step of such an algorithm, first '''H''' is fixed and '''W''' found by a non-negative least squares solver, then '''W''' is fixes and '''H''' is found analogously. The procedures used to solve for '''W''' and '''H''' may be the same<ref name="lin07"/> or different, as some NMF variants regularize one of '''W''' and '''H'''.<ref name="hoyer02"/> Specific approaches include the projected [[gradient descent]] methods,<ref name="lin07">{{cite doi|10.1162/neco.2007.19.10.2756}}</ref><ref>{{cite doi|10.1109/TNN.2007.895831}}</ref> the [[active set]] method,<ref name="kim2008nonnegative">{{Cite journal
| |
| | author = Hyunsoo Kim and Haesun Park
| |
| | title = Nonnegative Matrix Factorization Based on Alternating Nonnegativity Constrained Least Squares and Active Set Method
| |
| | journal = [[SIAM Journal on Matrix Analysis and Applications]]
| |
| | volume = 30
| |
| | issue = 2
| |
| | year = 2008
| |
| | pages = 713–730
| |
| | url = http://www.cc.gatech.edu/~hpark/papers/simax-nmf.pdf
| |
| }}</ref><ref name="gemulla"/> and the block principal pivoting method<ref name="kim2011fast">{{Cite journal
| |
| | author = Jingu Kim and Haesun Park
| |
| | title = Fast Nonnegative Matrix Factorization: An Active-set-like Method and Comparisons
| |
| | journal = [[SIAM Journal on Scientific Computing]]
| |
| | volume = 33
| |
| | issue = 6
| |
| | year = 2011
| |
| | pages = 3261–3281
| |
| | url = http://www.cc.gatech.edu/~jingu/docs/2011_paper_sisc_nmf.pdf
| |
| }}</ref> among several others.
| |
| | |
| The currently available algorithms are sub-optimal as they can only guarantee finding a local minimum, rather than a global minimum of the cost function. A provably optimal algorithm is unlikely in the near future as the problem has been shown to generalize the k-means clustering problem which is known to be [[NP-complete]].<ref>{{Cite journal
| |
| | title = On the equivalence of nonnegative matrix factorization and spectral clustering
| |
| | author = Ding, C. and He, X. and Simon, H.D.,
| |
| | journal = Proc. SIAM Data Mining Conf
| |
| | volume = 4
| |
| | pages = 606–610
| |
| | year = 2005
| |
| }}</ref> However, as in many other data mining applications, a local minimum may still prove to be useful.
| |
| | |
| Arora, Ge, Kannan, and Moitra (2012) have given an algorithm for exact NMF that provably runs in polynomial-time (i.e., always halts with the correct answer) if one of the factors W satisfies the separability condition. This condition is observed to hold in factorizations empirically found in many settings. They also give a more precise analysis of the complexity of NMF as a function of the dimension of '''W''' and '''H'''.<ref>{{Cite conference
| |
| | author = Sanjeev Arora and Rong Ge and Ravi Kannan and Ankur Moitra
| |
| | title = Computing a Nonnegative Matrix Factorization ---Provably.
| |
| | url = http://arxiv.org/abs/1111.0952
| |
| | booktitle = ACM STOC
| |
| | year = {2012}
| |
| }}</ref>
| |
| | |
| == Relation to other techniques ==
| |
| | |
| In ''Learning the parts of objects by non-negative matrix factorization'' Lee and Seung proposed NMF mainly for parts-based decomposition of images. It compares NMF to [[vector quantization]] and [[principal component analysis]], and shows that although the three techniques may be written as factorizations, they implement different constraints and therefore produce different results.
| |
| | |
| [[Image:Restricted Boltzmann machine.svg|thumb|NMF as a probabilistic graphical model: visible units (''V'') are connected to hidden units (''H'') through weights ''W'', so that ''V'' is [[Generative model|generated]] from a probability distribution with mean <math>\sum_a W_{ia}h_a</math>.<ref name="lee-seung"/>{{rp|5}}]]
| |
| It was later shown that some types of NMF are an instance of a more general probabilistic model called "multinomial PCA".<ref>{{Cite conference
| |
| | author = Wray Buntine
| |
| | url = http://cosco.hiit.fi/Articles/ecml02.pdf
| |
| |format=PDF| title = Variational Extensions to EM and Multinomial PCA
| |
| | booktitle = Proc. European Conference on Machine Learning (ECML-02)
| |
| | series = LNAI
| |
| | volume = 2430
| |
| | pages = 23–34
| |
| | year = 2002
| |
| }}</ref>
| |
| When NMF is obtained by minimizing the [[Kullback–Leibler divergence]], it is in fact equivalent to another instance of multinomial PCA, [[probabilistic latent semantic analysis]],<ref>{{Cite conference
| |
| | author = Eric Gaussier and Cyril Goutte
| |
| | year = 2005
| |
| | url = http://eprints.pascal-network.org/archive/00000971/01/39-gaussier.pdf
| |
| |format=PDF| title = Relation between PLSA and NMF and Implications
| |
| | booktitle = Proc. 28th international ACM SIGIR conference on Research and development in information retrieval (SIGIR-05)
| |
| | pages = 601–602
| |
| }}</ref>
| |
| trained by [[maximum likelihood]] estimation.
| |
| That method is commonly used for analyzing and clustering textual data and is also related to the [[latent class model]].
| |
| | |
| It has been shown <ref>Chris Ding, Xiaofeng He, and Horst D. Simon (2005).
| |
| "[http://ranger.uta.edu/~chqding/papers/NMF-SDM2005.pdf On the Equivalence of Nonnegative Matrix Factorization and Spectral Clustering]". Proc. SIAM Int'l Conf. Data Mining, pp. 606-610. May 2005</ref><ref>Ron Zass and Amnon Shashua (2005). "[http://www.cs.huji.ac.il/~zass/papers/cp-iccv05.pdf A Unifying Approach to Hard and Probabilistic Clustering]". International Conference on Computer Vision (ICCV) Beijing, China, Oct., 2005.</ref> NMF is equivalent to a relaxed form of [[K-means clustering]]: matrix factor '''W''' contains cluster centroids and '''H''' contains cluster
| |
| membership indicators, when using the least square as NMF objective. This provides theoretical foundation for using NMF for data clustering.
| |
| | |
| NMF extends beyond matrices to tensors of arbitrary order.<ref>{{Cite journal
| |
| | author = [[Pentti Paatero]]
| |
| | title = The Multilinear Engine: A Table-Driven, Least Squares Program for Solving Multilinear Problems, including the n-Way Parallel Factor Analysis Model
| |
| | journal = [[Journal of Computational and Graphical Statistics]]
| |
| | volume = 8
| |
| | issue = 4
| |
| | pages = 854–888
| |
| | year = 1999
| |
| | doi = 10.2307/1390831
| |
| | jstor = 1390831
| |
| }}</ref><ref>{{Cite journal
| |
| | author = Max Welling and Markus Weber
| |
| | year = 2001
| |
| | title = Positive Tensor Factorization
| |
| | journal = [[Pattern Recognition Letters]]
| |
| | volume = 22
| |
| | issue = 12
| |
| | pages = 1255–1261
| |
| | doi = 10.1016/S0167-8655(01)00070-8
| |
| }}</ref><ref>{{Cite conference
| |
| | author = Jingu Kim and Haesun Park
| |
| | title = Fast Nonnegative Tensor Factorization with an Active-set-like Method
| |
| | publisher = Springer
| |
| | pages = 311–326
| |
| | url = http://www.cc.gatech.edu/~hpark/papers/2011_paper_hpscbook_ntf.pdf
| |
| | year = 2012
| |
| | booktitle = High-Performance Scientific Computing: Algorithms and Applications }}
| |
| </ref> This extension may be viewed as a non-negative version of, e.g., the [[PARAFAC]] model.
| |
| | |
| Other extensions of NMF include joint factorisation of several data matrices and tensors where some factors are shared. Such models are useful for sensor fusion and relational learning.<ref>{{Cite conference
| |
| | author = Kenan Yilmaz and A. Taylan Cemgil and Umut Simsekli
| |
| | title = Generalized Coupled Tensor Factorization
| |
| | url = http://books.nips.cc/papers/files/nips24/NIPS2011_1189.pdf
| |
| | booktitle = NIPS
| |
| | year = {2011}
| |
| }}
| |
| </ref>
| |
| | |
| NMF is an instance of the nonnegative [[quadratic programming]] ([[NQP]]) as well as many other important problems including the [[support vector machine]] (SVM). However, SVM and NMF are related at a more intimate level than that of NQP, which allows direct application of the solution algorithms developed for either of the two methods to problems in both domains.<ref>{{Cite conference
| |
| | author = Vamsi K. Potluru and Sergey M. Plis and Morten Morup and Vince D. Calhoun and Terran Lane
| |
| | title = Efficient Multiplicative updates for Support Vector Machines
| |
| | year = 2009
| |
| | booktitle = Proceedings of the 2009 SIAM Conference on Data Mining (SDM)
| |
| | pages = 1218–1229
| |
| }}</ref>
| |
| | |
| == Uniqueness ==
| |
| The factorization is not unique: A matrix and its [[inverse matrix|inverse]] can be used to transform the two factorization matrices by, e.g.,<ref>{{Cite conference
| |
| | author = Wei Xu, Xin Liu & Yihong Gong
| |
| | title = Document clustering based on non-negative matrix factorization
| |
| | publisher = [[Association for Computing Machinery]]
| |
| | location = New York
| |
| | year = 2003
| |
| | booktitle = Proceedings of the 26th annual international ACM SIGIR conference on Research and development in information retrieval
| |
| | pages = 267–273
| |
| | url = http://portal.acm.org/citation.cfm?id=860485
| |
| }}</ref>
| |
| : <math>\mathbf{WH} = \mathbf{WBB}^{-1}\mathbf{H}</math>
| |
| If the two new matrices <math>\mathbf{\tilde{W} = WB}</math> and <math>\mathbf{\tilde{H}}=\mathbf{B}^{-1}\mathbf{H}</math> are [[non-negative matrix|non-negative]] they form another parametrization of the factorization.
| |
| | |
| The non-negativity of <math>\mathbf{\tilde{W}}</math> and <math>\mathbf{\tilde{H}}</math> applies at least if '''B''' is a non-negative [[monomial matrix]].
| |
| In this simple case it will just correspond to a scaling and a [[permutation]].
| |
| | |
| More control over the non-uniqueness of NMF is obtained with sparsity constraints.<ref>Julian Eggert, Edgar Körner, "[http://dx.doi.org/10.1109/IJCNN.2004.1381036 Sparse coding and NMF]", ''Proceedings. 2004 IEEE International Joint Conference on Neural Networks, 2004., pp. 2529-2533, 2004.</ref>
| |
| | |
| == Applications ==
| |
| | |
| === Text mining ===
| |
| NMF can be used for [[text mining]] applications.
| |
| In this process, a [[document-term matrix|''document-term'' matrix]] is constructed with the weights of various terms (typically weighted word frequency information) from a set of documents.
| |
| This matrix is factored into a ''term-feature'' and a ''feature-document'' matrix.
| |
| The features are derived from the contents of the documents, and the feature-document matrix describes [[Data clustering|data clusters]] of related documents.
| |
| | |
| One specific application used hierarchical NMF on a small subset of scientific abstracts from [[PubMed]].<ref>{{Cite journal
| |
| | last1 = Nielsen
| |
| | first1 = Finn Årup
| |
| | last2 = Balslev
| |
| | first2 = Daniela
| |
| | last3 = Hansen
| |
| | first3 = Lars Kai
| |
| | title = Mining the posterior cingulate: segregation between memory and pain components
| |
| | journal = [[NeuroImage]]
| |
| | volume = 27
| |
| | issue = 3
| |
| | pages = 520–522
| |
| | year = 2005
| |
| | doi = 10.1016/j.neuroimage.2005.04.034
| |
| | pmid = 15946864
| |
| }}</ref>
| |
| Another research group clustered parts of the [[Enron]] email dataset<ref>{{Cite web
| |
| | last1 = Cohen
| |
| | first1 = William
| |
| | authorlink = William Cohen (disambiguation)
| |
| | title = Enron Email Dataset
| |
| | url = http://www.cs.cmu.edu/~enron/
| |
| | date = 2005-04-04
| |
| | accessdate = 2008-08-26
| |
| }}</ref>
| |
| with 65,033 messages and 91,133 terms into 50 clusters.<ref>{{Cite journal
| |
| | last1 = Berry
| |
| | first1 = Michael W.
| |
| | last2 = Browne
| |
| | first = Murray
| |
| | title = Email Surveillance Using Non-negative Matrix Factorization
| |
| | journal = [[Computational and Mathematical Organization Theory]]
| |
| | volume = 11
| |
| | issue = 3
| |
| | pages = 249–264
| |
| | year = 2005
| |
| | doi = 10.1007/s10588-005-5380-5
| |
| | first2 = Murray
| |
| }}</ref>
| |
| NMF has also been applied to citations data, with one example clustering [[Wikipedia]] articles and [[scientific journal]]s based on the outbound scientific citations in Wikipedia.<ref>{{Cite conference
| |
| | last1 = Nielsen
| |
| | first = Finn Årup
| |
| | title = Clustering of scientific citations in Wikipedia
| |
| | conference = [[Wikimania]]
| |
| | year = 2008
| |
| | url = http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=5666
| |
| }}</ref>
| |
| | |
| Arora, Ge and Moitra (2012) have given polynomial-time algorithms to learn topic models using NMF. The algorithm assumes that the topic matrix satisfies a separability condition that is often found to hold in these settings.
| |
| <ref>{{Cite conference
| |
| | author = Sanjeev Arora and Rong Ge and Ankur Moitra
| |
| | title = Learning Topic Models---Going beyond SVD.
| |
| | url = http://arxiv.org/abs/1204.1956
| |
| | booktitle = arxiv
| |
| | year = {2012}
| |
| }}
| |
| </ref>
| |
| | |
| === Spectral data analysis ===
| |
| NMF is also used to analyze spectral data; one such use is in the classification of space objects and debris.<ref name="BerryM2006Algorithm">{{Cite journal
| |
| | author = Michael W. Berry, et al.
| |
| | title = Algorithms and Applications for Approximate Nonnegative Matrix Factorization
| |
| | year = 2006
| |
| }}</ref>
| |
| | |
| === Scalable Internet distance prediction ===
| |
| NMF is applied in scalable Internet distance (round-trip time) prediction. For a network with <math>N</math> hosts, with the help of NMF, the distances of all the <math>N^2</math> end-to-end links can be predicted after conducting only <math>O(N)</math> measurements. This kind of method was firstly introduced in Internet
| |
| Distance Estimation Service (IDES).<ref name="IDES_Mao06">{{Cite journal
| |
| | author = Yun Mao, Lawrence Saul and Jonathan M. Smith
| |
| | title = IDES: An Internet Distance Estimation Service for Large Networks
| |
| | journal = [[IEEE Journal on Selected Areas in Communications]]
| |
| | volume = 24
| |
| | issue = 12
| |
| | pages = 2273–2284
| |
| | year = 2006
| |
| | doi = 10.1109/JSAC.2006.884026
| |
| }}</ref> Afterwards, as a fully decentralized approach, Phoenix network coordinate system
| |
| <ref name="Phoenix_Chen11">{{Cite journal
| |
| | author = Yang Chen, Xiao Wang, Cong Shi, and et al.
| |
| | url = http://www.cs.duke.edu/~ychen/Phoenix_TNSM.pdf
| |
| | format=PDF
| |
| | title = Phoenix: A Weight-based Network Coordinate System Using Matrix Factorization
| |
| | journal = [[IEEE Transactions on Network and Service Management]]
| |
| | volume = 8
| |
| | issue = 4
| |
| | pages = 334–347
| |
| | year = 2011
| |
| }}</ref>
| |
| is proposed. It achieves better overall prediction accuracy by introducing the concept of weight.
| |
| | |
| === Non-stationary speech denoising ===
| |
| | |
| Speech denoising has been a long lasting problem in [[audio signal processing]]. There are lots of algorithms for denoising if the noise is stationary. For example, the [[Wiener filter]] is suitable for additive [[Gaussian noise]]. However, if the noise is non-stationary, the classical denoising algorithms usually have poor performance because the statistical information of the non-stationary noise is difficult to estimate. Schmidt et al.<ref>Schmidt, M.N., J. Larsen, and F.T. Hsiao. (2007). "Wind noise reduction using non-negative sparse coding", ''Machine Learning for Signal Processing, IEEE Workshop on'', 431–436</ref> use NMF to do speech denoising under non-stationary noise, which is completely different than classical statistical approaches.The key idea is that clean speech signal can be sparsely represented by a speech dictionary, but non-stationary noise cannot. Similarly, non-stationary noise can also be sparsely represented by a noise dictionary, but speech cannot.
| |
| | |
| The algorithm for NMF denoising goes as follows. Two dictionaries, one for speech and one for noise, need to be trained offline. Once a noisy speech is given, we first calculate the magnitude of the Short-Time-Fourier-Transform. Second, separate it into two parts via NMF, one can be sparsely represented by the speech dictionary, and the other part can be sparsely represented by the noise dictionary. Third, the part that is represented by the speech dictionary will be the estimated clean speech.
| |
| | |
| === Bioinformatics ===
| |
| | |
| NMF has been successfully applied to [[bioinformatics]].<ref>{{Cite journal
| |
| | author = Devarajan, K.
| |
| | title = Nonnegative Matrix Factorization: An Analytical and Interpretive Tool in Computational Biology
| |
| | journal = [[PLoS Computational Biology]]
| |
| | volume = 4
| |
| | issue = 7
| |
| | year = 2008
| |
| }}</ref><ref name="kim2007sparse">{{Cite journal
| |
| | author = Hyunsoo Kim and Haesun Park
| |
| | title = Sparse non-negative matrix factorizations via alternating non-negativity-constrained least squares for microarray data analysis
| |
| | journal = [[Bioinformatics (journal)|Bioinformatics]]
| |
| | volume = 23
| |
| | issue = 12
| |
| | pages = 1495–1502
| |
| | year = 2007
| |
| | doi = 10.1093/bioinformatics/btm134
| |
| | url = http://bioinformatics.oxfordjournals.org/cgi/content/abstract/23/12/1495
| |
| | pmid = 17483501
| |
| }}</ref>
| |
| | |
| == Current research ==
| |
| Current{{When|date=April 2011}} research in nonnegative matrix factorization includes, but not limited to,
| |
| | |
| (1) Algorithmic: searching for global minima of the factors and factor initialization.<ref>{{Cite journal
| |
| | author = C. Boutsidis and E. Gallopoulos
| |
| | title = SVD based initialization: A head start for nonnegative matrix factorization
| |
| | journal = Pattern Recognition
| |
| | volume = 41
| |
| | issue = 4
| |
| | pages = 1350–1362
| |
| | year = 2008
| |
| | doi = 10.1016/j.patcog.2007.09.010
| |
| }}</ref>
| |
| | |
| (2) Scalability: how to factorize million-by-billion matrices, which are commonplace in Web-scale data mining, e.g., see Distributed Nonnegative Matrix Factorization (DNMF)<ref>{{Cite journal
| |
| | author = Chao Liu, Hung-chih Yang, Jinliang Fan, Li-Wei He, and Yi-Min Wang
| |
| | title = Distributed Nonnegative Matrix Factorization for Web-Scale Dyadic Data Analysis on MapReduce
| |
| | journal = Proceedings of the 19th International World Wide Web Conference
| |
| | year = 2010
| |
| | url = http://research.microsoft.com/pubs/119077/DNMF.pdf
| |
| }}</ref>
| |
| | |
| (3) Online: how to update the factorization when new data comes in without recomputing from scratch.
| |
| | |
| ==See also==
| |
| *[[Online NMF]] (Online non-negative matrix factorization)
| |
| *[[Multilinear algebra]]
| |
| *[[Multilinear subspace learning]]
| |
| *[[Tensor]]
| |
| *[[Tensor decomposition]]
| |
| *[[Tensor software]]
| |
| | |
| == Sources and external links ==
| |
| | |
| === Notes ===
| |
| {{Reflist|2}}
| |
| | |
| === Others ===
| |
| <div class="references-small">
| |
| * {{Cite journal
| |
| | author = J. Shen, G. W. Israël
| |
| | title = A receptor model using a specific non-negative transformation technique for ambient aerosol
| |
| | journal = [[Atmospheric Environment (journal)|Atmospheric Environment]]
| |
| | volume = 23
| |
| | issue = 10
| |
| | pages = 2289–2298
| |
| | year = 1989
| |
| | doi = 10.1016/0004-6981(89)90190-X
| |
| }}
| |
| * {{Cite journal
| |
| | author = [[Pentti Paatero]]
| |
| | title = Least squares formulation of robust non-negative factor analysis
| |
| | journal = [[Chemometrics and Intelligent Laboratory Systems]]
| |
| | volume = 37
| |
| | issue = 1
| |
| | pages = 23–35
| |
| | year = 1997
| |
| | doi = 10.1016/S0169-7439(96)00044-5
| |
| }}
| |
| * {{Cite journal
| |
| | author = Raul Kompass
| |
| | title = A Generalized Divergence Measure for Nonnegative Matrix Factorization
| |
| | journal = [[Neural Computation]]
| |
| | volume = 19
| |
| | issue = 3
| |
| | year = 2007
| |
| | pages = 780–791
| |
| | pmid = 17298233
| |
| | doi = 10.1162/neco.2007.19.3.780
| |
| }}
| |
| * {{Cite journal
| |
| | title=Nonnegative Matrix Factorization and its applications in pattern recognition
| |
| | author=Liu, W.X. and Zheng, N.N. and You, Q.B.
| |
| | journal=[[Chinese Science Bulletin]]
| |
| | volume=51
| |
| | pages=7–18
| |
| | year=2006
| |
| | url = http://www.springerlink.com/index/7285V70531634264.pdf
| |
| | doi=10.1007/s11434-005-1109-6
| |
| | issue=17–18
| |
| }}
| |
| * {{Cite arxiv
| |
| | author = Ngoc-Diep Ho, Paul Van Dooren and Vincent Blondel
| |
| | title = Descent Methods for Nonnegative Matrix Factorization
| |
| | year = 2008
| |
| | eprint = 0801.3199
| |
| | class = cs.NA
| |
| }}
| |
| * {{Cite journal
| |
| | author = Andrzej Cichocki, Rafal Zdunek and Shun-ichi Amari
| |
| | title = Nonnegative Matrix and Tensor Factorization
| |
| | journal = [[IEEE Signal Processing Magazine]]
| |
| | volume = 25
| |
| | issue = 1
| |
| | year = 2008
| |
| | pages = 142–145
| |
| | doi = 10.1109/MSP.2008.4408452
| |
| }}
| |
| * {{Cite journal
| |
| | title = Nonnegative Matrix Factorization with the Itakura-Saito Divergence: With Application to Music Analysis
| |
| | author = Cédric Févotte, Nancy Bertin, and Jean-Louis Durrieu
| |
| | journal = [[Neural Computation]]
| |
| | volume = 21
| |
| | issue = 3
| |
| | year = 2009
| |
| | pmid=18785855
| |
| | doi=10.1162/neco.2008.04-08-771
| |
| | pages=793–830
| |
| }}
| |
| * {{Cite journal
| |
| | author = Ali Taylan Cemgil
| |
| | title = Bayesian Inference for Nonnegative Matrix Factorisation Models
| |
| | journal = [[Computational Intelligence and Neuroscience]]
| |
| | volume = 2009
| |
| | issue = 2
| |
| | year = 2009
| |
| | doi = 10.1155/2009/785152
| |
| | url = http://www.hindawi.com/journals/cin/2009/785152.abs.html
| |
| | pages = 1
| |
| | pmid = 19536273
| |
| | pmc = 2688815
| |
| }}
| |
| </div>
| |
| | |
| === Software ===
| |
| * [http://code.google.com/p/beta-ntf/ beta_ntf] Python module for Nonnegative Tensor Factorization. Supports tensors of arbitrary shape.
| |
| * [http://www.cs.virginia.edu/~jdl/nmf/ Routines] for performing Weighted Non-Negative Matrix Factorzation
| |
| * [http://www.cc.gatech.edu/~hpark/nmfsoftware.php Fast Non-negative Matrix Factorization] [[MATLAB]] software by Haesun Park's group.
| |
| * [http://cran.r-project.org/web/packages/NMFN/index.html Non-negative Matrix Factorization] [[R (programming language)]] implementation by Suhai (Timothy) Liu.
| |
| *[http://code.google.com/p/pymf/ PyMF] A Python module that includes several matrix factorization methods.
| |
| *[http://isp.imm.dtu.dk/toolbox/ NMF toolbox] implemented in Matlab. Developed at IMM DTU.
| |
| * [http://scgroup.hpclab.ceid.upatras.gr/scgroup/Projects/TMG/ Text to Matrix Generator (TMG)] MATLAB toolbox that can be used for various tasks in text mining (TM) specifically i) indexing, ii) retrieval, iii) dimensionality reduction, iv) clustering, v) classification. Most of TMG is written in MATLAB and parts in Perl. It contains implementations of LSI, clustered LSI, NMF and other methods.
| |
| * [http://nimfa.biolab.si Nimfa] A Python library for non-negative matrix factorizations, various initialization methods and factorization quality measures.
| |
| * [https://sites.google.com/site/nmftool The non-negative matrix factorization toolbox for biological data mining].
| |
| * [http://sites.google.com/a/uw.edu/isdl/projects/cmf-toolbox Complex Matrix Factorization Toolbox].
| |
| | |
| [[Category:Linear algebra]]
| |
| [[Category:Matrix theory]]
| |
| [[Category:Multivariate statistics]]
| |
| [[Category:Machine learning algorithms]]
| |