|
|
| Line 1: |
Line 1: |
| '''Probabilistic latent semantic analysis (PLSA)''', also known as '''probabilistic latent semantic indexing''' ('''PLSI''', especially in information retrieval circles) is a [[statistical technique]] for the analysis of two-mode and co-occurrence data. In effect, one can derive a low dimensional representation of the observed variables in terms of their affinity to certain hidden variables, just as in [[latent semantic analysis]]. PLSA evolved from [[latent semantic analysis]].
| | Naval Architect Bykowski from Newcastle, likes handwriting analysis, property developers [http://thediamondentity.com/?option=com_k2&view=itemlist&task=user&id=17560 new condo launches in singapore] singapore and history. Of late had a family journey to Kernave Archaeological Site (Cultural Reserve of Kernave). |
| | |
| Compared to standard [[latent semantic analysis]] which stems from [[linear algebra]] and downsizes the occurrence tables (usually via a [[singular value decomposition]]), probabilistic latent semantic analysis is based on a mixture decomposition derived from a [[latent class model]].
| |
| | |
| ==Model==
| |
| [[Image:Plsi 1.svg|thumb|300px|[[Plate notation]] representing the PLSA model ("asymmetric" formulation). <math>d</math> is the document index variable, <math>c</math> is a word's topic drawn from the document's topic distribution, <math>P(c|d)</math>, and <math>w</math> is a word drawn from the word distribution of this word's topic, <math>P(w|c)</math> . The <math>d</math> and <math>w</math> are [[observable variable]]s, the topic <math>c</math> is a [[latent variable]].]]
| |
| | |
| Considering observations in the form of co-occurrences <math>(w,d)</math> of words and documents, PLSA models the probability of each co-occurrence as a mixture of conditionally independent [[multinomial distribution]]s:
| |
| | |
| : <math>P(w,d) = \sum_c P(c) P(d|c) P(w|c) = P(d) \sum_c P(c|d) P(w|c)</math>
| |
| | |
| The first formulation is the ''symmetric'' formulation, where <math>w</math> and <math>d</math> are both generated from the latent class <math>c</math> in similar ways (using the conditional probabilities <math>P(d|c)</math> and <math>P(w|c)</math>), whereas the second formulation is the ''asymmetric'' formulation, where, for each document <math>d</math>, a latent class is chosen conditionally to the document according to <math>P(c|d)</math>, and a word is then generated from that class according to <math>P(w|c)</math>. Although we have used words and documents in this example, the co-occurrence of any couple of discrete variables may be modelled in exactly the same way.
| |
| | |
| So, the number of parameters is equal to <math>cd + wc</math>. The number of parameters grows linearly with the number of documents. In addition, although PLSA is a generative model of the documents in the collection it is estimated on, it is not a generative model of new documents.
| |
| | |
| Their parameters are learned using the [[EM algorithm]].
| |
| | |
| == Application ==
| |
| PLSA may be used in a discriminative setting, via [[Fisher kernel]]s.<ref>Thomas Hofmann, [http://www.cs.brown.edu/people/th/papers/Hofmann-NIPS99.ps ''Learning the Similarity of Documents : an information-geometric approach to document retrieval and categorization''], [[Advances in Neural Information Processing Systems]] 12, pp-914-920, [[MIT Press]], 2000</ref>
| |
| | |
| PLSA has applications in [[information retrieval]] and [[information filtering|filtering]], [[natural language processing]], [[machine learning]] from text, and related areas.
| |
| | |
| It is reported that the [[aspect model]] used in the probabilistic latent semantic analysis has severe [[overfitting]] problems.<ref>{{cite journal|title=Latent Dirichlet Allocation|journal=Journal of Machine Learning Research|year=2003|first=David M.|last=Blei|coauthors=Andrew Y. Ng, Michael I. Jordan|volume=3|pages=993–1022|id= |url=http://jmlr.csail.mit.edu/papers/volume3/blei03a/blei03a.pdf|doi=10.1162/jmlr.2003.3.4-5.993}}</ref>
| |
| | |
| In 2012, pLSA has also been used in the [[bioinformatics]] context, for prediction of [[Gene Ontology]] biomolecular annotations.<ref>[http://home.dei.polimi.it/chicco/Wcci2012_DavideChicco_et_al.pdf ''"Probabilistic Latent Semantic Analysis for prediction of Gene Ontology annotations"'', Marco Masseroli, Davide Chicco, Pietro Pinoli. IEEE WCCI 2012 - the 2012 IEEE World Congress on Computational Intelligence proceedings. Brisbane, Australia, June 2012. (.pdf)]</ref>
| |
| | |
| ==Extensions ==
| |
| * Hierarchical extensions:
| |
| ** Asymmetric: MASHA ("Multinomial ASymmetric Hierarchical Analysis") <ref>Alexei Vinokourov and Mark Girolami, [http://citeseer.ist.psu.edu/rd/30973750,455249,1,0.25,Download/http://citeseer.ist.psu.edu/cache/papers/cs/22961/http:zSzzSzcis.paisley.ac.ukzSzvino-ci0zSzvinokourov_masha.pdf/vinokourov02probabilistic.pdf A Probabilistic Framework for the Hierarchic Organisation and Classification of Document Collections], in ''Information Processing and Management'', 2002</ref>
| |
| ** Symmetric: HPLSA ("Hierarchical Probabilistic Latent Semantic Analysis") <ref>Eric Gaussier, Cyril Goutte, Kris Popat and Francine Chen,
| |
| [http://www.xrce.xerox.com/Research-Development/Publications/2002-0044 A Hierarchical Model for Clustering and Categorising Documents], in "Advances in Information Retrieval -- Proceedings of the 24th [[Information Retrieval Specialist Group|BCS-IRSG]] European Colloquium on IR Research (ECIR-02)", 2002</ref>
| |
| | |
| * Generative models: The following models have been developed to address an often-criticized shortcoming of PLSA, namely that it is not a proper generative model for new documents.
| |
| ** [[Latent Dirichlet allocation]] - adds a [[Dirichlet distribution|Dirichlet]] prior on the per-document topic distribution
| |
| | |
| * Higher-order data: Although this is rarely discussed in the scientific literature, PLSA extends naturally to higher order data (three modes and higher), i.e. it can model co-occurrences over three or more variables. In the symmetric formulation above, this is done simply by adding conditional probability distributions for these additional variables. This is the probabilistic analogue to non-negative tensor factorisation.
| |
| | |
| ==History==
| |
| It was introduced in 1999 by [[Thomas Hofmann]],<ref>Thomas Hofmann, [http://www.cs.brown.edu/~th/papers/Hofmann-SIGIR99.pdf ''Probabilistic Latent Semantic Indexing''], Proceedings of the Twenty-Second Annual International [[Special Interest Group on Information Retrieval|SIGIR]] Conference on Research and Development in [[Information Retrieval]] (SIGIR-99), 1999</ref> and it is related <ref>Chris Ding, Tao Li, Wei Peng (2006). "[http://www.aaai.org/Papers/AAAI/2006/AAAI06-055.pdf Nonnegative Matrix Factorization and Probabilistic Latent Semantic Indexing: Equivalence Chi-Square Statistic, and a Hybrid Method. AAAI 2006 ]</ref> to [[non-negative matrix factorization]].
| |
| | |
| == References and notes ==
| |
| {{reflist}}
| |
| | |
| == See also ==
| |
| * [[Compound term processing]]
| |
| * [[Latent Dirichlet allocation]]
| |
| * [[Latent semantic analysis]]
| |
| * [[Pachinko allocation]]
| |
| * [[Vector space model]]
| |
| | |
| ==External links==
| |
| *[http://www.cs.brown.edu/people/th/papers/Hofmann-UAI99.pdf Probabilistic Latent Semantic Analysis]
| |
| *[http://www.semanticsearchart.com/researchpLSA.html Complete PLSA DEMO in C#]
| |
| | |
| {{DEFAULTSORT:Probabilistic Latent Semantic Analysis}}
| |
| [[Category:Statistical natural language processing]]
| |
| [[Category:Categorical data]]
| |
| [[Category:Latent variable models]]
| |
Naval Architect Bykowski from Newcastle, likes handwriting analysis, property developers new condo launches in singapore singapore and history. Of late had a family journey to Kernave Archaeological Site (Cultural Reserve of Kernave).