|
|
| Line 1: |
Line 1: |
| In [[computer science]], '''approximate string matching''' (often colloquially referred to as '''fuzzy string searching''') is the technique of finding [[String (computing)|strings]] that match a [[pattern]] approximately (rather than exactly). The problem of approximate string matching is typically divided into two sub-problems: finding approximate [[substring]] matches inside a given string and finding dictionary strings that match the pattern approximately.
| | The name of the author is Jayson. Alaska is the only place I've been residing in but now I'm considering other options. Doing ballet is some thing she would never give up. Distributing manufacturing is how he tends to make a living.<br><br>Feel free to surf to my webpage - free online tarot card readings ([http://1.234.36.240/fxac/m001_2/7330 their website]) |
| | |
| == Overview ==
| |
| The closeness of a match is measured in terms of the number of primitive operations necessary to convert the string into an exact match. This number is called the [[edit distance]] between the string and the pattern. The usual primitive operations are:{{ref|CRMN01}}
| |
| * insertion: ''cot'' → ''co'''a'''t''
| |
| * deletion: ''co'''a'''t'' → ''cot''
| |
| * substitution: ''co'''a'''t'' → ''co'''s'''t''
| |
| These three operations may be generalized as forms of substitution by adding a NULL character (here symbolized by *) wherever a character has been deleted or inserted:
| |
| * insertion: ''co'''*'''t'' → ''co'''a'''t''
| |
| * deletion: ''co'''a'''t'' → ''co'''*'''t''
| |
| * substitution: ''co'''a'''t'' → ''co'''s'''t''
| |
| | |
| Some approximate matchers also treat ''transposition'', in which the positions of two letters in the string are swapped, to be a primitive operation. Changing ''cost'' to ''cots'' is an example of a transposition.{{ref|CRMN01}}
| |
| | |
| Different approximate matchers impose different constraints. Some matchers use a single global unweighted cost, that is, the total number of primitive operations necessary to convert the match to the pattern. For example, if the pattern is ''coil'', ''foil'' differs by one substitution, ''coils'' by one insertion, ''oil'' by one deletion, and ''foal'' by two substitutions. If all operations count as a single unit of cost and the limit is set to one, ''foil'', ''coils'', and ''oil'' will count as matches while ''foal'' will not.
| |
| | |
| Other matchers specify the number of operations of each type separately, while still others set a total cost but allow different weights to be assigned to different operations. Some matchers permit separate assignments of limits and weights to individual groups in the pattern.
| |
| | |
| == Problem formulation and algorithms ==
| |
| One possible definition of the approximate string matching problem is the following: Given a pattern string <math>P = p_1p_2...p_m</math> and a text string <math>T = t_1t_2\dots t_n</math>, find a substring <math>T_{j',j} = t_{j'}\dots t_j</math> in ''T'', which, of all substrings of ''T'', has the smallest edit distance to the pattern ''P''.
| |
| | |
| A brute-force approach would be to compute the edit distance to P for all substrings of T, and then choose the substring with the minimum distance. However, this algorithm would have the running time [[Big O notation|''O'']](''n''<sup>3</sup> ''m'').
| |
| | |
| A better solution, which was proposed by Sellers {{ref|Sel80}}, relies on [[dynamic programming]]. It uses an alternative formulation of the problem: for each position ''j'' in the text ''T'' and each position ''i'' in the pattern ''P'', compute the minimum edit distance between the ''i'' first characters of the pattern, <math>P_i</math>, and any substring <math>T_{j',j}</math> of ''T'' that ends at position ''j''.
| |
| | |
| For each position ''j'' in the text ''T'', and each position ''i'' in the pattern ''P'', go through all substrings of ''T'' ending at position ''j'', and determine which one of them has the minimal
| |
| edit distance to the ''i'' first characters of the pattern ''P''. Write this minimal distance as ''E''(''i'', ''j''). After computing ''E''(''i'', ''j'') for all ''i'' and ''j'', we can easily find a solution to the original problem: it is the substring for which ''E''(''m'', ''j'') is minimal (''m'' being the length of the pattern ''P''.)
| |
| | |
| Computing ''E''(''m'', ''j'') is very similar to computing the edit distance between two strings. In fact, we can use the [[Levenshtein distance#Computing Levenshtein distance|Levenshtein distance computing algorithm]] for ''E''(''m'', ''j''), the only difference being that we must initialize the first row with zeros, and save the path of computation, that is, whether we used ''E''(''i'' − 1,''j''), E(''i'',''j'' − 1) or ''E''(''i'' − 1,''j'' − 1) in computing ''E''(''i'', ''j'').
| |
| | |
| In the array containing the ''E''(''x'', ''y'') values, we then choose the minimal value in the last row, let it be ''E''(''x''<sub>2</sub>, ''y''<sub>2</sub>), and follow the path of computation backwards, back to the row number 0. If the field we arrived at was ''E''(''x''<sub>1</sub>, 0), then ''T''[''x''<sub>1</sub> + 1] ... ''T''[''y''<sub>2</sub>] is a substring of T with the minimal edit distance to the pattern ''P''.
| |
| | |
| Computing the ''E''(''x'', ''y'') array takes [[Big O notation|''O'']](''mn'') time with the dynamic programming algorithm, while the backwards-working phase takes [[Big O notation|''O'']](''n'' + ''m'') time.
| |
| | |
| ==On-line versus off-line==
| |
| Traditionally, approximate string matching algorithms are classified into two categories: on-line and off-line. With on-line algorithms the pattern can be processed before searching but the text cannot. In other words, on-line techniques do searching without an index. Early algorithms for on-line approximate matching were suggested by Wagner and Fisher{{ref|WF74}} and by Sellers. {{ref|Sel80}} Both algorithms are based on [[dynamic programming]] but solve different problems. Sellers' algorithm searches approximately for a substring in a text while the algorithm of Wagner and Fisher calculates [[Levenshtein distance]], being appropriate for dictionary fuzzy search only.
| |
| | |
| On-line searching techniques have been repeatedly improved. Perhaps the most
| |
| famous improvement is the [[bitap algorithm]] (also known as the shift-or and shift-and algorithm), which is very efficient for relatively short pattern strings. The Bitap algorithm is the heart of the [[Unix]] searching [[programming tool|utility]] [[agrep]]. A review of on-line searching algorithms was done by G. Navarro.{{ref|Nav01}}
| |
| | |
| Although very fast on-line techniques exist, their
| |
| performance on large data is unacceptable.
| |
| Text preprocessing or [[index (search engine)|indexing]] makes searching dramatically faster.
| |
| Today, a variety of indexing algorithms have been presented. Among them are [[suffix tree]]s{{ref|Gus97}}, [[metric tree]]s{{ref|NB98}} and [[n-gram]] methods.{{ref|NBST01}}{{ref|Zob95}}
| |
| A detailed survey of indexing techniques that allows one to find an arbitrary substring in a text is given by Navarro ''et al.''{{ref|NBST01}}. A computational survey of dictionary methods (i.e., methods that permit finding all dictionary words that approximately match a search pattern) is given by Boytsov {{ref|B11}}.
| |
| | |
| == Applications ==
| |
| The most common application of approximate matchers until recently has been [[spell checking]].{{ref|Gus97}} With the availability of large amounts of DNA data, matching of [[nucleotide]] sequences has become an important application.{{ref|CRMN01}} Approximate matching is also used to identify pieces of music from small snatches and in [[spam filtering]].{{ref|Gus97}}
| |
| | |
| == See also ==
| |
| * [[String metric]]
| |
| * [[Locality-sensitive hashing]]
| |
| * [[Needleman–Wunsch algorithm]]
| |
| * [[Smith–Waterman algorithm]]
| |
| * [[Levenshtein distance]]
| |
| * [[Concept Search]]
| |
| * [[Regular expression#Fuzzy regular expressions|Approximate matching]] with addition of regular expressions ability
| |
| * [[Regular expressions]] for non-fuzzy (exact) matching
| |
| * [[Metaphone]]
| |
| * [[Soundex]]
| |
| * [[Agrep]]
| |
| * [[Plagiarism detection]]
| |
| | |
| == External links ==
| |
| * [http://flamingo.ics.uci.edu Flamingo Project]
| |
| * [http://www.cse.unsw.edu.au/~weiw/project/simjoin.html Efficient Similarity Query Processing Project] with recent advances in approximate string matching based on an edit distance threshold.
| |
| * [http://rockymadden.com/stringmetric/ StringMetric project] a [[Scala programming language|Scala]] library of string metrics and phonetic algorithms
| |
| * [https://github.com/NaturalNode/natural Natural project] a [[JavaScript]] natural language processing library which includes implementations of popular string metrics
| |
| | |
| ==References==
| |
| {{reflist}}
| |
| {{refbegin}}
| |
| * {{note|BN96}} {{cite conference |author=Baeza-Yates R, Navarro G |title=A faster algorithm for approximate string matching |editor=Dan Hirchsberg, Gene Myers |booktitle=Combinatorial Pattern Matching (CPM'96), LNCS 1075 |pages=1–23 |date=June 1996 |location=Irvine, CA }}
| |
| * {{note|NB98}} {{cite conference |author=Baeza-Yates R, Navarro G |title=Fast Approximate String Matching in a Dictionary |booktitle=Proc. SPIRE'98 |pages=14–22 |publisher=IEEE CS Press |url=http://reference.kfupm.edu.sa/content/f/a/fast_approximate_string_matching_in_a_di_105948.pdf}}
| |
| * {{note|B11}} {{cite journal |author=Boytsov, Leonid|title=Indexing methods for approximate dictionary searching: Comparative analysis|journal=Jea Acm|volume=16 |issue=1 |pages=1–91 |year=2011|doi=10.1145/1963190.1963191}}
| |
| *{{note|CRMN01}}{{cite book |last=Cormen |first=Thomas |authorlink=Thomas Cormen |coauthors=Leiserson, Rivest |title=Introduction to Algorithms |edition=2nd |year=2001 |publisher=MIT Press |isbn=0-262-03293-7 |pages=364–7}}
| |
| *{{note|GLL01}}{{cite book |author=Galil, Zvi; Apostolico, Alberto |title=Pattern matching algorithms |publisher=Oxford University Press |location=Oxford [Oxfordshire] |year=1997 |isbn=0-19-511367-5 }}
| |
| * {{note|Gus97}}{{cite book |author=Gusfield, Dan |title=Algorithms on strings, trees, and sequences: computer science and computational biology |publisher=Cambridge University Press |location=Cambridge, UK |year=1997 |isbn=0-521-58519-8 }}
| |
| * {{note|M99}} {{cite journal |author=Myers G |title=A fast bit-vector algorithm for approximate string matching based on dynamic programming |journal=Journal of the ACM |volume=46 |issue=3 |pages=395–415 |date=May 1999 |doi=10.1145/316542.316550 }}
| |
| * {{note|Nav01}} {{cite journal |author=Navarro, Gonzalo |title=A guided tour to approximate string matching |journal=ACM Computing Surveys |volume=33 |issue=1 |pages=31–88 |year=2001 |id={{citeseerx|10.1.1.96.7225}} |doi=10.1145/375360.375365}}
| |
| * {{note|NBST01}} {{cite journal |author=Navarro, Gonzalo, Ricardo Baeza-Yates, E. Sutinen and J. Tarhio |title=Indexing Methods for Approximate String Matching |journal=IEEE Data Engineering Bulletin |volume=24 |issue=4 |pages=19–27 |year=2001 |url=http://www.dcc.uchile.cl/~gnavarro/ps/deb01.pdf}}
| |
| * {{note|Sel80}} {{cite journal |author=Sellers, Peter H. |title=The Theory and Computation of Evolutionary Distances: Pattern Recognition |journal=Journal of Algorithms |volume=1 |pages=359–73 |year=1980 |doi=10.1016/0196-6774(80)90016-4 |issue=4 }}
| |
| *{{note|SKN01}}{{cite book |last=Skiena |first=Steve |title=Algorithm Design Manual |edition=1st |year=1998 |publisher=Springer |isbn=978-0-387-94860-7}}
| |
| * {{note|Ukk85}} {{cite journal |author=Ukkonen E |title=Algorithms for approximate string matching |journal=Information and Control |volume=64 |pages=100–18 |year=1985 |doi=10.1016/S0019-9958(85)80046-2 }}
| |
| * {{note|WF74}} {{cite journal |author=Wagner R, Fischer M |title=The string-to-string correction problem |journal=Journal of the ACM |volume=21 |pages=168–73 |year=1974 |url=http://portal.acm.org/citation.cfm?id=321811 |doi=10.1145/321796.321811}}
| |
| * {{note|Zob95}} J. Zobel, P. Dart. Finding approximate matches in large lexicons. Software-Practice & Experience 25(3), pp 331–345, 1995.
| |
| {{refend}}
| |
| | |
| {{DEFAULTSORT:Approximate String Matching}}
| |
| [[Category:String matching algorithms| ]]
| |
| [[Category:Pattern matching]]
| |
| [[Category:Dynamic programming]]
| |