|
|
| Line 1: |
Line 1: |
| In [[computer science]], '''all-pairs testing''' or '''pairwise testing''' is a [[combinatorial]] method of [[software testing]] that, for ''each pair'' of input parameters to a system (typically, a [[software]] [[algorithm]]), tests all possible discrete combinations of those parameters. Using carefully chosen [[test vector]]s, this can be done much faster than an exhaustive search of [[Software testing#Input combinations and preconditions|all combinations]] of all parameters, by "parallelizing" the tests of parameter pairs.
| | Luke Bryan is usually a superstar in the producing as well as profession progress 1st next to his third hotel recording, & , will be the resistant. He broken on the scene in 2011 along with his unique combination [http://minioasis.com luke bryan tickets phoenix] of downward-residence luke bryan luke bryan luke bryan ([http://lukebryantickets.neodga.com neodga.com]) ease of access, video star fantastic seems and words, is defined t [http://lukebryantickets.iczmpbangladesh.org where to buy luke bryan tickets] inside a significant way. The newest recor in the nation graph or chart and #2 in the pop graphs, creating it the second maximum debut at that time of 2005 for a nation designer. <br><br>The boy of your , understands patience and dedication are important elements in relation to an effective profession- . His initial recording, Stay Me, generated the very best hits “All My Friends Say” and “Country Person,” although his hard work, Doin’ Thing, found the performer-about luke bryan in concert ([http://www.museodecarruajes.org just click lukebryantickets.flicense.com]) three direct No. 6 single men and women: Else Phoning Is a Good Factor.”<br><br>During the slip of 2011, Tour: Bryan And which had a remarkable selection of , which includes Downtown. “It’s much like you’re acquiring a authorization to go to the next level, claims these musicians that had been a part of the Tourmore than in to a larger sized degree of designers.” It twisted as one of the most successful tours in the 15-season background.<br><br>Here is my web-site - where luke bryan is from [[http://lukebryantickets.flicense.com just click lukebryantickets.flicense.com]] |
| | |
| ==Rationale==
| |
| Assume that the test function{{clarify|date=June 2012}} has <math>N</math> parameters given in a set <math>\{ P_i\} = \{ P_1 , P_2 , ... , P_N \}</math>.
| |
| The range of the parameters are given by <math>R(P_i)= R_i</math>.
| |
| Let's assume that <math>|R_i|= n_i</math>.
| |
| We notice that the set of choices of ranges <math>X = \{ n_i \}</math> can be a [[multiset]] {{clarify|date=June 2012}}, because there can be multiple
| |
| parameters having same number of choices.
| |
| | |
| Let's define <math>max(S)</math> as one of the maximum of the multiset <math>S</math>.
| |
| Then, the number of pair-wise test cases on this test function would be:-
| |
| | |
| <math>
| |
| T = max(X) \times max ( X \setminus max(X) )
| |
| </math>
| |
| | |
| Plainly that would mean, if the <math> n = max(X) </math> and <math> m = max ( X \setminus max(X) ) </math>
| |
| then the number of tests is typically O(''nm''). Where ''n'' and ''m'' are the number of possibilities for each of the two parameters with the most choices.
| |
| | |
| {| class="wikitable"
| |
| |-
| |
| ! Parameter Name !! Value 1 !! Value 2 || Value 3 || Value 4
| |
| |-
| |
| | Enabled || True || False || * || *
| |
| |-
| |
| | Choice Type || 1 || 2 || 3 || *
| |
| |-
| |
| | Category || a || b || c || d
| |
| |}
| |
| | |
| In this case the parameters are Enabled with choices range of 2, Choice Type with 3, and Category with 4.
| |
| That would mean:
| |
| <math> | |
| X = \{ 2, 3 , 4 \}
| |
| </math> | |
| Hence, n = 4, m = 3 and number of tests would be 12.
| |
| The pict tool generated pairwise test cases on the input looks like:- | |
| | |
| {| class="wikitable"
| |
| |-
| |
| ! Enabled !! Choice Type !! Category
| |
| |-
| |
| | True || 3 || a
| |
| |-
| |
| | True || 1 || d
| |
| |-
| |
| |False || 1 || c
| |
| |-
| |
| |False || 2 || d
| |
| |-
| |
| |True || 2 || c
| |
| |-
| |
| |False || 2 || a
| |
| |-
| |
| |False || 1 || a
| |
| |-
| |
| |False || 3 || b
| |
| |-
| |
| |True || 2 || b
| |
| |-
| |
| |True || 3 || d
| |
| |-
| |
| |False || 3 || c
| |
| |-
| |
| |True || 1 || b
| |
| |}
| |
| | |
| The below table would generate a multiset :
| |
| | |
| {| class="wikitable"
| |
| |-
| |
| ! Parameter Name !! Value 1 !! Value 2 || Value 3 || Value 4
| |
| |-
| |
| | Enabled || True || False || * || *
| |
| |-
| |
| | Choice Type || 1 || 2 || 3 || 4
| |
| |-
| |
| | Category || a || b || c || d
| |
| |}
| |
| | |
| In this case the parameters are Enabled with choices range of 2, Choice Type with 4, and Category with 4.
| |
| That would mean:-
| |
| <math>
| |
| X = \{ 2, 4 , 4 \}
| |
| </math>
| |
| and it is a multiset. | |
| Hence, n = 4, m = 4 and number of tests would be 16, which are shown in the below table:-
| |
| {| class="wikitable"
| |
| |-
| |
| ! Enabled !! Choice Type !! Category
| |
| |-
| |
| |True||3||b
| |
| |-
| |
| |False||3||d
| |
| |-
| |
| |True||2||a
| |
| |-
| |
| |True||1||c
| |
| |-
| |
| |False||4||b
| |
| |-
| |
| |False||1||a
| |
| |-
| |
| |False||2||b
| |
| |-
| |
| |True||4||d
| |
| |-
| |
| |True||4||a
| |
| |-
| |
| |True||2||d
| |
| |-
| |
| |False||4||c
| |
| |-
| |
| |False||1||b
| |
| |-
| |
| |True||2||c
| |
| |-
| |
| |False||1||d
| |
| |-
| |
| |False||3||a
| |
| |-
| |
| |True||3||c
| |
| |}
| |
| | |
| The reasoning behind all-pairs testing is this: the simplest bugs in a program are generally triggered by a single input parameter. The next simplest category of bugs consists of those dependent on interactions between pairs of parameters, which can be caught with all-pairs testing.<ref>{{cite book | last = Black | first = Rex | title = Pragmatic Software Testing: Becoming an Effective and Efficient Test Professional | location = New York | publisher = [[John Wiley & Sons|Wiley]] | year = 2007 | isbn = 978-0-470-12790-2 | page = 240}}</ref> Bugs involving interactions between three or more parameters are progressively less common,<ref>{{cite journal | author=D.R. Kuhn, D.R. Wallace, A.J. Gallo, Jr. | title=Software Fault Interactions and Implications for Software Testing | journal=IEEE Trans. on Software Engineering |volume=30 | issue=6 |date=June 2004 | url=http://csrc.nist.gov/groups/SNS/acts/documents/TSE-0172-1003-1.pdf}}</ref> while at the same time being progressively more expensive to find by exhaustive testing, which has as its limit the exhaustive testing of all possible inputs.<ref>{{cite report| title=Practical Combinatorial Testing. SP 800-142. | publisher=Natl. Inst. of Standards and Technology | year=2010 |url=http://csrc.nist.gov/groups/SNS/acts/documents/SP800-142-101006.pdf }}</ref>
| |
|
| |
| This can be further generalized.{{cn|date=July 2012}}
| |
| The idea is to apply [[sorting]] to the set <math>X = \{ n_i \}</math> so that <math>P = \{ P_i \}</math> gets ordered too.
| |
| Let the sorted set be a <math>N</math> tuple :-
| |
| | |
| <math>
| |
| P_s = < P_i > \; ; \; i < j \implies |R(P_i)| < |R(P_j)|
| |
| </math>
| |
| | |
| Now we can take the set <math>X(2) = \{ P_{N-1} , P_{N-2} \}</math> and call it the pairwise testing.
| |
| Generalizing further we can take the set <math>X(3) = \{ P_{N-1} , P_{N-2} , P_{N-3} \}</math> and call it the 3-wise testing.
| |
| Eventually, we can say <math>X(T) = \{ P_{N-1} , P_{N-2} , ... , P_{N-T} \}</math> T-wise testing.
| |
| | |
| The N-wise testing then would just be, all possible combinations from the above formula.
| |
| | |
| One of the main strengths of combinatorial technique is that it enables a significant reduction of the number of test cases without compromising functional coverage.<ref>{{cite web|url=http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=4578383|title= IEEE 12. Proceedings from the 5th International Conference on Software Testing and Validation (ICST). Software Competence Center Hagenberg. "Test Design: Lessons Learned and Practical Implications. }}</ref> Many testing methods regard all-pairs testing of a system or subsystem as a reasonable cost-benefit compromise between often computationally infeasible higher-order combinatorial testing methods, and less exhaustive methods which fail to exercise all possible pairs of parameters. Because no testing technique can find all bugs, all-pairs testing is typically used together with other [[quality assurance]] techniques such as [[unit testing]], [[symbolic execution]], [[fuzz testing]], and [[code review]].
| |
| | |
| ==Notes==
| |
| | |
| {{reflist}}
| |
| | |
| == See also ==
| |
| {{portal|Software Testing}}
| |
| * [[Software testing]]
| |
| * [[Orthogonal array testing]]
| |
| | |
| == External links ==
| |
| * [http://www.combinatorialtesting.com Combinatorialtesting.com; Includes clearly written introductions to pairwise and other, more thorough, methods of combinatorial testing]
| |
| * [http://hexawise.com/ Hexawise.com - Pairwise test case generating tool with both free and commercial versions (also provides more thorough 3-way, 4-way, 5-way, and 6-way coverage solutions)]
| |
| * [http://testcover.com/pub/background/stareast2008.ppt Pairwise Testing Comes of Age - Review including history, examples, issues, research]
| |
| * [http://www.pairwise.org/ Pairwise Testing: Combinatorial Test Case Generation]
| |
| * [http://www.developsense.com/testing/PairwiseTesting.html Pairwise testing]
| |
| * [http://www.mcdowella.demon.co.uk/allPairs.html All-pairs testing]
| |
| * [http://csrc.nist.gov/acts/ Pairwise and generalized t-way combinatorial testing]
| |
| * [http://testapi.codeplex.com TestApi - .NET API library for testing, providing a variation generation API]
| |
| * [http://github.com/sageserpent-open/NTestCaseBuilder NTestCaseBuilder - another .NET API library; focussed purely on combinatoric testing and scalability of test case generation]
| |
| * [http://code.google.com/p/jcombinatorial/ JCombinatorial - an open-source library that facilitates all-pairs testing with JUnit]
| |
| * {{cite web |url=http://www.phadkeassociates.com/index_rdexperttestplanning.htm |title=rdExpert Software for Orthogonal Array Testing |publisher=Phadke Associates, Inc. |quote=Commercial toolset for Orthogonal Array and PairWise Testing.}}
| |
| * [http://msdn.microsoft.com/en-us/library/cc150619.aspx Pairwise Testing in the Real World: Practical Extensions to Test-Case Scenarios]
| |
| | |
| {{DEFAULTSORT:All-Pairs Testing}}
| |
| [[Category:Software testing]]
| |
| [[Category:Combinatorics]]
| |
| [[Category:Design of experiments]]
| |
Luke Bryan is usually a superstar in the producing as well as profession progress 1st next to his third hotel recording, & , will be the resistant. He broken on the scene in 2011 along with his unique combination luke bryan tickets phoenix of downward-residence luke bryan luke bryan luke bryan (neodga.com) ease of access, video star fantastic seems and words, is defined t where to buy luke bryan tickets inside a significant way. The newest recor in the nation graph or chart and #2 in the pop graphs, creating it the second maximum debut at that time of 2005 for a nation designer.
The boy of your , understands patience and dedication are important elements in relation to an effective profession- . His initial recording, Stay Me, generated the very best hits “All My Friends Say” and “Country Person,” although his hard work, Doin’ Thing, found the performer-about luke bryan in concert (just click lukebryantickets.flicense.com) three direct No. 6 single men and women: Else Phoning Is a Good Factor.”
During the slip of 2011, Tour: Bryan And which had a remarkable selection of , which includes Downtown. “It’s much like you’re acquiring a authorization to go to the next level, claims these musicians that had been a part of the Tourmore than in to a larger sized degree of designers.” It twisted as one of the most successful tours in the 15-season background.
Here is my web-site - where luke bryan is from [just click lukebryantickets.flicense.com]