Differences

This shows you the differences between two versions of the page.

Link to this comparison view

vv-lab:reorder [2015/02/18 22:59] (current)
egm created
Line 1: Line 1:
 +[[Concurrency Tool Comparison]]
  
 +== Model Description ==
 +
 +The benchmark contains a data race. There is a check whether the data race causes an inconsistency in the data values. When such an inconsistency is discovered an exception is raised. The benchmark contains two kinds of threads: '''​setter'''​ and '''​getter'''​which is the source of the data race.  The '''​getter'''​ thread also checks for the inconsistency in the data values that are caused by the
 +data race. As we increase the number of '''​setter'''​ threads while keeping the '''​getter'''​ thread constant, the semantic measure of the hardness in error discovery increases (i.e., get harder) as shown
 +in [[Hardness for Explicit State Software Model Checking Benchmarks]] paper.
 +
 +
 +Classes: 4
 +
 +SLOC: 44
 +
 +==Summary Results Table==
 +
 +{| align=center | border=1
 +|- bgcolor="​white"​ |
 +| colspan=5 align="​center"​ | reorder (1,1)
 +|- bgcolor="​lightblue"​
 +! Tool
 +! Trials (successful)
 +! Time
 +! Transition (paths)
 +! Max Depth (error depth)
 +|-
 +| JPF "​Randomized DFS"
 +| 100 (100) 1.00
 +| align="​right"​| 0.78s
 +| 83.69 (0.00)
 +| 18.32 (15.64)
 +|-
 +| JPF Stateless Random Walk
 +| 10125 (307) 0.03
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| CHESS
 +| 1 (1) 1.00
 +| align="​right"​| 0.78s
 +| 95.00 (5.00)
 +| 19.00 (NA)
 +|-
 +| ConTest
 +| 1000 (61) 0.06
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| CalFuzzer
 +| 100 (6) 0.06
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| JPF "​Abstraction Guided Refinement"​
 +| 1 (1) 1.00
 +| align="​right"​| 2.67s
 +| 29.00 (NA)
 +| 12.00 (12.00)
 +|- bgcolor="​white"​ |
 +| colspan=5 align="​center"​ | reorder (1,5)
 +|- bgcolor="​lightblue"​
 +! Tool
 +! Trials (successful)
 +! Time
 +! Transition (paths)
 +! Max Depth (error depth)
 +|-
 +| JPF "​Randomized DFS"
 +| 100 (100) 1.00
 +| align="​right"​| 16.61s
 +| 238450.80 (0.00)
 +| 52.94 (25.52)
 +|-
 +| JPF Stateless Random Walk
 +| 10128 (438) 0.04
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| CHESS
 +| 1 (0) 0.00
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| ConTest
 +| 1000 (285) 0.28
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| CalFuzzer
 +| 100 (8) 0.08
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| JPF "​Abstraction Guided Refinement"​
 +| 1 (1) 1.00
 +| align="​right"​| 2.67s
 +| 29.00 (NA)
 +| 12.00 (12.00)
 +|- bgcolor="​white"​ |
 +| colspan=5 align="​center"​ | reorder (10,1)
 +|- bgcolor="​lightblue"​
 +! Tool
 +! Trials (successful)
 +! Time
 +! Transition (paths)
 +! Max Depth (error depth)
 +|-
 +| JPF "​Randomized DFS"
 +| 5000 (4) 0.00
 +| align="​right"​| 2414.82s
 +| 38022689.25 (0.00)
 +| 89.75 (65.25)
 +|-
 +| JPF Stateless Random Walk
 +| 10127 (0) 0.00
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| CHESS
 +| 1 (0) 0.00
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| ConTest
 +| 1000 (0) 0.00
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| CalFuzzer
 +| 100 (9) 0.09
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| JPF "​Abstraction Guided Refinement"​
 +| 1 (1) 1.00
 +| align="​right"​| 1.67s
 +| 236.00 (NA)
 +| 30.00 (30.00)
 +|- bgcolor="​white"​ |
 +| colspan=5 align="​center"​ | reorder (2,1)
 +|- bgcolor="​lightblue"​
 +! Tool
 +! Trials (successful)
 +! Time
 +! Transition (paths)
 +! Max Depth (error depth)
 +|-
 +| JPF "​Randomized DFS"
 +| 100 (100) 1.00
 +| align="​right"​| 0.61s
 +| 560.58 (0.00)
 +| 28.33 (21.24)
 +|-
 +| JPF Stateless Random Walk
 +| 1000 (2) 0.00
 +| align="​right"​| 0.28s
 +| 27.50 (NA)
 +| 0.00 (NA)
 +|-
 +| CHESS
 +| 1 (1) 1.00
 +| align="​right"​| 0.44s
 +| 2600.00 (100.00)
 +| 26.00 (NA)
 +|-
 +| ConTest
 +| 1000 (23) 0.02
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| CalFuzzer
 +| 100 (12) 0.12
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| JPF "​Abstraction Guided Refinement"​
 +| 1 (1) 1.00
 +| align="​right"​| 1.67s
 +| 44.00 (NA)
 +| 14.00 (14.00)
 +|- bgcolor="​white"​ |
 +| colspan=5 align="​center"​ | reorder (3,1)
 +|- bgcolor="​lightblue"​
 +! Tool
 +! Trials (successful)
 +! Time
 +! Transition (paths)
 +! Max Depth (error depth)
 +|-
 +| JPF "​Randomized DFS"
 +| 100 (100) 1.00
 +| align="​right"​| 1.21s
 +| 3867.01 (0.00)
 +| 36.96 (27.26)
 +|-
 +| JPF Stateless Random Walk
 +| 1000 (1) 0.00
 +| align="​right"​| 0.21s
 +| 39.00 (NA)
 +| 0.00 (NA)
 +|-
 +| CHESS
 +| 1 (1) 1.00
 +| align="​right"​| 8.55s
 +| 66000.00 (2000.00)
 +| 33.00 (NA)
 +|-
 +| ConTest
 +| 1000 (7) 0.01
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| CalFuzzer
 +| 100 (11) 0.11
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| JPF "​Abstraction Guided Refinement"​
 +| 1 (1) 1.00
 +| align="​right"​| 1.67s
 +| 61.00 (NA)
 +| 16.00 (16.00)
 +|- bgcolor="​white"​ |
 +| colspan=5 align="​center"​ | reorder (3,3)
 +|- bgcolor="​lightblue"​
 +! Tool
 +! Trials (successful)
 +! Time
 +! Transition (paths)
 +! Max Depth (error depth)
 +|-
 +| JPF "​Randomized DFS"
 +| 100 (100) 1.00
 +| align="​right"​| 24.10s
 +| 216430.64 (0.00)
 +| 57.72 (29.79)
 +|-
 +| CHESS
 +| 1 (1) 1.00
 +| align="​right"​| 3454.61s
 +| 33516000.00 (684000.00)
 +| 49.00 (NA)
 +|-
 +| ConTest
 +| 1000 (20) 0.02
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| CalFuzzer
 +| 100 (36) 0.36
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| JPF "​Abstraction Guided Refinement"​
 +| 1 (1) 1.00
 +| align="​right"​| 1.67s
 +| 61.00 (NA)
 +| 16.00 (16.00)
 +|- bgcolor="​white"​ |
 +| colspan=5 align="​center"​ | reorder (4,1)
 +|- bgcolor="​lightblue"​
 +! Tool
 +! Trials (successful)
 +! Time
 +! Transition (paths)
 +! Max Depth (error depth)
 +|-
 +| JPF "​Randomized DFS"
 +| 100 (100) 1.00
 +| align="​right"​| 7.35s
 +| 33895.58 (0.00)
 +| 46.61 (34.29)
 +|-
 +| JPF Stateless Random Walk
 +| 1000 (0) 0.00
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| CHESS
 +| 1 (1) 1.00
 +| align="​right"​| 140.53s
 +| 1200000.00 (30000.00)
 +| 40.00 (NA)
 +|-
 +| ConTest
 +| 1000 (3) 0.00
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| CalFuzzer
 +| 100 (12) 0.12
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| JPF "​Abstraction Guided Refinement"​
 +| 1 (1) 1.00
 +| align="​right"​| 1.67s
 +| 80.00 (NA)
 +| 18.00 (18.00)
 +|- bgcolor="​white"​ |
 +| colspan=5 align="​center"​ | reorder (4,2)
 +|- bgcolor="​lightblue"​
 +! Tool
 +! Trials (successful)
 +! Time
 +! Transition (paths)
 +! Max Depth (error depth)
 +|-
 +| JPF "​Randomized DFS"
 +| 100 (100) 1.00
 +| align="​right"​| 19.46s
 +| 170146.30 (0.00)
 +| 56.39 (36.93)
 +|-
 +| JPF Stateless Random Walk
 +| 1000 (0) 0.00
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| CHESS
 +| 1 (1) 1.00
 +| align="​right"​| 2836.43s
 +| 26640000.00 (555000.00)
 +| 48.00 (NA)
 +|-
 +| ConTest
 +| 1000 (0) 0.00
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| CalFuzzer
 +| 100 (10) 0.10
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| JPF "​Abstraction Guided Refinement"​
 +| 1 (1) 1.00
 +| align="​right"​| 1.67s
 +| 80.00 (NA)
 +| 18.00 (18.00)
 +|- bgcolor="​white"​ |
 +| colspan=5 align="​center"​ | reorder (5,1)
 +|- bgcolor="​lightblue"​
 +! Tool
 +! Trials (successful)
 +! Time
 +! Transition (paths)
 +! Max Depth (error depth)
 +|-
 +| JPF "​Randomized DFS"
 +| 100 (100) 1.00
 +| align="​right"​| 9.98s
 +| 107466.97 (0.00)
 +| 49.72 (37.57)
 +|-
 +| JPF Stateless Random Walk
 +| 10128 (0) 0.00
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| CHESS
 +| 1 (1) 1.00
 +| align="​right"​| 2275.93s
 +| 20633000.00 (439000.00)
 +| 47.00 (NA)
 +|-
 +| ConTest
 +| 1000 (3) 0.00
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| CalFuzzer
 +| 100 (12) 0.12
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| JPF "​Abstraction Guided Refinement"​
 +| 1 (1) 1.00
 +| align="​right"​| 1.67s
 +| 101.00 (NA)
 +| 20.00 (20.00)
 +|- bgcolor="​white"​ |
 +| colspan=5 align="​center"​ | reorder (5,2)
 +|- bgcolor="​lightblue"​
 +! Tool
 +! Trials (successful)
 +! Time
 +! Transition (paths)
 +! Max Depth (error depth)
 +|-
 +| JPF "​Randomized DFS"
 +| 100 (100) 1.00
 +| align="​right"​| 85.43s
 +| 996758.65 (0.00)
 +| 65.17 (42.90)
 +|-
 +| JPF Stateless Random Walk
 +| 1000 (0) 0.00
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| CHESS
 +| 1 (0) 0.00
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| ConTest
 +| 1000 (10) 0.01
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| CalFuzzer
 +| 100 (24) 0.24
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| JPF "​Abstraction Guided Refinement"​
 +| 1 (1) 1.00
 +| align="​right"​| 1.67s
 +| 101.00 (NA)
 +| 20.00 (20.00)
 +|- bgcolor="​white"​ |
 +| colspan=5 align="​center"​ | reorder (5,5)
 +|- bgcolor="​lightblue"​
 +! Tool
 +! Trials (successful)
 +! Time
 +! Transition (paths)
 +! Max Depth (error depth)
 +|-
 +| JPF "​Randomized DFS"
 +| 117 (2) 0.02
 +| align="​right"​| 2206.74s
 +| 31429172.50 (0.00)
 +| 86.00 (44.00)
 +|- bgcolor="​white"​ |
 +| colspan=5 align="​center"​ | reorder (6,1)
 +|- bgcolor="​lightblue"​
 +! Tool
 +! Trials (successful)
 +! Time
 +! Transition (paths)
 +! Max Depth (error depth)
 +|-
 +| JPF "​Randomized DFS"
 +| 100 (100) 1.00
 +| align="​right"​| 73.05s
 +| 833413.61 (0.00)
 +| 64.44 (46.18)
 +|-
 +| JPF Stateless Random Walk
 +| 1000 (0) 0.00
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| CHESS
 +| 1 (0) 0.00
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| ConTest
 +| 1000 (0) 0.00
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| CalFuzzer
 +| 100 (10) 0.10
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| JPF "​Abstraction Guided Refinement"​
 +| 1 (1) 1.00
 +| align="​right"​| 1.67s
 +| 124.00 (NA)
 +| 22.00 (22.00)
 +|- bgcolor="​white"​ |
 +| colspan=5 align="​center"​ | reorder (7,1)
 +|- bgcolor="​lightblue"​
 +! Tool
 +! Trials (successful)
 +! Time
 +! Transition (paths)
 +! Max Depth (error depth)
 +|-
 +| JPF "​Randomized DFS"
 +| 5000 (5000) 1.00
 +| align="​right"​| 213.90s
 +| 3586188.11 (0.00)
 +| 65.62 (48.65)
 +|- bgcolor="​white"​ |
 +| colspan=5 align="​center"​ | reorder (8,1)
 +|- bgcolor="​lightblue"​
 +! Tool
 +! Trials (successful)
 +! Time
 +! Transition (paths)
 +! Max Depth (error depth)
 +|-
 +| JPF "​Randomized DFS"
 +| 5000 (5000) 1.00
 +| align="​right"​| 1225.99s
 +| 20078513.07 (0.00)
 +| 73.59 (54.07)
 +|-
 +| JPF Stateless Random Walk
 +| 10126 (0) 0.00
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| CHESS
 +| 1 (0) 0.00
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| ConTest
 +| 1000 (0) 0.00
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| CalFuzzer
 +| 100 (10) 0.10
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| JPF "​Abstraction Guided Refinement"​
 +| 1 (1) 1.00
 +| align="​right"​| 1.67s
 +| 176.00 (NA)
 +| 26.00 (26.00)
 +|- bgcolor="​white"​ |
 +| colspan=5 align="​center"​ | reorder (9,1)
 +|- bgcolor="​lightblue"​
 +! Tool
 +! Trials (successful)
 +! Time
 +! Transition (paths)
 +! Max Depth (error depth)
 +|-
 +| JPF "​Randomized DFS"
 +| 5000 (593) 0.12
 +| align="​right"​| 2497.98s
 +| 38704112.56 (0.00)
 +| 81.67 (59.64)
 +|-
 +| JPF Stateless Random Walk
 +| 10124 (0) 0.00
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| CHESS
 +| 1 (0) 0.00
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| ConTest
 +| 1000 (10) 0.01
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| CalFuzzer
 +| 100 (10) 0.10
 +| align="​right"​| NA
 +| NA (NA)
 +| NA (NA)
 +|-
 +| JPF "​Abstraction Guided Refinement"​
 +| 1 (1) 1.00
 +| align="​right"​| 1.67s
 +| 205.00 (NA)
 +| 28.00 (28.00)
 +|}
 +==Chess==
 +
 +* ChessBound=2
 +* ChessMonitorVolatiles=true
 +
 +{| align=center | border=1 | -bgcolor=grey
 +! Reorder Params !! Paths !! Found Error !! Time Taken !! Steps (Depth)
 +|- 
 +| 1,1 || 5     || Yes || 0.78 secs || 19
 +|-
 +| 2,1 || >=100 || Yes || 0.436 secs || 26
 +|-
 +| 3,1 || >=2000 || Yes || 8.549 secs || 33 
 +|-
 +| 4,1 || >=30,000 || Yes || 140.526 secs || 40
 +|- 
 +| 5,1 || >​=439,​000 || Yes || 2275.930 secs || 47
 +|-
 +| 4,2 || >= 555,000 || Yes || 2836.426 secs || 48
 +|-
 +| 3,3 || >= 684,000 || Yes || 3454.611 secs || 49 
 +|-
 +| 1,5 || >= 708,000 || No || 3600.909 secs || 51 
 +|-
 +| 6,1 || >​=652,​000 || No || 3603.998 secs || 54
 +|-
 +| 5,2 || >​=602,​000 || No || 3600.612 secs  || 55
 +|-
 +| 8,1 || >​=579,​000 || No || 3605.120 secs || 68
 +|-
 +| 9,1 || >​=538,​000 || No || 3605.823 secs || 75
 +|-
 +| 10,1 || >​=499,​000 || No || 3600.534 secs || 82
 +|}
 +
 +==Java Pathfinder==
 +===Stateless Random Walk===
 +
 +* 10,000 Trials of Random Walk
 +* The ratio of the error discovery trials over the total number of trials is the path error density. ​
 +
 +{| align=center | border=1 | -bgcolor=grey
 +! Reorder Params !! Path Error Density
 +|- 
 +| 1,1 || 0.030
 +|-
 +| 1,5 || 0.043
 +|-
 +| 5,1 || 0.0
 +|-
 +| 8,1 || 0.0
 +|-
 +| 9,1 || 0.0
 +|-
 +| 10,1|| 0.0
 +|}
 +
 +===Stateful Randomized Depth-first search===
 +
 +* 100 randomized depth-first search trial, where each trial is time-bounded at
 +one hour. A total of 100 computation hours dedicated in discovering the error. ​
 +
 +* 2 MB of RAM allocated on each machine
 +
 +* The Error Discovering Trials is the number of trials that discovered an error
 +among the initially launched trials (100). ​
 +
 +{| align="​center"​ | border=1
 +|- bgcolor="​lightblue"​
 +! Parameters
 +! Trials (successful)
 +! Time
 +! Transition (paths)
 +! Max Depth (error depth)
 +!  abs time (ms)
 +!  rel. time (ms)
 +!  search depth
 +!  errorDepth
 +!  new states
 +!  revisited states
 +!  end states
 +!  backtracks
 +!  processed states
 +!  restored states
 +!  total memory (kB)
 +!  max total memory
 +!  free memory (kB)
 +!  heap objects
 +!  max heap objects
 +
 +|-
 +|(1,1)
 +| 100 (100) 100.00%
 +| align="​right"​| 0.78s
 +| 83.69 (0.00)
 +| 18.32 (15.64)
 +| align="​right"​ | 783.57
 +| align="​right"​ | 783.57
 +| align="​right"​ | 18.32
 +| align="​right"​ | 15.64
 +| align="​right"​ | 55.16
 +| align="​right"​ | 28.53
 +| align="​right"​ | 0.99
 +| align="​right"​ | 67.05
 +| align="​right"​ | 37.53
 +| align="​right"​ | 0.00
 +| align="​right"​ | 14001.28
 +| align="​right"​ | 6990528.00
 +| align="​right"​ | 0.00
 +| align="​right"​ | 174.00
 +| align="​right"​ | 174.00
 +|-
 +|(1,5)
 +| 100 (100) 100.00%
 +| align="​right"​| 16.61s
 +| 238450.80 (0.00)
 +| 52.94 (25.52)
 +| align="​right"​ | 16608.03
 +| align="​right"​ | 5007.11
 +| align="​right"​ | 52.94
 +| align="​right"​ | 25.52
 +| align="​right"​ | 49151.07
 +| align="​right"​ | 189299.73
 +| align="​right"​ | 0.98
 +| align="​right"​ | 238424.28
 +| align="​right"​ | 49123.57
 +| align="​right"​ | 0.00
 +| align="​right"​ | 244161.92
 +| align="​right"​ | 6990528.00
 +| align="​right"​ | 80162.71
 +| align="​right"​ | 185.26
 +| align="​right"​ | 185.26
 +|-
 +|(8,1)
 +| 5000 (5000) 100.00%
 +| align="​right"​| 1225.99s
 +| 20078513.07 (0.00)
 +| 73.59 (54.07)
 +| align="​right"​ | 1225990.15
 +| align="​right"​ | 5039.40
 +| align="​right"​ | 73.59
 +| align="​right"​ | 54.07
 +| align="​right"​ | 2874234.15
 +| align="​right"​ | 17204278.92
 +| align="​right"​ | 1.00
 +| align="​right"​ | 20078458.00
 +| align="​right"​ | 2874178.08
 +| align="​right"​ | 0.00
 +| align="​right"​ | 426544.46
 +| align="​right"​ | 932096.00
 +| align="​right"​ | 187472.32
 +| align="​right"​ | 202.00
 +| align="​right"​ | 202.00
 +|-
 +|(2,1)
 +| 100 (100) 100.00%
 +| align="​right"​| 0.61s
 +| 560.58 (0.00)
 +| 28.33 (21.24)
 +| align="​right"​ | 613.41
 +| align="​right"​ | 613.41
 +| align="​right"​ | 28.33
 +| align="​right"​ | 21.24
 +| align="​right"​ | 255.66
 +| align="​right"​ | 304.92
 +| align="​right"​ | 1.00
 +| align="​right"​ | 538.34
 +| align="​right"​ | 233.00
 +| align="​right"​ | 0.00
 +| align="​right"​ | 22846.08
 +| align="​right"​ | 932096.00
 +| align="​right"​ | 0.00
 +| align="​right"​ | 252.46
 +| align="​right"​ | 252.46
 +|-
 +|(9,1)
 +| 5000 (593) 11.86%
 +| align="​right"​| 2497.98s
 +| 38704112.56 (0.00)
 +| 81.67 (59.64)
 +| align="​right"​ | 2497980.25
 +| align="​right"​ | 5102.36
 +| align="​right"​ | 81.67
 +| align="​right"​ | 59.64
 +| align="​right"​ | 5223853.76
 +| align="​right"​ | 33480258.80
 +| align="​right"​ | 1.00
 +| align="​right"​ | 38704051.92
 +| align="​right"​ | 5223792.11
 +| align="​right"​ | 0.00
 +| align="​right"​ | 475022.89
 +| align="​right"​ | 932096.00
 +| align="​right"​ | 198919.75
 +| align="​right"​ | 206.00
 +| align="​right"​ | 206.00
 +|-
 +|(3,1)
 +| 100 (100) 100.00%
 +| align="​right"​| 1.21s
 +| 3867.01 (0.00)
 +| 36.96 (27.26)
 +| align="​right"​ | 1211.19
 +| align="​right"​ | 1211.19
 +| align="​right"​ | 36.96
 +| align="​right"​ | 27.26
 +| align="​right"​ | 1285.54
 +| align="​right"​ | 2581.47
 +| align="​right"​ | 1.00
 +| align="​right"​ | 3838.75
 +| align="​right"​ | 1256.97
 +| align="​right"​ | 0.00
 +| align="​right"​ | 30968.96
 +| align="​right"​ | 932096.00
 +| align="​right"​ | 0.00
 +| align="​right"​ | 255.03
 +| align="​right"​ | 255.03
 +|-
 +|(3,3)
 +| 100 (100) 100.00%
 +| align="​right"​| 24.10s
 +| 216430.64 (0.00)
 +| 57.72 (29.79)
 +| align="​right"​ | 24102.54
 +| align="​right"​ | 6396.58
 +| align="​right"​ | 57.72
 +| align="​right"​ | 29.79
 +| align="​right"​ | 47500.16
 +| align="​right"​ | 168930.48
 +| align="​right"​ | 1.00
 +| align="​right"​ | 216399.85
 +| align="​right"​ | 47468.86
 +| align="​right"​ | 0.00
 +| align="​right"​ | 111149.44
 +| align="​right"​ | 932096.00
 +| align="​right"​ | 48094.12
 +| align="​right"​ | 272.41
 +| align="​right"​ | 278.00
 +|-
 +|(4,1)
 +| 100 (100) 100.00%
 +| align="​right"​| 7.35s
 +| 33895.58 (0.00)
 +| 46.61 (34.29)
 +| align="​right"​ | 7348.58
 +| align="​right"​ | 6948.40
 +| align="​right"​ | 46.61
 +| align="​right"​ | 34.29
 +| align="​right"​ | 9516.88
 +| align="​right"​ | 24378.70
 +| align="​right"​ | 1.00
 +| align="​right"​ | 33860.29
 +| align="​right"​ | 9481.27
 +| align="​right"​ | 0.00
 +| align="​right"​ | 61336.96
 +| align="​right"​ | 932096.00
 +| align="​right"​ | 879.70
 +| align="​right"​ | 274.16
 +| align="​right"​ | 274.16
 +|-
 +|(4,2)
 +| 100 (100) 100.00%
 +| align="​right"​| 19.46s
 +| 170146.30 (0.00)
 +| 56.39 (36.93)
 +| align="​right"​ | 19455.02
 +| align="​right"​ | 4850.36
 +| align="​right"​ | 56.39
 +| align="​right"​ | 36.93
 +| align="​right"​ | 37562.55
 +| align="​right"​ | 132583.75
 +| align="​right"​ | 1.00
 +| align="​right"​ | 170108.37
 +| align="​right"​ | 37524.15
 +| align="​right"​ | 0.00
 +| align="​right"​ | 98801.92
 +| align="​right"​ | 932096.00
 +| align="​right"​ | 42491.61
 +| align="​right"​ | 279.55
 +| align="​right"​ | 279.93
 +|-
 +|(10,1)
 +| 5000 (4) 0.08%
 +| align="​right"​| 2414.82s
 +| 38022689.25 (0.00)
 +| 89.75 (65.25)
 +| align="​right"​ | 2414821.25
 +| align="​right"​ | 3641.25
 +| align="​right"​ | 89.75
 +| align="​right"​ | 65.25
 +| align="​right"​ | 4947236.75
 +| align="​right"​ | 33075452.50
 +| align="​right"​ | 1.00
 +| align="​right"​ | 38022623.00
 +| align="​right"​ | 4947169.50
 +| align="​right"​ | 0.00
 +| align="​right"​ | 471712.00
 +| align="​right"​ | 932096.00
 +| align="​right"​ | 212632.75
 +| align="​right"​ | 210.00
 +| align="​right"​ | 210.00
 +|-
 +|(5,1)
 +| 100 (100) 100.00%
 +| align="​right"​| 9.98s
 +| 107466.97 (0.00)
 +| 49.72 (37.57)
 +| align="​right"​ | 9984.68
 +| align="​right"​ | 4383.55
 +| align="​right"​ | 49.72
 +| align="​right"​ | 37.57
 +| align="​right"​ | 23411.50
 +| align="​right"​ | 84055.47
 +| align="​right"​ | 1.00
 +| align="​right"​ | 107428.40
 +| align="​right"​ | 23371.93
 +| align="​right"​ | 0.00
 +| align="​right"​ | 147628.80
 +| align="​right"​ | 6990528.00
 +| align="​right"​ | 45776.70
 +| align="​right"​ | 190.00
 +| align="​right"​ | 190.00
 +|-
 +|(5,2)
 +| 100 (100) 100.00%
 +| align="​right"​| 85.43s
 +| 996758.65 (0.00)
 +| 65.17 (42.90)
 +| align="​right"​ | 85426.04
 +| align="​right"​ | 5003.09
 +| align="​right"​ | 65.17
 +| align="​right"​ | 42.90
 +| align="​right"​ | 184139.71
 +| align="​right"​ | 812618.94
 +| align="​right"​ | 1.00
 +| align="​right"​ | 996714.75
 +| align="​right"​ | 184095.52
 +| align="​right"​ | 0.00
 +| align="​right"​ | 155769.60
 +| align="​right"​ | 932096.00
 +| align="​right"​ | 45871.92
 +| align="​right"​ | 281.11
 +| align="​right"​ | 281.67
 +|-
 +|(5,5)
 +| 117 (2) 1.71%
 +| align="​right"​| 2206.74s
 +| 31429172.50 (0.00)
 +| 86.00 (44.00)
 +| align="​right"​ | 2206738.50
 +| align="​right"​ | 2245.00
 +| align="​right"​ | 86.00
 +| align="​right"​ | 44.00
 +| align="​right"​ | 4352520.00
 +| align="​right"​ | 27076652.50
 +| align="​right"​ | 1.00
 +| align="​right"​ | 31429127.50
 +| align="​right"​ | 4352474.00
 +| align="​right"​ | 0.00
 +| align="​right"​ | 2576352.00
 +| align="​right"​ | 6990528.00
 +| align="​right"​ | 179173.00
 +| align="​right"​ | 201.00
 +| align="​right"​ | 201.00
 +|-
 +|(6,1)
 +| 100 (100) 100.00%
 +| align="​right"​| 73.05s
 +| 833413.61 (0.00)
 +| 64.44 (46.18)
 +| align="​right"​ | 73054.57
 +| align="​right"​ | 4832.70
 +| align="​right"​ | 64.44
 +| align="​right"​ | 46.18
 +| align="​right"​ | 155272.79
 +| align="​right"​ | 678140.82
 +| align="​right"​ | 1.00
 +| align="​right"​ | 833366.43
 +| align="​right"​ | 155225.40
 +| align="​right"​ | 0.00
 +| align="​right"​ | 157378.56
 +| align="​right"​ | 932096.00
 +| align="​right"​ | 49872.47
 +| align="​right"​ | 280.73
 +| align="​right"​ | 280.73
 +|-
 +|(7,1)
 +| 5000 (5000) 100.00%
 +| align="​right"​| 213.90s
 +| 3586188.11 (0.00)
 +| 65.62 (48.65)
 +| align="​right"​ | 213896.72
 +| align="​right"​ | 4963.18
 +| align="​right"​ | 65.62
 +| align="​right"​ | 48.65
 +| align="​right"​ | 579468.72
 +| align="​right"​ | 3006719.39
 +| align="​right"​ | 1.00
 +| align="​right"​ | 3586138.46
 +| align="​right"​ | 579418.07
 +| align="​right"​ | 0.00
 +| align="​right"​ | 369729.37
 +| align="​right"​ | 932096.00
 +| align="​right"​ | 172409.01
 +| align="​right"​ | 198.00
 +| align="​right"​ | 198.00
 +|}
 +
 +
 +==ConTest==
 +
 +* 1000 independent trials
 +
 +* The ratio of the error discovery trials over the total number of trials is the 
 +path error density. ​
 +
 +{| align=center | border=1 | -bgcolor=grey
 +! Reorder Params !! Path Error Density
 +|- 
 +| 1,1 || 0.061
 +|-
 +| 2,1 || 0.023
 +|-
 +| 3,1 || 0.007
 +|-
 +| 4,1 || 0.003
 +|-
 +| 5,1 || 0.003
 +|-
 +| 6,1 || 0.000
 +|-
 +| 1,5 || 0.285
 +|-
 +| 4,2 || 0.00
 +|-
 +| 3,3 || 0.02
 +|-
 +| 5,2 || 0.01
 +|-
 +| 8,1 || 0.00
 +|-
 +| 9,1 || 0.01
 +|-
 +| 10,1 || 0.00
 +|}
 +
 +==CalFuzzer==
 +
 +* 100 independent trials
 +
 +* The ratio of the error discovery trials over the total number of trials is the 
 +path error density. ​
 +
 +{| align=center | border=1 | -bgcolor=grey
 +! Reorder Params !! Path Error Density
 +|- 
 +| 1,1 || 0.06
 +|-
 +| 2,1 || 0.12 
 +|-
 +| 3,1 || 0.11
 +|-
 +| 4,1 || 0.12
 +|-
 +| 4,2 || 0.10
 +|-
 +| 5,1 || 0.12
 +|-
 +| 3,3 || 0.36
 +|-
 +| 6,1 || 0.10
 +|-
 +| 5,2 || 0.24
 +|-
 +| 1,5 || 0.08
 +|-
 +| 8,1 || 0.10
 +|-
 +| 9,1 || 0.10
 +|-
 +| 10,1 || 0.09
 +|}
vv-lab/reorder.txt ยท Last modified: 2015/02/18 22:59 by egm
Back to top
CC Attribution-Share Alike 4.0 International
chimeric.de = chi`s home Valid CSS Driven by DokuWiki do yourself a favour and use a real browser - get firefox!! Recent changes RSS feed Valid XHTML 1.0