[[Concurrency Tool Comparison]]

== Model Description ==

The benchmark contains a data race. There is a check whether the data race causes an inconsistency in the data values. When such an inconsistency is discovered an exception is raised. The benchmark contains two kinds of threads: '''setter''' and '''getter'''which is the source of the data race.  The '''getter''' thread also checks for the inconsistency in the data values that are caused by the
data race. As we increase the number of '''setter''' threads while keeping the '''getter''' thread constant, the semantic measure of the hardness in error discovery increases (i.e., get harder) as shown
in [[Hardness for Explicit State Software Model Checking Benchmarks]] paper.


Classes: 4

SLOC: 44

==Summary Results Table==

{| align=center | border=1
|- bgcolor="white" |
| colspan=5 align="center" | reorder (1,1)
|- bgcolor="lightblue"
! Tool
! Trials (successful)
! Time
! Transition (paths)
! Max Depth (error depth)
|-
| JPF "Randomized DFS"
| 100 (100) 1.00
| align="right"| 0.78s
| 83.69 (0.00)
| 18.32 (15.64)
|-
| JPF Stateless Random Walk
| 10125 (307) 0.03
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| CHESS
| 1 (1) 1.00
| align="right"| 0.78s
| 95.00 (5.00)
| 19.00 (NA)
|-
| ConTest
| 1000 (61) 0.06
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| CalFuzzer
| 100 (6) 0.06
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| JPF "Abstraction Guided Refinement"
| 1 (1) 1.00
| align="right"| 2.67s
| 29.00 (NA)
| 12.00 (12.00)
|- bgcolor="white" |
| colspan=5 align="center" | reorder (1,5)
|- bgcolor="lightblue"
! Tool
! Trials (successful)
! Time
! Transition (paths)
! Max Depth (error depth)
|-
| JPF "Randomized DFS"
| 100 (100) 1.00
| align="right"| 16.61s
| 238450.80 (0.00)
| 52.94 (25.52)
|-
| JPF Stateless Random Walk
| 10128 (438) 0.04
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| CHESS
| 1 (0) 0.00
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| ConTest
| 1000 (285) 0.28
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| CalFuzzer
| 100 (8) 0.08
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| JPF "Abstraction Guided Refinement"
| 1 (1) 1.00
| align="right"| 2.67s
| 29.00 (NA)
| 12.00 (12.00)
|- bgcolor="white" |
| colspan=5 align="center" | reorder (10,1)
|- bgcolor="lightblue"
! Tool
! Trials (successful)
! Time
! Transition (paths)
! Max Depth (error depth)
|-
| JPF "Randomized DFS"
| 5000 (4) 0.00
| align="right"| 2414.82s
| 38022689.25 (0.00)
| 89.75 (65.25)
|-
| JPF Stateless Random Walk
| 10127 (0) 0.00
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| CHESS
| 1 (0) 0.00
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| ConTest
| 1000 (0) 0.00
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| CalFuzzer
| 100 (9) 0.09
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| JPF "Abstraction Guided Refinement"
| 1 (1) 1.00
| align="right"| 1.67s
| 236.00 (NA)
| 30.00 (30.00)
|- bgcolor="white" |
| colspan=5 align="center" | reorder (2,1)
|- bgcolor="lightblue"
! Tool
! Trials (successful)
! Time
! Transition (paths)
! Max Depth (error depth)
|-
| JPF "Randomized DFS"
| 100 (100) 1.00
| align="right"| 0.61s
| 560.58 (0.00)
| 28.33 (21.24)
|-
| JPF Stateless Random Walk
| 1000 (2) 0.00
| align="right"| 0.28s
| 27.50 (NA)
| 0.00 (NA)
|-
| CHESS
| 1 (1) 1.00
| align="right"| 0.44s
| 2600.00 (100.00)
| 26.00 (NA)
|-
| ConTest
| 1000 (23) 0.02
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| CalFuzzer
| 100 (12) 0.12
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| JPF "Abstraction Guided Refinement"
| 1 (1) 1.00
| align="right"| 1.67s
| 44.00 (NA)
| 14.00 (14.00)
|- bgcolor="white" |
| colspan=5 align="center" | reorder (3,1)
|- bgcolor="lightblue"
! Tool
! Trials (successful)
! Time
! Transition (paths)
! Max Depth (error depth)
|-
| JPF "Randomized DFS"
| 100 (100) 1.00
| align="right"| 1.21s
| 3867.01 (0.00)
| 36.96 (27.26)
|-
| JPF Stateless Random Walk
| 1000 (1) 0.00
| align="right"| 0.21s
| 39.00 (NA)
| 0.00 (NA)
|-
| CHESS
| 1 (1) 1.00
| align="right"| 8.55s
| 66000.00 (2000.00)
| 33.00 (NA)
|-
| ConTest
| 1000 (7) 0.01
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| CalFuzzer
| 100 (11) 0.11
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| JPF "Abstraction Guided Refinement"
| 1 (1) 1.00
| align="right"| 1.67s
| 61.00 (NA)
| 16.00 (16.00)
|- bgcolor="white" |
| colspan=5 align="center" | reorder (3,3)
|- bgcolor="lightblue"
! Tool
! Trials (successful)
! Time
! Transition (paths)
! Max Depth (error depth)
|-
| JPF "Randomized DFS"
| 100 (100) 1.00
| align="right"| 24.10s
| 216430.64 (0.00)
| 57.72 (29.79)
|-
| CHESS
| 1 (1) 1.00
| align="right"| 3454.61s
| 33516000.00 (684000.00)
| 49.00 (NA)
|-
| ConTest
| 1000 (20) 0.02
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| CalFuzzer
| 100 (36) 0.36
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| JPF "Abstraction Guided Refinement"
| 1 (1) 1.00
| align="right"| 1.67s
| 61.00 (NA)
| 16.00 (16.00)
|- bgcolor="white" |
| colspan=5 align="center" | reorder (4,1)
|- bgcolor="lightblue"
! Tool
! Trials (successful)
! Time
! Transition (paths)
! Max Depth (error depth)
|-
| JPF "Randomized DFS"
| 100 (100) 1.00
| align="right"| 7.35s
| 33895.58 (0.00)
| 46.61 (34.29)
|-
| JPF Stateless Random Walk
| 1000 (0) 0.00
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| CHESS
| 1 (1) 1.00
| align="right"| 140.53s
| 1200000.00 (30000.00)
| 40.00 (NA)
|-
| ConTest
| 1000 (3) 0.00
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| CalFuzzer
| 100 (12) 0.12
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| JPF "Abstraction Guided Refinement"
| 1 (1) 1.00
| align="right"| 1.67s
| 80.00 (NA)
| 18.00 (18.00)
|- bgcolor="white" |
| colspan=5 align="center" | reorder (4,2)
|- bgcolor="lightblue"
! Tool
! Trials (successful)
! Time
! Transition (paths)
! Max Depth (error depth)
|-
| JPF "Randomized DFS"
| 100 (100) 1.00
| align="right"| 19.46s
| 170146.30 (0.00)
| 56.39 (36.93)
|-
| JPF Stateless Random Walk
| 1000 (0) 0.00
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| CHESS
| 1 (1) 1.00
| align="right"| 2836.43s
| 26640000.00 (555000.00)
| 48.00 (NA)
|-
| ConTest
| 1000 (0) 0.00
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| CalFuzzer
| 100 (10) 0.10
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| JPF "Abstraction Guided Refinement"
| 1 (1) 1.00
| align="right"| 1.67s
| 80.00 (NA)
| 18.00 (18.00)
|- bgcolor="white" |
| colspan=5 align="center" | reorder (5,1)
|- bgcolor="lightblue"
! Tool
! Trials (successful)
! Time
! Transition (paths)
! Max Depth (error depth)
|-
| JPF "Randomized DFS"
| 100 (100) 1.00
| align="right"| 9.98s
| 107466.97 (0.00)
| 49.72 (37.57)
|-
| JPF Stateless Random Walk
| 10128 (0) 0.00
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| CHESS
| 1 (1) 1.00
| align="right"| 2275.93s
| 20633000.00 (439000.00)
| 47.00 (NA)
|-
| ConTest
| 1000 (3) 0.00
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| CalFuzzer
| 100 (12) 0.12
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| JPF "Abstraction Guided Refinement"
| 1 (1) 1.00
| align="right"| 1.67s
| 101.00 (NA)
| 20.00 (20.00)
|- bgcolor="white" |
| colspan=5 align="center" | reorder (5,2)
|- bgcolor="lightblue"
! Tool
! Trials (successful)
! Time
! Transition (paths)
! Max Depth (error depth)
|-
| JPF "Randomized DFS"
| 100 (100) 1.00
| align="right"| 85.43s
| 996758.65 (0.00)
| 65.17 (42.90)
|-
| JPF Stateless Random Walk
| 1000 (0) 0.00
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| CHESS
| 1 (0) 0.00
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| ConTest
| 1000 (10) 0.01
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| CalFuzzer
| 100 (24) 0.24
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| JPF "Abstraction Guided Refinement"
| 1 (1) 1.00
| align="right"| 1.67s
| 101.00 (NA)
| 20.00 (20.00)
|- bgcolor="white" |
| colspan=5 align="center" | reorder (5,5)
|- bgcolor="lightblue"
! Tool
! Trials (successful)
! Time
! Transition (paths)
! Max Depth (error depth)
|-
| JPF "Randomized DFS"
| 117 (2) 0.02
| align="right"| 2206.74s
| 31429172.50 (0.00)
| 86.00 (44.00)
|- bgcolor="white" |
| colspan=5 align="center" | reorder (6,1)
|- bgcolor="lightblue"
! Tool
! Trials (successful)
! Time
! Transition (paths)
! Max Depth (error depth)
|-
| JPF "Randomized DFS"
| 100 (100) 1.00
| align="right"| 73.05s
| 833413.61 (0.00)
| 64.44 (46.18)
|-
| JPF Stateless Random Walk
| 1000 (0) 0.00
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| CHESS
| 1 (0) 0.00
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| ConTest
| 1000 (0) 0.00
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| CalFuzzer
| 100 (10) 0.10
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| JPF "Abstraction Guided Refinement"
| 1 (1) 1.00
| align="right"| 1.67s
| 124.00 (NA)
| 22.00 (22.00)
|- bgcolor="white" |
| colspan=5 align="center" | reorder (7,1)
|- bgcolor="lightblue"
! Tool
! Trials (successful)
! Time
! Transition (paths)
! Max Depth (error depth)
|-
| JPF "Randomized DFS"
| 5000 (5000) 1.00
| align="right"| 213.90s
| 3586188.11 (0.00)
| 65.62 (48.65)
|- bgcolor="white" |
| colspan=5 align="center" | reorder (8,1)
|- bgcolor="lightblue"
! Tool
! Trials (successful)
! Time
! Transition (paths)
! Max Depth (error depth)
|-
| JPF "Randomized DFS"
| 5000 (5000) 1.00
| align="right"| 1225.99s
| 20078513.07 (0.00)
| 73.59 (54.07)
|-
| JPF Stateless Random Walk
| 10126 (0) 0.00
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| CHESS
| 1 (0) 0.00
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| ConTest
| 1000 (0) 0.00
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| CalFuzzer
| 100 (10) 0.10
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| JPF "Abstraction Guided Refinement"
| 1 (1) 1.00
| align="right"| 1.67s
| 176.00 (NA)
| 26.00 (26.00)
|- bgcolor="white" |
| colspan=5 align="center" | reorder (9,1)
|- bgcolor="lightblue"
! Tool
! Trials (successful)
! Time
! Transition (paths)
! Max Depth (error depth)
|-
| JPF "Randomized DFS"
| 5000 (593) 0.12
| align="right"| 2497.98s
| 38704112.56 (0.00)
| 81.67 (59.64)
|-
| JPF Stateless Random Walk
| 10124 (0) 0.00
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| CHESS
| 1 (0) 0.00
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| ConTest
| 1000 (10) 0.01
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| CalFuzzer
| 100 (10) 0.10
| align="right"| NA
| NA (NA)
| NA (NA)
|-
| JPF "Abstraction Guided Refinement"
| 1 (1) 1.00
| align="right"| 1.67s
| 205.00 (NA)
| 28.00 (28.00)
|}
==Chess==

* ChessBound=2
* ChessMonitorVolatiles=true

{| align=center | border=1 | -bgcolor=grey
! Reorder Params !! Paths !! Found Error !! Time Taken !! Steps (Depth)
|- 
| 1,1 || 5     || Yes || 0.78 secs || 19
|-
| 2,1 || >=100 || Yes || 0.436 secs || 26
|-
| 3,1 || >=2000 || Yes || 8.549 secs || 33 
|-
| 4,1 || >=30,000 || Yes || 140.526 secs || 40
|- 
| 5,1 || >=439,000 || Yes || 2275.930 secs || 47
|-
| 4,2 || >= 555,000 || Yes || 2836.426 secs || 48
|-
| 3,3 || >= 684,000 || Yes || 3454.611 secs || 49 
|-
| 1,5 || >= 708,000 || No || 3600.909 secs || 51 
|-
| 6,1 || >=652,000 || No || 3603.998 secs || 54
|-
| 5,2 || >=602,000 || No || 3600.612 secs  || 55
|-
| 8,1 || >=579,000 || No || 3605.120 secs || 68
|-
| 9,1 || >=538,000 || No || 3605.823 secs || 75
|-
| 10,1 || >=499,000 || No || 3600.534 secs || 82
|}

==Java Pathfinder==
===Stateless Random Walk===

* 10,000 Trials of Random Walk
* The ratio of the error discovery trials over the total number of trials is the path error density. 

{| align=center | border=1 | -bgcolor=grey
! Reorder Params !! Path Error Density
|- 
| 1,1 || 0.030
|-
| 1,5 || 0.043
|-
| 5,1 || 0.0
|-
| 8,1 || 0.0
|-
| 9,1 || 0.0
|-
| 10,1|| 0.0
|}

===Stateful Randomized Depth-first search===

* 100 randomized depth-first search trial, where each trial is time-bounded at
one hour. A total of 100 computation hours dedicated in discovering the error. 

* 2 MB of RAM allocated on each machine

* The Error Discovering Trials is the number of trials that discovered an error
among the initially launched trials (100). 

{| align="center" | border=1
|- bgcolor="lightblue"
! Parameters
! Trials (successful)
! Time
! Transition (paths)
! Max Depth (error depth)
!  abs time (ms)
!  rel. time (ms)
!  search depth
!  errorDepth
!  new states
!  revisited states
!  end states
!  backtracks
!  processed states
!  restored states
!  total memory (kB)
!  max total memory
!  free memory (kB)
!  heap objects
!  max heap objects

|-
|(1,1)
| 100 (100) 100.00%
| align="right"| 0.78s
| 83.69 (0.00)
| 18.32 (15.64)
| align="right" | 783.57
| align="right" | 783.57
| align="right" | 18.32
| align="right" | 15.64
| align="right" | 55.16
| align="right" | 28.53
| align="right" | 0.99
| align="right" | 67.05
| align="right" | 37.53
| align="right" | 0.00
| align="right" | 14001.28
| align="right" | 6990528.00
| align="right" | 0.00
| align="right" | 174.00
| align="right" | 174.00
|-
|(1,5)
| 100 (100) 100.00%
| align="right"| 16.61s
| 238450.80 (0.00)
| 52.94 (25.52)
| align="right" | 16608.03
| align="right" | 5007.11
| align="right" | 52.94
| align="right" | 25.52
| align="right" | 49151.07
| align="right" | 189299.73
| align="right" | 0.98
| align="right" | 238424.28
| align="right" | 49123.57
| align="right" | 0.00
| align="right" | 244161.92
| align="right" | 6990528.00
| align="right" | 80162.71
| align="right" | 185.26
| align="right" | 185.26
|-
|(8,1)
| 5000 (5000) 100.00%
| align="right"| 1225.99s
| 20078513.07 (0.00)
| 73.59 (54.07)
| align="right" | 1225990.15
| align="right" | 5039.40
| align="right" | 73.59
| align="right" | 54.07
| align="right" | 2874234.15
| align="right" | 17204278.92
| align="right" | 1.00
| align="right" | 20078458.00
| align="right" | 2874178.08
| align="right" | 0.00
| align="right" | 426544.46
| align="right" | 932096.00
| align="right" | 187472.32
| align="right" | 202.00
| align="right" | 202.00
|-
|(2,1)
| 100 (100) 100.00%
| align="right"| 0.61s
| 560.58 (0.00)
| 28.33 (21.24)
| align="right" | 613.41
| align="right" | 613.41
| align="right" | 28.33
| align="right" | 21.24
| align="right" | 255.66
| align="right" | 304.92
| align="right" | 1.00
| align="right" | 538.34
| align="right" | 233.00
| align="right" | 0.00
| align="right" | 22846.08
| align="right" | 932096.00
| align="right" | 0.00
| align="right" | 252.46
| align="right" | 252.46
|-
|(9,1)
| 5000 (593) 11.86%
| align="right"| 2497.98s
| 38704112.56 (0.00)
| 81.67 (59.64)
| align="right" | 2497980.25
| align="right" | 5102.36
| align="right" | 81.67
| align="right" | 59.64
| align="right" | 5223853.76
| align="right" | 33480258.80
| align="right" | 1.00
| align="right" | 38704051.92
| align="right" | 5223792.11
| align="right" | 0.00
| align="right" | 475022.89
| align="right" | 932096.00
| align="right" | 198919.75
| align="right" | 206.00
| align="right" | 206.00
|-
|(3,1)
| 100 (100) 100.00%
| align="right"| 1.21s
| 3867.01 (0.00)
| 36.96 (27.26)
| align="right" | 1211.19
| align="right" | 1211.19
| align="right" | 36.96
| align="right" | 27.26
| align="right" | 1285.54
| align="right" | 2581.47
| align="right" | 1.00
| align="right" | 3838.75
| align="right" | 1256.97
| align="right" | 0.00
| align="right" | 30968.96
| align="right" | 932096.00
| align="right" | 0.00
| align="right" | 255.03
| align="right" | 255.03
|-
|(3,3)
| 100 (100) 100.00%
| align="right"| 24.10s
| 216430.64 (0.00)
| 57.72 (29.79)
| align="right" | 24102.54
| align="right" | 6396.58
| align="right" | 57.72
| align="right" | 29.79
| align="right" | 47500.16
| align="right" | 168930.48
| align="right" | 1.00
| align="right" | 216399.85
| align="right" | 47468.86
| align="right" | 0.00
| align="right" | 111149.44
| align="right" | 932096.00
| align="right" | 48094.12
| align="right" | 272.41
| align="right" | 278.00
|-
|(4,1)
| 100 (100) 100.00%
| align="right"| 7.35s
| 33895.58 (0.00)
| 46.61 (34.29)
| align="right" | 7348.58
| align="right" | 6948.40
| align="right" | 46.61
| align="right" | 34.29
| align="right" | 9516.88
| align="right" | 24378.70
| align="right" | 1.00
| align="right" | 33860.29
| align="right" | 9481.27
| align="right" | 0.00
| align="right" | 61336.96
| align="right" | 932096.00
| align="right" | 879.70
| align="right" | 274.16
| align="right" | 274.16
|-
|(4,2)
| 100 (100) 100.00%
| align="right"| 19.46s
| 170146.30 (0.00)
| 56.39 (36.93)
| align="right" | 19455.02
| align="right" | 4850.36
| align="right" | 56.39
| align="right" | 36.93
| align="right" | 37562.55
| align="right" | 132583.75
| align="right" | 1.00
| align="right" | 170108.37
| align="right" | 37524.15
| align="right" | 0.00
| align="right" | 98801.92
| align="right" | 932096.00
| align="right" | 42491.61
| align="right" | 279.55
| align="right" | 279.93
|-
|(10,1)
| 5000 (4) 0.08%
| align="right"| 2414.82s
| 38022689.25 (0.00)
| 89.75 (65.25)
| align="right" | 2414821.25
| align="right" | 3641.25
| align="right" | 89.75
| align="right" | 65.25
| align="right" | 4947236.75
| align="right" | 33075452.50
| align="right" | 1.00
| align="right" | 38022623.00
| align="right" | 4947169.50
| align="right" | 0.00
| align="right" | 471712.00
| align="right" | 932096.00
| align="right" | 212632.75
| align="right" | 210.00
| align="right" | 210.00
|-
|(5,1)
| 100 (100) 100.00%
| align="right"| 9.98s
| 107466.97 (0.00)
| 49.72 (37.57)
| align="right" | 9984.68
| align="right" | 4383.55
| align="right" | 49.72
| align="right" | 37.57
| align="right" | 23411.50
| align="right" | 84055.47
| align="right" | 1.00
| align="right" | 107428.40
| align="right" | 23371.93
| align="right" | 0.00
| align="right" | 147628.80
| align="right" | 6990528.00
| align="right" | 45776.70
| align="right" | 190.00
| align="right" | 190.00
|-
|(5,2)
| 100 (100) 100.00%
| align="right"| 85.43s
| 996758.65 (0.00)
| 65.17 (42.90)
| align="right" | 85426.04
| align="right" | 5003.09
| align="right" | 65.17
| align="right" | 42.90
| align="right" | 184139.71
| align="right" | 812618.94
| align="right" | 1.00
| align="right" | 996714.75
| align="right" | 184095.52
| align="right" | 0.00
| align="right" | 155769.60
| align="right" | 932096.00
| align="right" | 45871.92
| align="right" | 281.11
| align="right" | 281.67
|-
|(5,5)
| 117 (2) 1.71%
| align="right"| 2206.74s
| 31429172.50 (0.00)
| 86.00 (44.00)
| align="right" | 2206738.50
| align="right" | 2245.00
| align="right" | 86.00
| align="right" | 44.00
| align="right" | 4352520.00
| align="right" | 27076652.50
| align="right" | 1.00
| align="right" | 31429127.50
| align="right" | 4352474.00
| align="right" | 0.00
| align="right" | 2576352.00
| align="right" | 6990528.00
| align="right" | 179173.00
| align="right" | 201.00
| align="right" | 201.00
|-
|(6,1)
| 100 (100) 100.00%
| align="right"| 73.05s
| 833413.61 (0.00)
| 64.44 (46.18)
| align="right" | 73054.57
| align="right" | 4832.70
| align="right" | 64.44
| align="right" | 46.18
| align="right" | 155272.79
| align="right" | 678140.82
| align="right" | 1.00
| align="right" | 833366.43
| align="right" | 155225.40
| align="right" | 0.00
| align="right" | 157378.56
| align="right" | 932096.00
| align="right" | 49872.47
| align="right" | 280.73
| align="right" | 280.73
|-
|(7,1)
| 5000 (5000) 100.00%
| align="right"| 213.90s
| 3586188.11 (0.00)
| 65.62 (48.65)
| align="right" | 213896.72
| align="right" | 4963.18
| align="right" | 65.62
| align="right" | 48.65
| align="right" | 579468.72
| align="right" | 3006719.39
| align="right" | 1.00
| align="right" | 3586138.46
| align="right" | 579418.07
| align="right" | 0.00
| align="right" | 369729.37
| align="right" | 932096.00
| align="right" | 172409.01
| align="right" | 198.00
| align="right" | 198.00
|}


==ConTest==

* 1000 independent trials

* The ratio of the error discovery trials over the total number of trials is the 
path error density. 

{| align=center | border=1 | -bgcolor=grey
! Reorder Params !! Path Error Density
|- 
| 1,1 || 0.061
|-
| 2,1 || 0.023
|-
| 3,1 || 0.007
|-
| 4,1 || 0.003
|-
| 5,1 || 0.003
|-
| 6,1 || 0.000
|-
| 1,5 || 0.285
|-
| 4,2 || 0.00
|-
| 3,3 || 0.02
|-
| 5,2 || 0.01
|-
| 8,1 || 0.00
|-
| 9,1 || 0.01
|-
| 10,1 || 0.00
|}

==CalFuzzer==

* 100 independent trials

* The ratio of the error discovery trials over the total number of trials is the 
path error density. 

{| align=center | border=1 | -bgcolor=grey
! Reorder Params !! Path Error Density
|- 
| 1,1 || 0.06
|-
| 2,1 || 0.12 
|-
| 3,1 || 0.11
|-
| 4,1 || 0.12
|-
| 4,2 || 0.10
|-
| 5,1 || 0.12
|-
| 3,3 || 0.36
|-
| 6,1 || 0.10
|-
| 5,2 || 0.24
|-
| 1,5 || 0.08
|-
| 8,1 || 0.10
|-
| 9,1 || 0.10
|-
| 10,1 || 0.09
|}