MiniZinc - Challenge 2017 Results

MiniZinc Challenge 2017 Results

CP 2017 presentation slides of the MiniZinc Challenge 2017 (results).

Entrants

The entrants for this year (with their descriptions, when provided):

Choco 4 (description). A Java FD solver.
Choco 5 (see Choco 4 for description). A Java FD solver.
Concrete (description). A CP system written in Scala.
HaifaCSP (description). A C++ FD solver using SAT solving algorithms.
iZplus (description).
JaCoP (description). A Java FD solver.
Mistral 2.0 (description). A C++ FD solver.
OR-Tools CP (description).
OR-Tools LCG (description).
OR-Tools LCG Core (description).
OscaR/CBLS (description). A constraint-based local search solver written in Scala.
Picat CP (description).
Picat SAT (description).
SICStus Prolog (description). A Prolog development environment with a FD constraint programming module.
sunny-cp^— (see sunny-cp for description). A variant of sunny-cp only using Choco, Gecode, HaifaCSP, JaCoP, iZplus, MinisatID, Mistral, Opturion CPX, OR-Tools, Picat.
Yuck (description). A local search solver written in Scala.

In addition, the challenge organisers entered the following FlatZinc and MiniZinc implementations:

Chuffed (description). A C++ FD solver using lazy clause generation.
G12/FD. A Mercury FD solver (the G12 FlatZinc interpreter's default solver).
Gecode (description). A C++ FD solver.
LCG-Glucose (description). A C++ FD solver using lazy clause generation and it is based on Glucose 3.01 with modification along the lines of Chuffed and CPX.
MZN/Cbc. Translates to MILP (description), uses Cbc version 2.9.
MZN/Gurobi. Translates to MILP (description), uses Gurobi Optimizer version 7.5.1.
sunny-cp (description). A multi-threaded CP portfolio solver using 13 different CP and MIP solvers incl. Chuffed, G12 solvers, Gecode.

As per the challenge rules, these entries are not eligible for prizes, but do modify the scoring results.

Furthermore, entries in the FD search category (Gecode, JaCoP, Picat CP, SICStus Prolog) were automatically included in the free search category, while entries in the free search category (Choco 5, Chuffed, Concrete, G12/FD, HaifaCSP, Mistral, OR-Tools LCG, OR-Tools LCG Core, OscaR/CBLS, Picat SAT, Yuck and promoted FD entries except Gecode) were automatically included in the parallel search category. Lastly, all entries in the parallel search category and promoted entries into that category were automatically included in the open search category.

Summary of Results

The results for the MiniZinc Challenge 2017 are

Category	Gold	Silver	Bronze
Fixed	OR-Tools LCG	JaCoP	Choco 4
Free	iZplus	OR-Tools LCG	Picat SAT
Parallel	Choco 4	iZplus	OR-Tools LCG
Open	sunny-cp—	Choco 4	OR-Tools LCG
Local Search	iZplus	Yuck	OscaR/CBLS

Description of Results

All times are given in milliseconds.

A score of 0.0 indicates a worse answer in quality (worse objective, no proof of optimality, or no answer for satisfaction problems), 1.0 a better solution in quality. When the quality is the same, the 1.0 purse is split with respect to time used.

If a promoted entry does not recognize an option (or states that it is just ignored), times and solutions from the previous category are used for scoring. The suffixes -fd, -free, -par or -open (for the parallel portfolio solver entered) at the end of the solver names indicate which configuration the solvers were run with.

mzn2fzn was run with the same time and memory limits as the solvers.

In the Status column:

S indicates that a solution was found,
C indicates that the search was complete,
ERR indicates an incorrect answer or the solver aborted,
ERR indicates that flattening aborted (time-out or out-of-memory),
UNK indicates that no answer was returned in the time limit.

Download all problems

All problems are available in a zipped tar-ball here.

Selection:

Select a list of solvers and benchmarks and click on “Compute Results” to score the solvers against each other on the selected benchmarks. The entrants for each of the fd search, free search and parallel search categories can be selected with the corresponding buttons.

Solver selection:	Problem selection:

Plot settings:
Create plots for each instance Note that plotting takes time. Time axis scale: Default Linear Log Plot lines shape: Steps Lines
Area scoring settings:
Normalise

Summary:

Total per problem:

Solver	Score	Score Incomplete	Score Area

Problem	Solver	Score	Score Incomplete	Score Area

Individual results:

Problem	Instance	Solver	Status	Time	Objective	Score	Score Incomplete	Score Area

Objective plots:

Problem	Instance	Plot

Global constraint per model

The following table lists the global constraints used by each model in this year's challenge. In addition, the columns RC and SBC, respectively, indicate whether the model contains redundant or/and symmetry breaking constraints.

Problem	Type	Kind	RC	SBC	MiniZinc Globals
cargo	real	min			cumulative, diffn
city-position	combi	min		✓
community-detection	combi	max			value_precede_chain, global_cardinality_low_up
crosswords	puzzle	max			all_different
gbac	combi	min			bin_packing_load, global_cardinality_low_up_closed
groupsplitter	combi	max	✓	✓	count, table
hrc	real	min
jp-encoding	real	min			count
ma-path-finding	combi	min
mario	puzzle	max			subcircuit
opd	combi	min		✓	lex_greatereq
opt-cryptanalysis	real	min			table
rcpsp-wet	combi	min	✓		cumulative
rel2onto	real	min	✓		all_different
road_construction	combi	min
routing-flexible	real	min
steelmillslab	real	min		✓	bin_packing_load
tc-graph-color	combi	min
tdtsp	real	min	✓		inverse
traveling-tppv	combi	min			all_different, regular

The files on this page are for MiniZinc version 2.1.5.