MiniZinc Challenge 2020 Results

CP 2020 presentation slides of the MiniZinc Challenge 2020 will be available here.

Entrants

The entrants for this year (with their descriptions, when provided):

• Choco 4 (description). A Java FD solver.
• Flatzingo (description).
• Mistral 2.0 (description).
• JaCoP (description). A Java FD solver.
• OptiMathSAT (description).
• OR-Tools (description).
• OscaR/CBLS (description). A constraint-based local search solver written in Scala.
• PicatSAT (description).
• SICStus Prolog (description). A Prolog development environment with a FD constraint programming module.
• sunny-cp^— (description). A variant of sunny-cp only using Choco, Gecode, HaifaCSP, JaCoP, iZplus, MinisatID, Mistral, Opturion CPX, OR-Tools, Picat.
• Yuck (description). A local search solver written in Scala.

In addition, the challenge organisers entered the following FlatZinc and MiniZinc implementations:

• Chuffed (description). A C++ FD solver using lazy clause generation.
• Geas (description). A C++ lazy clause generation solver with an OCaml FlatZinc frontend.
• Gecode (description). A C++ FD solver.
• MZN/Cbc. Translates to MILP (description), uses Cbc version 2.10.4.
• MZN/CPLEX. Translates to MILP (description), uses IBM ILOG CPLEX Optimizer version 12.10.0.0.
• sunny-cp (description). A multi-threaded CP portfolio solver using 13 different CP and MIP solvers incl. Chuffed, G12 solvers, Gecode.

As per the challenge rules, these entries are not eligible for prizes, but do modify the scoring results.

Furthermore, entries in the FD search category (Gecode, JaCoP, SICStus Prolog) were automatically included in the free search category, while entries in the free search category (Chuffed, OscaR/CBLS and promoted FD entries except Gecode) were automatically included in the parallel search category. Lastly, all entries in the parallel search category and promoted entries into that category were automatically included in the open search category.

The Choco entry submitted by the final deadline had a bug that prevented it from competing in the free and parallel categories. We entered an updated version that was provided after the deadline and fixed those problems, but we treated it the same as the other internal entries: Choco was therefore ineligible to win prizes in the free and parallel categories, but its performance modifies the scoring results.

Summary of Results

The results for the MiniZinc Challenge 2020 are

Category	Gold	Silver	Bronze
Fixed	SICStus Prolog	JaCoP	Choco 4
Free	OR-Tools	PicatSAT	Mistral 2.0
Parallel	OR-Tools	PicatSAT	Mistral 2.0
Open	OR-Tools	sunny-cp—	PicatSAT
Local Search	Yuck	OscaR/CBLS

Description of Results

All times are given in milliseconds.

A score of 0.0 indicates a worse answer in quality (worse objective, no proof of optimality, or no answer for satisfaction problems), 1.0 a better solution in quality. When the quality is the same, the 1.0 purse is split with respect to time used.

If a promoted entry does not recognize an option (or states that it is just ignored), times and solutions from the previous category are used for scoring. The suffixes -fd, -free, -par or -open (for the parallel portfolio solver entered) at the end of the solver names indicate which configuration the solvers were run with.

The time limit includes both MiniZinc compilation and solving.

In the Status column:

• S indicates that a solution was found,
• C indicates that the search was complete,
• ERR indicates an incorrect answer or the solver aborted,
• ERR indicates that flattening aborted (time-out or out-of-memory),
• UNK indicates that no answer was returned in the time limit.

Download all problems

All problems are available in a zipped tar-ball here.

Selection:

Select a list of solvers and benchmarks and click on "Compute Results" to score the solvers against each other on the selected benchmarks. The entrants for each of the fd search, free search and parallel search categories can be selected with the corresponding buttons.

Solver selection: Problem selection:

Plot settings: Note that plotting takes time. Create plots for each instance Time axis scale: Default Linear Log Plot lines shape: Steps Lines Area scoring settings: Normalise

Summary:	Total per problem:
Solver Score Score Incomplete Score Area	Problem Solver Score Score Incomplete Score Area

Individual results:

Problem	Instance	Solver	Status	Time	Objective	Score	Score Incomplete	Score Area

Objective plots for each instance:

Problem	Instance	Plot

Score development over time:

Global constraint per model

The following table lists the global constraints used by each model in this year's challenge. In addition, the columns RC and SBC, respectively, indicate whether the model contains redundant or/and symmetry breaking constraints.

Model	RC	SBC	Global constraints used
bnn-planner
cable_tree_wiring	X		all_different
code-generator	X	X	all_different, diffn_nonstrict, diffn, maximum, table, cumulative, member, value_precede_chain, minimum, decreasing
collaborative-construction	X	X
gbac			global_cardinality_low_up_closed, bin_packing_load
hoist-benchmark-for-minizinc		X
is	X		table, circuit
lot-sizing	X	X	global_cardinality, at_least, at_most
minimal-decision-sets
p1f-pjs		X	circuit, all_different, inverse, lex_less
pentominoes			regular
pillars-and-planks		X	diffn
racp	X		cumulative
radiation
sdn-chain
skill-allocation
soccer-computational
stable-goods
tower_challenge			arg_max
whirlpool

---

The files on this page are for MiniZinc version 2.4.3.