Four Theory Solvers
Inside LRA, LIA, bitvectors, and arrays. The first three each have their own decision procedure. The last one cheats.
Last week we opened one theory solver all the way. Equality with uninterpreted functions had four axioms, one algorithm, and a worked trace you ran by hand on . Today we open four more, at deliberately uneven depth. LRA goes deepest because its geometry is the most visually rewarding decision procedure in the course. LIA and BV each get one idea. Arrays is the payoff: its decision procedure is the same congruence closure you already know.
The solver-interface contract
Every theory solver has the same contract:
The theory-solver contract
Given a conjunction of literals in my theory, I return sat (with a model) or unsat.
A literal is an atomic constraint or its negation. The LRA literal , the bitvector literal , and the EUF literal are all literals. A conjunction of literals is a formula built from literals with only between them. No , no outer . The negation is already absorbed into each literal.
The boolean envelope around the literals is handled above the theory solvers by a separate engine. That envelope is the ands, ors, and nots that connect literals across the whole formula. The theory solvers cooperate only through that engine. How they cooperate is next week's story. Today we stay inside each one.
LRA: Linear Real Arithmetic
Signature
Constants are rationals. Variables range over the rationals (Z3 works in rationals internally; for this fragment, rationals and reals are interchangeable). The operators are the linear ones:
where is a rational constant and is a variable. The relations are all the linear comparisons:
What is not in the signature: multiplication of two variables. The product is nonlinear arithmetic, a different and much harder theory. "Linear" here means "linear combinations of variables with rational coefficients, nothing else."
What LRA decides
A conjunction of LRA literals is a system of linear inequalities. Each single inequality is a half-plane in the space of variables: the set of points on one side of a line. In dimensions it becomes a half-hyperplane, but the 2D picture carries the intuition.
The conjunction is the intersection of all those half-planes. That intersection has a geometric name: a convex polytope. It is the feasible region of the system.
The whole LRA decision problem collapses to one geometric question:
LRA as geometry
Is the polytope empty?
Empty polytope: unsat. Non-empty polytope: sat, and any point inside is a model.
Simplex: walking corners
How do you find a point inside a polytope when you do not know where one is? The classical answer, introduced by Dantzig in 1947 and still the workhorse SMT solvers run today, is simplex.
Simplex leans on a specific fact about polytopes and linear objectives: if a linear objective has a minimum over a non-empty polytope, the minimum is achieved at a vertex. Not in the interior, not along an edge: at a corner. So the search is over vertices, not over the continuous interior.
The loop is short:
simplex(polytope, objective):
start at some vertex v
loop:
if some edge leaving v lowers the objective:
walk that edge to the adjacent vertex
v := that vertex
else:
return v # no improving edge; v is optimal
Three moves: pick a starting vertex, step to an adjacent vertex with lower cost, stop when no adjacent vertex is better. Each step is called a pivot.
Tracing simplex on the blending polygon
Back to the Practice polygon. Same blend LP, second view. After eliminating via , the feasible region sits in the plane and has four vertices. Call them V1 through V4. Their costs, under :
| Vertex | Cost | |
|---|---|---|
| V1 | $2.33 | |
| V2 | $2.50 | |
| V3 | $2.59 | |
| V4 | $2.10 |
We will start simplex at V2. That is the "pure beans" vertex, the same corner the LIA version of the solver picked in Practice when you rounded to integers. It is feasible (all three nutrition constraints are met) but it is not cheap.
Step 1: pivot from V2. V2 has two adjacent vertices on the polygon: V1 below it along the edge, and V3 along the blend cap. V1 costs $2.33; V3 costs $2.59. V1 is cheaper. Walk the edge V2 → V1.
Step 2: pivot from V1. V1's two adjacent vertices are V2 (uphill, $2.50) and V4 (downhill, $2.10). V4 is cheaper. Walk the edge V1 → V4 along the protein floor.
Step 3: stop at V4. V4's two adjacent vertices are V1 ($2.33, uphill) and V3 ($2.59, uphill). No edge leaving V4 lowers the objective. V4 is optimal.
Two pivots. Z3 reported the same corner in Practice because Optimize ran this walk.
The starting vertex is not special. Simplex would have reached V4 from any corner of the polygon; the details of which edges get pivoted depend on where you start and on the pivot rule (which adjacent vertex to pick when more than one improves). The outcome does not.
Feasibility is what the theory solver actually does
Simplex with an objective finds the cheapest vertex. Simplex without an objective is called general simplex, and it just finds any vertex, or reports that the polytope is empty. Both variants share the same vertex walk.
SMT theory solvers run general simplex by default, typically the incremental variant from Dutertre and de Moura's 2006 paper A Fast Linear-Arithmetic Solver for DPLL(T), designed for fast add and remove of bounds as the surrounding SAT search evolves. The theory-solver contract is feasibility, not optimization: the solver is asked whether a conjunction of literals is satisfiable, not which model is cheapest. Your Practice demo used Optimize, which wraps simplex in the cost-directed mode we just traced. If you had written Solver() with the same constraints and called check(), Z3 would have returned sat with some feasible point, not the cheapest one. Optimization is a capability LRA happens to support cleanly because the underlying algorithm already walks vertices.
Complexity
Linear programming sits in P. The first polynomial-time algorithm was Khachiyan's ellipsoid method (1979); Karmarkar's interior-point method (1984) is faster and more robust. Simplex itself is worst-case exponential: there are pathological inputs (the Klee-Minty cube is the textbook example) that force it to visit exponentially many vertices. Those inputs are rare, and on realistic LPs simplex is dominant. Every major SMT solver uses it.
The fact worth remembering:
LRA is the one polynomial-time theory today
LRA is in P. The other three theories we look at today, LIA, bitvectors, and arrays, are all NP-complete.
Where LRA belongs, and where it stops
LRA is one of the oldest applied mathematical theories: logistics, operations research, scheduling, and finance have relied on linear programming since the 1940s. When your problem is linear in continuous quantities, reach for LRA. The moment any variable has to be an integer, the geometry changes. The feasible region stops being the polygon you just walked and becomes only the integer lattice points sitting inside it, and that single change takes the problem out of P and into NP-complete. The next section is why.
LIA: Linear Integer Arithmetic
Signature
The LIA signature is the LRA signature, with one change: variables range over the integers instead of the rationals. Operators and relations are the same: addition, subtraction, multiplication by a constant, and the full set of linear comparisons.
Nothing syntactic distinguishes an LRA formula from an LIA formula. The difference is semantic. Int('i') ranges over ; Real('i') ranges over . The same inequalities mean different things depending on which sort the variables have.
What LIA decides
A conjunction of LIA literals describes the same polytope LRA would describe. What changes is the notion of feasibility. In LRA, feasibility is "is there a real point in the polytope?" In LIA, feasibility is "is there a lattice point, a point whose coordinates are all integers, inside the polytope?"
That single change creates a gap.
The relaxation lies
Consider the system
The LP relaxation (drop the integrality requirement) is feasible. The thin strip between the lines and has plenty of real-valued points inside the bounding box.
Zoom into the strip. Every integer lattice point in the square sits outside it. The reason is arithmetic: at integer , the quantity is always a multiple of 3. It cannot equal 1 or 2. So no lattice point lies inside a strip defined by .
Non-empty polytope. Empty integer hull.
The LP relaxation lies
LRA feasibility is a necessary condition for LIA feasibility (the LP polytope must be non-empty) but not sufficient. A non-empty LP polytope may contain zero integer points.
That gap is where LIA's hardness lives. LRA answers "does the polytope contain any point?" in polynomial time by running simplex. LIA has to answer a harder question: does the polytope contain any point whose coordinates are all integers? That question is NP-complete.
Branch-and-bound
The classical LIA decision procedure reuses the LRA machinery. Run the LP relaxation. If it returns a feasible witness that happens to be an integer point, we are done: the LIA formula is satisfiable. If the witness is fractional, pick one fractional variable and split the problem on that variable. One subproblem forces the variable to be at most its floor; the other forces it to be at least its ceiling. Neither subproblem loses any integer solution, because no integer lies strictly between consecutive integers. Recurse.
branch_and_bound(S):
solve the LP relaxation of S
if LP is infeasible:
return UNSAT
v := an LP witness assignment
if every value in v is an integer:
return SAT with model v
pick some variable x_i whose value v_i is not an integer
left := branch_and_bound(S ∧ x_i ≤ floor(v_i))
right := branch_and_bound(S ∧ x_i ≥ ceil(v_i))
if left returns SAT or right returns SAT:
return SAT
return UNSAT
Three moves: LP-relax, pick a fractional variable, split. The LRA solver does the heavy lifting; the outer loop handles the integrality.
Two details matter.
First, the LP layer must return a witness vertex, not just a sat/unsat bit. Branch-and-bound needs the fractional assignment in order to know which variable to split on. A feasibility-only LP with no model output would be useless here. This is one reason the general-simplex method we traced in the previous section returns a point, not just a flag.
Second, the two subproblems never lose an integer solution. Every integer feasible for the parent problem satisfies either or , because there is no integer strictly inside the open interval between floor and ceiling. The split is complete.
Termination is the subtle part
Branch-and-bound as stated above is not guaranteed to terminate. The same strip from the figure is the classical witness. Take
alone, without the bounding box. It has infinitely many real solutions (the strip extends forever along the direction ) and zero integer solutions. Naive branch-and-bound, handed this system, picks a fractional LP witness, splits on one variable, and recurses into two new strips that still have infinitely many real solutions and zero integer solutions. The recursion never bottoms out.
The fix is a small-model bound: every integer-feasible variable can be bounded by a quantity polynomial in the size of the formula. For an integer coefficient matrix with entries bounded by , if an integer solution exists, one exists with every . That bound is astronomically large in absolute terms but it is finite. Adding it to the original system forces termination. The branch-and-bound recursion tree is now bounded, so if no integer solution exists, the search exhausts and returns UNSAT in finite time. Kroening and Strichman's Decision Procedures §5.3 states the formal algorithm (their 5.3.1) and the bound argument in full.
Without the bound, branch-and-bound can loop forever
The unbounded strip has no integer solutions but infinitely many real solutions. Naive branch-and-bound never detects UNSAT. Every real LIA implementation adds a small-model bound to guarantee termination.
Worked example
Practice traced a strides-3-and-5 loop whose dependence question collapsed to the Diophantine equation inside the iteration box . At Z3 returned unsat. What ran under the hood was branch-and-bound, and here is the first step.
Start with the LP relaxation (take the same constraints with Real variables and drop the integrality). Simplex returns a feasible vertex:
Both coordinates are fractional, so the LP does not directly answer the integer question. Pick one fractional variable to split on. Take , whose value sits between 2 and 3. Create two subproblems:
- Left: the parent constraints plus .
- Right: the parent constraints plus .
Neither subproblem discards any integer solution the parent had, because no integer lies strictly between 2 and 3.
graph TD
accTitle: One branch-and-bound step for 3i minus 5j prime equals 2 at N equals 4
accDescr: Root node shows the LP relaxation witness i equals twenty-four elevenths and j prime equals ten elevenths, both fractional. Two child nodes branch on the variable i. The left child adds the constraint i at most two and its LP witness is two and four fifths. The right child adds the constraint i at least three and its LP witness is three and seven fifths. Each child has a dotted descendant indicating further branching below.
root["LP: i = 24/11, j' = 10/11
(fractional; branch on i)"]
root -->|"i ≤ 2"| L["LP: i = 2, j' = 4/5
(still fractional)"]
root -->|"i ≥ 3"| R["LP: i = 3, j' = 7/5
(still fractional)"]
L -.-> Ld["..."]
R -.-> Rd["..."]
classDef node fill:#fffde7,stroke:#8a6d00
classDef more fill:#f5f5f5,stroke:#aaa,color:#777
class root,L,R node
class Ld,Rd moreRecurse. The left subproblem's new LP witness is ; still fractional in , so it branches again. The right subproblem's LP is ; also fractional. Both sides keep splitting. Every subproblem eventually becomes integer-infeasible under its accumulated bounds, and the procedure returns unsat for the root. No pair of distinct iterations in a box collides on this stride pattern.
Contrast at . The first branching step reaches an integer vertex (the LP witness is still fractional, but one of the deeper subproblems hits the integer solution on the line before the tree grows). LIA returns sat with that witness. Same algorithm, different tree shape. The answer depends on the iteration box, exactly as Practice showed.
Callback to Practice
When demo_box_too_small returned unsat at , Z3's LIA solver ran branch-and-bound in the shape above. Its internal statistics report eleven arith-branch events and nine arith-conflicts on that single query: the concrete work a single check() call hides.
Complexity
LIA is NP-complete. The relaxation-lies gap above shows where the hardness lives. The LP polytope can be non-empty while the integer hull is empty, and deciding which case you are in requires combinatorial search.
Branch-and-bound with a small-model bound is correct but slow when the integer hull is deep inside a large polytope. Real solvers combine branch-and-bound with Gomory cuts. After each LP relaxation, generate a new inequality that every integer solution satisfies but the current fractional LP witness violates. Adding the cut tightens the polytope without discarding any lattice points. The combined procedure is called branch-and-cut, and it is what Z3's LIA solver actually runs. Gomory cuts (Gomory 1963) were considered impractical for large problems for decades and revived in the 1990s; they are now standard.
One alternative procedure is worth naming. The Omega test (Pugh 1991) extends Fourier-Motzkin elimination to integers and is the workhorse inside compiler dependence analyzers, where the systems are small and the coefficients are small.
Forward pointer
LIA assumes integer arithmetic with unbounded precision. Production code does not. A 32-bit signed int holds values, not infinitely many, and it wraps when you overflow. The next section covers the theory that models that.
BV: Bitvectors
Signature
Fixed-width bitvector sorts, one per width . In Z3, BitVec('x', 32) declares a 32-bit variable. Each value of type BV_n is a specific pattern of bits.
That same bit pattern carries two possible integer meanings:
- Unsigned (binary): .
- Signed (two's complement): .
The 8-bit pattern 11001000 is 200 unsigned and −56 signed. The bits are the same; the reading is what changes.
Arithmetic is always modulo . That wraparound is the defining feature of the theory, and it is why BV can catch bugs that Int cannot: a 32-bit a + b is not in , it is .
The signature includes arithmetic (bvadd, bvsub, bvmul, bvudiv, bvsdiv, signed and unsigned remainders), bitwise operators (bvand, bvor, bvxor, bvnot), shifts (bvshl, and logical vs. arithmetic right shifts bvlshr and bvashr), comparisons (unsigned bvult/bvule, signed bvslt/bvsle), and width-changing operators (extract, concat, sign_extend, zero_extend).
What BV decides
LRA and LIA textbook treatments decide conjunctions of literals. BV is different.
BV handles arbitrary boolean combinations
The BV theory decides satisfiability of arbitrary boolean combinations of BV literals, not just conjunctions. The reason comes out of the decision procedure: once every BV variable has been replaced by a vector of Booleans, the whole formula is a Boolean formula and CDCL handles the ands, ors, and nots natively.
NP-complete.
Bit-blasting: just encode circuits
The decision procedure is bit-blasting (sometimes called flattening). Three moves:
- Every BV variable becomes a vector of Booleans. A
BitVec('x', n)becomes fresh Boolean variables . Bit 0 is the least significant. - Every BV operator becomes a Boolean circuit on those bits. The circuit's output wires become a new Boolean vector representing the operator's result; the internal wires become fresh Boolean variables tied to the inputs by the circuit's logic.
- The whole formula becomes CNF. Collect the Boolean equivalences for every circuit (flattened via Tseitin's encoding), conjoin them with the original formula's boolean structure, and hand the result to the SAT engine from Week 2.
The engine is CDCL. Nothing new under the hood. The work is in step 2, in choosing good circuits.
One adder, scaled up
Addition is the canonical circuit. A full adder takes three input bits, , , and a carry-in , and produces two output bits:
Chain full adders with the carry-out of stage wired to the carry-in of stage , and you have an -bit ripple-carry adder. At 4 bits:
bvadd on a 32-bit BV is this same circuit, scaled to 32 full adders and 33 carry wires. Every bit of the result is a Boolean formula over the input bits and the carries above it. Together the whole adder is a block of roughly CNF clauses at width .
Every other BV operator has an analogous circuit.
- Subtraction
bvsub(a, b)uses the same adder with bitwise-negated and the initial carry set to 1 (two's complement: ). - Comparison
bvult(a, b)computes and inspects the carry-out; the pattern " iff the subtractor overflows" compiles to one extra clause. - Shifts
bvshl,bvlshr,bvashrby a variable amount use a barrel shifter: stages, where stage either shifts by bits or passes through, controlled by one bit of the shift amount. Total size . - Bitwise operators
bvand,bvor,bvxor,bvnotare pointwise: the -th output bit is a single gate over the -th input bits.
Multiplication is where bit-blasting hurts. A shift-and-add multiplier has gates. Kroening and Strichman measure the flattened CNF directly:
| Multiplier width | Boolean variables | CNF clauses |
|---|---|---|
| 8 | 313 | 1,001 |
| 16 | 1,265 | 4,177 |
| 32 | 5,089 | 17,057 |
| 64 | 20,417 | 68,929 |
Size alone is not the problem. The deeper issue is that a multiplier's circuit has a dense symmetry of XOR chains that modern CDCL decision heuristics do not latch onto well. Multiplier formulas routinely defeat SAT solvers that handle much larger but structurally less symmetric CNFs.
Worked example: bit-blasting the Practice midpoint
Practice asked Z3 whether
can ever land outside the interval when . Bit-blasting produces, piece by piece:
- 64 fresh Boolean variables for the two 32-bit inputs.
- A 33-wire ripple-carry adder computing (32 sum bits plus the carry-out, the extra bit that lets us see overflow).
- A right-shift-by-one, which is free at the circuit level: it is a pure renaming that wires for and ties to 0.
- Two 33-wire subtractor circuits for the two
ULEcomparisons against and . - A small layer of Boolean wiring for the conjunction and negation at the top of the formula.
The total is a few hundred Boolean variables and a few thousand clauses. CDCL solves it in milliseconds. When it returns sat, the satisfying assignment to the 64 input bits reads out as a concrete pair of 32-bit integers, exactly the overflow witness the Practice page tabulated: , .
Complexity
BV is NP-complete. For most operators at realistic widths, that upper bound is not a practical ceiling. CDCL handles bit-blasted circuits for 32- and 64-bit addition, subtraction, comparison, bitwise ops, and shifts routinely, in milliseconds.
The pain point is multiplication. The 64-bit multiplier from the table above is exactly what L03's sq-vs-sqabs demo built when you used full BV semantics, and it is why that full-BV version was slow. Replacing the multiplication with an uninterpreted function umul plus one axiom () took the problem out of BV and into EUF. Congruence closure from last week decided the reformulated problem in milliseconds because it never built the multiplier circuit at all.
That tradeoff is a recurring lesson of solver-aided work. When full BV is slow, it is usually because CDCL is drowning in a multiplier's XOR chains. When a UF abstraction is fast, it is because the problem never needed the multiplier's internal structure in the first place.
Modern BV solvers pursue the same idea systematically. Kroening and Strichman's incremental bit-blasting (their Algorithm 6.3.1) starts with the cheap bitwise constraints and only adds multiplier constraints when the SAT solver cannot close the case without them. Z3 and CVC5 implement variations.
Forward pointer
Three theories down, three different decision procedures. The fourth cheats.
Arrays: Reduce to EUF
Signature
The theory of arrays is parameterized by an index sort and a value sort . An Array[I, V] is an object modeling a map from indices to values. Z3's Array('a', IntSort(), IntSort()) declares an integer-indexed array of integers; other combinations (bit-vector indices, array-valued elements, and so on) are all permitted.
The theory has two operators:
select(a, i): read the value stored at index of array . Returns an element of .store(a, i, v): return a new array that agrees with everywhere except at index , where it takes the value . The input array is unchanged.
Arrays are functional, not imperative. A store does not mutate its input; it produces a fresh array and leaves the original intact. That matches Z3's semantics, and it matches how verification conditions use arrays. Treating store as a mutation is a common beginner mistake, and the theory is easier to reason about once you drop the mental model of "an array variable that gets overwritten."
The one axiom: read-over-write
The whole theory is a single axiom, due to McCarthy (1962):
Reading right after writing at the same index gives you what you wrote. Reading at a different index gives you the original. Every true fact about select and store follows from this axiom plus the equality axioms from L03.
There is a second axiom called extensionality that says two arrays are equal if they agree at every index. We set it aside today. Extensionality is an optional add-on that SMT solvers include or omit; most common array queries do not need it, and the Practice section did not.
The decision procedure: rewrite, then hand to EUF
Arrays do not have their own search-based algorithm. The decision procedure is a rewrite followed by a call to the congruence-closure procedure from last week.
The rewrite uses the read-over-write axiom as a directed term-level rule:
Apply it anywhere the pattern matches. Each application eliminates one store from inside a select by turning it into an if-then-else on the two possible index comparisons. Repeat until no select contains a store.
What remains after full rewriting is a formula in which every select is applied to a base array variable, the array symbols you originally declared. In that remaining formula, select is an uninterpreted function symbol. Nothing in the logic interprets it; the solver treats it as a fresh function and reasons only by equality and congruence. The residual formula is pure EUF. Hand it to the congruence-closure procedure.
Worked example: writes commute
Practice Demo 4 Case 1 asked whether two orders of updates to distinct indices produce the same array:
The query is for every index . Negate it and hand the negation to the solver.
Apply the rewrite to each side, outermost store first. For :
Apply again to the inner select:
So the left side rewrites to . Two applications on the right give . Same two cases, checked in the opposite order.
Case-split on to collapse the ite nests. Three possibilities:
graph TD
accTitle: Arrays decision procedure on the writes-commute example
accDescr: Root node holds the negated query asking whether Select of a2 at i equals Select of b2 at i. A single arrow labeled rewrite read-over-write four times leads to a node labeled case analysis on i. That node has three children. The first child, on the edge i equals three, is the EUF equality ten equals ten. The second child, on the edge i equals five, is twenty equals twenty. The third child, on the edge otherwise, is select of x at i equals select of x at i. Each leaf is marked as decided by congruence closure.
root["Select(a_2, i) = Select(b_2, i) ?"]
root -->|"rewrite (RoW × 4)"| eq["case analysis on i"]
eq -->|"i = 3"| c1["10 = 10"]
eq -->|"i = 5"| c2["20 = 20"]
eq -->|"otherwise"| c3["select(x, i) = select(x, i)"]
classDef node fill:#fffde7,stroke:#8a6d00
classDef leaf fill:#e8f5e9,stroke:#2e7d32
class root,eq node
class c1,c2,c3 leafEach leaf is an EUF equality in which both sides are the same term. The negated query at each leaf becomes a disequality between a term and itself. Congruence closure decides each one unsat immediately: both sides sit in the same congruence class (trivially, since they are the same term), and the disequality check fires. Every case closes. The original equality holds at every , and the two arrays are indistinguishable.
The payoff
The algorithm you traced by hand last week on is also the array decision procedure. You just have to rewrite the formula first.
EUF was not simply the first theory we met. It is the engine underneath arrays, too.
Complexity and lazy instantiation
The rewrite is the source of hardness. Each application of the RoW rule introduces a case split on whether the two indices are equal. Many store and select interactions cause those case splits to multiply, and the rewritten formula's size can grow exponentially in the number of array operations.
For that reason, the theory of arrays is NP-complete, even though the EUF engine underneath is polynomial. The decision problem moved up one complexity class because the reduction is exponential in the worst case.
Real array solvers do not apply RoW eagerly. Instead they use lazy instantiation: expand a case split only once the SAT engine asks which branch holds, not before. That keeps the rewritten formula compact on problems that do not exercise every case. The technique carries directly into how solvers cooperate across theories, which is L05's territory.
Two small notes on scope. Full array logic, with unrestricted universal and existential quantifiers over indices, is undecidable (Kroening and Strichman, Problem 7.2, reduces from the two-counter-machine halting problem). The decidable fragment is the quantifier-free slice Practice exercises. For bounded quantification (statements like "this array is sorted"), Bradley, Manna, and Sipma identified a decidable extension called the array property fragment.
Where arrays belong
Modeling indexed mutable state without committing to a concrete layout. Memory inside program verifiers (Klee, Dafny, CBMC), the heap in symbolic execution, filesystems, key-value stores, database records. Writing the model in select and store and letting the solver handle the rewriting is the usual move. The alternative, bit-blasting a flat address space, scales poorly.
What's next
You now know five theory solvers: EUF, LRA, LIA, bitvectors, and arrays. Next week, quantifiers come back, and we see how the boolean envelope talks to all five at once for formulas with real boolean structure.