Why

The first step groups neighboring pixels of similar color into regions called superpixels; we aim for about a thousand by default. Working with regions rather than single pixels keeps the outline readable. Each pixel is described by its color in Lab (more on that below) and its position. The studio offers three main approaches:

• Quickshift (the default, Vedaldi & Soatto, 2008) estimates how crowded each pixel's neighborhood is with similar colors, then links every pixel to the nearest pixel that is more crowded than itself, as long as it's within a maximum distance $\tau$:
  $$P_i = \sum_j \exp\!\Big(-\frac{d(i,j)^2}{2\sigma^2}\Big), \qquad \operatorname{parent}(i) = \operatorname*{arg\,min}_{j \,:\, P_j > P_i} d(i,j) \quad \text{if } d(i,j) < \tau$$
  Following the links uphill, pixels that reach the same peak form a region. Regions follow the photo's edges and can be long and thin, so they stay faithful to detail, but grouping them into colors leaves more specks too small to paint. The defaults ($\sigma = 5,\ \tau = 11.5$) give 925 regions for the slug.
• SLIC (Achanta et al., 2012) is k-means clustering on those five numbers per pixel, starting from a grid of centers $S$ pixels apart, with a distance that weighs color against position:
  $$D = \sqrt{d_{\text{color}}^2 + \Big(\frac{d_{\text{position}}}{S}\Big)^2 m^2}$$
  The compactness $m$ keeps regions compact and roughly even in size, with no specks; long, thin things like a twig can get absorbed into their neighbors. The default asks for 1,150 regions and gets 1,018 for the slug.
• No superpixels skips regions altogether: k-means groups the pixels themselves by color. The painting looks closest to the photo (for the slug, an average color difference of 7.6 against about 10 with regions) but it's a weak guide. Nothing decides where paint goes, so the colors scatter into over 15,000 specks too small to paint.

Compared at the same number of regions, neither SLIC nor Quickshift wins outright: Quickshift follows edges better and is often a little more faithful on busy photos, while SLIC is tidier and easier to paint.

Each region $r$ is reduced to the average color of its pixels $p$:

The average is taken in CIELAB ("Lab"), a color space built so that straight-line distances roughly match how different two colors look to people: $L$ is lightness, $a$ runs green to red, and $b$ runs blue to yellow. Averaging the raw RGB numbers instead gives subtly wrong, often muddier colors.

Now k-means runs again, on color alone. It looks for $k$ group colors $\mu_1, \dots, \mu_k$ ($k$ will be the size of your pallete) and an assignment $g(r)$ of regions to groups that keep every region close to its group's color:

It starts from $k$ guesses spread out on purpose (k-means++), then repeats two moves (the studio runs 10 rounds): every region joins its nearest group ($g(r) = \operatorname*{arg\,min}_j \lVert \bar c_r - \mu_j \rVert$) and every group color moves to the average of its regions. Each group's final color is the average of all its pixels. Every region takes its group's number. That's the color guide.

Finding which paints make a color is an inverse problem. Often, it is easier to solve the forward direction. It is easier to figure out If I mix these paints, what color will I get? than to answer What paints do I need to mix this particular color?

For light, mixing is simple addition. Paint is different: pigments absorb and scatter light (the Kubelka–Munk model), so blue and yellow paint make green, while averaging their RGB numbers gives a dull olive.

paintrecipe uses Mixbox (Sochorová & Jamriška, 2021), a pigment mixing model built on Kubelka–Munk. Mixbox maps each color to seven numbers, the amounts of four base pigments plus a small correction, $z = \varphi(\text{rgb}) \in \mathbb{R}^7$, chosen so that mixing paint is just a weighted average of those numbers:

The physics stays inside Mixbox. What's left is linear algebra.

Mixbox hands us the forward direction: pick amounts of each paint, average their seven numbers and convert back. In math terms, the mix is a linear function of the amounts. Double one tube's share and its contribution to the mix doubles; combine two recipes and their numbers simply add.

That's what makes the inverse easy. If mixing were a tangled, nonlinear process, the only way to find a recipe would be to search: try some amounts, mix, compare, adjust, repeat. That's guess and check again, just done by a computer. A linear forward direction turns the question into a system of linear equations, one of the most thoroughly solved problems in mathematics. There's no searching: the standard tools go straight to the answer. They can also tell us when it's the best one possible.

Now bring in your paints. Each tube in your set has its own seven numbers; stack them as the columns of a matrix $C$. How much of each tube goes into a mix is a vector $w$:

Finding a recipe means solving $C\,w = z_t$ for $w$, where $z_t = \varphi(\text{target})$. Each tube is a point in seven dimensions. Any eight or more such points are linearly dependent: some tube can always be written as a combination of the others (using positive and negative amounts). What matters is the rank of $C$, the number of independent directions its columns span. Seven or more tubes of different colors almost always give full rank, 7, so $C\,w = z_t$ has an exact solution for any target. Our default set's 12 tubes leave 12 − 7 = 5 spare directions, so there are endlessly many exact answers. So have we done it? Can we make any color we desire? Not quite…

Real paint follows two rules: no negative amounts and amounts that add up to the whole mix. In symbols, $w$ must lie on the simplex $\Delta = \{\, w : w \ge 0,\ \mathbf{1}^\top w = 1 \,\}$. So mixing is averaging. The mixes your paints can make form the convex hull of their latents:

No mix is lighter than your lightest paint or more vivid than your most vivid one.

You'll never make white without white.

So the question becomes: what's the closest mix the rules allow? Below, four methods go from ignoring the rules to respecting both.