Tackling Customer Segmentation

A business held a book of roughly seventeen thousand customers and wanted to group them so that pricing could be set by group rather than one account at a time. The request sounds like bookkeeping. It is not. Deciding which group each customer belongs to — and first deciding what the groups even are — is the entire problem, and the obvious answer is wrong.

The obvious answer is the label each customer already wears. Every account in this data carried an assigned industry — automotive, machinery, chemical, food, and so on. It is tempting to price by that label. The analysis below shows why doing so would price on a dimension that does not describe how customers actually behave.

The problem is clustering, not classification

Two tasks are easy to confuse. Classification assigns each item to a group that is already known and labeled: you have examples of each bin, and you place new items into them. Clustering is the harder, unsupervised task: the groups are not known in advance. You have to discover them from the structure of the data itself, then decide how many there are.

Grouping customers for pricing is clustering. Nobody handed over a correct set of pricing tiers with labeled examples. The tiers had to be found. That distinction governs every decision that follows.

Choosing a method

Several established methods cluster data, each with a known weakness:

• K-Means partitions records into k groups by minimizing distance to a group center. It is the most widely used method, but it requires the number of groups, k, to be fixed in advance, and it measures distance with Euclidean geometry — which is defined only for numeric values.
• Hierarchical clustering builds a tree of nested groups but offers no objective rule for where to cut it into a final number of clusters.
• DBSCAN finds dense regions and handles noise, but struggles when clusters have different densities and is highly sensitive to its two parameters.
• Gaussian Mixture Models assign soft probabilities but give no clean measure of how strongly a point belongs to its cluster.

K-Means is the standard choice and the one adopted here — with one modification that matters.

The data is mixed, so the method has to be

K-Means fails on this data as-is, because customer records are not all numbers. They hold two kinds of field at once:

• Continuous numeric fields that have magnitude — the count of distinct HTC codes a customer imports (the Harmonized Tariff Code is the international system that assigns every traded good a numeric classification), the number of vendors, the number of products, the count of customs entries and line items. These can be added, averaged, and measured by distance.
• Categorical fields that have no magnitude — the dominant mode of transport (truck, ocean, air, rail), a top-100 flag. “Ocean” is not larger than “truck”; these are names, not quantities, and Euclidean distance is undefined for them.

The fix is KPrototypes, a variant of K-Means introduced by Huang (1998). It measures numeric fields with squared Euclidean distance, measures categorical fields by counting mismatches, and combines the two into a single dissimilarity score using a weight, γ (gamma), that sets how much a categorical disagreement counts against a numeric one. A cluster’s prototype is the mean of its numeric fields and the mode of its categorical fields. This is what lets a single model cluster on magnitude and on category together.

How many groups? The elbow method

K-Means and KPrototypes both require k, the number of clusters, before they run — which is awkward, because finding the groups is the point. The number has to be chosen deliberately.

The elbow method is the standard technique. Run the clustering for a range of k values and record, for each, the total within-cluster cost — the summed dissimilarity of every point to its own cluster’s prototype. That cost always falls as k rises: more clusters always fit the data more tightly, and at k equal to the number of points the cost reaches zero. So the goal is not the lowest cost. It is the point where adding another cluster stops buying much — the bend, or “elbow,” in the curve, after which each new cluster yields only a marginal improvement.

On this data the cost falls steeply from k=2 to k=4, then flattens: the drop is roughly 11,700 from k=2 to k=3, but under 2,000 per step past k=5. The bend sits at about four to five clusters.

The elbow is intuitive but not the only option. The gap statistic, the silhouette score, the Calinski–Harabasz score, and the Davies–Bouldin score each formalize “how well-separated are the clusters” differently, and are worth consulting when the elbow is ambiguous. Here the elbow was clear enough; the model was fit at five clusters.

What the groups turned out to be

The five discovered clusters are not arbitrary. Ordered by activity, they form a clean ladder in which each tier is roughly an order of magnitude above the one below it, on every measure at once. The median customer in the smallest tier imports one distinct HTC code from one vendor; the median customer in the largest imports 191 distinct codes from 126 vendors and files tens of thousands of customs lines. The regularity of the ladder is evidence that the tiers are real: the data genuinely separates into levels of engagement.

The categorical field earns its place here. Mode of transport is not a size, so K-Means could not have used it, but KPrototypes can. Across the whole book, trucking carries about 85% of shipments, ocean about 9%, air under 6%. The mix shifts slightly by tier — the largest customers lean marginally more on trucking — a modest signal, but a real one that a numeric-only method would have discarded.

The finding: the industry label does not predict behavior

Now the result that justifies the whole exercise. Compare the discovered behavioral tiers against the industry categories the customers already carried.

They do not agree. Every industry spreads across all five tiers — and in nearly the same proportions. Automotive customers split roughly 51 / 20 / 13 / 9 / 7 percent across the tiers from smallest to largest. Machinery, chemical, textile, food, lumber — each shows almost the identical internal spread. Knowing a customer’s industry tells you almost nothing about how large or active that customer is.

This is the point the industry label misses. The categories are a real classification — they are just the wrong axis for pricing. A pricing scheme built on industry would charge an automotive account and treat it as a coherent group, when automotive accounts in fact range from a single-entry importer to one filing tens of thousands of lines. The behavioral tiers, discovered from actual activity, are the coherent groups. They are the correct basis for adjusting price.

Why it matters

The clustering bought three concrete things.

Because every customer is scored by the same rule, the whole book can be grouped at once and re-grouped the moment new figures arrive, without anyone re-arguing what a tier should mean. The rule decides, identically, every time — and it applies to a new customer on day one, from the same handful of measured fields.

Because a few observed features place a customer on the ladder, pricing and processing effort can be estimated rather than guessed. And because the discovered tiers cut across the legacy industry categories, the analysis also delivers a warning: the grouping the business already had was not describing the behavior it wanted to price on.

None of this survives on its own. A grouping is only as good as the records under it and the yardstick used to judge it. The clusters should be re-checked as new customers, new goods, and new codes arrive, and the underlying data kept clean from the start — an ignored mess does not stay the same size.

References

1. Huang, Z. (1998). Extensions to the k-Means Algorithm for Clustering Large Data Sets with Categorical Values. Data Mining and Knowledge Discovery, 2(3), 283–304. — the origin of the k-prototypes method.
2. de Vos, N. J. kmodes: Python implementations of the k-modes and k-prototypes clustering algorithms. github.com/nicodv/kmodes
3. Aprilliant, A. The k-prototype as Clustering Algorithm for Mixed Data Type (Categorical and Numerical). Towards Data Science.
4. Thorndike, R. L. (1953). Who Belongs in the Family? Psychometrika, 18(4), 267–276. — an early statement of the elbow idea.
5. Rousseeuw, P. J. (1987). Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20, 53–65.
6. Tibshirani, R., Walther, G., & Hastie, T. (2001). Estimating the number of clusters in a data set via the gap statistic. Journal of the Royal Statistical Society: Series B, 63(2), 411–423.
7. Caliński, T., & Harabasz, J. (1974). A dendrite method for cluster analysis. Communications in Statistics, 3(1), 1–27.
8. Davies, D. L., & Bouldin, D. W. (1979). A Cluster Separation Measure. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-1(2), 224–227.
9. World Customs Organization. Harmonized Commodity Description and Coding System (HS). — the HTC classification underlying the tariff-code features.