My default has long been the empirical CDF, or eCDF. Once you get used to reading them, eCDFs are almost always cleaner: no bin widths, percentiles are directly visible, and zooming in does not make you forget about the tails.

An eCDF plots the fraction of observations less than or equal to each value. Formally, for a sample $x_1,\ldots,x_n$, it is

In plain English: the x-axis is the value, and the y-axis is the cumulative share of the data. If the curve is at 0.8 when $x = 12$, then 80% of the observations are at or below 12.

Here is a simple example. The sample mostly comes from a central distribution, with a visible right-tailed outlier component. Draw a few random samples, change the histogram bin width, and toggle the x-axis zoom. The histogram keeps asking you to make plotting choices; the eCDF answers distribution questions directly.

No bins

The first advantage is that eCDFs have no bin width.

With a histogram, the shape can change depending on where the bins start and how wide they are. Make the bins too narrow and the plot looks noisy. Make them too wide and you smooth away real structure. There are rules of thumb for choosing bin widths, but the fact that you need a rule of thumb is already a smell.

An eCDF has no bins. Every observation is used directly, and the plot means exactly what it says. There is no hidden smoothing parameter quietly changing the visual impression.

Percentiles are the native units

The second advantage is that percentiles are trivial to read. Want the median? Look at where the curve crosses 0.5. Want the 90th percentile? Look at where it crosses 0.9. Want to compare two distributions? Plot two eCDFs and see which curve reaches a given cumulative share sooner. This makes the plot especially nice for questions like “how bad is the worst 10%?” or “what share is under this threshold?”

Histograms are worse for this because they show local mass rather than cumulative probability. That is sometimes what you want, but often you end up mentally integrating the bars anyway. If the question is about ranks, percentiles, thresholds, or tail probabilities, the eCDF is already in the right units.

Zooming without losing the tails

The third advantage is that eCDFs behave much better when there are outliers.

With a histogram, a few extreme values can stretch the x-axis so far that the relevant part of the distribution gets crushed into a small region. You can zoom in, but then the tails disappear from the plot entirely. So you are forced to choose between seeing the main body clearly and remembering that the tails exist.

With an eCDF, zooming into the relevant x-range does not have the same failure mode. If you cut off the x-axis at, say, the 99th percentile, the curve simply stops near 0.99. The missing 1% is still visible as missing vertical distance. You can focus on the region where most of the data lives without pretending that the tail mass is zero.

This is a nice separation of concerns. The x-axis can focus on the region you care about, while the y-axis still accounts for the whole sample. If the curve ends at 0.96, then 4% of the sample remains above the visible range. That is much harder to miss than a zoomed histogram with chopped-off bars.

The familiarity trap

The real advantage of histograms is familiarity. People know how to read them because they have seen them a thousand times. That is not nothing, and in narrow cases it may be decisive. Otherwise, “histogram” mostly means “the plot everyone is used to”, which is not a statistical argument.

For routine distribution checks, histograms are usually a bad bargain. They make you choose bins, they hide percentiles, and they handle outliers poorly. If the question is about thresholds, ranks, quantiles, tail probabilities, or comparing distributions, a histogram is forcing the reader to do extra work. The eCDF is already showing the relevant object.

The standard example is something like fraud detection: suppose only 1% of transactions are fraudulent. People see the 99/1 split and immediately reach for over-sampling, under-sampling, SMOTE, class weights, or some other intervention to “fix” the training set. But in many cases there is nothing to fix. The imbalance is not a bug in the data. It is a fact about the world that the model is supposed to learn.

This is especially true if the classifier is meant to produce probabilities. If the true probability of fraud is usually small, then a good model should usually predict small probabilities. That is not a failure mode. That is the correct answer.

The loss function matters here. For a binary classifier that predicts $p_i = Pr[y_i = 1 \mid x_i]$, the negative log-likelihood, aka log loss, is

This loss already accounts for both classes. A positive example contributes $-\log(p_i)$; a negative example contributes $-\log(1-p_i)$. If positives are rare, there are fewer positive terms in the sum because positives really are rare. The maximum likelihood estimate is trying to fit the conditional probability distribution in the population you sampled from. It is not secretly confused by the fact that one class occurs more often than the other.

In fact, the imbalance is often exactly what determines the intercept, or more generally the baseline probability. If you train a logistic regression model on representative data where positives occur 1% of the time, the model has to learn that the base rate is around 1%. If you over-sample the positives until the training data is 50/50, you have told the model a different story about the world. Unless you correct for that later, the predicted probabilities will be too high.

This is the main point that gets lost in the usual discussion: resampling changes the objective. Over-sampling the minority class is equivalent to giving those examples extra weight. Under-sampling the majority class is equivalent to throwing away information from the common class. Both can be useful if you deliberately want a different objective, but they are not neutral preprocessing steps. They change what the model is optimizing.

Here is a simulated logistic regression example. The model is well-specified and the test set has the real base rate. Changing the positive-class weight mostly shifts the log-odds, so the ranking survives while the probabilities get worse. The calibration plot compares mean predicted probability on the x-axis with the observed positive rate on the y-axis; the dashed diagonal is where predictions match reality:

This matters because log loss is a proper scoring rule. In plain English, that means the loss is minimized by telling the truth: if the conditional probability is 0.03, the best prediction under log loss is 0.03. But if you train on an artificially balanced dataset, the empirical distribution no longer has the same base rate as the real distribution. The model may still learn a ranking that is useful for discrimination, but its raw probabilities will generally be miscalibrated. And calibration is exactly what you want if the downstream decision depends on expected value, risk, or any comparison of probabilities.

There are reasonable caveats. If the rare class is extremely rare, you may not have enough examples to learn much about it. That is a sample size problem, not a class imbalance problem. The solution is usually more data, better features, pooling across related cases, or stronger regularization. Also, if your optimization procedure struggles because mini-batches rarely contain positives, there may be engineering tricks that help training. But those tricks should be understood as optimization aids, not as corrections to the statistical target.

Similarly, if your goal is not calibrated probabilities but a specific decision rule, then asymmetric costs may justify a weighted loss. For example, missing a fraud case may be much worse than annoying a customer with a review. But then the class weights should come from the costs of the decision problem, not from the class frequencies themselves. The fact that fraud is rare does not automatically imply that fraudulent examples deserve 99 times the weight.

The clean separation is:

1. Use log loss if you want calibrated probabilities.
2. Choose a threshold using the costs and benefits of action.
3. Evaluate calibration and decision quality on data with the real base rate.

The common mistake is to mix these steps together. People see an imbalanced training set, modify it to make the classes look balanced, train a classifier, and then wonder why the predicted probabilities are off. But the probabilities are off because the model was trained on a distorted version of reality.

Class imbalance can be a warning sign that you need more data or that naive accuracy will be a useless metric. It is not, by itself, a reason to resample the training set. If the training set is representative and the model is trained with log loss, the imbalance is already part of the likelihood. Treat it as signal, not contamination.

For example, this question comes up with prediction markets, where the market price can be interpreted as the best estimate of the probability of the event $X$ occuring. But there are no prediction intervals around this market price. Or consider models for classification, such as logistic regression or other machine learning algorithms, which produce predicted probabilities for each possible class (e.g., $X_i =$ “transaction $i$ is fraudulent”). In both of these cases, we face the same issue with representing uncertainty — how does the market/model express confidence in its predicted probabilities?

In this post, I’ll explain why this question stems from a fundamental confusion: it’s a misconception to think that a predicted probability is a point estimate that doesn’t convey any uncertainty. Below, I’ll show that there are two distinct sources of uncertainty that are being conflated here, and that one or both can be used to express uncertainty.

Two types of uncertainty

The key distinction here is between:

1. Uncertainty over the outcome, $X$ vs $\neg X$
   • Also known as aleatoric uncertainty
   • (FYI: the symbol "$\neg$" is the negation operator and can be read as “not”)
2. Uncertainty over model parameters which are used to generate a prediction for the outcome
   • Also known as epistemic uncertainty

Let’s unpack each case in depth.

Uncertainty over outcomes

When we aren’t working with a model, we only have the first source of uncertainty to deal with. But it isn’t obvious where the uncertainty lies: if we say $Pr[X] = 0.02$, it may appear that we’ve just given a point estimate. But recall that this is a binary outcome space, i.e., the only possible outcomes are $X$ or $\neg X$. So the full probability distribution (over the two possible outcomes) can be summarized by one probability, $p := Pr[X]$ (which implies $1-p = Pr[\neg X]$). As we’ve provided a full probability distribution over the outcome space, it’s not possible to say anything more — any uncertainty must be embedded in this distribution.

Intuitively, probabilities near 0 or 1 reflect a high degree of certainty. A prediction without any uncertainty at all would just be a yes or no answer, i.e., a predicted probability of 0 or 1. It would just state which outcome will occur, with no notion of uncertainty or hedging.

More precisely, confidence in a probability prediction is reflected by how extreme it is relative to a baseline or prior belief. To see this, suppose that there is an event $X$ that is very likely to occur, and that a prediction market has given $X$ a predicted probability of 0.97. If you are maximally uncertain/ignorant about $X$, what probability do you assign? Intuitively, you hedge your bets and stick to 0.97. Here, 0.97 is the baseline, which you can treat as your prior probability. Given this prior information, a prediction of 0.97 reflects maximal uncertainty. (If you didn’t have any prior information whatsoever, you would go with 0.5.)

Then, if you have some new information about $X$, you can update your prior to get a posterior. If your information provides strong evidence in favor of $X$, then your posterior probability might jump up to, say, 0.997. On the other hand, if your information strongly supports $\neg X$, then your posterior might drop to, say, 0.78. Thus, your degree of confidence is revealed by the degree to which your probability moves away from the baseline and toward 0 or 1.

You can use Bayes’ Theorem to play with some numbers yourself. Denote your prior by $p := Pr[X]$, and assume you’ve used your information $D$ to compute the likelihoods $q(X) := Pr[D \mid X]$ and $q(\neg X) := Pr[D \mid \neg X]$. Denote the likelihood ratio by $\lambda := q(X) / q(\neg X)$.

Then compute the posterior and rearrange in terms of the likelihood ratio:

Note that the posterior can be expressed purely in terms of the prior and the likelihood ratio (i.e., it doesn’t depend on the individual likelihoods). This means that the magnitudes of the likelihoods don’t matter; all that matters is their ratio, which indicates how much the information $D$ favors $X$ relative to $\neg X$.

If you play with this formula, you’ll get a sense of how the information in the likelihoods updates the prior to a posterior probability. Notice that when $\lambda = 1$, the posterior reduces to $p$, the prior. In other words, when $D$ is uninformative about $X$, it leaves your prior belief unchanged. Furthermore, for any prior belief $p$, if $\lambda > 1$, then your posterior will be pushed upward from your prior (and vice versa for $\lambda < 1$). That is, any information in favor of $X$ will increase your confidence in $X$ — even if $p=0.999$!

Judging the quality of probability predictions is simple: just check that they’re calibrated. For example, predictions made with, 80% confidence should be correct 80% of the time. With enough completed predictions, you can plot a reliability diagram to assess the calibration of the predictions.

Uncertainty over models

When we are working with a model, we also have a second source of uncertainty — that of the model. This uncertainty is reflected in the posterior distribution over the parameters (at least for Bayesians — frequentists would use the sampling distribution of the parameter estimator to derive confidence intervals / standard errors). Because these parameters are used to produce predictions, their uncertainty propagates through to produce additional uncertainty over the outcome.

This is meta-uncertainty: uncertainty over the model which produces the uncertain prediction of the outcome. (In fact, you can have higher levels of meta-uncertainty by including uncertainty over any hyperparameters of the model.)

For example, here’s a specification for a Bayesian logistic regression model, where I’ve put an informative prior on the model coefficients:

You can see how the uncertainty from the model’s prior (Normal distribution) propagates through, adding to the uncertainty in the likelihood (Bernoulli distribution). As a result, we get a full density for $p_i$ over the interval $(0,1)$. The spread of this density reflects model uncertainty, whereas the distance from the prior distribution reflects the degree of confidence in the prediction of the outcome. The key thing to realize is that these two sources of uncertainty are orthogonal.

For example, you could simultaneously have a very confident prediction of the outcome but with a lot of model uncertainty — a widely spread posterior distribution that is far away from the prior distribution:

Or you could have a very unconfident prediction of the outcome with very little model uncertainty — a tightly distributed posterior distribution that remains close to the prior distribution:

Summing up

The argument I’ve made here can be summed up as:

1. A probability prediction is a full probability distribution and so it inherently quantifies uncertainty — it’s a misconception to think that you need a prediction interval to express uncertainty
2. Model uncertainty propagates through to produce an additional (but orthogonal) layer of uncertainty over the outcome

This may seem obvious in retrospect, but it’s always good to gain clarity on the fundamentals, where confused intuitions may lurk unnoticed. Here, intuitions such as “a scalar prediction must be a point estimate” and “you need a confidence/prediction interval to express uncertainty” are highly misleading.

In this case, understanding the distinct sources of uncertainty has resolved some confusion I had about prediction markets and machine learning model predictions. The upshot is that (if I’m comfortable ignoring model uncertainty), I need only be concerned that the probability predictions are calibrated. This is straightforward to check: for prediction markets, you just need some historical data on outcomes; for classifier models, you can check on a holdout dataset. Then you can be comfortable that the predictions are accurately quantifying uncertainty.

It’s an idealistic vision — it ignores the political economy that sustains the gigantic inefficiencies he decries — yet it’s an overwhelmingly sensible one. Perhaps I’m biased towards the views of pro-growth economists, but this essay was packed with an uncommon amount of common sense (which is a pretty low bar when it comes to the topic of policy).

The essay makes a two-part argument: that growth should be a very high policy priority, and that there’s a lot of inefficient regulation that hinders growth. I’ll quote a bunch of highlights below if you don’t want to read the whole thing.

In the long run, nothing but growth matters. Small differences in growth rates lead to dramatically different outcomes, because of compounding.

If the US economy had grown at 2% rather than 3.5% since 1950, income per person by 2000 would have been $23,000 not$50,000. That’s a huge difference. Nowhere in economic policy are we even talking about events that will double, or halve, the average American’s living standards in the next generation.

The primacy of economic growth should be a really obvious point, but it’s easy to lose sight of the big picture. If you think about it, economic growth is the only thing that has lifted nations out of poverty.

Nothing other than productivity matters in the long run. A factor of three increase in income in 50 years, and the much larger rise in income and health since the dawn of the industrial age, dwarfs what unions bargaining for better wages, progressive taxes or redistribution, monetary, fiscal or other stimulus programs, minimum wage laws or other Federal regulation of labor markets, price caps and supports, subsidies, or much of anything else the government can do.

And the primacy of growth should be non-controversial:

38% more income — or 26% less income — drives just about any agenda one could wish for, from strong defense, to environmental protection, to the affordability of social programs, to the welfare of any segment of the population, to public investments, health, and fundamental research.

Dumb regulation hinders growth. To increase growth, there’s a lot of obviously wasteful regulation that could be made a lot smarter. Unfortunately, that regulation is there because of problems of political economy, and so it’s not as easy to fix as it looks.

When the average person (voter) expresses concern over inequality, what they really mean is that they are concerned that average people are not getting ahead economically. If the average person were getting ahead, whether some big shot CEOs fly on private jets or not would make little difference. Conversely, the average voter, if not the average left-wing pundit, does not support equality of misery. If the average person continues to do poorly, it would bring them little solace for the government to tax away the lifestyles of the rich and famous.

On the problems of health insurance regulation:

The central problem of preexisting conditions was an artifact of regulation. In the ideal form of health insurance, you buy cheap catastrophic insurance when young, but the insurance policy can follow you as you age, change jobs, and move from state to state, and does not radically increase premiums if you get sick. […] We need to allow simple, portable, largely catastrophic, lifelong, guaranteed-renewable health insurance to emerge. Right now it’s illegal.

On energy policy:

The poster child for inefficiency may well be the mandate for gasoline producers to use ethanol. Corn ethanol, it turns out, does nothing to help the environment: It takes nearly as much petroleum energy to produce it as it contains, in the form of fertilizer, transport fuel and so on; it uses up valuable land, which directly emits greenhouse gases, and contributes to erosion and runoff; it drives up the price of food.

If you are serious about carbon, let the words “nuclear power” pass your lips. We have sitting before us a technology that can easily supply our electricity and many transport needs, with zero carbon or methane emissions. New designs, if only they could pass the immense regulatory hurdle, would be much safer than the 1950s Soviet technology that failed at Chernobyl or the 1960s technology that failed at Fukushima. We are now operating antiques. And even with this rate of accident, nuclear power has caused orders of magnitude less human or environmental suffering than any other fuel.

Similarly, the most environmentally friendly way for people to live is in tightly packed cities, fed by genetically modified foods which yield more per acre of farmland and require fewer fertilizers and pesticides, from laser-leveled fields run efficiently by large corporations in the highest productivity locations. Federal policies to the contrary are not just anti-growth, they’re anti-environment too. When Federal policy can say these things in public, it will have a bit more standing to invoke the name of “science.”

On taxation:

Often, however, tax reform proposals sacrifice too quickly the principles of what a good tax system should be with perceived political accommodations to powerful interest groups. Economists should not play politician. We should always start with “in a perfect world, here is what the tax code should look like,” and accommodate political constraints only when asked to. Political constraints change quickly. Economic fundamentals do not.

The right corporate tax rate is zero. Corporations never pay taxes. Every dollar of taxes that a corporation pays comes from higher prices of their products, lower wages to their workers, or lower returns to their owners. […] For all these reasons, eliminating the corporate tax is as likely to be more rather than less progressive. The higher prices a corporation charges hurt everyone. The lower wages corporations pay hurt workers. The income it passes along to its owners is subject to our highly progressive tax system.

When we say broaden the base by removing deductions and credits, we should be serious about that. Thus, even the holy trinity of mortgage interest deduction, charitable donation deduction, and employer provided health insurance deduction should be scrapped. The extra revenue could finance a large reduction in marginal rates. Why? Consider the mortgage interest deduction. Imagine that in the absence of the deduction, Congress proposes to send a check to each homeowner, in proportion to the interest he or she pays on money borrowed against the value of the house. Furthermore, rich people, people who buy more expensive houses, people who borrow lots of money, and people who refinance often to take cash out get bigger checks than poor people, people who buy smaller houses, people who save up and pay cash, or people who pay down their mortgages. A rich person buying a huge house in Palo Alto, who pays 40% marginal income tax rate, gets a check for 40% of his huge mortgage. A poor person buying a small house in Fresno, who pays a 10% income tax, gets a check for 10% of his much smaller mortgage. There would be riots in the streets before this bill would pass. Yet this is exactly what the mortgage interest deduction accomplishes.

On labor markets:

Start, of course, with taxes: income taxes and payroll taxes are primarily taxes on employment. But the regulatory burdens of employment are larger still, as anyone who has tried to get a nanny legal will attest. Minimum wages, occupational licensing, anti-discrimination laws, laws regulating hours people can work, benefits they must receive, leave they must be given, fear of lawsuits if you fire someone, and so forth all impede the labor market.

The usual argument is that workers need protection of all these laws. Well, the supposed protections do cost economic growth, and they do reduce employment. How much do they actually protect workers? The strongest force for worker protection is a vibrant labor market — if you don’t like this job, go take another. The tightly regulated labor market makes it much harder to get a new job, and thus, paradoxically, lowers your bargaining power in the old one.

An exceptional section on immigration. Had a hard time not quoting the whole thing:

Immigrants contribute to economic growth. Even if income per capita is unchanged, imagine how much better off our social security system, our medicare system, our unfunded pension promises, and our looming deficits and debt would be, if America could attract a steady flow of young, hard-working people who want to come and pay taxes. Aha, we can attract them! They’re beating the doors down to come. But then we keep them out.

Allowing free migration is, by many estimates the single policy change that would raise world GDP the most. If you believe in free trade in goods, and free investment, then you have to believe that free movement of people has the same benefits.

On education:

The culprit is easy to find: awful public schools run by and for the benefit of politically powerful teachers’ and administrators’ unions. (Don’t forget the latter! Teachers account for only half of typical public school expenses.) Education poses a particularly large tradeoff between profits to incumbents and economic growth, since education lies at the foundation of higher productivity. In addition, the costs of awful schools fall primarily on lower-income people who cannot afford to get out of the system. It is one of the major contributors to inequality. The solution is simple as well: widespread financing by vouchers and charter schools. As with health care, a vibrant market demands that people control their spending, and can move it to where they get better results. As with health care, the government does not have to directly provide a service in order to help people to pay for that service. But as with health care, a healthy market also demands supply competition, that new schools be allowed to start and compete for students.

As exciting as this vision is, I’m not very optimistic about any of these prescriptions being implemented, at least not in the near term. But it’s good to know that we have a lot of simple solutions if we ever get desperate. And I hope that some of these proposals (like immigration) can gain enough momentum to be implemented in the medium term.

-Richard West

I'm no expert on this topic, but it seems to me that the harms of tobacco are from smoking (and chewing) and not from the nicotine itself (apart from the addictiveness). So nicotine administered with gum/patch/vaping would be somewhat more like caffeine in harmfulness (loads of people are addicted to caffeine).

Nicotine also seems to have similar useful effects. From Wikipedia: "Nicotine appears to have significant performance enhancing effects, particularly in fine motor skills, attention, and memory." A Martian scientist comparing the harms and benefits (for humans) of drinking coffee vs vaping nicotine might think that both drugs are safe and have productivity-boosting effects, and wouldn't see a reason for only one of them to be socially acceptable.

So why is nicotine held to a much higher standard? Well, the public health effort to end smoking became a moral crusade. As Jonathan Haidt puts it, morality binds and blinds. The anti-smoking crusade gained enough strength that it created a hated enemy. Smokers in North America are now a low-status out-group from the perspective of most non-smokers. As usual, when activists morally charge their case by painting something as sacred or evil, they commit themselves to not giving up an inch. Smoking is pure evil, and nicotine is central to smoking, so nicotine must be evil too. Any admission that nicotine might not be horrible is helping the enemy.

Caffeine, on the other hand, did not get loaded with negative affect because it had a safe delivery system. Imagine if nicotine had historically been consumed in drink form instead of smoking -- in such a world, I imagine that it would just be another stimulant drink like coffee or tea.

It's a real shame that e-cigarettes (nicotine) are being morally condemned along with cigarettes (tobacco). If smokers can switch to vaping, they can avoid all the harm and still get their nicotine. From a public health perspective, getting smokers to switch is a huge win. Doctors should be prescribing e-cigs to smokers, government should promote switching. Unfortunately, since nicotine has been painted evil by association, it's probably not going to happen.

The point I want to make here is that the emphasis on prediction is just a convenient special case of a more general principle: that a theory should correspond to reality, just as a map corresponds to the territory. Reality can be observed past, present, and future. It is just as well to vet a theory against past observations as against future observations. The reason the scientific method favors prediction is that it prevents the scientist from concocting a 'just so' story that too neatly fits the existing facts ("overfitting"). An idealized honest scientist can test a theory against any empirical evidence.

Darwin's theory is so amazing because it makes sense of so many existing facts. (And you can still make predictions about historical facts ("retrodictions"), such as the famous quip that evolution would be falsified by finding fossil rabbits in the precambrian.) Of course, if you have a beautiful theory that has no connection to reality, then you're not doing science. Science is concerned with explaining reality, and so scientific theories must say things about reality – things which can (in principle) be empirically checked. It ultimately doesn't matter where that evidence is temporally located.