Peeking
Peeking is not a strategy
A product lead once told us they “just glance” at the feature-flag dashboard every morning so they can kill a bad variant early. That glance is a look. Looks spend alpha. Pretending otherwise does not make the false-positive rate smaller.
In Variant Readout Desk we treat peeking as a design choice, not a moral failing. Feature-flag tools make daily charts easy. PMs are paid to be impatient. The combination is predictable: a test declared for 21 days dies on day six because a line went green on a high-traffic Friday.
What the glance actually does
Each unplanned look is another chance to stop for a fluke. If you check every morning for three weeks, you have not run one test. You have run a sequence of tests that share a sample. The p-value on the screen was not built for that sequence. It will flatter you.
We ask students to count the looks they already took on a past flag. Most underestimate. The flag platform’s “always-on chart” counts even when nobody opened a notebook. If the chart existed, someone looked.
Pre-commit or do not look
Sequential Peeking Clinic teaches two honest options. One: no interim looks until the pre-declared runtime ends, holidays excluded. Two: a small number of pre-committed looks with an alpha spend written down before first exposure. “We will check on day 7 and day 14, and only then” is a design. “We will check if the PM asks” is not.
Thai calendars make this sharper. A glance across Songkran is not the same glance as a quiet Tuesday. If your only green day sat inside a tourism spike, you did not learn about the feature. You learned about the island.
What we do not teach
We will not show you how to p-hack a flag or how to dress a daily peek as “Bayesian updating” without changing the decision rule. If you need to stop early for safety — crashes, payments, legal — that is a kill switch, not an experiment readout. Write it as a kill. Do not file it as a win.
The memo lab scores whether you named your looks. Students who hide them fail the module even when the primary metric moved. That rule has cost us a few glowing reviews. We kept it.