10-12% Off Our SDK's Time to Interactive, and the A/B Skills That Measured It

TL;DR: A run of performance changes to Rokt's web SDK, two of which turned out to be outliers, took 10-12% off our time to interactive, and the win held from p50 to p95. Writing the code was the easier half. The harder half was measuring what it was worth, so I packaged that method into two Claude skills, ab-setup and ab-diagnose. Setting up and reading the next one is now a step rather than a project.

Where the 10-12% Landed

Rokt's web SDK renders offers inside partners' checkout and confirmation pages. Within a month, I shipped a group of performance changes. Most made small improvements. Two were outliers and accounted for most of the measurable reduction in startup time. I could not have told you in advance which two they would be.

Seeing the same proportional improvement at p95 was the part I was happiest about. Those sessions are the ones most affected by slow devices and networks, and the absolute time saved was larger.

I Came Back With a List

I'm a boomerang, and I came back with a shortlist of performance problems I had noticed the first time around, which gave me somewhere practical to start instead of a blank profiler trace.

Rokt talks a lot about “speed of iteration,” and I took that literally: test an idea quickly, measure it, keep what worked, and move on from what didn't.

The first change was not glamorous: a bundle-size budget in CI. Every pull request now shows its size delta, so regressions are visible during review rather than months later. That bit of plumbing made every change after it easier to judge.

Two Ways We Cut Startup Time

I grouped the runtime work into two categories: reduce the JavaScript needed before render, and remove unnecessary delays from startup.

Load Less JavaScript Before Render

• Code-split features that most sessions do not need during startup.
• Removed legacy polyfills after defining an explicit browser-support floor, then consolidated duplicated helpers.
• Stopped loading code for features disabled in configuration.

Remove Startup Delays

• Ran independent async work concurrently and used the earliest readiness signal that each task required.
• Split bootstrap into work required before render and work that could happen afterwards.

Smaller changes helped too: memoizing hot factories, making event subscriptions lazy, cancelling superseded requests, and removing speculative preloads that no longer paid off. None produced a headline number alone, but I would ship every one of them again.

Leverage First, Delegate Second

AI helped me move through the list much faster. While an idea was still unproven, I kept the work interactive: trace this await chain, show what enters this bundle, and challenge this hypothesis. Once I chose an approach, I delegated the repetitive call-site edits and tests. That is the split I describe in AI-Assisted Coding Workflows: Delegating vs Leveraging.

What the Two Outliers Had in Common

Neither of the two largest gains made an individual calculation faster. One reduced how much JavaScript had to arrive and be parsed before render. The other moved nonessential work until after render. Both shortened the startup path that users wait for.

Existing timing markers already surrounded those phases, which made the gains measurable. Other changes reduced CPU or memory within a phase and did not necessarily move a phase-boundary metric. For every experiment, we wrote down which timing should change before looking at the result.

Now, How Do We Prove It's Faster?

So we'd made it faster. That felt good, but a latency chart only answers half the question. I also wanted to know whether the change appeared where the code said it should and what those milliseconds might be worth.

We're a Guest on Someone Else's Page

The SDK runs as third-party code on partner pages, so its startup time comes directly from their page-load budget. Rokt had also run a controlled delay study that estimated how revenue changes with each added second of latency. We could use that relationship in reverse to estimate the value of time saved.

The Measurement Was Harder Than the Code

The main result came from a combined holdback. Traffic was split 50/50: one cohort ran with the performance changes and one without them. We analyzed a closed window instead of checking the result each day and stopping when it looked favorable. That is where the 10-12% comes from, and it is the number I would put my name on.

The latency difference appeared quickly, but we ran the test for several days to check that assignment stayed balanced and daily results were consistent. Revenue needed much more data because the expected change was only a fraction of a percent.

One of the two main changes also had its own randomized arm. That let us check whether the improvement appeared in the phase changed by the code:

The framework phase improved across all three percentiles. A later phase, untouched by the code, stayed flat in the same analysis. That mechanism check gave us better evidence than a top-level latency chart alone.

What Could Faster Mean for Revenue?

This was the number I really wanted to understand. With the delay study as a conversion between seconds and revenue, the latency result became something we could estimate in business terms.

The model estimated a 0.3–0.5% revenue improvement at p50 and 1.4–1.9% at p95. The p95 range is higher because the model uses seconds, not percentages. Saving 12% at p95 removes more milliseconds than saving 11% at p50.

In the holdback itself, revenue per transaction moved +0.20%. Seeing that was exciting. The direction matched the model, but the result was not statistically significant at this sample size, so I treat it as supporting evidence rather than a proven revenue increase.

For context, Rokt reported US$834 million in revenue last year. That figure shows the scale involved, but it is not a base I can multiply by 0.20%. The SDK covers only part of company revenue, and the observed movement was not conclusive.

The Skills We Built So the Next One Is Easier

Proving the result took more work than writing either optimization. I did not want anyone, including future me, to reconstruct the assignment, power, and analysis rules from old queries six months later. So I split the method into two Claude Skills, one for each side of the experiment.

How the Next Experiment Connects Performance to Impact

Say the next idea is to move another startup task until after render. Before writing the experiment plumbing, I can ask ab-setup to prepare the test around that change.

I give it the proposed treatment, the timing phase expected to move, the business outcome connected to that timing, and the available traffic. The skill returns the assignment design, enrollment marker, control behavior, primary metric, guardrails, power check, and removal plan. If the expected business effect needs more traffic than we can reasonably collect, that becomes clear before the experiment starts. We can make the performance metric the primary claim instead of pretending the test can prove more than the sample allows.

After the test has run for its pre-registered window, ab-diagnose checks the production markers and sample ratio first. It then rebuilds the cohorts from enrollment, applies the agreed exclusions, and compares the target timing with the nearby metric that should remain flat. Finally, it connects the measured time saved to the delay model and reports each conclusion at the level the evidence supports: measured performance, modeled impact, or directional business movement.

Input to ab-setup
├─ Treatment: move task X until after render
├─ Expected mechanism: framework startup should fall
├─ Primary metric: SDK time to interactive
├─ Impact model: revenue sensitivity to time saved
├─ Guardrail: downstream render phase stays flat
└─ Available traffic and test window

Output from ab-diagnose
├─ Assignment and sample-ratio checks
├─ Measured timing effect by percentile
├─ Mechanism and guardrail results
├─ Uncertainty and power
└─ Measured performance, modeled impact, and directional business evidence

The skills do not decide whether an experiment succeeded or manufacture a business result. They make sure we ask the same questions, in the same order, and preserve the line between a faster system and the impact we can defensibly connect to it.

A document still depends on someone remembering to find it. A skill is available when the experiment starts and can generate the setup, checks and analysis from the same rules each time.

Closing Thoughts

This work improved two systems. The first was the SDK itself: it now sends less JavaScript during startup and no longer assumes that nonessential work must finish before rendering. That gave users a faster experience from p50 through p95 and left the startup path easier for the next engineer to change.

The second was how we measure impact. A latency result tells us the system is faster. The mechanism check tells us the improvement came from the code we changed. The delay model helps us estimate what the saved time could be worth. The holdback tells us whether the business outcome moved in the same direction. Those are related signals, but they are not interchangeable, and the strength of the claim should match the strength of each one.

That connection should be designed with the optimization, not assembled when someone asks for an impact summary at the end. ab-setup starts with the expected system change, its user-facing metric, and the business outcome it could influence. ab-diagnose follows that chain back through the data and reports exactly where the evidence stops.

The code changes improve the product today. The Skills make the next improvement easier to measure and easier to connect to impact. I'm proud of the 10-12%, but the number carries more weight because we can explain where it came from, what it might be worth, and what we have not yet proved.

The numbers matter. So does earning the right to quote them.