Alexey Elkin

Our payment infrastructure had no way to detect when Stripe and our database fell out of sync. Stripe processes a payment, we write the result to our database, and if that second step fails for any reason there's no error, no alert, nothing. The two systems just quietly disagree. I noticed this while reviewing our payment flows and started mapping out where it could happen.

I found four places. An operation that touched Stripe and our database in sequence, with nothing watching what happened in between. A refund could process on Stripe's side and never land in our records. A payment could go through but fail to create an order on our end. Creator earnings could simply never be calculated if a specific payment event was missed. And a payout could be marked as completed in our system before the actual bank transfer was attempted, leaving money stuck in a permanent "completed" state that never actually moved. None of it produced a visible error.

Patching each flow individually would have been the wrong call. The real problem is structural: you can't make two external systems agree at the exact moment of writing. The right answer is to accept that inconsistencies will occasionally happen and catch them after the fact.

I built a background job that runs on a schedule, looks back over the last 24 hours, and compares what Stripe says happened against what our database recorded, across payments, refunds, invoices, fees, and payouts. Any gap triggers an immediate Slack alert. The job only detects and reports, it never silently corrects, because you want to understand a new failure mode before you automate a fix for it.

Refine's product is a search engine, which means the database was ingesting a continuous stream of search events from customer sites around the clock. ClickHouse, which is built for analytical queries over large volumes of time-series data, was storing all of it. The analytics dashboard was reading from it directly, and the problem was that every single load was scanning the full event history to compute what to show. Roughly 50 million records on every request. You could see it in how long the page took to come up.

The first thing I changed was how the data was structured for querying. Instead of aggregating across the raw event history on every request, I set up materialized views that pre-aggregated the data by hour, day, and week as events came in. A query for "last 7 days" went from scanning tens of millions of rows to reading a few hundred pre-computed ones. That handled most of the latency.

The second change was caching. The results of a dashboard query for a given time window don't change until new data arrives, so hitting ClickHouse on every load was unnecessary. I added a Redis cache in front of the queries so that repeated requests for the same window returned instantly. Between the two changes, response times for uncached requests came down from 300ms to around 50ms, and cached requests returned instantly.

When I built the promotions system at Terra, the first question wasn't what features it needed. It was what a discount actually does to the payment stack. It changes what Stripe charges, what the creator gets paid, and how refunds are calculated if the order is returned. Getting any of those wrong means money ends up in the wrong place.

At Terra, creators sell products through their own storefronts. The core design question was who absorbs the cost of a discount. The answer we landed on was simple: the creator always takes the hit. If a creator issues a 20% off code, their payout is reduced by that amount. Terra's cut stays fixed. The Stripe charge reflects the discounted total, and the payout calculation works backwards from there.

The harder problem was what happens when a creator wants to discount beyond their margin. We allowed it, but Terra can't lose money in the process, so any order where the discount would have eaten into Terra's cut gets an automatic adjustment. The creator sees exactly what they're agreeing to before they publish the code.

Stacking was the other non-trivial piece. Creators could run sitewide promotions alongside individual coupon codes, so we had to make explicit decisions about when they could be combined and which took priority when multiple qualified at once.

One bug that came up in production: address updates at checkout were silently wiping the coupon discount from the order. The customer had seen the discounted total, but by the time the order confirmed the discount was gone. Once we identified the cause we fixed it and ran a backfill job to correct every affected order.