There was this decision from a few years ago I’ve carried as a quiet regret, the call to build a custom solution rather than adopting an industry-standard managed service. At the time, the team treated it as a win, and I let myself be talked into the same view, even though something felt off. We finally walked it back recently. I wish I’d trusted that instinct sooner.

The story about making bad calls on build vs buy is never new. I’ve also seen the other side of the coin: teams that bought an expensive platform, used 40% of its features, then couldn’t unwind the contract when their needs changed.

It’s one of the most common decisions engineering leaders make, and one of the most consistent misses. This experience taught me an expensive lesson: it’s almost never because the math is hard. It’s because the question was framed wrong, or because the loudest voice in the room had the strongest preference, not the strongest reasoning.

I would like to break this into few chapters:

1. Why are we getting this wrong?

2. The questions that actually matter

3. What does this look like at different scales?

4. The traps that we watched people fall into

5. A practical way to actually decide

6. A closing honest thought

Why are we getting this wrong?

Before the framework, it helps to be honest about who’s pulling on what.

Engineers tend to lean towards Build. Building is more interesting than integrating. It’s better on your resume, and it carries a sense of ownership. I’m also guilty of this.

Leaders on the other hand, tend to lean towards buying. Buying spreads accountability. A vendor letting you down is an easier conversation than your team failing. Procurement feels safer than a headcount ask, and vendors are very good at making “Buy” feel like the obvious adult decision.

Vendors will tell you that maintenance is your real cost. They’re right.

Internal advocates will tell you that licenses compound forever. They’re also right.

Both can be true. The job is figuring out which one matters more for your specific situation, and not letting the people with the strongest preferences sway the decision.

The questions that actually matter

The wrong question is: “Is it cheaper to build or buy?” Cost matters, but it’s downstream of bigger things. Four questions move me more than the spreadsheet does.

Is this a core differentiating factor?

As engineering leaders, we need to be ruthless about where we spend our team’s cycles. If the business doesn’t care how a system works, they only care that it delivers data reliably, then you should default to “Buy”.

Take a data catalog. Engineering teams often want to build a custom internal discovery platform because stitching together metadata from the warehouse and transformation layers is a genuinely interesting technical challenge. But no business user cares about the elegant architecture behind your internal catalog. They just want to find their tables and trust their metrics so they can make decisions.

The data models and the insights they generate are your differentiator. The catalog is just the search bar. Buy the tool off the shelf, and keep your engineering cycles focused on building the data products that actually power the business.

Do we have team capacity and continuity for the next 3 to 5 years?

This is the question most often skipped, and the one that kills the most Build decisions.

Building is never expensive, owning it is.. Owning means on-call rotations, documentation, onboarding new engineers, security patches, dependency upgrades, and the slow drift of “the original team has all left and nobody really knows why this works.”

This one hits especially hard for stretched teams. If your engineers are already firefighting and context-switching, adding a custom-built system to their plate doesn’t save money. It just hides the cost in burnout and missed roadmap commitments.

If you can’t credibly commit to staffing the maintenance team for three to five years, don’t build.

Is our scale or constraint profile unique enough that vendors fail us?

This is the legitimate “Build” case.

If you process petabytes a day and the vendor’s pricing assumes terabytes, the math falls apart. This is when building makes more sense.

This is why hyperscalers (cloud services) build so much. At their scale, vendor solutions either don’t exist or cost more than building. There might be one, but hopefully it’s not an overkill for your business.

Notice the inversion, though: most organizations citing this reason aren’t actually at that scale. They’re using “scale” as a justification for a decision they already made for other reasons, where in reality it is never the case.

How reversible is this decision?

The best way to evaluate this is the “one-way vs. two-way door” framework. Two-way doors are reversible. If you don’t like what’s on the other side, you just walk back. They deserve a fast decision. One-way doors are practically irreversible and demand careful debate.

Buying a SaaS tool you can swap out next quarter is a two-way door. Hand-rolling a custom architecture that becomes the spine of your data stack is a one-way door.

The harder it is to back out, the more conservative your default should be. Always look for the two-way door first: Buy the tool, integrate it, and learn your actual edge cases. If it proves insufficient, you can still
“Build” later.

Living with a vendor solution for a year gives you the exact blueprint you need, making any future “Build” dramatically cheaper.

What does this look like at different scales?

The same framework lands in different places depending on where you sit.

Bigger organization build a lot because scale breaks vendors, they have the engineering bench, and the differentiation case is genuinely real for most of the stack. But even they don’t build everything. They buy CRMs, HR systems, and most of their corporate IT. The discipline isn’t “build everything.” It’s drawing the boundary clearly and resisting the urge to redraw it because a senior engineer fancies an interesting project.

Mid size companies usually do best buying about 80% of the data stack and building the 20% that genuinely differentiates them. The failure mode here is creep: each individual Build decision looks reasonable, but they accumulate into a homegrown stack the team can no longer maintain. By year three, half the engineering org is stuck keeping the lights on for tools nobody else uses.

SMEs should buy almost everything. The cost of context-switching between maintaining custom tools and delivering business value is brutal at small scale. The exception is when no vendor exists for the specific thing you need, or when the thing is so centric to your product. Otherwise: buy, integrate well, and focus your engineering energy on what your business actually does.

A useful gut check for smaller teams: if you have to ask whether you should build it, the answer is almost certainly no. Building should feel obviously right, not like a close call.

The traps that we watched people fall into

A few patterns show up again and again, regardless of company size.

The “we’re special” trap.

Almost every team I’ve worked with has, at some point, argued their requirements are too unique for any vendor. About 10% of the time this is true. The other 90%, the requirements are normal but emotionally familiar, which makes them feel unique. To solve this, get a third party opinion before betting on this.

The “open source is free” trap.

Self-hosting an open-source tool is not Buy. It’s Build with a head start. You still own the operations, upgrades, security posture, and the deep expertise required to debug it at 2 a.m. People treating it as a free Buy are usually about to learn an expensive lesson.

The “engineer ego” trap.

Sometimes a Build proposal is really a senior engineer asking for an interesting project, dressed up as a business case. This isn’t malicious. It’s human. But the cost doesn’t disappear because the motivation was personal. Ask: would we still want to build this if it had to be owned by our most junior engineer for the next three years?

The “outsource accountability” trap.

The mirror image: sometimes a Buy decision is risk-aversion in a suit. The vendor becomes the throat to choke when things go wrong, even when Build would have been the right call. Buying doesn’t make a project succeed. It just changes who gets blamed when it doesn’t.

A practical way to actually decide

When I’m sitting in front of one of these decisions, I work through something like this.

Sketch the three-year total cost of ownership for both paths.

Include people, not just dollars. For Build, count the engineers you’ll need to maintain it, not just ship v1. For Buy, count the integration work, vendor management overhead, and a realistic price increase at renewal. Most renewal increases are not 5%.

Write down what you’d need to believe for each option to be right.

If the Build case requires believing you can hire and retain three specialized engineers for five years, and you’re already struggling to fill the openings you have, that’s a signal. If the Buy case requires believing a particular vendor will still be competitive in three years, look at their funding, their churn, and how often they’ve been acquired in their category.

Talk to two or three peers who’ve made the same decision.

Not vendors, not your account executive. Actual engineering leaders who’ve owned the outcome. The hour you spend on those calls is more valuable than any analyst report.

Default to Buy unless Build is obviously the right answer.

This is the part most engineering cultures get wrong. The default in our industry is “we can build it” because we can. The question isn’t whether we can. It’s whether we should.

Build is the obvious answer.

There are times regardless of whether you want to buy or get a vendor to build it, after checking the vendor’s pricing and negotiations, the price just doesn’t make sense and you are asked to build it yourself.

We just had one recently! One of our most used services that we would like to procure, has a unique pricing mechanism and forces user-based pricing, rather than consumption-based. When it doesn’t make sense to buy while we can still survive after the above four questions, we can just continue to maintain it (and still cost less, with minimal maintenance).

A closing honest thought

The reason this decision is hard isn’t that the framework is complicated. It’s that the framework forces you to be honest about things that are uncomfortable: how special your team really is, how long your engineers will actually stay, how much you really know about your future requirements, whose ego is in the room.

When I was asked to build a framework around this, it mostly landed on a set of questions and with scoring on every question that contributed to whether to build or buy, the theme is almost the same, just different ways of asking and looking at the big picture.

I’m currently working with the best leaders I could ask for, who don’t have a bias toward one or the other. They have a bias toward asking the harder questions before either side wins the argument. Most of the time, the right answer becomes obvious once everyone in the room is willing to say what they actually think.

Irregardless as engineering leader, we should own whichever decision, clearly have the documentation on why such decisions were made, and when the “why” are no longer important or solved, it’s time to reconsider it again.

Here’s what I learned along the way.

The early bias

When I started out, I didn’t really understand what an Engineering Manager does. My main exposure to “management” was through Product Managers, and in those early days it felt like my best ideas kept getting shot down. I was proposing what I thought were elegant technical solutions, however what I got from them was a straight no.

This resulted in me building a quiet, defensive belief: Managers are the people who push back on innovation.

“We don’t see things as they are; we see them as we are.” – Anaïs Nin

Looking back, my ideas weren’t being shot down. They were being weighed against a reality I couldn’t yet see. A multi-week delay against a critical launch. A long-term roadmap that would make my “elegant” fix irrelevant in three months. A customer problem I hadn’t yet framed as one. I was solving for technical elegance. The business was solving for survival.

That gap wasn’t a lack of support. It was a lack of shared context, and honestly, a lack of effort on my part to look for it. The pandemic-era remote setup certainly didn’t help, but I also wasn’t proactive enough to close the loop.

The pushback I felt wasn’t resistance; it was context I hadn’t yet uncovered.

The shift

My view started changing as I worked with managers who didn’t just say no but also explained the whys.

In one role, even when my ideas weren’t picked, I got clear, business-aligned reasons most of the time. 1:1s shifted from status updates into deep dives on technical problems and personal growth. I won’t pretend I instantly came around. I still quietly believed some of my “optimal” solutions were being passed over. But I also got to ship a few of them, which built trust on both sides. And a few years later, I watched some of those same world-class solutions of mine failed catastrophically in production. That was its own kind of education.

In a later role with a different manager, the conversation evolved again. We stopped talking only about my work and started talking about the team. Bottlenecks ahead, focus areas, team health. The dynamic became collaborative rather than one-directional. I could surface frustrations openly and stress-test whether something was a real problem or just my pessimism looking for one.

Management, I realized, wasn’t about stopping ideas. It was about curating them, so the team kept solving the right problems.

Trying it out

When the chance to take on direct reports came up, I took it. Partly to test my own assumptions, partly because I was genuinely curious whether the version of management I’d come to respect was something I could practice myself.

It wasn’t easy. I leaned heavily on my own manager when I felt stuck, and was lucky to be given room to try my own approach before anyone stepped in. Over the years, a few things became clear:

1. Business context isn’t a tax on engineering. It’s the input that makes engineering decisions actually good.

2. Soft skills do real, measurable work. Most of what I once dismissed as “management overhead” was actually trust-building.

3. The big picture is easier to see, and harder to act on, than I’d assumed.

4. A single template doesn’t work across direct reports. The job is to figure out what each person needs, then adapt.

I’ll go deeper on these in a follow-up post. For now, the short version: I didn’t dislike management. That realization made my next decision harder, not easier.

Why I’m choosing IC route

Despite enjoying the people side, I noticed where my real flow state lived: heads-down on a hard technical problem, designing a system from scratch, untangling a gnarly bug. That granular satisfaction wasn’t something I could replicate with the overhead of people management challenges.

I’m returning to IC path not because I disliked management, but because I still believe the staff engineering path is equally vital, and more aligned with the work I want to be remembered for. I can still lead, influence, and mentor without a roster of direct reports. The leverage just looks different.

I want my legacy to be the systems I built, the standards I set, and the engineers I influenced.

The manager’s detour wasn’t a failed experiment. It’s the thing that makes me a better staff engineer than I would have been otherwise. I now understand the trade-offs I used to push against, the context my managers were operating in, and the reasons “no” sometimes is the right answer.

Resources that helped

If you’re weighing the same choice, these shaped my thinking:

1. Path to staff engineer substack: Excellent deep dives into the nuances of senior technical roles

2. Soft Skills Engineering Podcasts: A great resource for navigating the “human” side of technical work, regardless of title.

3. The Staff Engineer’s Path by Tanya Reily: A must-read for anyone who wants to lead without leaving the keyboard.

That’s it for now!

I remember debugging a broken pipeline and tracing it back to a commit message that just said “fix”

Many of us are familiar with standards like Conventional Commits that define what a good commit message should look like. But in practice, people still rush through this step. By the time the code is ready, nobody wants to spend extra effort describing it properly.

As a result, commit history often ends up being far less useful than it should be. Instead of helping with debugging, code review, and understanding why a change happened, it becomes another thing engineers have to work around.

That is why using AI to generate commit messages is a practical use case as it helps to reduce the friction of writing commit messages that are actually useful.

Where AI Actually Helps

AI does not magically make commit messages important. It just removes the worst part of writing them: starting from nothing.

The model only needs to read a diff, identify the dominant change, summarize it into a sentence, and format it consistently.

Instead of this:

You get this:

AI drafts the message, you review it.

Tooling Options

You do not need to overthink the tooling. Pick the path that matches your workflow.

• Copilot in VS Code if you want the easiest built-in experience.

• AICOMMITS if you want to use your own API key, choose your provider, or run against local backends.

• GitLens if you already use it heavily, but note that AI-generated commit messages are part of the paid Pro tier.

Do not skip the instructions - the tool matters less than the instructions.

Even a good model will generate average commit messages if you do not tell it what good looks like.

Here is a simple example:

Generate a commit message using Conventional Commits.

Format:
<type>(<scope>): <subject>

Optional body:
- Include only if extra context is useful
- Keep it brief
- Focus on why the change was made if it is obvious from the diff

Footer:
Jira: https://your-company.atlassian.net/browse/<TICKET-ID>

Rules:
- Allowed types: feat, fix, refactor, chore, docs, test, perf, ci, build
- Use a scope when relevant
- Keep the subject under 72 characters when possible
- Use imperative mood
- Do not end the subject with a period
- Avoid vague wording like "update", "change", "fix issue", or "misc"
- Prefer the user or system impact over file names
- Return only the final commit message in plain text

One thing worth checking before using any of these tools: Make sure your company allows code or diffs to be sent to external AI providers :)

Final Thought

This is not about replacing judgment. It is about making useful commit history easier to maintain.

Your future self might thank you when you are debugging a regression, and AI agents will have a better trail to follow when they need to figure out where a bug was introduced.

Stop settling for bad commit messages!

Once a pipeline hits production, the clock starts ticking. Data sources drift, requirements change, and scale eventually breaks your initial assumptions. The architecture that worked yesterday is almost immediately facing its next bottleneck.

In this field, success isn't about finding a silver bullet. It's about engineering for the chaos before it hits.

That’s why we’re calling this IncrementallyHQ. We’re here because we believe the only way to build anything worth keeping is to get 1% better every day. No giant leaps - just one PR, one query optimization, and one lesson learned the hard way.

We’re starting this newsletter to document our journey. We want to build in public, share the real-world, unpolished realities of working in data, the architecture wins, the debugging nightmares, and the lessons learned along the way.

Essentially, this is our brain dump - unfiltered.

Meet the Team

• Nazmi Asri: Our pragmatic architect who has spent a career building AI and data workflows from the ground up in the world of Fintech and Banking. He’s less about the hype and more about the real-world grind of making a modern tech stack actually play nice together. He’s the reason our architecture remains grounded in logic rather than just buzzwords. Outside of work, Nazmi is a bit of a technical article junkie, always hunting for the latest trends.

• Jeremy Tee: Our infrastructure anchor for all things streaming and cloud spend. Whether it’s a streaming pipeline having a meltdown in the middle of the night or our cloud spend suddenly trying to hit a new high score, Jeremy is the one who pulls us back from the edge. He’s the reason our systems stay both rock-solid and lean. Outside the terminal, he’s recently fallen deep into the running rabbit hole.

• Abid: Our heavy lifter for the weirdest technical bugs. If there’s a deep-seated technical bottleneck or an architectural knot that no one can untie, Abid is usually the one fixing it. He’s the reason our engineering standards stay high and our systems remain resilient as they scale. When the laptop is shut, he’s usually decompressing with a gaming stream or deep in books related to self-improvement and growth.

What’s Actually in This Newsletter?

We aren’t interested in rewriting documentation you can already find on Google. We want to talk about the “Synthesis” of our work:

• No-BS Architecture: The “teardowns” of our tech stacks - why we chose them, how we built them, and exactly where they let us down.

• The Cost of Data: Real talk on FinOps. How to keep the bill from spiraling while optimizing complex ETL pipelines for performance.

• The “Plumbing” for AI: The unglamorous infrastructure work required to make LLMs and agentic workflows actually useful in production.

• Scaling the Humans: Our notes on building teams - from hiring and setting engineering standards to the messy reality of jumping from IC to Lead.

• The Failure Archive: An honest look at the debugging nightmares and architectural dead-ends we’ve hit, so you don’t have to repeat them.

A Shared Space

We want this to be a collective home for fellow builders. If you’ve got a hard-won lesson or a nightmare debug you want to document, the door is wide open.

We’re excited to build this. Let’s get to work - incrementally.

— Nazmi, Jeremy, and Abid