I was wrong about DevSecOps

A quick note before we dive in: The journey I’m describing here is a personal one, synthesising lessons, stories, and observations from my entire career. Think of this as a “greatest hits” of hard-won lessons, not a diary entry about any single company, past or present. It’s all about the evolution of the practice (and the practitioner!).

The Unmoving Needle

I was reviewing a detailed dashboard of vulnerabilities in our Node codebase, and it concerned me. The needle had not moved down; the number of issues had not decreased over the last three months. I thought if I had implemented the tools, fine-tuned them, perfected the pipelines so the scans were quick and helpful, I’d see a drastic decrease in the risk in our estate. The needle didn’t move down; it was slowly moving upwards.

The reality of business is that fundamentally, it needs to make money, it needs to ship new features, and it needs to keep moving forward. Security tickets were always getting de-prioritised. This is the fact I was facing and what I was struggling against. The business — and, therefore, the engineers —are always against the clock, racing to ship this feature before their competitors. Engineers generally prefer designing new solutions over patching things they haven’t worked on in six months. They want to keep their managers happy and ship new features that will be used by actual people; it’s what keeps them going.

But I was stuck with questions that were left unanswered. Why was this happening? Why doesn’t anyone care about these security flaws but me? Why hasn’t the introduction of this new, sleek tool helped reduce our risk level?

And that’s when it hit me. I was so focused on introducing this new tooling, which would “super make everything better”, that I’d completely failed to understand why it was so painful for the developers to use it. I wasn’t solving their problem; I was just adding to it.

The Tool-Chain Fallacy

I had built the almost-perfect pipeline, the masterpiece - all of the cool tools - everything that was needed to make the most secure development experience. I was so focused on building the best possible solution, that I had lost sight of what was realistic. The mantra “perfection is the enemy of good” had never even crossed my mind. All of the promises that sales-people tell you, I had implemented.

Every time a developer wanted to release code, the code had to go through multiple security gates, in which code was checked for vulnerabilities with SCA (Static Composition Analysis) and would often fail. This was too late into development, when they had their pull request up and were ready to merge. I had SAST (Static Application Security Testing) set up that scanned their code each night, automatically creating JIRA tickets and adding them to the backlog. On the surface, it looked like this was working really well.

However.

Developers hated it.

Leadership were frustrated.

I seemed to be constantly adding exceptions to allow the code to deploy. We were shipping what appeared to be vulnerable code. I would fight my corner to try and get the big things patched, but ultimately, I would be overridden to ship the feature.

I didn’t understand at the time what was wrong, and it has taken me a long time to realise what had happened. I had become the exact thing I didn’t want to be - the department that says no.

The lookback

I was a developer for a long period of my career, so it should’ve been obvious what the problems were. To understand what was happening, I had to take a step back from looking at my security dashboard, and put myself back into the shoes of the developer. I stopped looking at my security dashboards and started having regular catch-ups with my peers. From their perspective, they didn’t see these checks as overly helpful - sometimes they did catch major problems, but more often than not, they were minor things that didn’t actually matter that much. For example, it was a vulnerable dependency in a service that wasn’t actually exposed on the internet, and the value of the feature would win a new customer. The pressure to deliver features, to keep building new things or putting out fires - I can’t blame them for getting frustrated with the tooling and trying to get around it.

This isn’t the culture I wanted - I wasn’t a partner in shipping great software - I was the inspector at the end of the line.

I had strayed from what I wanted DevSecOps to be - I ended up back in the traditional sense of a security team at the end of the process - saying no.

The awakening

My objective changed. It changed from having the most secure pipeline and feeding dashboards, worried about every security vulnerability, to how can I make this easy? How can I make the most secure path the easiest path? How can I once again become a partner in delivering secure software?

I used to think that security should be invisible in the background. I think invisible is the wrong terminology here - invisible doesn’t provide assurance and validation that it is working - it should be frictionless. When you d rive a car, you know the safety systems are there - the traction control, the airbags - but they don’t get in your way. They just work when they need to (although I hope you never have to use them!). This is what security should feel like.

Looking back at the younger me, I had built something great in theory - it would block any potentially vulnerable code from being deployed. But perfect is the enemy of done.

Large security initiatives will most likely fail. I had started a campaign to bring in a fundamental change, without getting everyone on board. Small, incremental improvements are far more valuable than a grand plan that never gets off the ground. Progress, not perfection.

Relationships

My first step wasn’t to go speak to all of the developers. It was to talk to my peers. It took a while to build the trust, but once that was there, I was able to get the whole truth and insight into what was happening. I learnt which tools were noisy, which processes were painful, and where the real friction was. After hearing the pain directly from them, I started to understand and change my mindset. If you’re in a similar situation and don’t have strong relationships with your peers, I’d invest the time to build them. Those relationships are possibly the most valuable tool you can have to get direct feedback. They are, in my experience, the only metric that can measure developer happiness. You can’t measure this with the amount of code written, deleted, pushed, or any DORA metrics. You have to rely on conversations and gut instinct. Are the conversations getting easier? Is there less pushback? That’s your real progress indicator.

But I know I do not have all of the answers. If you’ve found a better way to measure this, please tell me. I’m all ears!

Course correcting

Super corny image, I know!

I started out thinking my job was to build a perfect security machine. I learned that my real job is to be a partner to the people who build our products. There will always be some non-negotiables, but in reality, most things are not world-ending, and it’s important to get your company’s product out there. Ultimately, this is what pays your wages and the whole reason the business can keep doing business.

Security is a team sport, and we rely on our developers to keep us informed and to build the features. Without the developers, we wouldn’t have any security, nor any product! So keep them happy - build those relationships - it’s how the game is won.

It’s a marathon, not a sprint - let’s keep making those small wins and moving forwards - not stopping people doing what they do best.