This should not be controversial, yet here we are…

As a word to the nay sayers, yes there are Otaku¹ that code amazing things in their lairs and have no, or close to no, social interaction with their users. Or, even worst, are just rude to their users. I know a few. However, for everyone one of them, there are 1000s that are producing garbage and AI-slop.

99.9%² of software is produced by a team. That teams needs to communicate amongst themselves. In fact, both the Lean principles and Agile manifesto both have communication as a core value.

Communication in engineering is obvious

Let’s assume we have Alice, our principal developer. She is a good developer, using AI to amplify her hard earned skills and with decades of experience in producing quality software. In her daily work, she has to speaks to the following people:

1. Project managers (PM), who set the tasks and the deadlines. Or rather deal with things when the deadlines make a funny noise as they go past.
2. The three amigos. ‘nuff said.
3. Her fellow developers, who might require her input for a new feature or a bug fix. Or that require a PR/MR review, or just help understanding the legacy code base and why it is so clearly wrong. Well, it’s wrong now. It was the best thing some time ago and now needs refactoring. But there’s no time. True story.
4. Replying to comments left by others on her PR/MRs.
5. SQA / Testers, the people who break the lovely things Alice makes despite her using TDD/BDD and other best practices. There’s always something wrong: a bug per 10 lines of code.
6. Operations who will run her code in production. They have questions, requirements, and suggestions to make it easier to use. After all, they are the ones who will be calling Alice at 2AM on Saturday when her code breaks in production. Something Alice is very keen to avoid.
7. Security. Alice has enough experience to know that involving security early will lead to much less pain later. She’s familiar with all the OWASP and is pretty confident her code is rock solid. But, two pairs of eyes always beat one.
8. Design, because that’s where the magic happens. Always a conversation because some things are impossible — or so hard it might not be worth doing.
9. All the 5 Scrum ceremonies because…
10. Arguing with Claude, Gemini, or Copilot No, I am not counting this as a social thing.

To be honest, I am surprised that Alice has any time whatsoever to actually type code.😉

These interactions should be obvious, certainly with anyone doing secure SDLC, Agile/Lean, DevOps, or DevSecOps best practices. After all, this is how software is made.

But, does it stop here? As with any rhetorical question, no, it does not stop here.

Communication with other parts of the business is vital

Communication within the engineering teams is well defined, so it should be simple. But, what about the rest of the business?

How does it work? Who does Alice need to talk to on a regular basis and why?

First, there’s C-suite/the Board. They are the people with the vision and the power to get everyone there. They set the strategy and the direction of the company. While they mostly ask for things, they should also tell you what not to do. If you are never going to release for iPhones and Android, there’s no need to do certain things. Of course, a change of direction can always happen. But clear dos and don’ts are vital.

Second, there’s Sales. They are the ones that ultimately bring the money in. They (should) know what the current customers expect next, what their new customers love, and what feature(s) are missing from the product that cost sales. They promise software releases, patches, and demos that Alice and her development team really should know about more than two days before.

Third, there’s Marketing. New features get promoted to the market (hopefully after they are done!) and can cause surges in users. The latter is kinda important for both Operations and Security to know about.

Fourth, there’s Finance. If Alice wants a new laptop, or more tokens, or to buy a new set of tools (compilers, or editors), she has to make a case to finance. It’s much easier to do if she has good connections there.

A really smooth operation has all the engineering teams, sales, marketing, and finance working together in harmony. Easier said than done.

When is the last time anyone in your development, operations, or security teams had a meeting with marketing, sales, or C-suite?

Should this be a regular thing? Probably…

Finally, your Admin team might be invisible but who makes sure there’s coffee/tea and doughnuts in the meeting rooms? Who makes sure that the meetings rooms are not double booked? Who makes sure that all the tiny annoying impediments are just not there? That’s right, the admin team. You’ll know when you have a good one when nothing goes wrong.

An Ice Breaker

Here’s a way to break the ice between all those teams: Get two teams (does not matter which) into a room and have a 15 minutes presentation from each about what they do. Then, brainstorm what each team can do to make the other’s life easier. An hour, you can do that on Friday afternoon when everyone is pretty much done for the week.

Then, name a champion in each team (volunteer!) who will be the first point of contact and champion for the other team in their team.

Yes, it is that easy.

A pitfall of over communication…

Meetings are the real alternative to work…

Yeah. That’s the pitfall. Talking too much and inefficiently does lead to extraordinary waste. The cost is not just in resources (time and money), but also burn out leading to tiredness and low morale.

Meetings for meeting’s sake is an anti-pattern and should be avoided at all costs.

Booking some time in your calendar for focus is always a good idea. Make sure at least half your day is for doing, not talking about doing. As all guidelines, this one will vary according to your role and your environment.

For meetings, start with 30 mins max. Then, get efficient to reduce it to 20 then 15. No agenda, no meeting. Not talking about the agenda: leave. ¯\_(ツ)_/¯ Harsh? Maybe…

Effective communication is a skill. As all skills, it can be learned.

Conclusion

“Individuals and interactions over processes and tools”

This is one of the four pillars of Agile, written over two decades ago, to address the problems of software production at the time.

Frederick Brooks in the Mythical Man Month, first published in 1975, states that communication is vital in avoiding disaster. That was 51 years ago!

We can go even further back to 1936 (90 years ago) and How To Win Friends and Influence People by Dale Carnegie who teaches you how to communicate effectively.

This should not be news to anyone, and yet ask anyone in tech, or tech adjacent, and they will be surprised. 🙄

…

Imagine, if you will, a day when all the drama is removed from your software production: no panic, no crisis, just smooth software releases that exceed your customer’s expectations. This is what we have done in the past and can do for you.

How about getting in touch to see how we can help you?

This place produced software for its own use - it ran a large and complicated website and that required data handling, user decision handling, content management, and many other pieces of software that heavily affected the flow of money.

One of the inevitabilities if you do this is you will need to revisit software that you wrote years (occasionally even just months) ago and change it. Sometimes circumstances have drifted so much that it is obviously time to rewrite it from scratch. Other times such software just needs a few tweaks to catch up.

An uncomfortable number of times, devs are torn between the two. This happens a lot because meddling with existing software, that is currently making money, is often left until it’s unavoidable, and that coincides with that uncertainty line.

But here’s the thing; devs (myself included) much prefer greenfield. Start again. Make something entirely new. A fully, empty playground for you to do whatever you want in. Maybe even branch out into a new language, new framework, new patterns.

Brownfield is none of those things.

So when I came to ask for somebody to work on some code that clearly needed improvements - operational improvements mainly, the functionality was largely good - I asked for a certain senior developer from the owning team to take it on. He was, to say the least, not thrilled.

So I had to sit him down and tell him why. Spread some wisdom. I’m old.

You see, the reason I want you to do this is because it’s brownfield, I explained. Yes, yes, he knew. It’s boring, can’t one of the more junior people do it?

Ah, there we come to it. You see, greenfield is fun because it’s easy. No constraints. Your choice of patterns. Nobody else’s thinking in the mix. Easy.

I want you because this is hard. This isn’t frolicking in the fields of code, this is juggling two chainsaws and a live badger in a lift surrounded by live wires. This has intermediate stages, every one of which needs to be both adding functionality and being backwards compatible, until it’s time to not be.

Brownfield isn’t boring, it’s frustrating, and hard, and that instinct to start again fresh (and to make all new mistakes that took months of pain and debugging to get out of the previous version) is made of a dislike of that frustration and difficulty.

One of the common questions non-senior devs would ask is, what does it take for me to get promoted to senior dev?

Well, there are various answers and some hurdles to pass, but let’s start with this: show me a good brownfield project.

Conclusion

Imagine, if you will, a day when all the drama is removed from your software production: no panic, no crisis, just smooth software releases that exceed your customer’s expectations. This is what we have done in the past and can do for you.

How about getting in touch to see how we can help you?

Firmamentum’s legal council asked me not to mention the date this story took place, who the people involved were, or the make/model of the car…

Once upon a time, a major car manufacturer was working on a new car. Their cars were expensive and prone to be stolen, so security was at the forefront of their mind. Their cars had a reputation of being especially safe, so the safety features would have to be better than the ones they had previously. Continuous improvements, right?

Some clever marketing person suggested that they hire a former car thief to test the security of their car. What a good idea. They found one, and all he requested was the manual for the car. Because a good show is always welcome, the demonstration of the attempted break in would be live. They were so sure. Can you see where this is going yet?

Come the day, the thief is introduced. The car is revealed. Oohs and Ahs abound. All is going so well. Then, the thief asks for a hammer. A simple humble hammer. No worries whatsoever, the glass is reinforced, there’s been testing, no problem. The thief circles the car, looks deep in thoughts, and slams the hammer into the front collision sensor of the car.

BOOM goes the air bag.

As a safety feature, all the doors open. The thief smile, gets in the care, and hot wires it.

Whoops…

How does this map to software?

There is a Unix philosophy that states: that a tool should do one and only one job. ls lists files; cd changes directories; find finds specific items; cat concatenates files; etc… The power comes from combinations of those different tools.

In operations, separation of concerns is also a must. One and only one service should run on a machine (or VM, or pod or container image, …) at a time. Partly, this is driven by security as well. It’s best not to have your customer database on the same machine that host your email and web servers.

Why?

Well, if a threat actor breaks into your web server, they have access to your full database. If they are two separate services, then there is a better chance of the threat actor having only restricted access to the database.

But, it’s not just a security concern.

Imaging you have your web server and your email server on the same machine:

• When an OS upgrade goes wrong, both your web server and your email server go down.
• If you get a DDOS attack on your web server, your email goes down too.
• If you get a massive influx of spam emails, your web server goes down too.
• If you want to have a second web server… Do you clone the machine and have two email ones?
• A web admin has the same access than an email admin. They can interfere with each other’s work.

Monitoring of resources is much easier if you have different machines doing different things. Upgrades are easier. Administration is easier. It’s a win-win.

However…

And this is a big one. In a car, you cannot have one safe and another one secure. This does not work. Neither does it work when you are programming giant underwater robots: if the robot’s legs are powered by 400 bar hydraulics, they will not notice a human body in the way. This is where communication comes in! You need to be clear where the dangers are, and what mitigations can be in place. Optimisation is done based on different, often contradictory goals.

In mathematics optimisation theory, we call those bilevel optimisation. For example, pick the best place to maximise a function while minimising a second one. The first function could be a tool charge, while the second is the travellers desire to move to an alternative route. Or safety vs security of the car…

Conclusion

How would you address the car’s safety vs security concern?

Maybe a piece of code that detects whether there is a person in the car, or blocks the engine, or have some good remote tracking, or what? What new problems does this cause?

This is the type of complexity that we’re looking for when we build software.

Imagine, if you will, a day when all the drama is removed from your software production: no panic, no crisis, just smooth software releases that exceed your customer’s expectations. This is what we have done in the past and can do for you.

How about getting in touch to see how we can help you?

Once upon a time, in a country far far away, there was a giant underwater robot being build. When I say, giant, I mean the robot was slightly bigger than a double decker bus.¹ It’s aim was to act as an emergency pipe repair. In case an underwater pipe started leaking gas or oil, it could be deployed quickly to clamp it and reduce the ecological impact of the leak. Then, planning could occur to properly fix it.

I was in charge of writing all the command and control software for the robot. As a note, I will be using the royal we from now on. Yes, there were others involved, I was not a lone wolf!

The problem(s)

The main difficulty was that the robot was being built at the same time as the software was being written. There was no way we could do hourly, or daily, or even weekly releases. Also, the robot in question lived in a warehouse in the far away country and most certainly did not have an internet connection. If you move a 400 bar hydraulic motor, you want to make sure there is no human in the way because the hydraulic is not going to notice crunching someone. Thus, no internet!

What we did have were electrical drawings showing which sensor was connected to which input and outputs. But, we knew that these things change. One of the classic example is leg up/down controls. Whatever the drawing says will have to pass mechanical constraints. Doing it the drawing way involves cables crossing and that means that the casing no longer locks so water can get in. So, they get swapped. The technical drawing should be updated, but …

I decided to do three things:

1. Write a hardware simulator with a world model. If an input meant the leg was going up, then the simulator would change the world model to reflect that and its impact would be felt in other aspects of the model.
2. Write TDD tests first for everything. Some of them were, by nature, not true unit tests since they had incestuous knowledge of the internals of the code. But, as the Way of Testivus states, better a bad test today than a perfect one tomorrow.
3. Write BDD stories for the each feature. The goal was to make sure that the most common operations were taken care of and were well defined.

It did take longer to write the software. But, was it worth it?…

The result

We went to the far away country and worked 12 hours shifts for two weeks, including weekends. This involved dozens of people: hardware engineering, mechanical engineering, electrical engineering, and a few divers as well as a crane to drop the robot in the water. Thankfully, there was no boat and no trip in the middle of the sea.

The above image is one of the cabins where we ran the software from. The red / orange suits were very fetching. After all, we needed to be seen in the warehouse.

By Wednesday of the first week, it was clear we would not be needed in week two. By Friday, we were on our way home.

60% time saved.

Nice.

What does this mean? It means, operation costs saved, time saved so the people can focus on different projects, and an assurance of quality. Probably several million euros saved right there.

Conclusion

Two side effects happened as a result of this. First, my company got a lovely letter from the project leadership saying it was the smoothest, best quality software they had ever got from us. This was timely since our contract was being renewed.

Second, several years after I left the company, a developer there reached out to me. They wanted to personally thank me for the work I did on the simulator. It was still being used to save time and test code years later. The initial layout of resources to code the simulator paid for itself in more than one project, over ten years².

I have done this before and I can do it again for your company.

Imagine, if you will, a day when all the drama is removed from your software production: no panic, no crisis, just smooth software releases that exceed your customer’s expectations. This is what we have done in the past and can do for you.

How about getting in touch to see how we can help you?

You have a great idea.

It solves a problem people have. You do your market research, and people want it. You even talk to your VC and Angel investors and they tell you it’s a great idea — Let me know when you’re raising funds they say. Perfect! Let’s go 🚀!

Your team is awesome. You have a fantastic CTO / founding engineer and Claude can prototype your application in days, not months. Your marketing team (also you?) works overtime to get those sales calls. All is going well, you’re doing Series A and B funding rounds.

But something is nagging you now. You’re not closing enough. Sales cycles are getting longer and longer. Your existing customers are leaving. What is going on?

Why are things falling apart?

There are many answers to this and this is but a short overview of the things we have seen. It’s also the things we help you fix if you find yourself in a similar situation.

As your team grows, new people join in and the culture changes massively. It stops from being a fun start-up where we break things and move fast to a grown up company. It is now impossible to do all the things: you must delegate, relinquish control, and learn how to do less yourself. This means letting go. It is hard when it’s your baby. Even if you hire the best of the best in their field. This is what our foundation of People is about.

Your developers are starting to drown in technical debt, which of course, is not a technical problem but a business one — Back to leadership! Features are taking longer and longer to write, the quality of the code goes down, and more and more time is spent on fixing new and old bugs. After all, for $1 spent on development, $0.50 is spent on bug fixing. 😱 This is what our foundation of Development is about.

Reliability failures lead to frequent crashes, bugs, or downtime which erode trust. Customers expect software to work seamlessly, and repeated failures push them to competitors. Small predictable downtimes are fine, but repeated errors are not. Observability goes hand in hand with this. It helps you see what goes wrong early, where to optimise, and why your AWS bill is over £100k per month. This is what our foundation of Operations is about.

Cybersecurity is another one. Most companies will evaluate yours using Security Scorecard or similar tools. Ours ranks as an A(100), if you wanted to check it out. Your customers will run a security scanner on your services, just to see what happens. And they might even ask for a SBOM report or require SOC-2 / ISO 27,001 compliance certification. It’s a cost which offsets the risk of incidents. We all know data breaches or weak security measures damage reputation and trust. Customers prioritize security and will leave if their data is at risk. This is what our foundation of Security is about.

AI (LLM) do enhance developers allowing them to write much faster code. Although, this moves the bottle neck in your pipeline elsewhere. And the quality of that code might not be as good as you think it is: 1.7 more bugs, whose impact is twice as serious might not be what you wanted when you got those Copilot and Claude subscriptions. This is what our foundation of AI is about.

Conclusion

Does this resonate with you?

If so, please book a call with us. We can help. It’s what we do.

Imagine, if you will, a day when all the drama is removed from your software production: no panic, no crisis, just smooth software releases that exceed your customer’s expectations. This is what we have done in the past and can do for you.

How about getting in touch to see how we can help you?

Artemis II is a spacecraft that took mankind back to the stars. The images can be seen here and eventually the mission will lead to a permanent presence of humanity on the Moon. The gates of space opera are no longer closed, but opened.

What has this to do with software I hear you say?

It is not about “I have two Microsoft Outlooks and neither one is working”, although this post could well be about that.

Software in Space

The Artemis II mission is a fine example of how software, operations, and security mingle together. Let’s take a walk through the process. And, before you ask, I had nothing to do with any of it. I never worked at NASA. ¯\_(ツ)_/¯

If your company was writing the software for the spacecraft, how would you write it? Would you vibe code the whole thing with Claude? Would you use the latest language du jour? Let’s go with Rust. Would there be any guidance as to how it is written? How much testing would you do? Is 70% test coverage good enough or would you want to see each line of code executed by two (or more) distinct tests?

How about operations? How do you test that your code works as expected on the hardware which is still being built? What happens if you need to patch things as they run? What metrics do you decide to show to the crew, which are are for computers to read, and which ones go into the black box? Which version of Kubernetes do you run locally? How about X-ray radiation impact on the hardware?

Do you run a firewall? More than one? How about the link between the space craft and the ground? Do you insist on a TLS certificate? When is that going to renew? Or is the comms okay unencrypted. What could go wrong?

In space, no one can hear you scream. If things go wrong, the crew perishes. The craft is lost. The mission is a failure. Do you have what it takes?

Conclusion

Does this appear to be daunting task? Are you not doing all this already? This is what we do: we plan and make sure that it’s boring:

No drama. No panic. No crisis.

Sounds like something you could do with?

Imagine, if you will, a day when all the drama is removed from your software production: no panic, no crisis, just smooth software releases that exceed your customer’s expectations. This is what we have done in the past and can do for you.

How about getting in touch to see how we can help you?

As I write, NASA is preparing to send 4 astronauts into lunar orbit. Fuel is the major component by weight, but I would wager that if measured by complexity, a major fraction or even a majority of the equipment on board is to do with crew survival. If a robot blows up, freezes and won’t wake, loses comms or just loses its way, well, that was expensive but we can just try again. Losing people is not acceptable. So… why are we sending anyone?

Why don’t we just send drones? Drones are now dominating the battlefield in several current wars. But once again, we are wary of fully autonomous drones, instead preferring FPV drones guided by an operator.

What is it that NASA and the world’s most advanced militaries are aware of, that apparently AWS - hardly a technological laggard - forgot last December?

It’s that writing good automation is very hard. People make complex decisions, weighing many subtle factors. Automation… not so much.

It’s painfully easy to get overtaken by the hype of how much money automation will save you, in salaries, training, seats, time. You name the resource, automation will improve things. Even computing power - a script needs less computer resources than an employee.

What those highly advanced organisations know is that if you attempt too much automation, you will be limited by needing to simplify your task to permit automation to succeed. If the task is simple, scripts, frameworks, even AI agents are fine. If not, we still need humans.

Is your business so simple that a script, an app, or an AI can run it? What does a CTO contribute there, exactly? Ha-ha, just kidding, I’m sure you could all tell me at length why your job cannot be automated. But, are you sure that when you look at your employees and see automation opportunities, that you take the time to find out why trying to automate them away is a mistake? Do you listen at length?

Conclusion

There are various symptoms of over-reliance on automation. Know that going too far will look great, until it doesn’t. Finding the right balance is hard.

…

Imagine, if you will, a day when all the drama is removed from your software production: no panic, no crisis, just smooth software releases that exceed your customer’s expectations. This is what we have done in the past and can do for you.

How about getting in touch to see how we can help you?

Lost in the details

Generally, this means being unable to grasp the overall situation because one is focused on the minor details. It happens to all of us, and despite really wanting not to repeat it, we find ourselves in the same situation again and again.

Like being lost in a forest full of ghosts¹, we get lost in the day to day hustle of creating software. There are always deadlines, be they self constrained, or something that Sales have sold and must be delivered by a specific date. Looking at the next round of funding, watching the runway get smaller and smaller. The pressure is unrelenting.

What goes wrong?

We focus on getting customers requirements, generating specification, or trying to articulate a coherent vision. Agile tells us to iterate a lot, but that’s not always possible, for example in safety critical industries.

We focus on optimising the code, writing it faster and faster with AI while getting as much testing done as possible. Code reviews become a distraction and a road block, leading to LGTM (Looks Good To Me) comments or worse, descending into tabs vs spaces flame wars.

Security can wait until we have a product. It’s a distraction and a road block. We can fix it later. Well, chances are you won’t have a choice if there is breach or you will have to refactor swaths of code.

How hard can operations be? It’s just a docker image, right? Throw it over the fence and let the DevOps/SRE worry about it. What else are they going to do? We can do multiple releases per day so what does it matter if one fails? AI can fix it for us.

Fix the current problem. No matter how many yaks one needs to shave, it is worth doing now. No, yesterday!

This is known as head-on-fire optimisation. And, we’ve all been there. We’ll be there again.

Aside in Mathematics

In Mathematical optimisation theory, there are two types of maxima, local and global ones.² When trying to optimise a function, one of the most common trap is to focus on the local ones. Sometimes, an optimisation technique will focus on that and never escape. Yes, the solution is good, but it’s not optimal.

This is exacerbated in bi-level optimisations, where you aim to reduce one function while maximising another. This is what happens when you try to set a congestion charge: Maximise the revenue while minimising the congestion. It is really hard.

Almost all optimisation algorithms have an inbuilt systems to try to escape local maxima. The details vary, but it’s (nearly) always there.

So, what’s the solution in the software world?

The Solution???

There is no one-size fits all solution.

There are things I do that helped me in the past. One that stands out is to step back, have a walk, and let my subconscious do the work for me. It’s quite good at coming up with weird and interesting solutions. A good night sleep helps too!

Another would be to take our MOT. It’s been designed based on our 100 years of experience with all the above in mind. It’s a gap analysis. It’s a health check. It’s therapy. It’s discovering what is wrong. This is why we sign an NDA! We want you to be able to say the truth otherwise, we cannot help.

Imagine, if you will, a day when all the drama is removed from your software production: no panic, no crisis, just smooth software releases that exceed your customer’s expectations. This is what we have done in the past and can do for you.

How about getting in touch to see how we can help you?

A beginning is a very delicate time. Know then, that it is the year 10191…¹

Err… I mean, 2024 and we were starting our new company. We were looking for a name that would be both memorable and would represent our values. Being from a theoretical astrophysics background, I suggest a latin word for the firmament of the sky… And then, we looked into more detail at the definition and the rest, as they say, was history.

Firmamentum

The definition of Firmamentum has three meanings:

1. A strengthening, support, prop, stay.
2. The firmament; the sky fixed above the earth.
3. The main point or crux (of an argument).

We chose the name as these three things relate perfectly to what we do.

First, a strengthening of you software functions, based on five foundations of people, development, security, operations, and AI,

We look at how the C-suite’s vision is communicated in the day-to-day work for clarity of purpose. Which development best practices suite your business and add a robust quality assurance, ensuring efficiency and predictability. Shift left of operations and security to ensure robustness and resiliency. And finally, how AI can be used to enhance those things to make your business more efficient and profitable.

Second, the firmament where, if you apply our proven advice, you will soar to. This is not idle talk. We have reduced testing times by 60%, slashed AWS bills by 40% while increasing reliability, and applied AI in medical devices to support medical staff.

Our MOT is a gap analysis of all the pillars and foundations which make your software. It is simple because we have spent over a century looking at it between us. Once you have de-risked, you can optimise which is what we do next! And this is what people think of us:

Finally, the crux of the argument: We disagree between ourselves. And those disagreements are where our true value shines. We have different experiences, different views, and different ideas. Like Forbes keeps saying, diverse teams are more effective. There is no one way of doing things, some things will make more sense for you than others. Hearing us speak about it will spark understanding.

Conclusion

So, what’s in a name? Everything!

Imagine, if you will, a day when all the drama is removed from your software production: no panic, no crisis, just smooth software releases that exceed your customer’s expectations. This is what we have done in the past and can do for you.

How about getting in touch to see how we can help you?

Technical debt is a techie problem, the business should not pay for it. If you have some technical debt, it’s because you hired some bad staff. The good ones get everything right.

I have heard this way too often from both CEO and COO who do not really get it. And that’s because technical debt is not a technical problem, yet by calling it technical, we create a barrier to understanding that it is a business problem. Let’s look at a tiny aspect of it, that of security…

Of course, no one seems to want to improve their security nowadays: ship fast, break things, and let the next person worry about it, right?

One person who does care is the hacker. They are going to make thousands from you as you lose millions in the process. Or worse, they’re just going to get kudos from their l337 h4ck3r¹ friends.

The Security Debt Problem

Applications continue to ship with known weaknesses even as development workflows speed up. A new Datadog State of DevSecOps 2026 report examines how dependency management and pipeline practices are influencing exposure across cloud native environments.

Across the environments studied, 87% of organizations run at least one exploitable vulnerability in production services, affecting 40% of those services. This condition points to a persistent accumulation of security debt inside deployed software stacks. Your dependencies are 278 days out of date and your pipelines aren’t protected.

What happens next? You get hacked. Your customer data is stolen and leaked on the dark web. Your customers curse you and leave in droves. Your name is in the news as the weekly company that treated security as an after thought. And everyone laughs behind your back.

How do you fix it?

We know it is possible as is shown here: UK reduces cyberattack fix times from two months to eight days. But, how?

First, patching is critical. You need to have a rolling update of all your dependencies. It costs resources and is not glamorous but it is needed. How it works is simple:

• Week 1: Patch all dependencies in development and fix all the new issues it causes.
• Week 2: The patches get into the testing environment and are explicitly tested.
• Week 3: The patches get into the staging environment and are monitored for failures.
• Week 4: The patches are deployed into production.

If you really want to be efficient, this means that every week your developers patch their dependencies.

Second, there are plenty of open sourced and free security scanners. There are, of course, many pay for tools as well — for example…. Running one of these in your CI/CD pipeline is a sure way to find out which need to be updated as a priority. Of course, this is only as good as those tools are good. Still, having multiple ones is not hard to set up and is a good way to catch those elusive vulnerabilities. One problem here is that of false positives: you must review it all which can lead to alert fatigue.

Finally, and most importantly, as technical leadership we need to make sure that the business at large understands that this is not a cost. It is protecting your revenue stream. It is the equivalent of locking your car and taking the keys with you, not leaving the key in the ignition in the car park. You need to set resources (technical, monetary, and time) to protect your revenue stream. How much is often dictated by your company’s risk appetite: How much would a breach cost you?

Conclusion

This is just a tiny aspect of the problem. It is an opportunity to change minds, to educate senior leadership who, for entirely understandable reasons, do not understand this aspect. This is what I have done many time in the past, and I can do again for you.

Imagine, if you will, a day when all the drama is removed from your software production: no panic, no crisis, just smooth software releases that exceed your customer’s expectations. This is what we have done in the past and can do for you.

How about getting in touch to see how we can help you?