Digital Economy Dispatch #144 -- Beware of Software Rust

Digital Economy Dispatch #144 -- Beware of Software Rust
13th August 2023

I’ve written a lot of software in my time. It has had many uses and taken many forms: Accounting and stock control systems in Basic for the IT departments of small and large businesses; Commercial software products and off-the-shelf software packages in Java; Research prototypes and advanced demonstrators in C and C++; and numerous toy systems for different organizations written in a myriad of programming languages long forgotten or overtaken by events. While they exhibited a wide variety of characteristics, there are a few things that have been consistent along this journey.

First and foremost, I have learned that for every minute I spent writing a new line of code, I spent at least 5 times that amount trying to figure out what someone else’s code did so that I could integrate with it, and 10 times as much time fixing their deployed software to add additional functionality, improve its performance, upgrade it to work in a new operating context, or simply remove errors. Just as “most writing is re-writing”, for software developers “most coding is re-coding”.

It all goes right back to my first day as a paid software developer working in a software consulting company in the 1980’s. With my freshly minted first class degree in computer science, I entered the workforce feeling like I had a good grounding in the principles of software engineering and was looking forward to designing well-crafted solutions and producing reams of elegant code. So, imagine my reaction when first thing that happened on the first Monday morning of my new career was that I was given a stack of printouts over a foot high of archaic, poorly written software and told to spend the rest of the week working out what it did and why it didn’t work. My request for any supporting documentation to guide me through the code was met with hollow laughter.

With this background, maybe you will forgive me when I tell you that I’m more than a little sceptical that the AI-powered digital revolution receiving so much attention today will go as fast or as smoothly as some people seem to expect. You see, my fear is that many of the main advocates, futurists, strategists, politicians, and decision makers in the public and private sector lack understanding of a key concept: The digital revolution is fundamentally driven by software…a lot of it…and software rusts.

What Lies Beneath

We all now accept that the digital revolution is built on software. The world is run by “software defined everything”. We are measured, monitored, guided and controlled by software in everyday devices such as our phones. Software is embedded in and around the many devices commonly in use in our homes, offices, factories, hospitals, supermarkets, and everywhere else. Often, a lot of software. For example, one estimate concluded that there is still over 220 billion lines of COBOL code currently in operational use. More software is still being created. It is estimated, for example, that it takes up to 150 million lines of software code to run a high-end car today. Even your smart doorbell or your internet-connected toothbrush may require tens of thousands of lines of code to operate effectively. That’s an awful lot of software to write, maintain, upgrade, and renew.

The so-called “AI revolution” will only add to this software stack. Large amounts of software will need to be created, adding to the challenge of not just creating systems that work, but also managing them as they are fixed, upgraded, evolved, and replaced.

All software of any size and complexity contains errors. That is now well researched and the reasons well documented. Unfortunately, much less well understood is how and why software decays and becomes unusable over time. In many projects, hardware upgrades and fixes tend to be easier to see, diagnose, and implement. Budgets can be determined and money set aside based on pre-defined schedules. Unfortunately, software issues can often be less obvious, and as a result their detection and repair are frequently under budgeted.

Furthermore, the decline of software can happen quickly and erratically. The quantity and effect of that decline is determined by at least 2 factors: The extent to which we understand the context in which the software is being used, and the predictability of changes in that operating environment. Right now, with rapid and disruptive shifts in every aspect of business and society, we may well be in a situation where that understanding has never been lower, and the certainty of dramatic change has never been higher.

We’re beginning to see the impacts of being “software driven”. For example, in the digitization of medical devices it is acknowledged that software is “something of a blind spot” for companies with a focus on hardware engineering. Software problems are now the leading cause of device recalls and failures. This is a concern that seems likely to get worse as the software embedded in such devices becomes more sophisticated, autonomous, interconnected, and complex.

Why Software Rusts

Unfortunately, the challenges adjusting to a software driven world lie much deeper. As Andy Kyte pointed out some years ago, it may be helpful to keep in mind that software rusts:

“It decays. It gets dilapidated, geriatric, arthritic, sclerotic, senile –
and eventually it dies and goes to software Valhalla.”

Software "rust" refers to the phenomenon where code gradually becomes outdated, less efficient, or even non-functional due to various factors. In a similar way to how metals corrode over time, software can also deteriorate, leading to decreased performance, security vulnerabilities, and overall system instability. Many reasons contribute to this decay.

One major factor contributing to software rust is technological advancement. As programming languages, frameworks, and libraries evolve, older codebases can struggle to keep up. Deprecated functions and outdated practices can accumulate, leading to compatibility issues and difficulties in maintenance. Much of the effort of software developers is simply updating deployed software to the latest releases of the software technology stack itself.

Another source of software rust is the accumulation of technical debt. The errors and limitations in software often exist because shortcuts were taken during development to meet deadlines or cut costs. Over time, these shortcuts can result in a complex and tangled codebase that becomes difficult to modify or expand. This not only impedes future development but can also introduce bugs that are hard to locate and fix.

Perhaps one obvious impact of many of these concerns is the risks to security due to vulnerabilities in the code. As new threats emerge, older software that hasn't been updated or patched becomes increasingly susceptible to attacks. Hackers exploit these vulnerabilities, compromising user data or system integrity. This remains the most effective target for ransomware attacks and other exploits.

Documentation and knowledge loss are further culprits for software rust. The documentation associated with many systems is frequently inadequate or out of date from the moment the software is deployed.  If a software project lacks proper documentation and key developers move on, it becomes challenging for newcomers to understand and contribute to the codebase effectively. This lack of understanding can lead to more mistakes and inefficient solutions, adding to the technical debt, increasing the fragility of the system, and accelerating the impact of software rust.

Staying Alive

In a world of “software defined everything”, leaders and decision makers in digital transformation programmes must accept the principle that software rusts. Preventing software rust requires a disciplined, proactive approach that focuses on maintaining code quality, staying current with technology, and ensuring ongoing security and functionality. There are three main strategies an organization should adopt to prevent software rust:

1. Continuously Maintain and Refactor Code: Regularly maintaining and refactoring the codebase is crucial to prevent the accumulation of technical debt. Encourage developers to revisit and improve existing code, eliminate redundant or outdated elements, and adhere to coding standards. This prevents the codebase from becoming convoluted and difficult to manage. Implement automated testing and continuous integration practices to catch issues early in the development process and ensure that changes don't introduce new bugs.

2. Schedule Updates and Upgrades: Stay current with the latest technologies, frameworks, libraries, and programming languages. Ensure software teams plan and schedule regular updates and upgrades to keep the software stack up-to-date. This helps prevent compatibility issues and ensures that security vulnerabilities are patched promptly. Additionally, staying updated enables the organization to leverage new features and improvements that can enhance the software's performance and functionality.

3. Keep Accurate Documentation and System Knowledge: Maintain comprehensive and up-to-date documentation for the software. This includes clear explanations of software architecture, design decisions, and use cases. Encourage a culture of knowledge sharing among developers to ensure that information isn't lost when team members move on or new members join. Proper documentation empowers developers to work effectively with the codebase, make informed decisions, and understand the context behind past choices.

Surrounding these core strategies, teams should be encouraged to fostering a culture of continuous learning, enhance collaboration among developers, and prioritize security practices such as regular security audits and penetration testing. By taking these steps, organizations can mitigate the risk of software rust, maintain the longevity of their software products, and ensure that they remain competitive in an ever-evolving technological landscape. Essential in a world deploying ever more software as it proceeds on its digitally transformation journey.