Digital Economy Dispatch #103 — The Inconvenient Truth about Software Engineering

Digital Economy Dispatch #103 — The Inconvenient Truth about Software Engineering30th October 2022

The Inconvenient Truth about Software Engineering

One of the most challenging questions I get is perhaps one of the most basic: “What do you do?”. It has always been something I’ve struggled with. These days I get away with a half-mumbled response about teaching at a university or being a researcher. That satisfies most people. But it gets more uncomfortable if they then insist on the details. How do I tell them that I am not really sure myself what I do?

Some time ago, in a previous life, I was bolder and would often reply “I’m a Software Engineer”. I’m not sure for most people this was particularly informative. However, I hoped that the juxtaposition of those two words would be sufficient to give them an impression of someone trained, skilled, and accomplished in the use of digital technology. Sometimes it worked; sometimes it didn’t.

I gave up using that title some time ago when two things dawned on me. First, it became clear to me that I have less and less to do with developing and delivering software, and spend more of my time talking about it. Second, and perhaps more worryingly, I realized that I am not sure I know what it means to be a software engineer. Sure, in the general sense of someone earning a living in the software industry or spending their time trying to solve problems with software-based solutions. But not in the details of what those two words mean. Indeed, as I looked at what was happening around me in the software industry, it felt like we had lost sight of the fact that the role of a software engineer requires much more than a focus on writing software, and much of what I witnessed was severely lacking in almost all the discipline of an engineering profession.

The Long and Winding Road

It seems that I am not alone. In Adam Barr’s book on “The Problems with Software”, he focuses on the issues we face when we write large, complex software systems. By drawing on his long-time experiences at Microsoft, he considers the practices commonly used today to create and maintain software and asks why it is often of low quality, takes too long, and cannot easily be evolved. His surprising conclusion is that in order to move forward we must first look back. Let me explain.

Without being overly dramatic, for those interested in building better software, we could describe the past 50 years as an unfulfilled search to define a discipline of software engineering. From the earliest days of programming the first computers, it was recognized that major challenges needed to be overcome if the power of these new computing machines were to be harnessed appropriately. With increasing availability of computers from the early 1960’s, the lack of skills and talent become known as a “software crisis”. This was highlighted in a series of NATO-sponsored conferences in 1968 where the term “software engineering” was originally defined as a way to galvanize the community into action.

In response, a great deal of attention and activity was focused over the next 3 decades on finding ways to design, implement, deliver, and maintain the software needed to make use of the powerful computer hardware that was emerging. Significant progress was made. Barr refers to this as “The Golden Age” for software engineering.

The Golden Age

Throughout the 1970s, 80s, and 90s there was a focus on finding ways to improve how computers could be used efficiently and effectively. It resulted in a wide variety of innovation in techniques and technologies to control and manipulate computers using new operating systems such as UNIX, programming languages like C, relational databases such as DB2, and a host of other advances. Undoubtedly, these made computers more accessible to many people and formed the backbone of a new generation of programmers, systems analysts, and operations managers.

However, Barr takes this retrospective analysis much further. He draws our attention to the mathematicians, computer scientists, and software practitioners that were developing insights about the disciplines of software engineering. Not just the creation of the tools and technologies making computers easier to program. But fundamental advances in how best to carry out the task of creating and managing software-intensive systems.

Amongst those he highlights are people such as Fred Books, Harlan Mills, Watts Humphrey, Winston Royce, Vick Basili, and Capers Jones. They were busy looking at how large-scale software was being produced in corporations such as IBM and Lockheed Martin. Then, they used their scientific background and training to document evidence of faults and failures, measure frequency of errors, investigate cause and effect, and so on. It was gathering data that mattered.

Similarly, Barr describes the pioneering work of other notable mathematicians and scientists such as David Parnas, Don Knuth, and Edward Dijkstra who were building the theories on which many of the new ideas about how best to build them and the assessment approaches for understanding the quality attributes of these software-intensive solutions. Their work created a shared language and formed a robust set of principles for this emerging discipline.

Lost in the Mists of Time

As Barr freely admits, most of these pioneers and their work were unknown to him and his colleagues at Microsoft during his time leading teams in designing and delivering large codebases. Not only was much of this work forgotten, in many cases it had never been highlighted in the first place. What happened?

There are many possible answers to why software engineering has lost touch with its roots. For Barr there are 2 important aspects to what went wrong.

The first part of the answer can be seen as far back as 1971 in Winberg’s seminal work, The Psychology of Computer Programming. In that work Weinberg describes how the close alignment between hardware and software was starting to be broken. Those responsible for software development did not grow out of a strong electronic engineering tradition, and consequently understood much less about how their programs affected the computer.

Furthermore, with the advent of time-shared computers and powerful software tools they were encouraged to participate in a frenzied cycle of throwing software at the computer to see what happens, fixing it quickly, and firing it off again without the time to analyze root causes of errors. This resulted in poor practices where software creators are encouraged to care less about the quality of their solutions as long as they “work” and rarely share them with others to build up skills in the most elegant, well-designed approaches to problems. Shared engineering best practice began to be lost.

Second, Barr blames the growing split between the academic practice in universities and the practical realities found in companies developing and delivering large software systems. In the “Golden Age” of software engineering, there was a close alignment between both communities and a strong synchronization between theory and practice. They relied intimately on each other.

To a large degree, Barr argues that this has been lost. Much of what we see today, he believes, involves programmers with no formal background in mathematics, statistics, or even computer science. They are frequently self-taught by collecting ideas and experiences wherever they can Today that increasingly means online learning from a variety of dubious sources and working in isolation in their bedrooms and basements. This not in itself bad. However, it often limits the formal training they have received to rather shallow programming practices.

Based on conversation with Don Knuth, one of the pioneers of the “Golden Age”, Barr observes that in the 1960s and 70s academics were good programmers and industry professionals were not. Yet, now this has reversed as the software industry matured and academics drifted out of sync with what was going on in industry. For a variety of practical and philosophical reasons we have seen academics focus on smaller programs and more niche areas of research to the point that much of their expertise and insight is no longer useful for generating advice for industry.

Furthermore, emerging software development best practices are too often promoted and adopted without sufficient evidence of their value, according to Barr. From object-oriented programming to agile delivery, we have been encouraged to use techniques that bring enormous benefits in productivity and vastly reduce time from idea to solution. Yet, comparatively little research is in place to support these techniques. Consequently, too often there is insufficient rigour in how these approaches are described, understood, applied, and evaluated.

The conclusion, from Barr’s perspective, is that the major push to expand software delivery at the end of the 20th Century may have produced many commercial successes. However, he worries about the software engineering price it has extracted:

“We threw the engineering discipline baby out with the mainframe bathwater. The challenge for software engineering is how to get it back.”

What’s to Be Done?

Given this backdrop, you might expect that Barr is pessimistic about the future of software engineering. Not so. He sees a number of reasons to look forward with optimism.

The first is that he views the move to cloud computing as a key driver for a new discipline in software engineering. The focus on service delivery and assembly of solutions by combining services means that there is much more emphasis on quality of service, availability of services, and mechanisms for building robust solutions by combining services from multiple sources. Additionally, a variety of emerging cloud-based data gathering approaches for online service use is fuelling analytical techniques that underpin software architectures and design methods for large-scale, fault tolerant solutions.

Second, Barr has high hopes that recent changes in team structures for more agile delivery will yield benefits in how organizations experience software design and delivery. As multi-disciplinary teams form around service-based systems, the team works together to solve problems, build better solutions, and manage their evolution as errors are found or needs change. This is a positive move forward, and much closer to the Fred Brooks’ view from the 1970’s of “surgical teams” than anything we have seen in the past 40 years.

Finally, Barr hopes that we will soon see a renaissance in software engineering education fundamentals in universities, schools, and elsewhere. He recognizes that there are many structural and cultural challenges here. However, he sees ways in which a different educational approach could be emphasized based on introducing students to large codebases, working cooperatively to understand and evolve that code, encouraging closer alignment between universities and the software industry, and bringing a broader perspective on the software problems by working with other disciplines. Businesses will demand more from educators to help close the gap that currently exists.

The final piece to the puzzle for Barr is to refocus on software engineering as its own discipline. This requires a re-emphasis on the core fundamentals of software engineering. It also requires recognition that certification is the heart of every engineering profession. Something that the software world has not yet broadly embraced. The need for more robust, dependable software-intensive systems will be a major spur for increasing the professionalism of all those involved.

Into the Storm

As reliance on digital systems grows, it is essential that we improve the quality and performance of software. To achieve this, we must accept an inconvenient truth that creating good software is much more than programming, and engineering demands we adopt a disciplined, professional approach. Adam Barr’s recent book on “The Problem with Software” reminds us that there is much we can learn for the future by reviewing the insights from software engineering pioneers over the past 50 years. They point the way to how to train the next generation of software engineers.