Digital Economy Dispatch #115 -- Intelligence Moves into the Open

Digital Economy Dispatch #115 -- Intelligence Moves into the Open22nd January 2023

Since the early days of computing, strategists and technologist have faced a major dilemma: How to create an open, accessible set of capabilities for the benefit of all while also protecting privacy and driving commercial interests. On one side of the argument is the need to ensure that the knowledge and systems in use are transparent and feely available to all. On the other sits the need for allowing personal freedoms and encouraging new investment without inappropriate barriers in place. Many aspects of these concerns are highlighted in the voices of key individuals across the years including Richard Stallman, Tim O’Reilly, Tim Berners-Lee, and many more.

This debate has taken several forms. Initially, much of the discussion involved the public standards that underpin the internet. It then evolved into efforts such as the Open Source Initiative and a wide variety of Open Source Foundations championing different technologies such as the Apache web server and the many flavours of the Unix operating system. In all cases the goals of these efforts have been to use “the power of the crowd” to increase innovation by creating a reusable technology base that encourages broad involvement in its evolution and the widest possible user base. Such ideas have a great deal of appeal across many domains.

Toward Open Data

While much of the previous focus has been on core software infrastructure, the concept of openness is an increasingly important topic in relation of data. Access and ownership of data have always been critical concerns. However, in this digital age the opportunities and challenges have magnified as individuals, companies, governments, and institutions are able to capture, store, analyze, and share more and more data from a wider spectrum of sources. Furthermore, digital advances make it easier to gather data in large amounts, in real-time, and even without the knowledge or consent of those involved.

In this context, how we understand and manage digitally derived data has become one of the most important questions we face today. It has led to concerns about the hoarding and exploitation of data by the BigTech companies, and by other public and private agencies. In response, groups such as the Open Data Institute (ODI) have proposed a different vision for open data in “a world where data works for everyone”.

In our current digital context, questions about who owns your data and how it is used become more relevant and complex. The digital data trail we create is substantial. It seems that everything we do becomes a permanent part of a digital footprint: The clicks you make in your internet browser, CCTV images as you arrive in a supermarket car park, your digital health records, every item you purchase online, and so on.

The challenge, as always, is finding the right approach for collecting and managing data “in the public interest”. There is no doubt that important steps have been taken through initiatives such as those in the UK Government for coordinating common standards for data management and highlighting publicly available data sources from government. However, even as these efforts come into effect , the fast-paced advance of digital technology raises new questions.

Open Source Intelligence

As the number of digital data sources expand, the amount of information available for analysis has increased dramatically. This raises the possibility that we can use that data to learn more about global events and use that information to understand and manage their path. For many people, this has been highlighted in two major world events:

• In addition to their many severe human consequences, the global covid pandemic forced the world to reconsider many aspects of how it tracks data about individuals and populations. The lessons from these activities have driven what some now call “a new era of digital epidemiology”.

• The conflict in Ukraine has expanded the discussion on open data and its effects. The terrible consequences of the fighting in that region have focused attention on ways to gather information and share it amongst its citizens and across the world.

In both of these situations, there have been important advances in data gathering and analysis to help understand more about situation as it evolved and to act on the information available. One important element of this has been the use of Open Source Intelligence (OSINT).

Broadly speaking, OSINT is the gathering of open data sources to support understanding and guide decision making. Often associated with military, security, or law enforcement activities, the use of public information has long been important in their work. With the wider availability of digital data sources, the opportunities for OSINT have significantly increased. Whether examining satellite images to understand troop movements or analyzing social media posts to review sentiment about government actions, digital sources are being used as important inputs to many kinds of strategic decision making.

However, the importance and impact of open source approaches to intelligence now go far beyond digitizing traditional intelligence operations. They are now seen as a major digital disruption in the conflict in Ukraine such that some describe it as a digital war being fought both on the ground and in cyberspace with OSINT a significant factor.

The Six Impacts of OSINT

This was highlighted in a recent speech by General Hockenhull, the UK’s head of Strategic Command. In his reflections on the conflict in Ukraine, he believes that OSINT has had a significant range of impacts in 6 key areas:

1. Anticipatory intelligence: Intentions and attitudes can be examined by collecting commercial data and analyzing public data from social media streams. From these sources, future actions can be predicted.

2. Shared Understanding: A more complete view can be created with shared data from a variety of sources. This builds a consistent picture and creates confidence across all stakeholders.

3. Disprove falsehoods: False statements and misunderstandings can quickly be rebutted with public data sources that have been gathered from many places and openly inspected.

4. Reinforcement and resilience: Many sources of data can be combined to ensure errors are reduced. Furthermore, by using multiple commercially technology providers, the data gathering is more resilient in case of the failure of any one piece of the system.

5. Expanded data gathering: Rather than rely on a small number of sources, an open approach supports wide data collection from many sources. This provides greater assurance in the completeness of the perspective created.

6. More pieces of the jigsaw: Much of the challenge in analysis is making sense of partial information being gathered across complex, fast-changing environments. Open source data provides an opportunity to deliver more of the jigsaw pieces needed to complete the picture.

As a result, OSINT is viewed as an important step forward in strategy and decision making. It encourages and supports a more open approach to deliver results, and ensures that your able to adapt to fast paced changes in your environment.

However, as General Hockenhull points out, there are also several dangers to this approach. Significantly, digital data can be overwhelming. So much is generated so quickly that systems are unable to process it with the speed and fidelity needed to make sense of what’s happening. So, while OSINT gives you more of the jigsaw pieces, General Hockenhull also admits that this can sometimes feel like all you have done is create an infinite set of new pieces, not gained a better sense of how they fit together. A challenge many will recognize from their own digital transformation activities.

Similarly, while OSINT is an exciting opportunity to bring more intelligence to strategy and decision making, it will only be useful if the data gathered is accurate and complete. Unfortunately, in many situations, particularly in government and public sector agencies, the data being make available is inaccurate or incomplete. Another major challenge that still needs urgent attention.

More is More

As many more open digital data sources emerge, we have the opportunity to use them to learn quickly about what is happening around us and to make better decisions about next steps. By using Open Source Intelligence (OSINT), more effective and adaptable approaches to strategy are possible. This is being seen in the conflict in Ukraine where OSINT is driving 6 important impacts that are critical for all organizations navigating uncertain times.