Digital Economy Dispatch #207 -- Enterprise AI and the Future of the Data Center

Digital Economy Dispatch #207 -- Enterprise AI and the Future of the Data Center
27th October 2024

While the excitement surrounding AI innovation is understandable, too much of what we see today is focused on the visible surface of delivering AI-at-scale. Perhaps the real story of enterprise AI lies far beneath, in the underlying infrastructure that enables AI's capabilities. Data centers, the physical backbone of the internet, face significant challenges in accommodating the growing demands of AI workloads. As we push the boundaries of AI, we must also address the limitations of these critical facilities. To understand these issues, we need to take a trip into the shadows to discover a darker side of the enterprise computing infrastructure powering AI.

The unprecedented acceleration of AI adoption is fundamentally reshaping enterprise computing infrastructure. As organizations deploy increasingly sophisticated AI models at scale, data centers and their associated technologies have emerged as a critical strategic consideration that will determine the success or failure of enterprise AI initiatives. Billions of dollars are now being spent on new data centers to support the new demands of AI.

Understanding the scale of this transformation is essential for technology leaders. Data centers, which have long served as the foundation of enterprise computing, are undergoing a profound evolution to meet the computational demands of modern AI workloads. These facilities, incorporating advanced power distribution, cooling systems, and specialized hardware configurations, represent both a significant challenge and opportunity for organizations pursuing AI initiatives.

What Lies Beneath

To understand where data centers fit in the landscape of delivering AI-at-scale, you need to know a little about how the internet works. It is not enough to focus on webpages and social media channels, APIs and data lakes, or web protocols and standards. You have to go below the surface – literally – to explore the physical foundations of what makes the internet work.

Despite its seemingly virtual nature, the Internet is in reality a tangible network of fiber optic cables, data centers, routers, and physical connections – or "tubes" as Senator Ted Stevens famously called them. And it is here you need to start the journey to understand the foundations of Enterprise AI.

The best description of this complex physical world that I have read comes from Andrew Blum’s excellent book, “Tubes: Behind the Scenes at the Internet". Through his global journey, he takes us on a tour of the massive data centers, undersea cable landing stations, and Internet exchange points where different networks connect. He visits critical facilities and follows the path of sub-sea cables that carry Internet traffic between continents.

What we get from the book is a fascinating portrait of the Internet as a physical place, maintained by real people working in unmarked buildings scattered around the world. The data centers, network connection hubs, and other facilities are intentionally kept in nondescript buildings for security reasons yet house the critical infrastructure that makes global digital communication possible. Critical decisions about where to place these facilities, how to connect them, and who controls them shape the modern Internet's architecture.

The Data Center Landscape

But of course, there is far more to AI than the tubes. What they connect is just as important. Data centers are highly engineered facilities that serve as the physical foundation of modern computing and the internet, housing vast arrays of servers, storage systems, and networking equipment in environmentally controlled spaces. These facilities function as the backbone of digital services – from cloud computing and website hosting to streaming media and enterprise applications – by providing the massive computing power, storage capacity, and network connectivity needed to handle global internet traffic. With redundant power supplies, cooling systems, and network connections ensuring near-constant uptime, data centers enable the reliable operation of the digital services that businesses and consumers rely on every day.

As organizations provide a variety of computing needs, the market has responded with differentiated data center solutions to meet these requirements. Broadly, we can distinguish three main categories of data centers:

• Hyperscale facilities offer the advantages of scale and specialized AI infrastructure, providing immediate access to advanced computing resources without the capital expenditure of building dedicated facilities. They offer large-scale cloud services (e.g., AWS, Azure). Ideal for businesses needing flexible, scalable infrastructure.

• Colocation services present a hybrid approach, allowing organizations to maintain control over their AI hardware while leveraging professional facility management and power infrastructure. They provide physical space, power, and cooling for customers' own equipment. They’re suitable for businesses requiring full control over hardware.

• Enterprise data centers, while requiring significant investment, offer maximum control over infrastructure optimization for specific AI workloads. Such facilities owned and operated by companies to house their IT infrastructure. They are designed for specific organizational needs.

The selection of appropriate data center strategies has become a critical decision point for technology leaders, directly impacting an organization's ability to execute on their enterprise computing initiatives effectively. This decision must balance immediate computational requirements against long-term scalability, cost considerations, and strategic control of their computing infrastructure. Consequently, risk averse large organizations favour hybrid solutions that combine elements of all three.

The Computational Burden of Modern AI

Yet, the meteoric rise of AI is fundamentally reshaping the data center landscape, creating unprecedented demands on computational infrastructure and forcing a rethink of traditional data center architectures. As organizations race to deploy increasingly sophisticated AI models, data centers face mounting pressure to deliver the massive compute power and storage capabilities required to train and run these systems at scale.

The computational requirements of AI systems, particularly large language models (LLMs), have grown exponentially in recent years. Training GPT-3, one of the earlier large language models, required an estimated 1,287 megawatt-hours of electricity and several thousand high-performance GPUs running continuously for months. More recent models like GPT-4 and Claude 3 are believed to have even greater computational demands, though exact figures remain proprietary.

These issues are highlighted in a new McKinsey report focused on how the rise of AI is driving data center demand. McKinsey forecast a significant increase in Europe's data center capacity driven by AI and digital transformation. Power consumption is expected to skyrocket from 10GW to 35GW by 2030. This trajectory raises significant questions about infrastructure readiness and capacity planning that technology leaders must address in their strategic planning.

The Evolution of Data Center Strategy in the AI Era

As enterprise AI deployment accelerates, technology leaders must navigate an increasingly complex landscape of infrastructure decisions. The traditional approaches to data center strategy and capacity planning are proving insufficient for the unique demands of AI workloads, necessitating a fundamental reconsideration of infrastructure strategy.

Current-generation AI workloads present distinct operational challenges that differ significantly from traditional enterprise computing requirements. The combination of high-density computing requirements, complex cooling demands, and dynamic resource utilization patterns requires a sophisticated approach to infrastructure planning and management.

Infrastructure modernization has become a central concern, with organizations implementing comprehensive strategies to address these challenges. Traditional power densities of 5-10 kW per rack are giving way to AI-optimized configurations requiring 30-50 kW or more, necessitating fundamental changes in power distribution and cooling architectures. The integration of specialized AI accelerators – from NVIDIA's H100 GPUs to custom ASICs – has become a critical consideration in infrastructure planning.

Resource optimization presents another significant challenge. Organizations must implement sophisticated workload scheduling systems capable of managing the burst patterns and intense resource requirements characteristic of AI workloads. This optimization extends beyond simple resource allocation to encompass complex decisions about workload placement across on-premises and cloud infrastructure.

Future Strategies for Enterprise AI

Looking forward to the continued acceleration of AI adoption, it is expected that several key trends will shape the evolution of data center strategy:

• Architectural Innovation: The emergence of modular design philosophies enables rapid scaling of AI compute capacity. Purpose-built AI data centers, optimized from the ground up for high-density workloads, represent a growing trend in infrastructure deployment. The development of advanced cooling technologies and power distribution systems specifically engineered for AI workloads continues to advance.

• Operational Transformation: Data center management is becoming increasingly automated, with AI systems optimizing facility operations in real-time. This automation extends to workload placement, power management, and cooling optimization, enabling more efficient operation at scale.

• Sustainability Integration: The environmental impact of AI workloads is driving sophisticated approaches to power management and cooling efficiency. Organizations are implementing comprehensive sustainability strategies that include renewable energy integration, advanced cooling technologies, and efficient resource utilization.

For technology leaders, navigating these trends will require a strategic approach to data center planning that considers both immediate requirements and long-term scalability. Success requires balancing multiple factors.

The first is to determine appropriate infrastructure flexibility. The ability to scale AI compute capacity rapidly while maintaining operational efficiency is crucial. This flexibility must extend to supporting diverse AI workloads and emerging technologies.

The second consideration is resource optimization. Sophisticated approaches to workload management and resource allocation become essential for controlling costs while maintaining performance. This includes intelligent distribution of workloads across hybrid infrastructure environments.

Finally, it is important to define an approach to strategic control over AI infrastructure while leveraging external resources effectively. This balance varies based on specific requirements for data security, compliance, and operational management.

With so many complex considerations in mind, successfully defining and executing an Enterprise AI strategy will not be easy. The resulting path forward will inevitably require hybrid approaches that combine traditional enterprise computing infrastructure with specialized AI facilities, managed through sophisticated orchestration systems. Organizations that successfully navigate these challenges while building robust foundations for AI infrastructure will be best positioned to capitalize on the transformative potential of AI.