The Impact of AI on the Network

In this AI-focused podcast episode of TeleGeography Explains the Internet, Seva Vayner, Product Director for Cloud Edge & AI at Gcore, joins host Greg Bryan to tackle what is perhaps the dominant topic in telecom today: the impact of AI on the network.

The pair distinguish between "AI for networking" and "networking for AI," and explore how AI training models are driving a dramatic increase in power consumption within data centers, with rack power capacity rapidly growing over the past few years.

• Seva explains how AI inferencing is creating a need for distributed network infrastructure, transforming the role of Content Delivery Networks (CDNs) from simply distributing content to enabling real-time interaction with AI models.

• He shares how Gcore is helping telcos and enterprises adopt AI solutions and exactly what that entails.

• We close out on the evolving definition of the "edge" as power becomes a bigger constraint than connectivity in facilitating AI workloads and what Seva sees for the near future of AI and networks.

Power density is exploding

The most immediate physical challenge is the sheer amount of electricity required to train models. Vayner notes that just a few years ago, a standard data center rack capacity was roughly 5 kilowatts (kW). By 2022, discussions shifted to 50 kW per rack, and today, densities are reaching 130 kW per rack, with future projections hitting as high as 600 kW. This exponential growth is driven by the shift toward high-performance GPU clusters, such as NVIDIA’s H100s, which are essential for training large models.

The shift from training to inference

While training models requires massive, centralized compute power with high "East-West" interconnectivity, the actual usage of these models—inference—requires a distributed approach. Vayner compares this evolution to the traditional Content Delivery Network (CDN) model. Just as CDNs were built to distribute video and static content closer to users to reduce latency, networks must now distribute compute power to handle real-time AI interactions.

For applications like voice assistants or future real-time video generation, latency is critical. This is creating a new role for CDNs, transforming them from content distributors into platforms enabling real-time, distributed AI inferencing.

The definition of "edge" is changing

Historically, the "edge" was defined by geography—placing servers in Tier 2 or Tier 3 cities to be closer to the user. However, power is becoming a bigger constraint than connectivity. Because high-end GPUs consume so much energy and generate so much heat (requiring liquid cooling), putting them in traditional "edge" locations, like office building closets, is becoming impossible. Consequently, the "edge" is now defined by where sufficient power and cooling can be secured, rather than just physical proximity.

Enterprise adoption and time-to-market

Enterprises are moving beyond public SaaS experiments toward building private AI solutions to protect their data security. However, building proprietary infrastructure from scratch is risky due to the speed of hardware innovation. Vayner points out that if a company spends a year building a data center, their GPUs may be obsolete by the time they launch. As a result, enterprises are increasingly turning to turnkey solutions that offer managed infrastructure and orchestration, allowing them to focus on business value rather than hardware maintenance.

As Vayner concludes, while the market is currently hyped, AI workloads will eventually become a commodity workload integrated into everyday life, much like standard CPU-based applications are today.