LAN Versus WAN Traffic Volumes: 80-20 Rule Probably Still Holds

Historically, more data has moved on local area networks of all types than across public and private data networks. That was true in the client-server era and arguably remains true in the cloud computing era. Even looking at consumer data consumption, which one might arguably claim is 80-20 in favor of WAN-delivered content versus local on-premise data sharing, most of that end-user-consumed or sent data is over private Wi-Fi--hence local area networks--even if the access connection implies that most data is device to cloud and cloud to device.

Given wide use of Wi-Fi as the LAN medium, device-to-cloud and cloud-to-device data transfers usually feature nearly-identical amounts of data movement "across the LAN" (Wi-Fi) as "across the WAN" (device to cloud to device).

The point is simply that growth of public network data volumes is a reflection of internet app use overall, and hence derivative of that use. That, in turn, has made data centers the focal point of data transfer requirements.

By some estimates there are 8,000 data centers globally, looking primarily at public (multi-tenant) and other large data centers supporting cloud computing. Of course, that only enumerates the big public and hyperscale data center sites. Forecasts suggest we are headed for more than 1,000 such centers by perhaps 2026.  

If we include all private enterprise and other entity data centers, there could be as many as seven million data centers operating globally. We sometimes forget how those facts have shaped global data networks. 

Some 50 or 60 years, global data traffic largely moved between telephone company central offices. Data network traffic was logically and often physically distinct, moving between private data centers on dedicated facilities and logical networks. 

source: Cisco 

These days, data moves between data centers and to end users of data from data centers. Central offices still function, but their contribution to volume is negligible. 

source: Statista 

Electricity consumption is one example of footprint, but the bigger implication for much of the connectivity business is the shaping of data communications volume. Traffic volume moving between data centers is just about equal to data served up to, and coming from, actual end users globally. 

Internet workloads in the cloud computing era drive traffic across networks, both between data centers and to end users who are the ultimate consumers. Far more traffic moves within each data center. 

source: IEA 

Edge computing might eventually shift some traffic off wide area networks and confine it locally. Just how much impact edge computing will have on WAN traffic is hard to determine right now. It might be quite fair to predict that cloud WAN traffic will continue to grow even if edge computing moves some workloads off remote data centers.  

At a high level, traffic moves between places where internet use is highest. Up to this point, that has meant east-west between Europe, North America and Asia. South-to-South and Europe-to-Africa are growth areas.

Edge Gets the Hype, But Cloud Data Centers Get the Money

Even as preparation for edge computing gets headlines, investment activity remains focused on hyperscale and regional data center investment by firms that formerly would have invested in other forms of real estate.

“Demand for data centers by global investors remained robust in the second half of 2021, with record-breaking merger and acquisition activity,” according to CBRE. “Nearly 95 percent of respondents to CBRE’s 2022 Global Data Center Investor Sentiment Survey, many of whom are the world’s largest institutional real estate investors, plan to increase their capital deployment in the data center sector,” says CBRE. 

In North America alone, transactions included:

• American Tower’s acquisition of CoreSite for $10.1 billion.

• Blackstone’s acquisition of QTS for $10 billion.

• Cyxtera’s SPAC merger with Starboard Value Acquisition Corp, valued at $3.4 billion.

• Mapletree’s acquisition of the Silas Realty Trust Portfolio of 29 data centers for $1.3 billion.

• Prudential & Digital Realty’s joint venture sale of 10 powered shell data centers for $581 million.

• DigitalBridge’s Vantage SDC acquisition of CA22, a 24 MW hyperscale data center, for $539 million.

Already in 2022, North American transactions include:

• KKR & Global Infrastructure Partners acquisition of CyrusOne for $15 billion.

• DataBank’s acquisition of four CyrusOne data centers in Houston for $670 million.

In the Europe Middle East Africa market, 2021 transactions include:

• IPI Partners acquired the dominant Nordic operator Digiplex.

• Iron Mountain acquired a turnkey facility in Frankfurt from Keppel Data Centers for €76 million.

• Antin Infrastructure Partners acquired leading U.K. managed-services provider Pulsant.

• Azrieli purchased colocation operator Green Mountain in Norway for 7.6 billion NOK.

• Blackstone acquired a triple-net facility leased to Equinix in London Docklands for £196.5 million.

• Keppel DC REIT acquired two triple-net facilities, one in the Netherlands and the other in the U.K. for a combined total of more than €100 million.

• Digital9 Infrastructure acquired Verne Global, a dominant Icelandic colocation provider for £231 million.

• AtlasEdge, the recently formed edge operator backed by Liberty Global, Digital Bridge and Digital Realty, acquired a portfolio of 12 assets from Colt Data Centers.

In 2022, transactions in EMEA include:

• KAO Data, backed by Legal & General Capital, Goldacre and Infratil, have acquired two data centers in West London via a sale and partial leaseback.

• Digital Realty agreed to acquire a majority stake in Teraco, Africa's leading carrier-neutral colocation provider, from a consortium of investors, including Berkshire Partners and Permira, as well as Medallion Data Centres in Nigeria.

In the Asia-Pacific region, 2021 transactions include:

• Vantage Data Centers and lead investor DigitalBridge acquired Hong Kong-based PCCW and Agile Data Centers.

• GLP acquired a 50% stake in Songjiang Internet Data Centre in Shanghai.

• Digital Edge acquired five data centers in Japan for $230 million.

• Equinix entered the India market through the acquisition of GPX India.

Already in 2022, APAC transactions include: 

• Equinix $525 million joint venture with GIC in South Korea.

• Keppel Capital to close $1.1 billion data center fund.

• Mitsui announced intention to invest $2.7 billion to develop data centers in Japan.

• GLP announced plans to develop 900 MW of data centers across Tokyo and Osaka and totaling $12 billion.

In Latin America, 2021 transactions include:

• Piemonte Holding acquired Globo’s Data Center in Rio de Janeiro.

• Squared Capital acquired KIO Networks, a leading data center operator in Mexico.

• Goldman Sachs Asset Management invested in Piemonte Holdings' Brazilian edge data center platform Elea Digital.

• EllaLink and Equinix delivered the first undersea cable between Europe and Latin America.

In Latin America, 2022 transactions include:

• Equinix opened a hyperscale facility in São Paulo and announced plans for two more in Mexico City and another in São Paulo.

• Ascenty launched two data centers in Rio de Janeiro and Hortolândia.

• DigitalBridge-owned Scala Data Centers began construction on two hyperscale data centers in Brazil, with one fully leased to a cloud provider.

• Tencent Cloud launched its first Brazilian data center.

• Ascenty plans to build its fourth data center in São Paulo.

• Microsoft announced plans to establish a Chilean data center region.

Data Center Architecture Hinges on Connectivity: N-S as well as E-W

Some adaptations of data center architecture in the cloud computing era have to do with connectivity architecture more than the way servers are cooled, the types of processors and specialized chips used in servers or the way servers are clustered. 

A cluster is a large unit of deployment, involving hundreds of server cabinets with top of rack (TOR) switches aggregated on a set of cluster switches. Meta has rearchitected its switch and server network into “pods,” in an effort to recast the entire data center as one fabric. 

source: Meta 

The point is to create a modularized compute fabric able to function with lower-cost switches. That, in turn, means the fabric is not limited by the port density of the switches. It also enables lower-cost deployment as efficient standard units are the building blocks. 

That also leads to simpler management tasks and operational supervision as well. 

And though  much of that architecture refers to “north-south” communications within any single data center, the value of the north-south architecture also hinges on the robustness of the east-west connections between data centers and the rest of the internet ecosystem. 

source: Meta 

In fact, it is virtually impossible to describe a data center architecture without reference to wide area and local area connectivity, using either the older three-tier or newer spine-leaf models. The point is that data centers require connectivity as a fundamental part of the computing architecture. 

source: Ultimate Kronos Group 

That will likely be amplified as data centers move to support machine learning operations as well as higher-order artificial intelligence operations. 

Still, in the cloud computing era, no data center has much value unless it is connected by high-bandwidth optical fiber links to other data centers and internet access and transport networks. 

“From a connectivity perspective, these networks are heavily meshed fiber infrastructures to ensure that no one server is more than two network hops from each other,” says Corning.

The other change is a shift in importance of north-south (inside the data center) and east-west (across the cloud) connections and data movement. As important as north-south intra-data-center traffic remains, communications across wide area networks assumes new importance in the AI era. 

The traditional core-spine-leaf connectivity architecture between switches and centers increasingly looks to be replaced by a two-layer spine-leaf design that reduces latency. In other words, in addition to what happens inside the data center, there will be changes outside the data center, in the connectivity network design. 

source: Commscope

In the above illustration, the Entrance Room (ER) is the entrance facility to the data center.

The Main Distribution Area (MDA) holds equipment such as routers, LAN and SAN switches. The Zone Distribution Area (ZDA) is a consolidation point for all the data center network cabling and switches.

The Equipment Distribution Area (EDA) is the main server area where the racks and cabinets are located.

 

So even if connectivity remains a separate business from cloud computing and the data center business. But all are part of a single ecosystem these days. Cloud computing requires data centers and data centers require good connectivity. 

How Will Edge Change Data Traffic Patterns?

Data center and consumer end user data usage are virtually mirror images of each other: most end user data consumption involves remote server access, while most data center data consumption is based on other local servers. In other words, consumer data usage now is WAN centric, while data center demand is LAN centric. 

How those patterns might change once edge computing becomes ubiquitous is not yet clear. By definition, more computing and server access will come from a local source (on the premises or within a metro area) rather than from a remote data center.

Computation-focused operations (image analysis, virtual reality, augmented reality) will still rely on external server access, only those servers will be closer to the end user.

Many other content-related operations will rely on the existing metro data center architecture (content delivery networks), while highly-customized operations relying on large data stores will still happen msotly remotely.

The huge amount of “within the data center traffic” is partly caused by applications that involve lots of queries. Many internet applications are extremely “chatty”. A single search query within the data center might involve hundreds of server requests, for example. 

A social networking transaction has a similar multiplier effect, as it draws in an entire social graph to respond to a single query. 

source: Cisco 

The architecture of data centers can contribute to the amount of traffic as well, using with separate storage arrays, development or  production server pods and application server clusters that all need to talk to one another.

Data center traffic moving to end users was a decade ago a larger percentage of total wide area network data volume. That has been steadily changing, with more traffic moving between data center locations.  

source: Cisco 

In 2021, the volume of data moving between data centers is about equal to the amount of data moving to end users. Content caching accounts for some of the data center to data center increase. Content mirroring accounts for an additional amount of inter-data-center traffic. 

source: Cisco 

Still, wide area network bandwidth now is about equally composed of traffic heading for end users and traffic moving between data centers, a trend itself driven by the dominance of content as a driver of network capacity. 

source: Telegeography 

Content drives as much as 83 percent of transAtlantic traffic and 66 percent of transPacific traffic, for example. 

source: Telegeography 

Edge Computing Necessarily Drives Additional Interconnection Demand

The definition of “interconnection,” and “interconnection revenue” in the data center space might be changing, especially as edge computing grows.

Traditionally, such functions might mostly have been applied to in-building connections between servers. In such instances, even when there was a fee for such connections, the operations fell clearly within the realm of local area networking. 

Edge computing pushed connectivity needs out from the hyperscale and tier-one data centers to many local edge connections, by definition increasing the necessity of additional interconnectiions. And it does not stretch credulity to argue that non-traditional connectivity service providers including data centers themselves will benefit.

Today, interconnection increasingly refers to operations and potential revenue connecting servers and buildings across public networks or wide area networks. Equinox, for example, reported 2022 full year interconnection revenues representing about 17.5 percent of total revenue. 

Granted, much of that revenue comes from in-building cross connects, not “outside the building” connections between data centers, peering or interconnect locations. In the following chart, amounts are in dollars billion. So Equinix in 2022 earned about $1.27 billion in interconnection revenue. 

Assume 47 percent of that was earned supplying cross connects inside Equinix facilities, suggesting $597 million in such revenue. That leaves about $673 million in wide area network or access network interconnection revenue. 

source: Equinix 

Back in 2020, Equinix said Equinix Fabric, one of the firm’s “outside the building” connectivity lines of business, represented about 10 percent of total interconnection revenues. But that part of the business has grown. “Virtual” connections were said to be growing at about 32 percent annually.  At such rates, 2022 contributions from that source could have reached 17 percent or so of total interconnection revenue. 

But Equinix has several product lines within the interconnection area. 

Equinix Fabric provides secure, on-demand, software-defined interconnection globally. 

The biggest current revenue driver is likely the Cross Connects service, which provides a point-to-point cable link between two Equinix customers in the same data center.

Equinix Internet Exchange enables networks, content providers and large enterprises to exchange internet traffic through peering points. 

Equinix Internet Access operates in 40 markets. Fiber Connect provides dark fiber links between customers and partners between multiple Equinix data centers.

Assume cross connects still represent as much as 47 percent of interconnection revenue. That also means as much as 53 percent of revenue comes from services that traditionally are provided by telcos, internet service providers or wide area networking specialists. 

Those products include lit fiber services, access to dark fiber, Ethernet transport or wavelength transport. 

The point is that Equinix earns a substantial amount of “connectivity” revenue as part of its data center services operations. 

source: Paolo Gorgo

Of note is the value Equinix, for example, seems to derive from its interconnection function. Where all “pure play”  connectivity suppliers grudgingly expect lower prices virtually every year as a main trend, Equinix seems to derive much more value from its interconnection services. 

To be sure, interconnection is not the primary driver of revenue. But Equinix experience also illustrates that some interconnections are deemed more valuable than others. 

source: Hoya Capital 

It is not unrealistic to predict that, with the growth of edge computing, interconnection will drive even more activity and revenue. 

 

According to Viavi, interconnection “refers to the technology used to link together two or more individual data centers to pool resources, balance employee workloads, replicate data, or implement disaster recovery plans and provide business data closer to the edge.”

Data center interconnections, in other words, are now distinct from in-building interconnections connecting campus locations up to about five kilometers; metro connections up to 100 km or traditional WAN connections spanning hundreds of kilometers. 

source: Viavi 

As always, infrastructure providers are direct beneficiaries of new needs for data center interconnections as they supply the appliances required to create the connections. Data center operators might monetize their DCI capabilities directly or indirectly, perhaps separately from in-building cross connects and peering. 

Tenants might be charged for use of ports or higher-level networking protocols, for example, whether direct (cross connects) or virtual (WAN connections).

Perhaps the point here is that interconnection revenue seems to have higher value in the context of application and computing performance, compared to more generic enterprise data networking. 

For an industry perpetually concerned with declining profit margins caused, in large part, by perceived commodity status, the higher value apparently connected with Equinix ecosystem connections is significant.