The Fabric network design standardizes on “layer 3” from the TOR switches to the network’s edge, supports IPv4 and IPv6, and uses Equal-Cost Multi-Path (ECMP) routing. What Andreyev mentions next is mind-boggling, “Together, pods and planes form a modular network topology capable of accommodating hundreds of thousands of 10G-connected servers, scaling to multi-petabit bisection bandwidth, and covering our data-center buildings with non-oversubscribed rack-to-rack performance.” Network operations Each fabric switch of each pod connects to each spine switch within its local plane.” Andreyev explains, “To implement building-wide connectivity, we created four independent ‘planes’ of spine switches, each scalable up to 48 independent devices within a plane. Andreyev says this is simpler (it is hard to imagine what it used to be like).įacebook engineers stayed with the 48-node theme when adding the spine switches. The next step is to connect all the fabric switches - the slide in Figure C depicts how that is accomplished. Each pod is identical with equal connectivity.Scalability is simplified and unlimited.This approach has the following advantages: Andreyev mentions, “Each TOR currently has 4 x 40G uplinks, providing 160G total bandwidth capacity for a rack of 10G-connected servers.” A single pod consists of 48 racks and TOR switches that are meshed to four fabric switches. The slide in Figure B depicts the new server-rack grouping called a pod. With those billions of requests each day as incentive, the engineers decided to eliminate the complicated, bandwidth-robbing, top-down network hierarchy and replace it with a new design called Fabric. “Even more difficult is maintaining an optimal long-term balance between cluster size, rack bandwidth, and bandwidth out of the cluster.” Fabric: the new network topology Additionally, the need for so many ports in a box is orthogonal to the desire to provide the highest bandwidth infrastructure possible,” explains Alexey Andreyev, Facebook network engineer. To build the biggest clusters we needed the biggest networking devices, and those devices are available only from a limited set of vendors. “First, the size of a cluster is limited by the port density of the cluster switch. This arrangement works, but presents Facebook engineers with several challenges. Also shown in the schematic is each rack’s Top of Rack (TOR) switch that acts as an intermediary between the servers and an upstream aggregation switch.

In real life, instead of three racks, there would be hundreds. The old Facebook networkīefore Altoona, Facebook engineers organized a data-center’s server racks into clusters similar to that shown in Figure A. It’s more like they rewrote the networking design guide. That said, tweaking might not be the right term for what the engineers came up with and implemented in the Altoona facility. So besides building mammoth data centers like the one in Altoona, Iowa, Facebook engineers are constantly optimizing the design of a data center’s network. For instance, a data center topology that was sufficient five months ago is now inadequate. All good for Facebook in general, but a challenge for Facebook’s networking crew. > There may be issues with Bandwidth as all the Server traffic is going via Access Switch to Distribution.Facebook Fabric: An innovative network topology for data centersįacebook's engineers introduced a revolutionary way of moving bits around a data center, and showcased it in their new Altoona, Iowa facility.įacebook data centers receive billions of user requests each day, and that number is increasing as the company adds members and introduces new features. > More Rackspace and Space in the Data Center is required in TOR. > Since very less number of cable running between the server and network rack, it is quite easy to locate the cable and make changes. > server-to-server traffic will be Layer 2 in the Top of Rack Design. > Device Count is increased in TOR as we require one or two access switches in each server rack. > Only a few cables going from Server Rack to the Network Rack as Servers are connected to Access Switches in Server Rack. > In TOR, there is one or two access switches are installed on the top of each server rack which provides network connectivity to the servers and then that access switch has the connections towards the Distribution Layer Switch which is located in the Network Rack. > When there are a lot of servers in a Data Center needs to be connected then we use two popular network designs used in such circumstances – TOR (Top of Rack) and EOR (End of Row)

Tor switch to another server