Beyond the Edge: High Speed, High Density Data Cabling for the Future Data Centre
Mon 18 Mar 2019 | Stephen Morris
The data centre is here, it’s just not evenly distributed. The drive for lower latency is fuelled by enablement of technologies such as 5G – which will push data centre’s closer to the edge, meaning compute power, storage and data centre facilities will need to physically reside closer to where those services are demanded
Applications such of driverless cars and augmented reality will need to be supported by systems that deliver sub 1ms of latency, end to end link, and this will transform the data centre landscape as we know it today.
Data centres are set to get physically larger and more bandwidth hungry. Mass introduction of small wireless cells to support 5G will be, in the main, linked with fibre cables – propelling the need for high density fibre solutions from the ‘macro’ cell site, where small cells are aggregated, right back to the supporting data centres.
We are poised for gigantic growth of data traffic transacting within the data centre. Known as ‘machine to machine’ (M2M) or ‘East to West’ communications, M2M is predicted to grow by 44% CAGR between 2015-2020 (source: Cisco). Partly meeting this development, 100G is forecast to represent over 50% of data centre optical transceiver transmission capacity by 2019 (source: Infonetics). Additionally, 400G deployment will gain momentum in the coming year and is forecast to accelerate at great pace, representing the majority of Ethernet port shipment by 2021. Developments in 400G bring significant value in terms of cost saving, 4x the density per RU and 4 x scalability. These innovations will also drive 2x power performance efficiency – making a positive contribution towards energy reduction targets that ‘Edge’ sets out to achieve.
Single Mode Fibre or Multi Mode Fibre?
In the past, selection between single mode fibre (SMF) and multimode fibre (MMF) solutions were relatively defined – the former provided highest bandwidth at a higher price, the latter a cost-efficient alternative in less speed and reach sensitive use installations.
Today critical factors in decision making include determining whether current bandwidth requirement can be economically met; reach/distance requirements including foreseen network architecture specific goals achieved; and whether future Ethernet and/or Fibre Channel speed can be supported or easily migrated to over the return of investment (ROI) period. These four inter-dependent criteria and must be considered to reach the optimal choice.
With seemingly infinite bandwidth single-mode fibre (SMF) has often been considered the best policy by network designers and data centres to ensure a future without bandwidth bottlenecks. Traditionally there have been high costs associated with SMF optical systems, this cost addition is attributed to the price of the optical transceiver modules and overall cost is less influenced by the price of the passive optical cabling infrastructure.
At the same time, it is acknowledged that despite the cost penalty, SMF technology does indeed hold advantages over MMF in terms of longer reach and bandwidth capability. But as companies embark on a transition towards high-speed data transport do alternative fibre cabling systems offer capabilities that previously only SMF could satisfy?
At this point, it is important to acknowledge that the paradigm has recently shifted in terms of decision making when determining which grade of fibre provides the best return on investment in your particular environment. Technological advance now places MMF at the top of the list in terms of making an informed choice for both present and future network needs.
Is the MMF data pipe large enough to support future generations of traffic?
Multimode fibre supports a large proportion of today’s applications and reaches in the data centre at significantly lower cost. Additionally, it has the capacity to meet future data capabilities of data centres with a roadmap that can feasibly support up to 800G Ethernet.
The bandwidth capabilities of MMF has grown exponentially with advances in optical transmission technology. More recent MMF transmission developments have seen the introduction of faster vertical cavity surface emitting laser (VCSEL) moving from 10G to 25G, doubling of the line rate (PAM4) and shortwave division multiplexing (SWDM).
With 40/100GE already being deployed and objectives for 25/50/200/400GE defined by the IEEE, Ethernet is considered the leading networking platform. It can deliver data centre architecture to meet future applications needs whilst providing low operating costs today.
Outside of the IEEE, the SWDM Alliance multisource agreement (MSA) sets out to enable 100G over two MMF’s (duplex transmission/fibre pair), supporting transmission over OM3, OM4, OM4+ and OM5 grades. Inability of SWDM MMF solutions to support ‘break-out’ mode (discussed later), is a limiting factor in terms of wide adoption of this technology. Further, it is anticipated line rates for MMF fibre will accelerate from 50G to 100G in the near future, meaning more efficient MMF applications (fewer fibre pairs required) and widening scope for MMF ‘duplex’ and ‘parallel’ optics, utilising a single wavelength vs narrowing scope for MMF applications requiring multiple wavelengths.
Short reach applications, such as switch to server and M2M interconnections is where multimode fibre adds real benefit inside the data centre. Industry standard fibre which is laser optimised now provides a scale of fibre capabilities to assist data centre operators to plan successful infrastructure installs and upgrades dependent of operational requirements.
Application driven decision making
The rise of software-defined networking (SDN) has caused data centre designers to realign from traditional three-layer topologies (Figure 1) to ‘spine and leaf’ architecture (figure 2).
Figure 1
Figure 2
Collapsing the 3 three-layer architecture, to spine-leaf, drives higher port/connector density at the spine as every leaf is connected to every spine switch. A popular practice to resolve higher density connectivity challenges and increase efficiencies at the spine is to use cable assemblies and equipment that can operate ‘break-out’ configuration (see figure 3).
Figure 3
Taking 40G MMF data centre deployments as an example, over 50% of 40G market adoption was believed to be related to ‘breakout’ configurations i.e. 40G to 4x10G.The key driver was not for 40G end-to-end, but to deliver efficiencies in distribution of 10G.
The same concept of continuous incremental efficiencies will drive the growth in 100G (4x25G) and 400G (4 x100G or 2x200G). However, it should be noted that the early generations of SWDM over MMF are unlikely to support ‘break-out’ mode/ configuration. This should be considered when evaluating the benefit of SWDM’ vs single wave length high speed transmission and network architecture design.
Example of the efficiencies a solution operating in ‘breakout mode’ can deliver (estimates only):
- 30% cost saving per port (transceiver cost)
- 2x power efficiency
- up to 4x density improvement per RU
- More space for revenue generating equipment
- Adds value where space is a premium/limited
Cable Reach
Hyperscale data centres are typically very large facilities that house >50,000 server ports. The reach (distance between server/switches) in such facilities often exceeds 150m. With such long reaches it is not unusual to see a mix of 80:20 in favour of single-mode fibre. The opposite can be said for Enterprise data centres, where typically 95% of reaches are <100m and the mix is more likely to represent 80:20 in favour of multimode fibre.
There are advanced grades of optical fibre that offer extended reach MMF far beyond the standards, likewise enhanced transceiver specification to achieve the same end. The chart (see Table 1) gives examples of link distances based on a standard (non-extended reach) transceivers.
Cost as an Incentive
MMF transceivers are manufactured with lower cost VSCEL compared to the higher cost lasers typically seen in SMF transceiver optics. These VCSEL take advantage of the larger MMF core/ cladding diameter (50/125 micron – OM3, OM4, OM4+ & OM5) making the overall solution a lower cost option.
The structured cabling element (which includes fibre cabling) of a network installation typically represents <5% of an IT manager’s budget. Whilst the cost of the fibre grade itself (SMF or MMF) represents a very small percentage of the cable system make-up when considering each fibre channel/ link. Taking the example of a 10G MMF channel and adding the cost of the transceivers to the passive fibre cable infrastructure, the cost of the transceivers would represent circa 80% of the total cost.
There is a desire to simplify and reduce the cost of single-mode transceiver optical modules, this may come with compromises, an example is IEEE 400GBASE-DR4, which has a reach objective of 500m – this is shorter than the >Km typically seen for SMF solutions today.
The gap between MMF and SMF is expected to narrow in future as faster optoelectronic technology matures and volumes increase, however, as a solution today, the price premium for SMF is typically 1.5x to 4x that of a MMF solution depending on data rates/transceiver module type used.
Conclusion
Single-mode fibre continues to play an important role where reaches extend beyond 400m (up to 10G) and typically beyond 150m (40G data rates and beyond) which can include applications across server aisles and across the site.
The introduction of 100G-BASE-PSM4 (parallel single-mode) brings with it a new breed of SMF transceiver, incorporating low cost silicon photonics. This is expected to drive closer price alignment between MMF and SMF solutions, thus, increase popularity of SMF – particularly for greenfield 100G deployments.
However, there is a significant pre-existing installed base of MMF (OM3, OM4, OM4+) in data centres and legacy network expansion represents circa 50% of market requirement. Both equipment manufacturers and customers have a vested interest in supporting and maximising the return on Investment (RoI) of this legacy infrastructure. Therefore, a mass migration towards SMF in brownfield sites is not expected anytime soon.
In any event, the bandwidth of today’s multimode fibre solutions far exceed their perceived limitations. This being the case, bandwidth should no longer be the barrier for consideration of MMF to meet current or future needs. In addition, MMF is the most cost effective choice for the majority of data centre uses where reach is typically <150m. The business case for MMF continues to strengthen as the applications, bandwidth and roadmap it supports extends.
Acknowledging data centres are expected to physically grow in size (see figure 4) and that, for both present and future multimode applications a key limiting factor in terms of performance, in particular, reach, is the fibre’s ability to address model/chromatic dispersion interplay, the time to consider more advanced fibre grades (that address interplay) such as Signature CoreTM OM4 (OM4+) and Signature Core OM5 (OM5+) is… NOW.
Figure 4
Ref: http://www.datacenterknowledge.com/afcom/state-data-center-industry-2018-where-we-are-and-what-expect
OM4 Signature Core (OM4+) offers the best performance for the overriding majority of applications, far outperforming OM5 in most cases.
It is no longer enough to simply pick the next generation of MMF fibre in the belief it returns best value – best value lies in [a] understanding today’s and tomorrow’s application needs and [b] acknowledging the path that advance modulation/line rate is expected to follow.
Panduit has launched a White Paper: Light into Money – The Future of Fibre in the Data Centre Network download a copy here https://pages.panduit.com/light_to_money.html
