When Infrastructure Doesn’t Scale, Automation Becomes the Bottleneck
Fri 6 Feb 2026
Managing infrastructure at scale rarely fails because teams lack ambition or tooling. More often, it breaks down when manual habits persist for too long, when environments grow faster than the systems designed to control them, or when automation is treated as an optimisation rather than a foundation. At scale, small inconsistencies compound quickly, turning operational complexity into fragility.
Ahead of Cloud & AI Infrastructure on 4–5 March at Excel London, Igor Kanyuka, Staff Software Engineer at Meta, reflects on what it really takes to operate and evolve infrastructure across millions of servers. Drawing on experience running global fleets where failure is measured in seconds, not days, he shares practical insights on automation-first thinking, observability at scale, and why reliability is ultimately engineered through code, not heroics.
In this interview, Kanyuka explores how principles born at hyperscale can help teams of any size move away from firefighting and toward systems that are predictable, resilient, and built to grow.
– – – – – –
Managing millions of servers is something very few organisations face. What principles from operating at this scale do you think are useful for companies managing far smaller environments?
I always advocate for automation and Infrastructure as Code; while it’s a matter of survival at our scale, it’s the ultimate growth engine for smaller environments by replacing manual error with documented repeatability. I also stress the importance of observability and standardisation—you can’t scale a ‘snowflake’ environment, and you need the data to solve root causes rather than just reacting to fires. By adopting these principles early, a team stops firefighting and starts focusing on high-value work and growth.
Reliability at scale often depends on deliberate engineering rather than manual effort. What shifts in thinking help teams move toward automation-first operations?
The biggest shift is making it a non-negotiable habit to never touch a host directly – even during a ‘fire,’ you commit to the repo and let the deployment system handle the distribution. We have to stop thinking about a list of actions to perform on a server and start thinking in terms of the desired state of the system. When you combine that mindset with canary rollouts, health checks, and deep observability, you gain the confidence to move fast because you’ve built the safety net to catch a failure before it spreads.
Large fleets undergo constant change across hardware, software, and configuration. What approaches help teams keep systems stable even as their environments evolve continuously?
To maintain stability amidst constant change, you have to aggressively minimise permutations through strict standardisation. My approach is to lean heavily on abstractions: we decouple applications from the underlying hardware and OS using containers, and we hide hardware-specific quirks behind standardised interfaces like Redfish. This ensures that even as the physical fleet evolves, the management layer remains consistent, predictable, and scalable.
With so many machines generating signals, observability becomes essential. What helps distinguish meaningful insights from noise in environments of this scale?
At this scale, you have to accept that fixing every individual server is impossible; instead, we prioritise symptom-based metrics like latency over specific causes to stay focused on service health. By using p90/p99 aggregations and dimensioning data with tags – like region or hardware type – we can filter out ‘long tail’ noise and focus our engineering effort on the systemic patterns that actually impact the business.
Automation can reduce operational burden, but it also introduces new responsibilities. What helps teams build and maintain confidence in automated workflows over time?
Confidence in automation isn’t a feeling – it’s a byproduct of a fail-safe architecture. We build that trust by mandating pre-merge testing and validating changes with shadow traffic or canary rollouts before moving to phased deployments. When the system is designed to automatically revert based on health checks and maintains strict backward compatibility, an ‘undo’ becomes just as reliable as the rollout itself, turning high-stakes changes into routine events.
“Self-healing” infrastructure can mean different things depending on an organisation’s maturity. How would you describe it in practical terms for teams beginning their automation journey?
In practical terms, self-healing isn’t about ignoring problems – it’s about decoupling the remediation from the investigation. It means building automated ‘first responders’ that handle known triggers, like clearing disk space or restarting a stalled service, before they impact the user. While this keeps the system running, the real maturity comes from treating every automated fix as a data point; we track these remediations to identify systemic patterns so that we can eventually deliver a permanent code fix rather than just repeatedly treating the symptom.
Retiring infrastructure responsibly becomes a major challenge at scale. What principles guide effective decommissioning when systems never stop growing or changing?
Decommissioning strategy is heavily dictated by supply chain logistics; hardware may arrive in staggered batches or massive bulk deployments, requiring a flexible approach. Ultimately, the goal is to maximise automation while minimising architectural drift. By enforcing hardware standardisation and maintaining software compatibility with old HW, coupled with rigorous continuous testing, you can ensure the decommissioning process is fast and painless. Having infrastructure as a code in place helps fast provisioning and traffic shift to the new systems.
Operating at global scale requires collaboration across hardware, networking, software, and automation teams. What helps build alignment across such diverse disciplines?
Building alignment at global scale rests on two critical pillars:
- Encapsulated Boundaries & API Contracts: We must establish clear boundaries between teams and systems, using well-defined APIs as the primary communication method. This decoupling allows individual teams to innovate and optimize their internal implementations without risk to the broader ecosystem.
- Cross-Functional Decision Making: Involving representatives from hardware, networking, and software early in the decision-making process ensures that new initiatives are compatible across the entire stack. This holistic involvement ensures we meet business requirements and that every layer – from the physical to the virtual – is ready for the next generation of infrastructure.
For organisations modernising their infrastructure, what early indicators suggest they’re ready to move toward more automated, observable, and governed operations?
As I mentioned, these principles aren’t just about scale; they are about confidence and reliability. To me, the definitive indicator is the moment a project moves from Proof of Concept into Production. Regardless of the size of the environment, once you have users depending on your service, automation and observability shift from being ‘extra work’ to being your primary insurance for rapid root-cause analysis and service stability.
Ahead of your session, what perspective or idea do you hope will stay with people as they think about infrastructure at large scale?
Ultimately, I want people to walk away realising that at scale, your infrastructure is only as strong as the code that manages it. Automation is the engine that makes the impossible, possible; it is the heartbeat of the operation, and without it, scaling is just a theory – automation is what makes it a reality.
