Features Hub

The bastion server that isn’t there

Mon 11 Feb 2019 | Joshua Kite

When Joshua Kite joined DAZN he had a vision to tackle the scaling and security deficiencies of the traditional approach to bastion servers. Ahead of DevOps Live 2019, we talk to Joshua about his new approach, and the lessons other engineers can learn from his experience

Bastion servers are a critical security component in a network configuration, they are essentially robust computers coupled to a proxy service and antivirus and anti-malware. Other security metrics in a network configuration rely on its data, so it’s an element that must be protected. However, Joshua Kite observed several deficiencies that can cause at-scale deployments of bastions to unravel. The traditional approach “simply doesn’t scale,” he says, leading to dreadful security risks.

Joshua realised that cloud-native platforms could benefit from providing bastion SSH as a stateless service on AWS with identity access management (IAM). Even though people are hesitant to use SSH in the cloud, there are always occasions where you need it, he says, especially in development with databases.

A solution of this kind was a natural fit for new sporting streaming service DAZN, where the theme is building stateless microservices and using hosted services for holding state. By exploiting AWS IAM, the DAZN team could also benefit from not having to manage identity management for its own services.

“This is something that I brought with me to DAZN, at an early and immature stage,” Joshua says.

“Because this service is stateless we can easily update the infrastructure as code immutably and deploy automatically – we don’t need to worry about preserving anyone’s stuff.”

Joshua took his proposed solution to his managers and directors, who then nurtured Joshua’s vision. After a short but intense period of development, they were able to deploy at an early stage.

“To some extent we grew with it,” he says. “There have been a number of code contributions from colleagues and the wider open source community.”

Outside the box

The key to enabling the final solution was an unorthodox use of docker containers. People don’t normally expect to use a docker container interactively, but Joshua’s critical insight was that he could use them to allow every DAZN user to have their own little server, just for them, unique for that session.

“They can set up whatever they like, without worrying about what the previous user did, who else is on the host or what the next user might need.”

One of the biggest challenges of deploying his solution at scale was culture. Subversions of traditional approaches are often met with resistance and inspire misconceptions, and the “bastion server that isn’t there” as Joshua likes to call it, jarred people because its style was so different to what people are used to.

Although the instance size and speed on the cloud hosts didn’t prove an issue (as they often do) the team soon found that they were maxing out their IAM limits.

In part, because they were also using another identity solution, from the Widdix project, for their cluster hosts. As it also hooked into IAM, before they knew it the volume of IAM queries coming in their direction was untenable.

“You should always reflect and question your approach and your tools whether they are your own or inherited from others”

“We were making more than 75000 IAM queries per account every 24 hours – this meant that some queries would fail. That was fun finding a solution to!”

The code is now available on GitHub and the Terraform Module Registry under an MIT licence. It has been in production for a year at DAZN with entirely automated deployment.

The long duration of habit

The lessons Joshua draws from his experience with the “phantom” bastion server are familiar. Don’t follow the same grooves etched by your predecessors; consider the lesser known powers of well-used tools, like docker containers for instance.

“It sounds cliche but I would always try to encourage people to try and find a different angle that they can view things from. You should always reflect and question your approach and your tools whether they are your own or inherited from others. Try to develop a vision for your solution to your problem.”

Nonetheless, he advocates following the well-etched wisdom of Roger Bacon (c.1214 – c.1294) who centuries before the silicon chip articulated four obstacles to truth, which include the “long duration of habit” and the “opinion of the ignorant masses.”

“I think that these are just as true today, almost 800 years later,” he says.

Join me at DevOps Live.

Joshua is presenting at DevOps Live, taking place at ExCeL London March 12-13th. DevOps Live and its colocated events attract over 20,000 IT industry professionals.

Experts featured:

Joshua Kite

Site Reliability Engineer


AWS Cloud DevOps engineering stateless
Send us a correction Send us a news tip