Frankfurt data centre issues cause connectivity headache for Linode
Mon 2 Oct 2017
Virtual private server provider Linode suffered failures at its Frankfurt data centre, causing connectivity issues and packet loss.
The New Jersey-based cloud provider faced numerous problems during the 1st and 2nd October and has confirmed that at around 2 pm UTC on Sunday it was dealing with connectivity issues which affected Linodes in its Frankfurt data centre.
The firm updated customers that both of its Frankfurt internet edge routers were experiencing problems with their forwarding tables. The status update page stated that this was causing sporadic blackholing of customer traffic.
An hour later, Linode engineers were still troubleshooting with Cisco which attributed the connectivity problems to a software bug.
By 3.32 am, the company stated: ‘After many hours of troubleshooting…it appears we have hit hardware and software failures on both internet facing edge routers simultaneously.’
It explained that the ‘A’ side router had had a ‘module degrade silently’, which meant that it did not fully fail and ’caused strange forwarding behaviour’, but did not notify the operating system of the failure. That module failure became apparent during the boot process once the ‘A’ side router was reloaded.
The failure occurred despite the availability of redundant routers, which Linode said also experienced the same undocumented forwarding table bug that caused the blackholing of customer traffic.
By the time the announcement was made, engineers had been able to ‘normalise connectivity’, though, in order to fully resolve the issue, the company noted that separate maintenance will have to be carried out. A full investigation and analysis will also be carried out in due course.
Linode suffered further issues around four hours later with another update. The company noted that packet loss was occurring in some routing paths between its Frankfurt data centre and other Linode data centres. This was eventually solved by 12.44 pm on October 2nd.
This is not the first time Linode has experienced connection difficulties. Last year, it suffered a ten-day downtime due to a large denial of service attack.
