15-05-2025
EXCLUSIVE: Cybersmart founder owns the internet outage
'This lesson could help the industry,' Cybersmart cofounder and CTO Laurie Fialkov told Daily Maverick in a candid interview as the national outage dust settles.
The criticism for the outage that brought Cybersmart customers to a digital standstill earlier this week, came swiftly. And so did an apology – and then, unusually, an unvarnished account of what really went wrong.
Cybersmart reached out to Daily Maverick to let founder and CTO Laurie Fialkov pull back the curtain on 39 sleepless hours that nearly broke the network, the business, and a few engineers along the way. (Spoiler: it was not a cable break.)
'We did screw up,' he admitted. 'That kind of outage is long in this industry.'
The Cape Town-based ISP and fibre network operator experienced a near-total service disruption starting around midday on Monday, 12 May. What followed was a cascade of misdiagnoses, desperate rewiring, and a rude awakening to the dangers of old hardware lurking in even the most redundant systems.
From café to carrier
Fialkov's internet journey began in a Sea Point internet café called Inthenet in 1996. Back then, a 33.6kbps modem counted as high-speed.
Cybersmart, as the business became known in 1998, grew slowly and steadily. It's now a national network operator with thousands of businesses and residential customers, a fleet of fibre infrastructure and what was – until this week – a 22-year record of uninterrupted uptime.
'We spend so much time trying to be the ISP that never goes down,' Fialkov said. 'We've got everything. Multiple cable systems, battery redundancy, ringed networks… and we just never had an outage.'
But this week, everything went down – silently.
Phantom signals
It started quietly. Customers called Fialkov directly – just a few at first. Network monitoring showed all systems green. 'I can reach the whole network. It's impossible that we are down,' he recalled thinking. 'But then you get 15 calls in five minutes, and it's got to be an issue.'
Outside-looking diagnostics (via 'looking glasses' – remote tools that simulate connectivity from various points) revealed the horror: Cybersmart's Autonomous System Number, AS36874, had essentially vanished from parts of the global internet.
'Like this shell on the internet – completely isolated.'
A Denial-of-Service attack seemed a likely culprit. NexusGuard, Cybersmart's DDoS mitigation partner, was called in – only to say: Not a DOS.
Then, the real enemy emerged: old gear.
The ghost in the chassis
The culprit? Legacy Cisco 6500 routers – high-end switches that formed the spine of Cybersmart's original network.
'These things have been working for 15 years. End of life, yes. But working. We were meant to replace them. But… if it ain't broke, right?'
Until it broke. Hard.
One router in Johannesburg froze. Another in Cape Town followed.
'Too coincidental,' Fialkov said. Then more went dark. 'It was like a cancer – six machines, six different places, all failing.'
The root cause? A global routing table explosion. New peers (China Telecom, Hurricane Electric, Saudi Telecom) dumped an extra 150,000 routes into the internet's core. The old routers couldn't cope – they choked and silently failed.
With no support (Cisco dropped them years ago) and no viable fix, the team made a call: rip them all out.
Wait, what is a 'routing table explosion'?
At the core of the internet is a global 'routing table' – a constantly updated map showing how data travels between networks.
On Monday, three major networks (China Telecom, Hurricane Electric, and Saudi Telecom) suddenly added around 150,000 new routes to that map.
The result? A routing table 'explosion'.
Older routers – like these legacy Cisco 6500s still used in parts of Cybersmart's network – couldn't cope. These machines rely on specialised memory with strict limits. When overloaded, they didn't crash loudly; they just stopped forwarding traffic, silently dropping data.
These left parts of the internet unreachable, even though the hardware appeared 'green' and online.
It wasn't a cyberattack or a power cut. Just old infrastructure overwhelmed by a sudden global change – and management not retiring it soon enough.
The first cut, and the deepest
That decision triggered a national network reconfiguration. More than 180 switches and 65 PPE servers had to be re-patched, reconfigured and brought back online.
Fialkov described the operation as 'cutting out the cancer'.
The operation took 39 hours and 16 minutes – a truly Herculean effort.
'There are some guys who haven't slept for 40 hours now. They really showed up for us,' he said.
By Wednesday morning, most services were restored. But not without a cost – to Cybersmart's reputation and its customers' businesses.
Lessons in humility
'This has been a life lesson,' Fialkov said. 'You get too arrogant. Twenty-two years without an outage, and you start to believe your own myth.'
He admitted that the company had been sitting on a known problem: ageing infrastructure, flagged for replacement years ago.
'We'd been looking at the same thing for four years. Working perfectly. Until it didn't. Took us out at the knees.'
Still, he insists the issue wasn't a lack of contingency. 'We've got spares. We've got redundancy. This was human complacency. We left something broken in the network for too long.'
Heartfelt apology to the 'R399'
Interestingly, he says the customers hardest hit weren't the big corporates – it was the small businesses.
'The R399 customer? That's the guy who might be running his whole business off one link,' Fialkov said. 'An outage like this could be the end of him.'
He told Daily Maverick how he spent 11 hours on a support call with one such customer, trying to assure them their business would survive.
That sobering reality drove home what Cybersmart had become. 'Ten years ago, no one would've noticed if we went down. Now? The whole country feels it.'
Where to now?
There are still issues being resolved, and some customers are wrongly blaming Cybersmart for unrelated faults. But for the most part, the network is back.
Fialkov's candour in this moment of failure is unusual in South Africa's telecoms industry – and maybe even refreshing.
'You owe your customers a service,' he said. 'And if you can't deliver it, you must be called out on that.'
And then, just like that, the fibre was (mostly) back – but the scar remains. DM
