Australian network operator Telstra has detailed the findings of an external investigation into its July 8 network outage.. The investigation, which was conducted by Technology Audit Partners (TAP), found that the carrier’s outage was caused by a network timing node issue on its network.. – Telstra.
The outage impacted thousands of the carrier’s mobile customers, plus the country’s Triple Zero service, which connects emergency calls, while the outage also led to train services being canceled.. Indeed, the investigation confirmed what was revealed at the time, when it was noted by Telstra that the outage was due to defects related to time-keeping servers at data centers in Sydney and Melbourne.. “The investigation confirmed the outage was triggered by a specific technical event, consistent with what we said at the time: incorrect date information propagated through parts of our mobile network following planned maintenance on the network timing system,” said Vicki Brady, CEO, Telstra.. “Most significantly, TAP found the outage was primarily a result of us not treating network timing as a critical capability within the network (or a ‘sovereign function’) requiring the highest levels of oversight and protection.”.
Brady apologized to the company’s customers, noting that the carrier had let them down, and explained that the carrier has taken “immediate action” to strengthen the resilience of its network.. “We have migrated services away from the previous network time protocol (NTP) servers to our strategic system across all three sites,” added Brady, who said Telstra has added additional monitoring and alarm capabilities in its network.. She said Telstra has also introduced additional testing of network changes in lab environments, and worked closely with our vendors to “continue uplifting our operational and change processes.”.
What went wrong?. Telstra said at the time that the outage started at around 4:30 am, though TAP’s investigation found it had actually started earlier, around 2:50 am.. Providing details as to why the network failed, TAP explained that it was caused at the completion of a planned NTP (Network Time Protocol) timing chassis change at 2:50 am when the chassis was powered up and back online..
According to TAP, this chassis was located in Melbourne, was part of the mobile timing architecture, and was designed to provide Stratum-2 and Stratum-3 NTP services.. However, when this server went back online, the GPS card in the chassis had reset during the change and began sending the incorrect date, 2006, to the network.. “The GPS card was actively used in that chassis and was missing a critical firmware update that was required to prevent this shift in time due to a known GPS rollover that was published in bulletins by the supplier.
While the planned change of the NTP server chassis was the trigger point to begin the outage, the steps leading to that point go back many years, as far back as 2020,” noted TAP.. The upgrades in 2020 would ultimately lead to an outage some six years, as engineers introduced a change to the NTP configuration that switched it a to “peering” configuration.
Previously, the servers were limited to seeking time from the twinned Stratum-2 servers.. To make the situation more difficult for Telstra, TAP’s report found that during the timing of the outage, the two engineers who had worked on the planned change on the chassis were on mandatory leave..
The issue may have been avoided after warning signs emerged last October, when personnel were alerted to an incident report that the NTP Stratum-3 clock was losing connectivity multiple times per day to its Stratum-2 source clock at Sydney, resulting in NTP alarms.. “Overall, there is not sufficient knowledgeable staff to do all the necessary work and peer review for NTP to fulfill all product ownership and support responsibilities. Our recommendation is to review the NTP staffing to ensure it is sufficient and bolster capability depth for NTP and timing overall,” was the recommendation from TAP.. “There was a failure to identify and manage this specific NTP risk despite evidence and concerns that were noted around the organization.
Failure to observe the pattern of risk associated with lost or unreliable sync issues and then act on them proactively was a contributor to the conditions which allowed the outage to occur, given that the GPS workaround was applied to address the stability issues.”. Brady and Telstra said the company accepts TAP’s review into what went wrong, and said it will complete a review of critical non-timing-related functions across our network to ensure the right level of priority has been applied..
A further investigation is still being carried out by the Australian Communications and Media Authority, which could see Telstra fined up to AU$30 million ($21.6m).. More in Carrier Networks. 02 Apr 2026.
07 Jul 2026. 13 Mar 2026. More in Outages.
24 Apr 2026. 15 Jul 2026. 24 Jul 2026
