What Do Car Dealerships, Advance Auto Parts, AT&T, and Airlines Have in Common?
The Hidden Risks of Behind-the-Scenes Software
It’s become common to hear about companies getting hacked or attacked by ransomware. For instance, MGM Resorts International was hit by a ransomware attack in 2023, the Los Angeles County Superior Court in 2024, and England’s NHS suffered massive disruptions in June 2024 due to a cyberattack. These incidents highlight the vulnerability of individual organizations to cyber threats.
However, what is often surprising is the role of behind-the-scenes software that spans vast numbers of organizations yet remains largely unknown to consumers. Disruptions to these services due to bugs or cyberattacks will have a far wider reach than a single organization.
The public became aware of three of these this year.
The CDK Global Ransomware Attack
Most consumers had never heard of CDK Global until a ransomware attack on this company in June 2024 crippled the operations of roughly 15,000 auto dealerships across the U.S. and Canada. Dealers had to process orders by hand and countless car sales were delayed. CDK was unable to restore operations quickly and ended up paying a $25 million ransom.
The initial breach took place through phishing campaigns to trick employees into installing malware or to obtain their credentials. After getting into a system, the attackers moved laterally to access other systems within the network.
The company was accused of implementing cost-cutting measures likely reduced IT and cybersecurity budgets, contributing to the company's inability to recover quickly from the attack. Additionally, CDK software allegedly had not been well maintained, further compounding the problem.
The Snowflake Data Breach
In July 2024, AT&T reported a massive data breach where hundreds of millions of phone call and text message records of all its customers were illegally downloaded. The breach did not occur directly at AT&T but within the servers of Snowflake, a data warehousing and analytics company.
Other companies that used Snowflake, like Santander, Ticketmaster, and Advance Auto Parts also had data stolen. With 9,437 companies using Snowflake’s platform, this breach exposed many organizations to significant risks. Most consumers were unaware of Snowflake’s existence and its role in storing their data.
Attackers logged in to Snowflake using stolen credentials. A requirement for multifactor authentication (MFA) would have prevented this. According to the Google Cloud Security 2H 2024 Threat Horizons Report, credential-use attacks accounted for 47% of breaches in the first half of 2024.
The CrowdStrike Update Failure
On July 19, 2024, Windows computers worldwide crashed due to a buggy software update from CrowdStrike, a provider of endpoint protection software. The failure affected 8.5 million servers worldwide, spanning industries that included auto manufacturing, healthcare, aviation, broadcasting, and banking. Businesses and consumers faced significant disruptions as their operations ground to a halt, including the cancelation of more than 6,700 flights in the U.S.
The bug in CrowdStrike’s software resulted in a NULL pointer dereference, which generally causes a program to crash. In this case, however, the software was running as a driver within the operating system, thus causing the operating system to crash.
This event, reported as the largest IT outage in history, required manual intervention to repair the damage. Administrators had to reboot into safe mode and delete the problematic file or else restore the entire operating system to a previous version. This process is labor-intensive and cannot be automated, making it particularly challenging for organizations with thousands of affected computers.
Common Traits and Risks
These events highlight the dangers of monopolization. A large set of companies relying on a single provider can lead to widespread disruptions. While the impact could be worse—imagine if Amazon Web Services were to shut down—the reliance of many enterprises on the same set of service providers poses significant risks. AT&T, Ticketmaster, and thousands of auto dealers may have had robust protections, but their operations depended on services from a small set of companies most consumers had never heard of. The chain of security is only as strong as its weakest link.
Monoculture and Its Dangers
The term monoculture in computing describes a scenario where an overwhelming portion of systems run the same software. This lack of diversity makes the entire ecosystem vulnerable to the same threats. When a critical vulnerability is discovered, it can spread rapidly, affecting many systems simultaneously. This is evident in the dominance of Windows in the desktop operating system market and the duopoly of Android and iOS in the mobile operating system market.
Monocultures arise because they offer significant advantages in terms of compatibility, support, and standardization. For example, Windows was widely adopted because it was compatible with virtually every PC, had a vast support network, and could run a wide range of software. Similarly, organizations prefer using mainstream software because it simplifies hiring, training, and obtaining support.
The Role of Cost Optimization
Cost optimization efforts often lead layoffs and hiring cheaper but less skilled staff. These employees may be more prone to introducing vulnerabilities and might require more time to develop and test software, which puts pressure on the schedule and can lead to cutting corners. Cost optimization can also result in avoiding extensive testing cycles and neglecting to test backup and recovery procedures. While these measures save costs upfront, they can result in significantly higher expenses if a failure occurs and the company must scramble to fix issues post-deployment.
In 2022, CDK Global was acquired by Brookfield Business Partners, a private equity firm. Private equity investments often lead to cost-cutting measures aimed at maximizing profits. These cuts frequently target IT and cybersecurity budgets, which are seen as cost centers. However, neglecting these areas can leave companies vulnerable to attacks and unable to respond effectively to incidents. Indeed, after CDK Global went private, employee reviews complained about the company focusing on cost-cutting and short-term profitability, resulting in layoffs and a loss of institutional knowledge. The CDK cyberattack serves as a stark reminder of the potential consequences of underinvestment in IT and cybersecurity.
Mitigating the Risks
Addressing the risks associated with behind-the-scenes software requires a multifaceted approach:
1. Avoid Monoculture: While avoiding monocultures can be challenging and economically unfeasible, especially during crises, diversification in systems and services can mitigate the risk of widespread failure. This can be difficult to do and is often not the best course of action. Because choosing a non-dominant service provider often requires more personal risk on the part of the CIO or CISO and more justification for the choice. The old adage, “nobody gets fired for buying IBM” applies here (to the current industry leaders), leading to a preference for established providers. Unfortunately, this attitude allows leading providers to gain an ever larger share of the market. When possible, companies should consider using multiple vendors and technologies to reduce their dependency on a single provider, but this assumes that other vendors exist and are equally capable. Although this approach might increase complexity, it can enhance resilience.
2. Invest in IT and Cybersecurity: IT and cybersecurity investments are often viewed as expenses with no direct return. However, these investments are crucial insurance policies against potential disasters and the erosion of consumer trust. Talented IT and cybersecurity professionals are costly but can identify vulnerabilities, implement robust security measures, and respond swiftly to incidents. Furthermore, cybersecurity insurance policies are increasingly requiring organizations to demonstrate good IT practices, making these investments essential for obtaining coverage. Cybersecurity also involves managing authentication, data privacy, access control, log retention, and a host of other items.
3. Thoroughly Test Software: Comprehensive testing for bugs and vulnerabilities is essential. The adage “good, fast, or cheap—pick two” often applies to software development, with testing being the area where shortcuts are frequently taken. However, the time and skill (which translates to cost for the time and the talented engineers) invested in thorough testing can prevent catastrophic failures. Companies should prioritize both automated and manual testing processes to identify and address potential issues before software is deployed.
4. Enhance Vendor Accountability: Organizations should maintain rigorous oversight of their vendors, especially those providing critical behind-the-scenes services. This includes conducting regular security audits, requiring vendors to adhere to strict security standards, and establishing clear incident response protocols. By holding vendors accountable, companies can ensure that their partners maintain high security and reliability standards.
5. Develop and Test Incident Response Plans: Preparedness is key to mitigating the impact of software failures and cyberattacks. Companies should develop and regularly update comprehensive incident response plans that outline steps to be taken in the event of a breach or failure. These plans should include communication strategies, technical procedures for containment and recovery, and roles and responsibilities for key personnel. Although difficult and disruptive, these plans should also be tested periodically to ensure that the documented procedures function as intended, backups can be restored, and failures can be contained.
Conclusion
The events described here could have happened to any number of other companies:
Employees will get tricked by social engineering attacks. People write buggy code. Credentials get stolen. The software will contain exploitable vulnerabilities. The goal is to reduce the likelihood of these problems and the damage that they may create.
In this case, the affected companies happened to be essential to the operations of many others, so the attacks had a multiplier effect. Each of the companies could have also done better: deploying multifactor authentication, using zero-trust principles to restrict lateral movement, and testing their code more rigorously before mass deployment.
Understanding and mitigating the risks associated with behind-the-scenes software is crucial in today's interconnected world. These systems provide essential services, and their failures can have far-reaching consequences. By avoiding monoculture, investing in IT and cybersecurity, thoroughly testing software, enhancing vendor management, and developing robust incident response plans, organizations can better protect themselves and their customers from the catastrophic effects of software failures and cyberattacks.
About Sanboca Insights
At Sanboca Insights, we specialize in tackling challenges unique to the automotive industry, enhancing the journey from concept to deployment. We partner with leading automakers across the Americas, Europe, and Asia, where we deliver strategic guidance to refine technology roadmaps, accelerate development, forge partnerships, and unveil new opportunities.
Are you ready to lead the charge in redefining the automotive user experience? Contact us today to start your journey toward a more integrated and data-driven future.