What took Google hours to fix a glitch that its engineers 'identified in 10 minutes'
Google has apologised for a major outage recently that impacted services running on Google Cloud across the globe recently. The widespread outage disrupted over 70 Google Cloud services, bringing down major third-party platforms like Cloudflare, OpenAI, and Shopify, as well as Google's own products, including Gmail, Google Calendar, Google Drive, and Google Meet.
In an incident report released late Friday (June 17), the company said, "Google Cloud, Google Workspace and Google Security Operations products experienced increased 503 errors in external API requests, impacting customers." Google attributed the hours-long downtime to a series of flawed updates, particularly a new 'quota policy checks' feature introduced in May. The feature, designed to evaluate automated incoming requests, was not adequately tested in real-world scenarios. This led to blank entries being sent across all Google Cloud data centers, triggering widespread system crashes.
What caused the Google Cloud outage
"On May 29, 2025, a new feature was added to Service Control for additional quota policy checks. This code change and binary release went through our region by region rollout, but the code path that failed was never exercised during this rollout due to needing a policy change that would trigger the code. As a safety precaution, this code change came with a red-button to turn off that particular policy serving path. The issue with this change was that it did not have appropriate error handling nor was it feature flag protected. Without the appropriate error handling, the null pointer caused the binary to crash. Feature flags are used to gradually enable the feature region by region per project, starting with internal projects, to enable us to catch issues. If this had been flag protected, the issue would have been caught in staging," said Google in the Incident report.
by Taboola
by Taboola
Sponsored Links
Sponsored Links
Promoted Links
Promoted Links
You May Like
Nejapa: Invierte $100 en AES CFD [Lee más]
ofertas especiales
Contáctanos
Undo
What took Google engineers hours to fix the issue and the way forward
According to the company, Google engineers identified the issue within 10 minutes, but the outage persisted for seven hours due to overloaded systems in larger regions. Google admitted it failed to use feature flags, a standard industry practice that could have caught the issue during a gradual rollout. " Within 10 minutes, the root cause was identified and the red-button (to disable the serving path) was being put in place. The red-button was ready to roll out ~25 minutes from the start of the incident. Within 40 minutes of the incident, the red-button rollout was completed, and we started seeing recovery across regions, starting with the smaller ones first. Within some of our larger regions, such as us-central-1, as Service Control tasks restarted, it created a herd effect on the underlying infrastructure it depends on (i.e. that Spanner table), overloading the infrastructure. Service Control did not have the appropriate randomized exponential backoff implemented to avoid this. It took up to ~2h 40 mins to fully resolve in us-central-1 as we throttled task creation to minimize the impact on the underlying infrastructure and routed traffic to multi-regional databases to reduce the load. At that point, Service Control and API serving was fully recovered across all regions. Corresponding Google and Google Cloud products started recovering with some taking longer depending upon their architecture," said Google in its Incident Report.
To prevent future incidents, Google said it will overhaul its system architecture to ensure operations continue even if one component fails. The company also committed to auditing its systems and improving both automated and human communication channels to keep customers informed during issues.
Google apologises for the outage
'We deeply apologize for the impact this outage has had,' Google wrote in the incident report. 'Google Cloud customers and their users trust their businesses to Google, and we will do better. We apologize for the impact this has had not only on our customers' businesses and their users but also on the trust of our systems. We are committed to making improvements to help avoid outages like this moving forward.'
CEO of Google's cloud unit Thomas Kurian, also posted about the outage in a Twitter post, saying 'we regret the disruption this caused our customers.'
AI Masterclass for Students. Upskill Young Ones Today!– Join Now
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Time of India
19 minutes ago
- Time of India
Aquarius Daily Horoscope Today, June 17, 2025: Celebrate quietly, even if alone
Today is about self-recognition. You've walked a long road, made hard choices, and learned quietly. Maybe others didn't clap, maybe no one noticed—but the stars see you. Celebrate how far you've come, even if no one else does. Don't wait for outside approval. Your growth is your own. Sit with a smile and thank yourself for surviving days you thought you couldn't. Today brings peaceful pride. Honour it in your own way. No spotlight is needed—only self-respect and quiet joy. Aquarius Love Horoscope Today In love, let appreciation begin with you. If you're in a relationship, notice how you've become more understanding, more balanced. Share that feeling with your partner, not through drama, but with calm presence. If single, don't feel you are lacking something. You've come far emotionally. Appreciate your healing, and don't settle just to feel accepted. The right person will celebrate you for your depth, not just your smile. Today, speak kindly to yourself. That energy will bring love closer. Aquarius Career Horoscope Today Career shows small signs of recognition today, even if not publicly. A senior or client may notice your consistency. Keep your focus. by Taboola by Taboola Sponsored Links Sponsored Links Promoted Links Promoted Links You May Like Free P2,000 GCash eGift UnionBank Credit Card Apply Now Undo You don't need applause to stay committed. Let your own sense of discipline be your motivation. If others are being celebrated, don't feel small. Your time is coming. Quiet workers often create lasting value. Take one more solid step forward. Your reward is in the progress you feel inside, not just in what the world sees. Aquarius Money Horoscope Today Money matters may not bring sudden gains, but your careful past actions are protecting you today. Maybe you chose not to overspend, maybe you saved quietly—today those choices matter. Give yourself credit. It's okay to feel satisfied with simple financial stability. You may also find a small way to treat yourself without guilt. Do it. Celebrate modest victories. No need to explain your spending or saving choices. Inner pride grows when you respect your own efforts. Aquarius Health Horoscope Today Health is steady but may feel slow. That's not a problem. Healing is happening, but quietly. Maybe your skin feels better, digestion is more stable, or sleep is deeper. These are wins. Celebrate them. If there's still something bothering your body, take comfort in the fact that you're listening now. Continue your slow routines—herbal drinks, warm food, light walks. Avoid comparing your energy to others. Your body is on its own timeline. Honour that. Healing doesn't need to be loud. Discover everything about astrology at the Times of India , including daily horoscopes for Aries , Taurus , Gemini , Cancer , Leo , Virgo , Libra , Scorpio , Sagittarius , Capricorn , Aquarius , and Pisces .
Mint
21 minutes ago
- Mint
Scale AI's Wang Brings to Meta Knowledge of What Everyone Else is Doing
(Bloomberg) -- Scale AI co-founder Alexandr Wang is such an enthusiastic networker that his former roommate — OpenAI Chief Executive Officer Sam Altman — once jokingly told him to tone it down a bit. Fortunately for Wang, he ignored the advice. Last week, the 28-year-old parlayed his ability to cultivate influential relationships into a $14.3 billion investment from Meta in Scale, and a new job for himself in Meta's 'superintelligence' group, reporting to CEO Mark Zuckerberg. His own cash and equity in the deal is worth more than $5 billion, according to the Bloomberg Billionaires Index. What Meta is buying: the one guy who knows what everyone else in the AI industry is doing. In recent months, Wang has become close with Zuckerberg and has spent time at the Meta CEO's houses in Tahoe and Palo Alto, California, discussing the future of AI, according to people familiar with the matter. Zuckerberg developed a strong admiration for him, the people said. Wang is also close with Altman, who before OpenAI used to run the elite startup accelerator Y Combinator. Scale went through YC in 2016, and the two men lived together for a period during the Covid-19 pandemic, according to other people who know the two men. At first glance, the Wang hiring looks a lot like the other expensive not-quite-acquisitions by Meta's big tech competitors. Microsoft, Amazon and Alphabet have also thrown gargantuan sums at AI researchers with advanced degrees and experience building cutting-edge models, sometimes by inking licensing deals with their companies. But Wang stands out because he's not an academic nor is he running a prominent AI model maker — he's just really good at knowing what's going on in the industry. His company, Scale, deals with the grunt work of human-powered data collection that's needed to build machine learning models, and most of the important companies have been his customers. With the Scale investment, Zuckerberg may gain leverage over some of his biggest AI rivals. Either those companies, including OpenAI and Google, continue to use Scale and help bolster a business Zuckerberg now owns 49% of, or they're forced to go find an alternative to Scale, which may slow their progress. (OpenAI intends to stick with Scale, while Google is reportedly planning to cut ties.) It's likely Wang also has valuable insights into Zuckerberg's competitors from his time partnering with them the last several network and reach extends beyond those firms, too. Since he co-founded Scale at 19, Wang has befriended other founders of billion-dollar startups such as Plaid, Figma and Brex. Wang has also curried favor with right-wing influencers and Republican lawmakers in Washington, in part by hammering concerns about China. In the process, Scale has racked up increasingly large government contracts. Going forward, those ties could also benefit Meta as it works to break into defense tech. Wang reaches out directly to keep up relationships with a dizzying number of people, in both junior and senior roles, and knows what they're working on and what they want to be doing, according to a close contact, who declined to be named speaking publicly about Wang. 'He knows everyone' in the AI world, that person said. Even Altman, an accomplished networker in his own right, teased Wang last year about his incessant need to be socializing. 'This is authoritative, truly no one flies in and out to more parties than you,' Altman tweeted at him. 'It looks like a full time job.' When Wang was born, in 1997, his Chinese immigrant parents named him Alexandr — missing the final e — because they wanted the numerologically lucky benefits of an eight-letter name, he once explained on a podcast. The number 8 is associated with wealth and prosperity in Chinese culture. Wang's parents were both physicists who worked at the Los Alamos National Lab in New Mexico; Wang inherited their braininess. He won math competitions in grade school, became a nationally ranked mathlete, and studied the violin. 'My sport was math,' he later recounted on a podcast. He finished high school early and enrolled at Massachusetts Institute of Technology but itched to start his own company. Even though his parents and two older brothers all have doctorates, he dropped out of college to move to San Francisco. 'I always wanted to be doing more, or wanted to be accomplishing more,' he said in a video for Forbes. He reassured his parents he was going to go back — 'a little white lie,' he said. In 2016, he teamed up with Lucy Guo, a Thiel Fellow, which means the billionaire Peter Thiel financially backed her choice to quit college because of her brilliance. The pair applied to YC with the idea for a doctor-booking app. During the 10-minute interview for YC, Wang came across as having a mix of potential and ego, YC partner Jessica Livingston recalled on the accelerator's podcast The Social Radars. 'May be arrogant or brilliant — I think the former,' she wrote in her notes at the time. 'But worth funding.' Other YC partners described the founder as 'high beta,' using an investing term to mean volatility. 'Describing a founder as 'high beta' means they're either going to flame out or take over the world,' YC founder Paul Graham posted about Wang. Within a few weeks of starting at YC, Wang and Guo realized the doctor-appointment idea was too steep a struggle and decided to pivot to a new idea. Wang had been inspired by recent advancements in machine learning; months earlier, the world's best human Go player had lost to a Go-playing AI. He and Guo decided to sell data-labeling services to other companies who needed to amass training data to feed into AI projects. Scale initially called itself an 'Uber for AI' because it offered contractors around the world the chance to make money by labeling data — for example, looking at imagery from self-driving cars and labeling obstacles, vehicles, signs and pedestrians. The startup quickly raised from top-tier investors. But two years after founding, the co-founders' relationship turned sour. After a dispute, Wang fired Guo and continued running the startup himself. As AI models continued to advance at a dramatic pace, Scale positioned itself as a key provider of the underlying data infrastructure. It was less flashy than building frontier models, but it was a lucrative business selling the work that still needed to be done by humans to the people building models that would replace them. Scale is not the only provider of data-labeling services, but Wang's star power, network and fundraising prowess helped make it the best known in the field. In the process, Wang and Scale also emerged as lightning rods for criticisms about the unseen workforce behind today's AI services. As Wang and Guo were on their way toward becoming billionaires, Scale worked with thousands of contractors in countries like Kenya and the Philippines who were paid relatively little to weed through reams of online data, with some saying they have suffered psychological trauma from the content they're asked to review. In a 2019 interview with Bloomberg, Wang said the company's contract workers earn 'good' pay — 'in the 60th to 70th percentile of wages in their geography.' Wang also pursued government contracts and impressed lawmakers with youth and confidence. In meetings in Washington, Wang came across as 'authoritative and direct' and made a concerted effort to meet with the Biden administration, said one former Biden staffer. Wang 'almost enjoyed' being underestimated because of his age, said a former Scale employee who accompanied him on trips to the capital. In 2020, the company landed a $90 million contract with the Defense Department; in 2022, that expanded to a deal to make its technology available to all federal agencies, under a $250 million purchase agreement. The outlet Semafor soon dubbed him 'Washington's AI Whisperer.' As Scale grew, so did Wang's wealth and profile. By 2022, at age 25, Forbes was calling him the youngest self-made billionaire. Scale threw posh private gatherings for the who's-who of AI, like a 2023 conference in the Utah mountains where, in between archery sessions and elevated dining, attendees discussed the threat of China's advancements in AI. Wang attended the Met Gala and frequently posted selfies hanging out with the Mad Men actress Kiernan Shipka. He spoke at the World Economic Forum in Davos, Switzerland, and was included on Time's 2024 list of the 100 most influential people. (Altman wrote the short blurb about Wang for the magazine's list, calling him a 'longtime friend.') Guo, Wang's ousted co-founder, is no longer involved in Scale, but her shares in the company also led Forbes to label her a self-made billionaire this year. She bought a luxury condo in Miami and threw a party with a rented lemur and snake, which provoked ire from her wealthy neighbors and prompted the New York Post to deem her 'Miami's No. 1 party girl.' In the past year, Wang has made overtures toward right-wing personalities and values. In June 2024, he announced that Scale's hiring policy would be based on 'MEI' — 'merit, excellence and intelligence,' a rebuttal to the diversity, equity and inclusion policies that were widely adopted by tech companies in the 2010s and have since been rejected and derided by many conservatives. In January, Wang attended Trump's inauguration and palled around with right-wing influencers such as the Paul brothers and the Nelk Boys. In one viral clip from the inauguration, comedian Theo Von falls backward and crashes into Logan Paul when his folding chair breaks; Wang and Altman are sitting a few feet away. After Wang met Von at the inauguration, Von had him as a guest on his popular podcast, one of several prominent shows that helped propel Trump to the White House. Wang, in a camo-print trucker hat with the word 'GODSPEED' embroidered across the front, explained to Von the basics of AI and pressed the point that the US risks losing its lead to China. 'Dude, you should be a superhero,' said Von, with awe in his voice. Wang replied, 'Math goes a long way,' to which Von added, 'Oh, hell yeah, dude — divide these nuts.' Wang has taken warnings about China's AI prowess to more serious forums, too. 'If we as Americans want to defend our way of life, our ideologies, and democracy broadly, it's important for us to stay ahead,' Wang told Bloomberg this winter at Davos. In January, Scale AI took out a full-page advertisement in the Washington Post stating: 'America must win the AI war.' That message has resonated in Washington as DeepSeek and other Chinese AI startups have released increasingly competitive AI models. Wang's recent moves echo those of Zuckerberg, who has also been cozying up to Trump. Meta rolled back its diversity efforts and weakened its hate-speech policies. They are starting to act alike in other ways; both Wang and Zuckerberg now sport curly mops of hair and favor streetwear fashion while appearing on right-leaning podcasts (Zuckerberg has been a guest with both Joe Rogan and Von). The two CEOs share the rare distinction of having both been, at one time or another, the world's youngest self-made billionaire. Now, one of them will be the other's boss. --With assistance from Jackie Davalos, Kurt Wagner, Riley Griffin, Rachel Metz and Shirin Ghaffary. More stories like this are available on
Time of India
25 minutes ago
- Time of India
No new tariffs after Bilateral Trade Agreement: India wants US assurance
India wants an assurance from the US that no additional tariffs will be imposed by the Donald Trump administration once the bilateral trade agreement (BTA) is finalised, said people familiar with the matter. The deal is in the final stages of negotiation and both sides are hopeful of reaching accord quickly. India has sought concessions for labour-intensive sectors such as leather and textiles. 'We have put everything on the table,' said one of the persons, adding that India wants stability in tariffs once a deal is rolled out. Trade agreements typically include renegotiation clauses, or compensation from the partner effecting a tariff increase. India would like the pact to provide for such a mechanism. 'This would ensure the agreement is ring-fenced from prospective changes,' said another person. The US President on April 2 announced a reciprocal tariff of 26% on goods imports from India as part of its trade levies all over the world. These tariffs were paused for 90 days until July 9. However, the baseline tariff of 10% remains in force. India and the US are looking to finalise the BTA ahead of the deadline. by Taboola by Taboola Sponsored Links Sponsored Links Promoted Links Promoted Links You May Like Elegant New Scooters For Seniors In 2024: The Prices May Surprise You Mobility Scooter | Search Ads Learn More Undo Timebound BTA Needed An assurance from Washington is required to ensure tariffs negotiated within the framework of the trade deal remain ring-fenced from any changes effected later by the US, said a third official detailing India's concern. There is also Trump's mercurial nature. Live Events Independent experts are agreed on the need for such a clause. 'The BTA should be timebound and not in perpetuity,' said an expert on trade issues. 'India must insist on a clawback provision — that it will withdraw benefits if the US raises tariffs or goes back on any of its commitments.' Negotiations Officials said Washington has to find a way to reduce tariffs as New Delhi has sought concessions for sectors such as textiles and leather, as mentioned above. The Trump administration currently requires approval from the US Congress to lower levy below most favoured nation (MFN) rates, but it does have the authority to scrap reciprocal tariffs. 'We want preferential and sustained trade advantages and have left it to the US to decide the route it takes to reduce its tariff barriers—whether through the Trade Promotion Authority or by seeking Congressional approval,' said one of the officials cited earlier. The government is also monitoring the impact of the 50% tariffs on steel and aluminium and the 25% tariffs on auto. While the auto components sector doesn't expect a big dent for now, there may be an impact if the tariffs persist. According to the official quoted, India will pursue a concession, if others get one. Early Harvest Both sides have had several rounds of discussions on the proposed trade deal and are hopeful of concluding at least an early harvest deal. 'We are working on the early tranche, and there is a date (July 9), before which we would like to conclude this. The progress is good,' commerce secretary Sunil Barthwal said on Monday. A US trade team is in India last week to firm up the contours. India has made it clear that any meaningful expansion in bilateral trade will require a significant reduction in US tariffs. Both sides aim to more than double such trade to $500 billion by 2030. However, a Delhi-based trade expert cautioned, 'India should rethink its proposed BTA with the US. The US has a history of incomplete trade deals, such as the Trans-Pacific Partnership and the Transatlantic Trade and Investment Partnership. It has retracted its WTO commitments also. Moreover, the future of the Indo-Pacific Economic Framework for Prosperity hangs in the balance with the US-China trade deal.'
