Data Engineering: Transforming The Backbone Of Modern Data Solutions
Mukul Garg is the Head of Support Engineering at PubNub , which powers apps for virtual work, play, learning and health. getty
In my journey through data engineering, one of the most remarkable shifts I've witnessed occurred during the integration of real-time data pipelines for a fast-growing SaaS platform. Initially, our data team was bogged down with batch processing and delayed analytics, which severely hindered decision-making speed.
However, when we implemented a real-time data architecture using technologies like Apache Kafka and cloud-native solutions, we were able to process and analyze data on the fly, dramatically increasing our business agility. This experience solidified my belief in the transformative power of modern data engineering.
This year, data engineering will have become the backbone of every digital transformation strategy. With the growing complexity of data sources and increasing demand for real-time analytics, companies are adopting cutting-edge technologies to build robust, scalable and efficient data infrastructures.
Data engineering today plays a pivotal role in unlocking the true value of data across industries, allowing businesses to harness their data in real time, improve decision-making and create personalized experiences for customers.
Several companies are at the forefront of implementing advanced data engineering solutions that set benchmarks for the industry:
• Netflix's Real-Time Data Pipelines: Netflix utilizes a data architecture that combines both batch and stream processing methods to handle massive quantities of data. This approach balances latency, throughput and fault tolerance by using batch processing for comprehensive views and real-time stream processing for immediate data insights.
• Uber's Predictive Analytics Engine: Uber has developed a sophisticated predictive analytics engine to optimize route planning and demand forecasting. By using real-time data processing, Uber can anticipate surge pricing and provide drivers with the most efficient routes in real time.
• Shopify's Automated Data Warehouse: Shopify recently moved to an automated data warehouse powered by cloud-native solutions like dbt (data build tool). This has allowed them to integrate sales, inventory and customer data more efficiently, resulting in quicker data-driven insights and better decision-making.
• Airbnb's Data Mesh Architecture: Airbnb has embraced a data mesh approach to scale its data infrastructure, decoupling data storage and processing across multiple teams. This approach enables each team to take ownership of its own data domain while using shared infrastructure, improving data discoverability and reducing bottlenecks. Benefits Of Modern Data Engineering
Modern data engineering offers several key benefits that have become essential for businesses today:
• Real-Time Analytics: With the advent of real-time data pipelines, companies can process and analyze data as it comes in. This has allowed businesses like Uber and Netflix to offer more timely, relevant insights and optimize decision-making in real time.
• Scalability: Data engineering solutions today can scale horizontally, handling increasing volumes of data without a corresponding increase in cost. Cloud data platforms like Snowflake and Google BigQuery are prime examples of scalable solutions that allow organizations to scale operations as they grow.
• Data Democratization: The rise of self-service data tools such as dbt and Looker has democratized data access, enabling teams across organizations to leverage data without needing deep technical expertise. This leads to faster decision-making across departments.
• Cost Efficiency: Cloud-native data solutions enable companies to optimize storage and compute costs by only paying for what they use, making it easier for small and medium-sized businesses to manage their data infrastructure without heavy upfront investments. Challenges And Considerations In Data Engineering
While the benefits are clear, there are also challenges in integrating modern data engineering solutions.
• Data Quality: Ensuring that the data being ingested into the system is clean, consistent and accurate is one of the most challenging aspects of data engineering. Poor data quality can lead to incorrect insights and missed opportunities.
• Data Privacy And Compliance: As data privacy regulations like GDPR continue to evolve, organizations need to ensure that their data pipelines comply with these regulations. This requires robust data governance and regular audits to maintain compliance.
• Integration Complexity: Integrating multiple data sources, especially from legacy systems, can be complex. Data engineering teams must ensure seamless integration while maintaining the integrity of the data and ensuring minimal latency.
• Maintaining Real-Time Performance: As real-time data processing becomes more prevalent, maintaining low-latency pipelines becomes increasingly difficult. Ensuring high throughput and minimal delays, especially with large datasets, requires careful infrastructure management. Maintaining Data Integrity And Security
In an age where data is the new currency, ensuring data integrity and security has never been more important. Implementing secure access controls, encrypted data pipelines and comprehensive monitoring systems can help safeguard data from potential breaches. Companies like Shopify and Airbnb are taking proactive steps to ensure their data infrastructures are both secure and resilient, using advanced data masking and encryption techniques to protect sensitive information. Future Outlook
Based on patterns observed in the industry, here are my predictions for data engineering in the next two to three years.
• Data Fabric And Data Mesh Expansion: The concept of a data fabric, which integrates disparate data sources into a unified layer, will continue to gain traction. Combined with data mesh architecture, organizations will see greater flexibility and scalability in their data operations, enabling more efficient collaboration across departments.
• Serverless Data Platforms: Serverless computing will take on an even greater role in data engineering. Companies will increasingly shift to serverless data architectures, reducing the overhead of managing infrastructure while focusing more on the logic of data processing.
• Data Privacy By Design: As privacy concerns grow, companies will build privacy-enhancing technologies into their data pipelines from the outset, ensuring compliance with global regulations without sacrificing performance. Conclusion
In 2025, data engineering is not just about building infrastructure—it's about creating agile, scalable and secure systems that can process vast amounts of data in real time. The future of data engineering looks bright as organizations continue to innovate, leveraging modern technologies to unlock new insights, improve decision-making and drive business growth. Companies that invest in the latest data engineering solutions should be well-positioned to gain a competitive edge in an increasingly data-driven world.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Yahoo
4 hours ago
- Yahoo
Why Shopify Stock Popped Today
A Wells Fargo analyst raised their price target on Shopify's stock today. They called the company their "signature pick" and believe it could be a lesser-known beneficiary of the AI revolution. Home to its Shopify Magic and Sidekick solutions, Shopify looks more poised to thrive amid AI's rise, rather than be disrupted by it. 10 stocks we like better than Shopify › Shares of North America's leading commerce enabler, Shopify (NASDAQ: SHOP), were 6% higher as of 3:30 p.m. ET Friday, according to data provided by S&P Global Market Intelligence. This rise stems from an analyst at Wells Fargo raising their price target on the stock from $107 to $125, naming Shopify a "signature pick." Although Nvidia, Palantir, OpenAI, and others capture most of the artificial intelligence (AI) fanfare, the analyst believes Shopify could prove to be a thematic AI story -- and I'd agree. In April this year, a leaked memo from Chief Executive Officer Tobi Lütke went viral. In it, he stated, "Before asking for more headcount and resources, teams must demonstrate why they cannot get what they want done using AI." Although this focuses on incorporating AI into Shopify's operations, the quote highlights that AI will be at the heart of what the company does going forward, whether internally or through its products. In 2023, the company launched Shopify Magic, a toolbox of AI-powered solutions (think AI-generated product descriptions or email campaigns, automated chat help, or image editing). Then, it launched Sidekick -- an AI-driven commerce assistant -- in 2024 to help with areas such as inventory optimization, pricing strategies, and gathering business insights. Just last quarter, Shopify launched This tool enables merchants to source products more effectively, allowing them to navigate the complex tariff environment in real time. Adding over 600 new product features for its merchants in the last two years alone, Shopify appears likely to remain an AI innovator rather than a disruptee, in my opinion. Though Shopify stock isn't cheap at 83 times cash from operations, its growth potential remains massive, holding only a 2% market share in its core geographies. Before you buy stock in Shopify, consider this: The Motley Fool Stock Advisor analyst team just identified what they believe are the for investors to buy now… and Shopify wasn't one of them. The 10 stocks that made the cut could produce monster returns in the coming years. Consider when Netflix made this list on December 17, 2004... if you invested $1,000 at the time of our recommendation, you'd have $674,395!* Or when Nvidia made this list on April 15, 2005... if you invested $1,000 at the time of our recommendation, you'd have $858,011!* Now, it's worth noting Stock Advisor's total average return is 997% — a market-crushing outperformance compared to 172% for the S&P 500. Don't miss out on the latest top 10 list, available when you join . See the 10 stocks » *Stock Advisor returns as of June 2, 2025 Wells Fargo is an advertising partner of Motley Fool Money. Josh Kohn-Lindquist has positions in Nvidia and Shopify. The Motley Fool has positions in and recommends Nvidia, Palantir Technologies, and Shopify. The Motley Fool has a disclosure policy. Why Shopify Stock Popped Today was originally published by The Motley Fool Error in retrieving data Sign in to access your portfolio Error in retrieving data Error in retrieving data Error in retrieving data Error in retrieving data
Yahoo
8 hours ago
- Yahoo
1 Megacap Tech Stock That Could Split Its Shares Next
Netflix shares trade significantly higher than the levels they did before the company's last stock split in 2015. The company's business is firing on all cylinders, making this a good time to split shares. Netflix expects double-digit revenue growth and significant operating margin expansion this year. 10 stocks we like better than Netflix › Netflix (NASDAQ: NFLX) stock has recently blown past $1,200 per share, making it hard to believe that shares traded at levels below $200 as recently as May of 2022. And the stock's momentum is strong this year, too. Shares are up about 40% this year alone, defying the market's sluggish return of less than 2% as of this writing. With a combination of a strong business, impressive stock price momentum, and a share price in the thousands, a stock split could be in the cards for the streaming giant soon. Netflix's performance has been stellar. In the first quarter of 2025, revenue rose 12.5% year over year to about $10.5 billion, and earnings per share soared 25.2%. Helping the company achieve such strong earnings-per-share growth is Netflix's widening operating margin. The key profitability metric hit 31.7% in the quarter, up from 28.1% in the year-ago period. The company also reported free cash flow of $2.7 billion, up 25% year over year. Netflix's business growth has been fueled primarily by three key tailwinds: membership growth, price increases, and a fast-growing advertising business. Importantly, the company believes all three of these catalysts have room to run. In its first-quarter update, management reaffirmed its guidance for full-year revenue to increase 11.5% to 14.1% year over year. This growth, management explained, "assumes healthy member growth, higher subscription pricing and a rough doubling of our ad revenue ... " Additionally, management continues to forecast a full-year operating margin of 29%, up substantially from 26.7% in 2024. Netflix hasn't split its shares since 2015. Back then, a 7-for-1 split lowered the stock price from about $700 to $100. Today, the share price is nearly double its pre-split peak. That alone doesn't guarantee a stock split. But historically, splits are more likely when a stock becomes expensive (in terms of the share price) relative to other megacaps and the company is on solid footing. Netflix checks both boxes. There's a sense of déjà vu with Netflix today. Just as has been the case recently for the company, it was experiencing strong subscriber growth, record earnings, and benefiting from strategic catalysts the last time it split its stock. Also strengthening the case for a stock split, Netflix shares currently trade far higher than other tech leaders like Microsoft, Meta Platforms, Apple, and Nvidia. Of course, a stock split would not affect the company's fundamentals, but it would lower the price per share and make Netflix more accessible to retail investors. But it's worth emphasizing that a stock split, in and of itself, isn't a reason to buy a stock. It is, however, often a symptom of strong underlying business momentum -- momentum strong enough to cause investors to bid up the share price to a level worthy of a stock split. It's also worth noting that even though Netflix's business is doing extraordinarily well, investors seem to already be pricing in this momentum. Shares trade at 59 times earnings. All else equal, this valuation multiple will likely come down meaningfully if the company delivers on its revenue growth and operating margin expansion targets for the full year. A combination of double-digit revenue growth and margin expansion should help earnings per share grow dramatically. But with a price-to-earnings multiple well in excess of even fast-growing tech giant Nvidia's, investors seem to be already betting on more staggering growth from the streaming giant. With a surging stock price, impressive revenue growth, and a nascent and fast-growing advertising business, Netflix is a top contender for the next big tech stock split. Though the company hasn't announced plans to split its shares, it's starting to look overdue. Before you buy stock in Netflix, consider this: The Motley Fool Stock Advisor analyst team just identified what they believe are the for investors to buy now… and Netflix wasn't one of them. The 10 stocks that made the cut could produce monster returns in the coming years. Consider when Netflix made this list on December 17, 2004... if you invested $1,000 at the time of our recommendation, you'd have $674,395!* Or when Nvidia made this list on April 15, 2005... if you invested $1,000 at the time of our recommendation, you'd have $858,011!* Now, it's worth noting Stock Advisor's total average return is 997% — a market-crushing outperformance compared to 172% for the S&P 500. Don't miss out on the latest top 10 list, available when you join . See the 10 stocks » *Stock Advisor returns as of June 2, 2025 Randi Zuckerberg, a former director of market development and spokeswoman for Facebook and sister to Meta Platforms CEO Mark Zuckerberg, is a member of The Motley Fool's board of directors. Daniel Sparks and his clients have no position in any of the stocks mentioned. The Motley Fool has positions in and recommends Apple, Meta Platforms, Microsoft, Netflix, and Nvidia. The Motley Fool recommends the following options: long January 2026 $395 calls on Microsoft and short January 2026 $405 calls on Microsoft. The Motley Fool has a disclosure policy. 1 Megacap Tech Stock That Could Split Its Shares Next was originally published by The Motley Fool Error in retrieving data Sign in to access your portfolio Error in retrieving data Error in retrieving data Error in retrieving data Error in retrieving data
Yahoo
8 hours ago
- Yahoo
Best Stock to Buy Right Now: Uber vs. Carvana
The world's mobility landscape is changing in several different ways. There may be room and reason to own both of these companies right now. One of these stocks, however, is experiencing oversized growth for reasons that aren't apt to persist. 10 stocks we like better than Uber Technologies › Uber Technologies (NYSE: UBER) and Carvana (NYSE: CVNA) aren't just two different companies. They're seemingly polar opposites. Ride-hailing giant Uber is thriving largely because owning and driving a car is an increasingly expensive hassle. Used car dealer Carvana, conversely, makes it easy and affordable to own your own automobile. One could reasonably argue there's solid -- even growing -- demand for both businesses. But it's difficult to deny the contrasted underpinnings of what make these two companies tick. Investors could understandably be confused. The good news is, one of these names is a clearly better bet than the other -- now and for the foreseeable future. You almost certainly know the companies. Uber Technologies isn't just a major personal mobility name, after all. It largely brought the domestic ride-hailing industry into existence, and now controls three-fourths of the U.S. market, according to data from Bloomberg. It's developing an international presence as well, even if it's not quite as dominant overseas. As for its fiscals, the $175 billion company's drivers provided over 11 billion rides last year, up 18% year over year, turning that into nearly $44 billion in revenue and almost $3 billion worth of operating net income. Nearly half of its revenue, however, came from deliveries and freight services rather than passenger trips. Carvana isn't quite as big, although its growth is just as impressive. The used car dealer reported $13.7 billion worth of revenue for 2024, up 27% year over year, generating a record-breaking $404 million in net income. Most of that profit came from sales of used vehicles to retail consumers, although wholesaling accounts for the bulk of its total unit transactions. A rebound from a inflation-crimped slide in demand after the peak of the COVID-19 pandemic helped drive these top and bottom lines higher. Last year's forward progress, however, also extends a choppy trend that's been in place for some time thanks to Carvana's clever marketing, and brilliant use of technology to establish scale. And yet, these two seemingly growing companies' stocks aren't exactly performing in tandem. Carvana shares are up by more than 200% for the past year, and testing their peak reached in 2021. Uber shares haven't made any real net progress since March of last year, upended multiple times by earnings reports marred by one modest shortfall or another. The market has largely lost its bigger-picture perspective on both companies, however. Take Uber Technologies' recent quarterly results as an example. Yes, since early 2024 either sales or earnings or guidance haven't always lived up to expectations. The company's never failed to make actual forward progress at any point during this stretch, however. Again, last year's revenue improved 18%, and is expected to repeat the feat this year. Although its sales growth rate is slowing, that's a purely mathematical matter. On an absolute basis, it's chugging along as well as it ever has. It's apt to continue doing so well into the foreseeable future, too. Uber is plugged into a serious secular trend. That is, although plenty of people still own and drive cars, there's a growing segment of the domestic and foreign population that doesn't want to do either anymore. A recent survey performed by Deloitte indicates that only 11% of people living in the United States over the age of 55 would consider giving up their vehicle, whereas 44% of people living in the U.S. under the age of 35 would consider doing so. And the disinterest is greater the younger the crowd. Thirty years ago roughly two-thirds of eligible teenagers held a driver's license. Today that figure's about one-third. The advent of a viable alternative like Uber is a key reason for this paradigm shift that's likely to remain in motion for years, as younger non-drivers grow up and pass along these new norms to their children. To this end, Straits Research believes the global ride-hailing market is set to grow at an average annual pace of more than 11% through 2033. Uber is well positioned to win more than its fair share of this growth. The food delivery industry is also set to grow at an annualized pace of 17%, by the way, according to an outlook from Precedence Research. This is another big growth opportunity for Uber, which already enjoys a strong presence within the market. But is Carvana just too promising to pass up? The bullish arguments hold water, to be sure. Chief among them is the sheer cost of a new car. Data from Kelley Blue Book indicates that as of April the average new automobile in the United States was being sold for a hefty $48,699. That puts the monthly payment well over $700, and subsequently, out of reach for most would-be buyers. Used cars are considerably more affordable though. Kelley Blue Book reports April's used car sales rolled in at an average price of $25,547 apiece, roughly halving the monthly payment as well. There's an important footnote to add to this dynamic, however. That is, the car business is incredibly cyclical. Once the current wave of updates and replacements has run its course, don't be surprised to start seeing Carvana underperform. Making matters even more challenging is the lack of new cars manufactured and sold in the U.S. between 2020 and 2022. As Edmunds points out, a three-year-old car (plus or minus a year) is Carvana's sweet spot, so to speak, where affordability and value intersect. Available inventory of these vehicles now stands at more than a decade low though, making it difficult for Carvana to offer what most consumers want. Never say never, of course. It's possible Carvana will continue to consolidate the country's highly fragmented and largely inefficient used car business, achieving growth through greater scale. There's certainly plenty of room for it. As it stands right now, Carvana estimates it only accounts for about 1% of the used car business. It's also possible Uber will run into an unexpected headwind sooner rather than later, even if it's not clear what that headwind might be. Be realistic though. Increasingly crowded urban and metropolitan streets and parking lots paired with car prices that aren't apt to abate anytime soon works in Uber Technologies' favor. Meanwhile, Carvana's recent sales strength doesn't reflect a new long-lived norm as much as it reflects the fact that the average vehicle being driven on U.S. roads is now 12.6 years, according to S&P Global Mobility, an age at which replacing it makes more sense than repairing it. The resulting demand for quality used cars isn't apt to last forever though. Now throw in the fact that Carvana's shares are currently trading 14% above analysts' consensus price while Uber's stock is 16% below analysts' (most of whom rate Uber as a strong buy at this time) average price target of $97.39, and there's little doubt that the ride-hailing powerhouse is a better stock to buy right now than the used car dealership chain's. Before you buy stock in Uber Technologies, consider this: The Motley Fool Stock Advisor analyst team just identified what they believe are the for investors to buy now… and Uber Technologies wasn't one of them. The 10 stocks that made the cut could produce monster returns in the coming years. Consider when Netflix made this list on December 17, 2004... if you invested $1,000 at the time of our recommendation, you'd have $669,517!* Or when Nvidia made this list on April 15, 2005... if you invested $1,000 at the time of our recommendation, you'd have $868,615!* Now, it's worth noting Stock Advisor's total average return is 792% — a market-crushing outperformance compared to 171% for the S&P 500. Don't miss out on the latest top 10 list, available when you join . See the 10 stocks » *Stock Advisor returns as of June 2, 2025 James Brumley has no position in any of the stocks mentioned. The Motley Fool has positions in and recommends Uber Technologies. The Motley Fool has a disclosure policy. Best Stock to Buy Right Now: Uber vs. Carvana was originally published by The Motley Fool Error in retrieving data Sign in to access your portfolio Error in retrieving data Error in retrieving data Error in retrieving data Error in retrieving data
