Latest news with #Appen
Forbes
08-07-2025
- Business
- Forbes
Why Human Evaluation Matters When Choosing The Right AI Model For Your Business
Ryan Kolln, CEO and Managing Director of Appen. As enterprises increasingly integrate AI across their operations, the stakes for selecting the right model have never been higher and many technology leaders lean heavily on standard industry benchmarks to guide their decisions. While these metrics are useful for early filtering, they don't tell the whole story. A model's leaderboard rank doesn't guarantee it will meet business needs. What's often missing is human evaluation—and, in many cases, customized, enterprise-specific benchmarks that reflect real-world usage and deployment requirements. In today's AI landscape, human insight is a necessary complement to automated benchmarking—essential not in isolation, but as part of a structured evaluation strategy. The Limits Of Standard Benchmarking Standard benchmarks—like MMLU, Humanity's Last Exam and MMMU—were designed to measure general model performance in controlled settings. When combined with metrics like F1 score (classification accuracy), BLEU (for translation tasks) or perplexity (for language models), standard benchmarks are useful for comparing general model performance in lab settings. But these benchmarks have limits. The complexity and diversity of business AI use cases are rapidly outgrowing the information reflected in standard benchmarking. As models approach saturation—where many achieve near-max scores—the value of standard benchmarks further diminishes. Standard benchmarking doesn't account for: • Context And Nuance: A model can perform well on a math Olympiad dataset and still fail to retrieve relevant insights from an enterprise knowledge base. • Alignment With Company Values: Standard benchmarks don't measure brand voice, regulatory compliance or cultural appropriateness. • Usability And Robustness: Metrics typically don't capture how users experience outputs—or how models perform under ambiguous or adversarial inputs. High scores on public leaderboards don't guarantee business success. Standard benchmarks are most valuable for filtering potential candidates in initial model selection; however, these metrics should be complemented by human evaluation to select the best model for your unique use cases. The Role Of Human Evaluation Human evaluation fills the gaps left by automated benchmarking. Through structured assessments, human reviewers—especially domain experts—can judge model outputs on critical dimensions that standard tests miss. Developing custom benchmarks tailored to your business' unique requirements can further enhance the accuracy of your model evaluation process. • Coherence: Are outputs logical, complete and contextually appropriate? • Bias And Fairness: Does the model treat different demographics equitably? • Task suitability: Can the model handle the complexity of business-specific tasks? Common human evaluation approaches include side-by-side comparisons (ranking two model outputs), rating scales for specific quality metrics (such as helpfulness or accuracy) and real-world task testing, where models are evaluated on actual business workflows. By embedding human judgment into model evaluation—and aligning it with custom benchmarks—companies gain a richer, more practical understanding of how an AI system will perform after deployment. Practical Approaches To Human Evaluation For organizations looking to implement human evaluation efficiently, several best practices can help: • Design structured review processes. Use standardized rubrics to assess outputs across key dimensions like accuracy, safety and tone. • Involve domain experts. Engage reviewers who understand your industry-specific language, compliance requirements and customer expectations. • Adopt hybrid evaluation models. Combine quantitative benchmark filtering with qualitative human review to balance scalability and depth. • Prioritize real-world tasks. Build custom test sets that mirror the scenarios your users will encounter, rather than relying solely on abstract prompts. • Leverage evaluation platforms. Deploy tooling that supports A/B testing, red teaming and rubric-based scoring to scale human evaluation across models. For example, a healthcare company evaluating AI for medical documentation might prioritize output accuracy, sensitivity to patient data privacy and alignment with clinical terminology—factors best judged by humans, not benchmarks alone. When Human Evaluation Is Mission-Critical Human evaluation is particularly vital in high-risk, high-compliance scenarios such as: • Financial decision support • Legal summarization • Customer service in regulated industries • Healthcare documentation These are domains where even subtle model failures can trigger outsized operational, financial, legal or reputational risks. Rethinking AI Evaluation In an environment where AI models are powerful but complex, human evaluation is no longer optional—it's essential. Business leaders must recognize that while public benchmarks help narrow model options, they are not definitive answers. A robust model selection strategy, complemented with human evaluation and enterprise-specific benchmarks, ensures that AI models meet business needs, align with brand and regulatory standards and deliver sustainable value. As AI adoption deepens, companies that integrate human-centred evaluation into their selection and monitoring processes will be better equipped to unlock AI's full potential while mitigating risks others may overlook. When choosing the right AI model for your enterprise, don't just ask how well it scores. Ask how well it works for your people, your customers and your mission. Human insight is the bridge between technical promise and real-world performance. Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
Yahoo
30-06-2025
- Business
- Yahoo
Scale AI's rivals say they're going hard to win its contractors and clients: 'Our servers are melting'
Scale AI's competitors say they are seeing an uptick in client inquiries after Meta's investment. AI training companies like Appen and Prolific are pitching themselves as neutral platforms. Rival companies also told BI they have seen a rise in interest from contractors on their platforms. Meta spent $14.3 billion to acquire nearly half of Scale AI and level up in the AI race — but the startup's rivals spy an opportunity as well. Executives at five Scale AI competitors told Business Insider that they have seen a big uptick in client inquiries and job interest since the Meta deal was announced on June 13. Meta now holds a 49% stake in a company handling AI training data for many of its competitors, like Google, OpenAI, and xAI. In response, those three companies paused at least some of their work with Scale AI. Independence from Big Tech has now become a core part of the pitch for rival AI training companies vying for those contracts. In a blog post following the Meta deal, Scale AI reassured clients that it remains a "neutral, independent partner." Ryan Kolln, the CEO of data annotation firm Appen, told BI the deal would "create a pretty big disruption to our industry and create huge opportunities for Appen and our peers to fill the hole that's going to be left by Scale." "The added pitch is, 'hey, we are a publicly listed company and we're really focused on data neutrality,'" added Kolln, whose company counts Amazon and Nvidia as clients. "Our customers are really evaluating their vendor ecosystem." UK-based Prolific, which provides vetted freelancers for academic and commercial AI research, is also using neutrality as a selling point, its CEO, Phelim Bradley, told BI. "We don't build models. We don't compete with our customers. We don't have conflicting incentives," Bradley said. He added that clients are now reluctant to go all in on a single AI training provider. Big companies often spread their work among vendors, like cloud providers. "Scale benefited a lot from their awareness and being synonymous with data labeling for Big Tech," Bradley said. "Now, it's a much easier question to answer: 'How are you different from Scale?'" A Scale AI spokesperson told BI that "nothing has changed" about its customer data protection. "A lot of this confusion is being driven by smaller competitors who seek to gain from promoting false claims," they added. "Security and customer trust have always been core to our business, and we will continue to ensure the right protections are in place to help safeguard all of our work with customers." Meta did not respond to a request for comment. Jonathan Siddharth, the CEO of Turing, which trains models for major AI labs including Meta, Anthropic, and Google, said that discussions with customers have increased tenfold as frontier labs realize they need "top talent and impartial partners." "Labs increasingly want a Switzerland-like collaborator — someone model-agnostic — who can help them win the AGI race, rather than being tied to a single player," he said, referring to artificial general intelligence. He added that data annotation companies are often working on the exact capabilities that differentiate one AI model from another. None of the Scale AI rivals that BI spoke with quantified the number of inquiries they have received from Big Tech companies since Meta's investment. Scale AI's competitors are also moving to pick up its freelance workers, some of whom have had projects they are working on paused after clients like Google halted them. Scale has at least 240,00 gig workers globally who conduct AI training projects, such as flagging harmful chatbot responses. After some of Scale AI's projects were paused, the market became flooded with freelancers looking for work. Sapien AI CEO Rowan Stone told BI that his company had 40,000 new annotators join within 48 hours of Meta's Scale AI deal. "Our servers are currently melting," Stone said last week. "Our engineering team spent the entire weekend bolstering load balancers, spinning up new infrastructure, and getting us ready for the load that we're seeing." Many of these new sign-ups were from India and the Philippines — regions where Scale AI had long been a leader, Stone added. "The change in user signup pattern coincides pretty neatly with the Scale news," he said. Mercor AI's head of product, Osvald Nitski, said that the startup has received applications from full-time Scale employees, adding, "Our hiring bar is extremely high — we're only taking the best people." Mercor says it works with six of the "Magnificent Seven" tech companies and is picking up projects from clients leaving Scale. In terms of contractors, Nitski said the company is focused on recruiting elite-level annotators, like International Math Olympiad medalists, Rhodes Scholars, and Ph.D. students. Nitski said it's been a busy two weeks at Mercor, as it has seen a sharp increase in inbound interest from major tech clients. "There was simply no time for podcasts and blog posts these past few weeks with all of the demand to be fulfilled," Nitski said. Read the original article on Business Insider Sign in to access your portfolio
Yahoo
24-06-2025
- Business
- Yahoo
Appen Limited's (ASX:APX) top owners are retail investors with 58% stake, while 29% is held by institutions
Significant control over Appen by retail investors implies that the general public has more power to influence management and governance-related decisions The top 25 shareholders own 42% of the company 29% of Appen is held by Institutions AI is about to change healthcare. These 20 stocks are working on everything from early diagnostics to drug discovery. The best part - they are all under $10bn in marketcap - there is still time to get in early. Every investor in Appen Limited (ASX:APX) should be aware of the most powerful shareholder groups. With 58% stake, retail investors possess the maximum shares in the company. In other words, the group stands to gain the most (or lose the most) from their investment into the company. And institutions on the other hand have a 29% ownership in the company. Generally speaking, as a company grows, institutions will increase their ownership. Conversely, insiders often decrease their ownership over time. Let's take a closer look to see what the different types of shareholders can tell us about Appen. See our latest analysis for Appen Institutions typically measure themselves against a benchmark when reporting to their own investors, so they often become more enthusiastic about a stock once it's included in a major index. We would expect most companies to have some institutions on the register, especially if they are growing. Appen already has institutions on the share registry. Indeed, they own a respectable stake in the company. This implies the analysts working for those institutions have looked at the stock and they like it. But just like anyone else, they could be wrong. If multiple institutions change their view on a stock at the same time, you could see the share price drop fast. It's therefore worth looking at Appen's earnings history below. Of course, the future is what really matters. Hedge funds don't have many shares in Appen. Looking at our data, we can see that the largest shareholder is UBS Asset Management AG with 4.8% of shares outstanding. For context, the second largest shareholder holds about 4.3% of the shares outstanding, followed by an ownership of 4.1% by the third-largest shareholder. On studying our ownership data, we found that 25 of the top shareholders collectively own less than 50% of the share register, implying that no single individual has a majority interest. While it makes sense to study institutional ownership data for a company, it also makes sense to study analyst sentiments to know which way the wind is blowing. There are plenty of analysts covering the stock, so it might be worth seeing what they are forecasting, too. The definition of an insider can differ slightly between different countries, but members of the board of directors always count. Company management run the business, but the CEO will answer to the board, even if he or she is a member of it. Most consider insider ownership a positive because it can indicate the board is well aligned with other shareholders. However, on some occasions too much power is concentrated within this group. We can see that insiders own shares in Appen Limited. In their own names, insiders own AU$28m worth of stock in the AU$283m company. This shows at least some alignment, but we usually like to see larger insider holdings. You can click here to see if those insiders have been buying or selling. The general public -- including retail investors -- own 58% of Appen. With this amount of ownership, retail investors can collectively play a role in decisions that affect shareholder returns, such as dividend policies and the appointment of directors. They can also exercise the power to vote on acquisitions or mergers that may not improve profitability. Our data indicates that Private Companies hold 3.5%, of the company's shares. It might be worth looking deeper into this. If related parties, such as insiders, have an interest in one of these private companies, that should be disclosed in the annual report. Private companies may also have a strategic interest in the company. It's always worth thinking about the different groups who own shares in a company. But to understand Appen better, we need to consider many other factors. For instance, we've identified 2 warning signs for Appen that you should be aware of. If you are like me, you may want to think about whether this company will grow or shrink. Luckily, you can check this free report showing analyst forecasts for its future. NB: Figures in this article are calculated using data from the last twelve months, which refer to the 12-month period ending on the last date of the month the financial statement is dated. This may not be consistent with full year annual report figures. Have feedback on this article? Concerned about the content? Get in touch with us directly. Alternatively, email editorial-team (at) article by Simply Wall St is general in nature. We provide commentary based on historical data and analyst forecasts only using an unbiased methodology and our articles are not intended to be financial advice. It does not constitute a recommendation to buy or sell any stock, and does not take account of your objectives, or your financial situation. We aim to bring you long-term focused analysis driven by fundamental data. Note that our analysis may not factor in the latest price-sensitive company announcements or qualitative material. Simply Wall St has no position in any stocks mentioned. Error in retrieving data Sign in to access your portfolio Error in retrieving data Error in retrieving data Error in retrieving data Error in retrieving data
Yahoo
24-06-2025
- Business
- Yahoo
Appen Limited's (ASX:APX) top owners are retail investors with 58% stake, while 29% is held by institutions
Significant control over Appen by retail investors implies that the general public has more power to influence management and governance-related decisions The top 25 shareholders own 42% of the company 29% of Appen is held by Institutions AI is about to change healthcare. These 20 stocks are working on everything from early diagnostics to drug discovery. The best part - they are all under $10bn in marketcap - there is still time to get in early. Every investor in Appen Limited (ASX:APX) should be aware of the most powerful shareholder groups. With 58% stake, retail investors possess the maximum shares in the company. In other words, the group stands to gain the most (or lose the most) from their investment into the company. And institutions on the other hand have a 29% ownership in the company. Generally speaking, as a company grows, institutions will increase their ownership. Conversely, insiders often decrease their ownership over time. Let's take a closer look to see what the different types of shareholders can tell us about Appen. See our latest analysis for Appen Institutions typically measure themselves against a benchmark when reporting to their own investors, so they often become more enthusiastic about a stock once it's included in a major index. We would expect most companies to have some institutions on the register, especially if they are growing. Appen already has institutions on the share registry. Indeed, they own a respectable stake in the company. This implies the analysts working for those institutions have looked at the stock and they like it. But just like anyone else, they could be wrong. If multiple institutions change their view on a stock at the same time, you could see the share price drop fast. It's therefore worth looking at Appen's earnings history below. Of course, the future is what really matters. Hedge funds don't have many shares in Appen. Looking at our data, we can see that the largest shareholder is UBS Asset Management AG with 4.8% of shares outstanding. For context, the second largest shareholder holds about 4.3% of the shares outstanding, followed by an ownership of 4.1% by the third-largest shareholder. On studying our ownership data, we found that 25 of the top shareholders collectively own less than 50% of the share register, implying that no single individual has a majority interest. While it makes sense to study institutional ownership data for a company, it also makes sense to study analyst sentiments to know which way the wind is blowing. There are plenty of analysts covering the stock, so it might be worth seeing what they are forecasting, too. The definition of an insider can differ slightly between different countries, but members of the board of directors always count. Company management run the business, but the CEO will answer to the board, even if he or she is a member of it. Most consider insider ownership a positive because it can indicate the board is well aligned with other shareholders. However, on some occasions too much power is concentrated within this group. We can see that insiders own shares in Appen Limited. In their own names, insiders own AU$28m worth of stock in the AU$283m company. This shows at least some alignment, but we usually like to see larger insider holdings. You can click here to see if those insiders have been buying or selling. The general public -- including retail investors -- own 58% of Appen. With this amount of ownership, retail investors can collectively play a role in decisions that affect shareholder returns, such as dividend policies and the appointment of directors. They can also exercise the power to vote on acquisitions or mergers that may not improve profitability. Our data indicates that Private Companies hold 3.5%, of the company's shares. It might be worth looking deeper into this. If related parties, such as insiders, have an interest in one of these private companies, that should be disclosed in the annual report. Private companies may also have a strategic interest in the company. It's always worth thinking about the different groups who own shares in a company. But to understand Appen better, we need to consider many other factors. For instance, we've identified 2 warning signs for Appen that you should be aware of. If you are like me, you may want to think about whether this company will grow or shrink. Luckily, you can check this free report showing analyst forecasts for its future. NB: Figures in this article are calculated using data from the last twelve months, which refer to the 12-month period ending on the last date of the month the financial statement is dated. This may not be consistent with full year annual report figures. Have feedback on this article? Concerned about the content? Get in touch with us directly. Alternatively, email editorial-team (at) article by Simply Wall St is general in nature. We provide commentary based on historical data and analyst forecasts only using an unbiased methodology and our articles are not intended to be financial advice. It does not constitute a recommendation to buy or sell any stock, and does not take account of your objectives, or your financial situation. We aim to bring you long-term focused analysis driven by fundamental data. Note that our analysis may not factor in the latest price-sensitive company announcements or qualitative material. Simply Wall St has no position in any stocks mentioned. Error in retrieving data Sign in to access your portfolio Error in retrieving data Error in retrieving data Error in retrieving data Error in retrieving data
Business Insider
19-05-2025
- Business
- Business Insider
Appen (APPEF) Receives a Buy from Canaccord Genuity
Canaccord Genuity analyst Conor O'Prey maintained a Buy rating on Appen (APPEF – Research Report) on May 16 and set a price target of A$2.35. The company's shares closed last Wednesday at $0.76. Confident Investing Starts Here: Easily unpack a company's performance with TipRanks' new KPI Data for smart investment decisions Receive undervalued, market resilient stocks straight to you inbox with TipRanks' Smart Value Newsletter According to TipRanks, O'Prey is a 3-star analyst with an average return of 2.9% and a 43.10% success rate. O'Prey covers the Technology sector, focusing on stocks such as Appen, Nextdc Limited, and Megaport. Currently, the analyst consensus on Appen is a Moderate Buy with an average price target of $1.51. Based on Appen's latest earnings release for the quarter ending December 31, the company reported a quarterly revenue of $182.84 million and a GAAP net loss of $3.41 million. In comparison, last year the company earned a revenue of $205.49 million and had a GAAP net loss of $114.59 million
