OpenAI launches program to design new 'domain-specific' AI benchmarks
OpenAI, like many AI labs, thinks AI benchmarks are broken. It says it wants to fix them through a new program.
Called the OpenAI Pioneers Program, the program will focus on creating evaluations for AI models that "set the bar for what good looks like," as OpenAI phrased it in a blog post.
"As the pace of AI adoption accelerates across industries, there is a need to understand and improve its impact in the world," the company continued in its post. "Creating domain-specific evals are one way to better reflect real-world use cases, helping teams assess model performance in practical, high-stakes environments."
As the recent controversy with the crowdsourced benchmark LM Arena and Meta's Maverick model illustrate, it's tough to know, these days, precisely what differentiates one model from another. Many widely-used AI benchmarks measure performance on esoteric tasks, like solving doctorate-level math problems. Others can be gamed, or don't align well with most people's preferences.
Through the Pioneers Program, OpenAI hopes to create benchmarks for specific domains like legal, finance, insurance, healthcare, and accounting. The lab says that, in the coming months, it'll work with "multiple companies" to design tailored benchmarks and eventually share those benchmarks publicly, along with "industry-specific" evaluations.
"The first cohort will focus on startups who will help lay the foundations of the OpenAI Pioneers Program," OpenAI wrote in the blog post. "We're selecting a handful of startups for this initial cohort, each working on high-value, applied use cases where AI can drive real-world impact."
Companies in the program will also have the opportunity to work with OpenAI's team to create model improvements via reinforcement fine tuning, a technique that optimizes models for a narrow set of tasks, OpenAI says.
The big question is whether the AI community will embrace benchmarks whose creation was funded by OpenAI. OpenAI has supported benchmarking efforts financially before, and designed its own evaluations. But partnering with customers to release AI tests may be seen as an ethical bridge too far.
This article originally appeared on TechCrunch at https://techcrunch.com/2025/04/09/openai-launches-program-to-design-new-domain-specific-ai-benchmarks/
Hashtags
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Yahoo
39 minutes ago
- Yahoo
Elon Musk biographer says major Tesla merger could be imminent: 'I think it's going to happen'
Could Tesla eventually merge with another of Elon Musk's companies, xAI? A prominent insider thinks such a move is increasingly likely. Walter Isaacson, the prominent Musk biographer who has received unprecedented access to Tesla's CEO, recently said he expects the two companies to eventually merge, Not a Tesla App reported. During a CNBC interview, Isaacson said combining Tesla and xAI would ultimately better serve each company's mission. "I think it's going to happen," Isaacson said, per "Because Musk, even in my book when he's starting xAI, [was] talking about [how] these chatbots are fine, but what you need is real-world AI. You need to be able to not only take all the texts and tweets that have ever been written, but all the videos from Teslas and all the Optimus robot [is] seeing and hearing." Tesla was a pioneer in electric vehicles and still has the top-selling vehicles in the space — although its sales numbers have dipped this year. But Musk has repeatedly said that the future of the company is tied in more than just cars, including "vast numbers of autonomous humanoid robots." That makes xAI seem like a natural partner for Tesla. It is behind the artificial intelligence assistant Grok, which will reportedly power Tesla's upcoming smart assistant. Musk has also said he expects Grok to be incorporated into Tesla's Optimus humanoid robots, with hopes of sending them to Mars in the near future. In May, Musk said in a CNBC interview, per Business Insider, that "there are no plans" to merge the companies, but that "it's not out of the question." Tesla's sales may have had a bumpy start to the year, but there's no denying the role it has played in bringing millions of cleaner cars onto roads around the world. Studies have shown that driving an EV can reduce carbon pollution by two-thirds compared to gas-powered cars. EVs can be even greener when paired with a renewable energy source for charging, such as solar. In addition, if you have solar panels, that energy is considerably cheaper than relying on the grid or public charging stations. EnergySage allows homeowners to save thousands on solar-panel installation costs by comparing quotes from local, vetted installers. And if the upfront costs of solar are too daunting, Palmetto's LightReach program allows people to lease solar panels, providing locked-in, low energy rates, and a lower carbon footprint, with no down payment. Do you think electric vehicles are efficient enough to replace gas cars? Totally Definitely not They're almost there They need a lot more work Click your choice to see results and speak your mind. Join our free newsletter for good news and useful tips, and don't miss this cool list of easy ways to help yourself while helping the planet.
Forbes
an hour ago
- Forbes
What Investors Should Know As Meta Gets (Back) Into Crypto
Meta (formerly Facebook) is getting back into crypto Markets and investment trends tend to move in cycles, and the cryptoasset sector is no exception to this rule of the marketplace. As TradFi institutions continue to deploy blockchain affiliated projects, including the launch of a stablecoin by SocGen running on the Ethereum blockchain, the adoption and acceleration of cryptoassets continues virtually unabated. Even as the sentiment toward crypto improves, prices of bitcoin and other cryptocurrencies increase, and the policy landscape pivots toward a pro-growth outlook there remain significant obstacles to mainstream utilization. For example, the tax treatment of crypto is an inhibitor to retail utilization of crypto as a method of payment, and the lack of insurance available for crypto and crypto-adjacent products can make it difficult for institutions to allocate substantial funds to cryptoassets. Against this landscape, exemplified both the increasing adoption and understanding of cryptoassets and applications with the continued limitations to institutional usage, one company stands apart for several reasons. Meta (formerly Facebook) recently has been questioned by Senators Warren and Blumenthal related to its support for the GENIUS Act, and specifically whether or not the firm would block a prohibition on Big Tech firms from owning stablecoin issuers. The specifics of the questioning by the senators will most assuredly change over time, but the letter that has been made publicly available detail that the senators desire specifics as to what the stablecoin plans for Meta are. Let's take a look at why this letter and these questions are important, not only for Meta, but for the cryptoasset marketplace at large. Meta, then operating as Facebook, already attempted to launch of a native stablecoin in 2019 via the Libra project which was subsequently rebranded as Diem. This previous effort occurred during an entirely different economic and policy landscape, and occurred as the organization was still contending with intense scrutiny following the 2016 U.S. Presidential election. Issues that were raised at the time dealt with the potential of a stablecoin issued by Facebook serving to weaken competition, compromise user privacy, and lead to continued fractionalization of which entity or organization sets policy for U.S. monetary and fiscal policy. While the cryptoasset landscape and policy outlook for crypto projects has definitively shifted to a more permissive stance the very same issues that were raised during 2019 loom large as Meta returns to the stablecoin marketplace. Specifically, the letter from the Senators cited the track record of privacy violations, scams, and fake news that continue to occur on the platform as risks that a native stablecoin could amplify. Even as stablecoins increasingly become more mainstream, and are approaching a market capitalization nearing $300 billion, Meta might find many of the same issues that stymied earlier efforts being dragged back to the surface. Since Meta is one of the few returning players to the stablecoin space this provides an opportunity for crypto native stablecoins such as Circle, which continues to ride high following its IPO in June. As Meta edges closer to launching its own stablecoin, the spotlight on Big Tech's role in digital money is about to get a lot brighter, especially as these same tech firms continue to invest billions in AI initiatives. For crypto-native firms like Circle, that's not a threat - it's an opportunity. Meta's sheer size and complicated history with data privacy all but guarantee it will draw intense regulatory scrutiny. And that scrutiny will set a new bar for how stablecoins are viewed and governed both in the U.S. and abroad. That's where Circle can shine. Unlike tech giants pivoting into payments, Circle was built in crypto — with regulatory engagement and transparency as core pillars. While Meta faces inevitable trust questions and regulatory hurdles, Circle can double down on its position as the safer, more compliant alternative. In the coming months, expect firms like Circle to lean into this advantage, especially as institutional partners and consumers alike grow more cautious about Big Tech controlling their money. Notably, the ongoing partnership between Circle and Coinbase – two of the largest crypto native firms that are publicly traded in the U.S. – can also serve to assuage concerns of policymakers. Regardless of this specific stablecoin project plays out the following reality is becoming increasingly clear, and some would say urgent, for the crypto marketplace. With tens of billions flowing into the sector, TradFi firms deploying blockchain based solutions and native stablecoins, and policymakers actively debating the GENIUS Act, the crypto audit and attestation narrative continues to seem stuck. While the AICPA continues to issue guidance and updates related to digital asset attestation, controls, and valuation, the authoritative standard setters remain behind the proverbial curve. As stablecoins become more important and integrated with payment, treasury, and lending systems the urgency for definitive and standardized auditing best practices will continue to elevate in importance.
Time Business News
2 hours ago
- Time Business News
TeamDesk Integration with OpenAI- A Guide to Unlocking AI-Powered Efficiency
In today's fast-paced digital world, businesses are constantly looking for ways to optimize workflows and improve decision-making. TeamDesk, a powerful online database system, enables businesses to organize and manage data effectively. However, integrating TeamDesk with OpenAI can take data management to a whole new level by introducing AI-powered automation, insights, and intelligent decision-making. This article explores the benefits of integrating TeamDesk with OpenAI, potential use cases, and step-by-step instructions on how to set up this integration for enhanced business performance. Benefits of Integrating TeamDesk with OpenAI By linking TeamDesk with OpenAI, businesses can unlock numerous advantages, such as: Automated Data Processing: AI models can analyze vast amounts of data stored in TeamDesk, extract insights, and generate reports automatically. Natural Language Processing (NLP): OpenAI's NLP capabilities allow businesses to interpret unstructured text data, classify customer queries, and provide sentiment analysis. Enhanced Customer Support: AI-powered chatbots can interact with customers using data stored in TeamDesk, ensuring accurate and contextual responses. Predictive Analytics: Businesses can forecast trends, customer behaviors, and sales patterns using OpenAI's predictive capabilities. Document Generation: AI can generate contracts, reports, and summaries based on data from TeamDesk. Task Automation: AI-driven automation reduces manual effort in processing and updating records within TeamDesk. Use Cases for TeamDesk and OpenAI Integration 1. AI-Powered Data Analysis Organizations can use OpenAI to analyze large datasets stored in TeamDesk and derive meaningful insights. For example, an HR department can process employee feedback and extract key sentiment trends. 2. Automated Report Generation Instead of manually compiling reports, OpenAI can generate executive summaries, sales reports, and performance evaluations based on structured data from TeamDesk. 3. Customer Support Automation Integrating OpenAI with TeamDesk allows businesses to deploy AI chatbots that fetch real-time data from the database, offering customers accurate and quick responses. 4. Email Automation OpenAI can draft and personalize email responses based on customer interactions stored in TeamDesk, ensuring better engagement and efficiency. 5. Fraud Detection & Risk Management With AI's pattern recognition capabilities, businesses can identify unusual activities and flag potential fraudulent transactions in their database. Steps to Integrate TeamDesk with OpenAI To successfully integrate TeamDesk with OpenAI, follow these steps: Step 1: Set Up API Access for TeamDesk TeamDesk provides API access to interact with external applications. To enable API access: Log into TeamDesk. Navigate to Setup > Integrations > API Access. Generate an API key to allow external applications (such as OpenAI) to interact with TeamDesk. Note down the API endpoint for retrieving and updating records. Step 2: Obtain OpenAI API Key To use OpenAI's capabilities: Visit OpenAI's website and sign up for an API key. Choose the AI model you want to use (GPT-4, GPT-3.5, or fine-tuned models). Save the API key securely for later use. Step 3: Connect TeamDesk and OpenAI Using a Middleware Since TeamDesk and OpenAI use different APIs, middleware such as Zapier, Make (formerly Integromat), or custom scripts can help connect them. Using Zapier: Create a Zap and select TeamDesk as the trigger. Choose an event (e.g., 'New Record Created' or 'Updated Record'). Select OpenAI as the action app and define the task (e.g., 'Summarize Text' or 'Generate Email Response'). Map the fields between TeamDesk and OpenAI. Activate the Zap to automate the workflow. Step 4: Develop a Custom Integration (Optional) For more control, businesses can develop a custom integration using Python or JavaScript. Example: Python Script for TeamDesk & OpenAI Integration import requests def get_teamdesk_data(): teamdesk_url = ' headers = {'Authorization': 'Bearer YOUR_TEAMDESK_API_KEY'} response = headers=headers) return def send_to_openai(prompt): openai_url = ' headers = {'Authorization': 'Bearer YOUR_OPENAI_API_KEY', 'Content-Type': 'application/json'} data = {'model': 'gpt-4', 'prompt': prompt, 'max_tokens': 200} response = headers=headers, json=data) return data = get_teamdesk_data() prompt = 'Summarize this customer feedback: ' + str(data) result = send_to_openai(prompt) print(result['choices'][0]['text']) Step 5: Test and Deploy After setting up the integration: Test the workflow to ensure seamless data transfer. Optimize the AI model's responses for accuracy. Deploy the integration and monitor performance. Conclusion Integrating TeamDesk with OpenAI enables businesses to leverage AI for automating workflows, enhancing customer service, and driving data-driven decision-making. Whether through no-code platforms like Zapier or custom-built Python scripts, this integration can transform business operations. By combining the power of structured databases with AI-driven insights, organizations can boost efficiency, reduce manual tasks, and enhance productivity. Ready to take your data management to the next level? Start integrating TeamDesk with OpenAI today! TIME BUSINESS NEWS
