California Supreme Court demands State Bar answer questions on AI exam controversy
The California Supreme Court urged the State Bar of California Thursday to explain how and why it utilized artificial intelligence to develop multiple-choice questions for its botched February bar exams.
California's highest court, which is responsible for overseeing the State Bar, disclosed Tuesday that its justices were not informed before the exam that the State Bar had allowed its independent psychometrician to use AI to develop some questions.
The Court demanded the State Bar to explain how it used AI to develop questions — and what actions it took to ensure the reliability of the questions.
The demand comes as the State Bar petitions the court to adjust test scores for hundreds of prospective California lawyers who complained of multiple technical problems and irregularities during the February exams.
The controversy is about more than the State Bar's use of artificial intelligence per se. It's about how the State Bar used AI to develop questions — and how rigorous its vetting process was — for a high stakes exam that determines whether hundreds of aspiring attorneys can practice law in California.
It also raises questions about how transparent State Bar officials were as they sought to ditch the National Conference of Bar Examiners' Multistate Bar Examination — a system used by most states — and roll out a new hybrid model of in-person and remote testing in an effort to cut costs.
In a statement Thursday, the court said it was seeking answers as to 'how and why AI was used to draft, revise, or otherwise develop certain multiple-choice questions, efforts taken to ensure the reliability of the AI-assisted multiple-choice questions before they were administered, the reliability of the AI-assisted multiple-choice questions, whether any multiple-choice questions were removed from scoring because they were determined to be unreliable, and the reliability of the remaining multiple-choice questions used for scoring.'
Last year, the Supreme Court approved the State Bar's plan to forge an $8.25 million, five-year deal with Kaplan to create 200 test questions for a new exam. The State Bar also hired a separate company, Meazure Learning, to administer the exam.
It was not until this week — nearly two months after the exam — that the State Bar revealed in a news release that it had deviated from its plan to use Kaplan Exam Services to write all the multiple-choice questions.
In a presentation, the State Bar revealed that 100 of the 171 scored multiple-choice questions were made by Kaplan and 48 were drawn from a first-year law students exam. A smaller subset of 23 scored questions were made by ACS Ventures, the State Bar's psychometrician, and developed with artificial intelligence.
'We have confidence in the validity of the [multiple-choice questions] to accurately and fairly assess the legal competence of test-takers,' Leah Wilson, the State Bar's executive director, said in a statement.
Alex Chan, Chair of the Committee of Bar Examiners, which exercises oversight over the California Bar Examination, initially played down the controversy, telling the Times that only a small subset of questions used AI — and not necessarily to create the questions.
Chan also noted that the California Supreme Court urged the State Bar in October to review 'the availability of any new technologies, such as artificial intelligence, that might innovate and improve upon the reliability and cost-effectiveness of such testing.'
'The court has given its guidance to consider the use of AI, and that's exactly what we're going to do,' Chan said.
But on Thursday Chan revealed to The Times that State Bar officials had not told the Committee of Bar Examiners ahead of the exams that it planned to use AI.
'The Committee was never informed about the use of AI before the exam took place, so it could not have considered, much less endorsed, its use,' Chan said.
Katie Moran, an associate professor at the University of San Francisco School of Law who specializes in bar exam preparation, said this begged a series of questions.
'Who at the State Bar directed ACS Ventures, a psychometric company with no background in writing bar exam questions, to author multiple-choice questions that would appear on the bar exam?' she said on LinkedIn. 'What guidelines, if any, did the State Bar provide?'
Mary Basick, assistant dean of academic skills at UC Irvine Law School, said it was a big deal that the changes in how the State Bar drafted its questions were not approved by the Committee of Bar Examiners or the California Supreme Court.
'What they approved was a multiple-choice exam with Kaplan-drafted questions,' she said. 'Kaplan is a bar prep company, so of course, has knowledge about the legal concepts being tested, the bar exam itself, how the questions should be structured. So the thinking was that it wouldn't be a big change.'
Any major change that could impact how test-takers prepare for the exam, she noted, requires a two-year notice under California's Business and Professions Code.
'Typically, these types of questions take years to develop to make sure they're valid and reliable and there's multiple steps of review,' Basick said. 'There was simply not enough time to do that.'
Basick and other professors have also raised concerns that hiring a non-legally trained psychometrist to develop questions with AI, as well as determine whether the questions are valid and reliable, represents a conflict of interest.
The State Bar has disputed that idea: 'The process to validate questions and test for reliability is not a subjective one, and the statistical parameters used by the psychometrician remain the same regardless of the source of the question,' it said in a statement.
On Tuesday, the State Bar told The Times that all questions were reviewed by content validation panels and subject matter experts ahead of the exam for factors including legal accuracy, minimum competence and potential bias.
The State Bar has yet to answer questions about why it deviated from its plan for Kaplan to draft all the exam multiple-choice questions. It has also not elaborated on how ACS Ventures used AI to develop its questions.
Hashtags
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Yahoo
4 days ago
- Yahoo
Anthropic co-founder on cutting access to Windsurf: 'It would be odd for us to sell Claude to OpenAI'
Anthropic Co-founder and Chief Science Officer Jared Kaplan said his company cut Windsurf's direct access to Anthropic's Claude AI models largely because of rumors and reports that OpenAI, its largest competitor, is acquiring the AI coding assistant. "We really are just trying to enable our customers who are going to sustainably be working with us in the future," said Kaplan during an onstage interview Thursday with TechCrunch at TC Sessions: AI 2025. "I think it would be odd for us to be selling Claude to OpenAI," Kaplan said. The comment comes just a few weeks after Bloomberg reported that OpenAI was acquiring Windsurf for $3 billion. Earlier this week, Windsurf said that Anthropic cut its direct access to Claude 3.5 Sonnet and Claude 3.7 Sonnet, two of the more popular AI models for coding, forcing the startup to find third-party computing providers on relatively short notice. Windsurf said it was disappointed in Anthropic's decision and that it might cause short-term instability for users trying to access Claude via Windsurf. Windsurf declined to comment on Kaplan's remarks, and an OpenAI spokesperson did not immediately respond to TechCrunch's request. The companies have not confirmed the acquisition rumors. Part of the reason Anthropic cut Windsurf's access to Claude, according to Kaplan, is because the company is quite computing-constrained today. Anthropic would like to reserve its computing for what Kaplan characterized as "lasting partnerships." However, Kaplan said the company hopes to greatly increase the availability of models it can offer users and developers in the coming months. He added that Anthropic has just started to unlock capacity on a new computing cluster from its partner, Amazon, which he says is "really big and continues to scale." As Anthropic pulls away from Windsurf, Kaplan said he's collaborating with other customers building AI coding tools, such as Cursor — a company Kaplan said Anthropic expects to work with for a long time. Kaplan rejected the idea that Anthropic was in competition with companies like Cursor, which is developing its own AI models. Meanwhile, Kaplan says Anthropic is increasingly focused on developing its own agentic coding products, such as Claude Code, rather than AI chatbot experiences. While companies like OpenAI, Google, and Meta are competing for the most popular AI chatbot platform, Kaplan said the chatbot paradigm was limiting due to its static nature, and that AI agents would in the long run be much more helpful for users.
Business Wire
4 days ago
- Business Wire
Kaplan Wins 'Test Prep Innovation of the Year' Designation in EdTech Breakthrough Awards Program for its Groundbreaking Partnership with the State of Illinois
FORT LAUDERDALE, Fla.--(BUSINESS WIRE)--Global educational services provider Kaplan's All Access License™ has been selected as the winner of the 'Test Prep Innovation of the Year' award in the 7 th annual EdTech Breakthrough Awards program, conducted by EdTech Breakthrough, a leading market intelligence organization that recognizes the top companies and solutions in the global educational technology market. Global educational services provider Kaplan's All Access License has been selected as the winner of the 'Test Prep Innovation of the Year' award in the 7th annual EdTech Breakthrough Awards program, conducted by EdTech Breakthrough. Kaplan specifically received this prestigious award this year for its innovative workforce development and higher education partnership with the State of Illinois, which provides free test prep courses to all students enrolled in Illinois' 12 public universities. Five Illinois community colleges are also included as part of a pilot program. More than 200,000 students now have free access to Kaplan's best-in-class preparation for graduate-level admissions exams and professional licensing exams, including comprehensive prep for the GRE ®, GMAT ®, LSAT ®, MCAT ®, NCLEX-RN ®, USMLE ®, Illinois State bar exam, real estate and securities exams, and more. They also have access to a suite of professional and academic skills development courses from Kaplan, including project management, data literacy, and critical thinking. To date, thousands of students have enrolled in one of more than 40 available Kaplan courses, collectively saving these aspiring professionals more than $10 million in out-of-pocket costs, which surpasses the original statewide investment. Kaplan was also the category winner in 2023 and 2024 for directly partnering with institutions of higher learning, including some of the most dynamic historically black colleges and universities (HBCUs), like Xavier University of Louisiana, Howard University, and Spelman College. By investing in a Kaplan All Access License™, state governments and colleges and universities can provide all of their students with access to Kaplan's industry-leading programs with zero out of pocket costs for students, helping them reach their career goals and reducing workforce shortagesin essential professions in the STEM, nursing, and teaching fields. Gregory Marino, CEO, Kaplan North America, said: "Kaplan is proud that our All Access License™ has again been honored as 'Test Prep Innovation of the Year' by EdTech Breakthrough. This recognition affirms our commitment to closing the opportunity gap and boosting economic mobility. By partnering with states and colleges and universities to offer universal test prep and workforce readiness programs, we're helping students achieve long-term career success and create a new generation of thriving professionals. As education leaders, we share a responsibility—and a real opportunity—to give as many students as possible an equal shot at success. And Kaplan's All Access License™ does just that.' Steve Johansson, managing director, EdTech Breakthrough, said: 'The breadth of product offerings with the All Access License™ is unique to Kaplan and enables institutions to support the broadest number of students. We know private test preparation is the best way to succeed when it comes to required high-stakes exams like the LSAT or other licensure exams. However, the expensive fees for test prep make its access unequal. In an era of resource constraints, the All Access License™ model from Kaplan simultaneously increases operational efficiencies while also increasing support for student excellence, resulting in immediate positive impact.' The mission of the EdTech Breakthrough Awards is to honor excellence and recognize innovation, hard work and success in a range of educational technology categories. This year's program attracted thousands of nominations from more than 15 different countries. Test names and other trademarks are the property of the respective trademark holders. About Kaplan Kaplan, Inc. is a global educational services company that helps individuals and institutions advance their goals in an ever-changing world. Our broad portfolio of solutions help students and professionals further their education and careers, universities and educational institutions attract and support students, and businesses maximize employee recruitment, retainment, and development. Stanley Kaplan founded our company in 1938 with a mission to expand educational opportunities for students of all backgrounds. Today, our thousands of employees working in 27 countries/regions continue Stanley's mission as they serve about 1.3 million students and professionals, 16,000 corporate clients, and 2,700 schools, school districts, colleges, and universities worldwide. Kaplan is a subsidiary of the Graham Holdings Company (NYSE: GHC). Learn more at
Yahoo
5 days ago
- Yahoo
Parental Guidance: A new front emerges in battle between far-right, LGBTQ+ themed books
Bill Bolin, left, and members of the Hartland Cromaine District Library Board of Trustees discuss policy at a special meeting on Tuesday, June 3, 2025 | Photo by Ben Solis Picture, if you will, a library with its books behind a plated glass cabinet, locked and out of reach to patrons of all ages, accessible only upon request to a library employee who has the appropriate key. Imagine a row of children's books and educational materials, deemed inappropriate or controversial by some, shoved away in an adult's only section – further out of reach than that imaginary cabinet – each with a warning that disseminating those books to a minor could be a crime. In some cities and townships across Michigan, library officials or the members of boards that oversee them, especially those who have expressed hostility to the LGBTQ+ community, are toying with the idea of making those barriers a reality. The effort has become a second front, so to speak, in the culture war over children's books and particularly those with LGBTQ+ content or themes. Much like the battles over prohibiting library books or criminalizing them for minors that have played out over the last few years, activists and attorneys across Michigan have said that similarly sequestering or restricting access to books runs afoul of the First Amendment. SUBSCRIBE: GET THE MORNING HEADLINES DELIVERED TO YOUR INBOX Jay Kaplan, a long-time staff attorney with the American Civil Liberties Union of Michigan staff specializing in LGBTQ+ rights, said in an interview with Michigan Advance that right-wing culture warriors were employing this new tactic because they were losing the overall campaign on outright book bans. 'They're losing in terms of the public opinion, [because] a vast majority of people disfavor this type of thing,' Kaplan said. 'And so they feel, 'We'll just try to move it to another place in the library. We'll try to put a label on it. We'll try and discourage people from being able to take them out.'' But federal courts have looked at the issue, and they have found, Kaplan said, that even when a community is not just removing the book entirely from the library, the act of putting a burden on the First Amendment right to receive information based on content amounted to the same. 'Particulary if what's motivating you to want to limit them is your disapproval of the subject matter of the book, or some of the contents of the book,' he said. 'That also violates the First Amendment.' For the Hartland Cromaine District Library in Livingston County, the conversation on labeling books started in 2022. Over time and with the election of new library Board of Trustees members, the conversation became much more pointed. Much of that had to do with the election of Bill Bolin, the pastor of the FloodGate Church in Brighton, and his elevation to the president of the Cromaine District Library board in January. Bolin and his church have been written about by various publications, including The Atlantic's Tim Alberta, detailing Bolin's mixture of right-wing conspiratorial politics and Christianity. Bolin also features throughout Alberta's 2023 book, 'The Kingdom, the Power, and the Glory: American Evangelicals in an Age of Extremism.' Some board members questioned Bolin's experience and qualifications for running the library board in a breezy rural district just before he was selected as its leader in January. But the board voted 5-2 to install him at the top. In February, Bolin, as the congregants of the FloodGate Church know him to do, began to speak out of order to introduce himself and detail exactly why it was that he sought the library board position and its leadership post, according to the approved minutes of the board's February 20 meeting. 'For those who do not know why I ran for this position, I desire to protect children from the harm that can befall them over coercive behavior,' Bolin said in a statement to the crowd. 'The approach I am suggesting, along with certain colleagues, is a commonsense approach to changing the sexual tone and nature of some library policies and practices.' Bolin said that the board would then discuss controversial items on the agenda, including the removal of June LGBTQ+ Pride displays, labeling certain books that may be deemed controversial, moving books to an age restricted area, providing supervision in the teen area to monitor 'behavior' and returning the Pledge of Allegiance to monthly meetings. Bolin then read from Michigan law regarding the displaying or disseminating of sexually explicit materials to minors, followed by a recitation of a potential warning label he had created warning adults of the dangers of providing such material to children. But Bolin wasn't talking about dirty magazines in a seedy retail store: he was talking about books within the community's public library. 'Someone needs to stand up for the children,' Bolin said. 'Those who serve on the library board and want to implement policies that restrict access to certain books and place content warning labels will be able to enact policies that reflect the community in which they serve the people of Cromaine Library District, either tonight by vote or later after referral to the appropriate committee for refinement of language and crafting of policies that state intention of the people.' Bolin added that a list of books that could be recommended for labeling was being compiled with at least 80 titles, minimum, to be presented to the librarian 'for labeling and movement into an age-appropriate section of the library.' Members of the community present at the February 20 meeting noted that the Top 10 challenged books in that list had LGBTQ+ characters or themes. Some questioned if the board had the legal authority to deem what was and wasn't sexual in nature about these books, while others praised the move. Among the latter was Livingston County Commissioner Wes Nakagiri, one of the architects of the conservative Tea Party movement in Michigan, which undoubtedly built the framework of the America First and MAGA movement that propelled President Donald Trump to power in 2016 and again in 2024. A day before that meeting, on February 19, the board sought and received a legal opinion from its corporate counsel at the Foster Swift law firm on whether it could move forward with restricting access to books or otherwise discouraging minor patrons from getting to them. In a memo provided to Michigan Advance, the law firm expressed concern that the library would be on shaky constitutional ground if it moved forward with a policy that would move books to an adult-only section, placing them behind glass, creating a separate section for controversial books or placing on them labels identifying them as obscene. The firm said each one of those actions might be unconstitutional based on similar policies already deemed unconstitutional by the courts. Although each of those actions came with significant risk of opening the library up to a lawsuit, the move to place warning labels on books indicating sexually explicit content and the possibility of criminal prosecution if the material was provided to a minor was noted as having 'several practical shortcomings that could lead to further violations of the law.' Not only would the labels constitute a burden on access to materials, the firm said courts would undoubtedly analyze the motives behind labeling some materials and not others. The firm added that the library would likely lose that challenge. Some community members have noted that communications between board members and residents or allies in the push to have some books labeled would only help a suing party win a case against the library. 'For example, if the 'sexually explicit material' labels tended to stigmatize protected speech or one viewpoint more heavily than another, like anti-LBGTQ+ messages vs. pro-LBGTQ+ messages, the library may be subject to liability,' the firm said in the February legal memo. 'Lastly, we note that there is current litigation in federal court involving an Alabama library's use of 'labels' that signaled that a material contained 'adult' themes and the library's prohibition on minors accessing those materials. In that case, the parties are awaiting a hearing and/or decision on the plaintiff's motion for preliminary injunction.' Yet Bolin and members of the board who supported the push continued on, and eventually sought a separate opinion from the Alliance Defending Freedom, a Christian legal advocacy group that works primarily to seed Christian religious beliefs, practices and customs in public schools or other government bodies. Bolin eventually brought a refined version of the proposed policy with labeling as the main avenue to keep LGBTQ+ themed and other materials deemed inappropriate to library staff and the board with notes from ADF attorney Paul Spena. Bolin was wary to name Spena as the attorney he was now working with to craft the policy but later relented when pressed on the issue. When asked to talk about the legal theory behind the labels and what case law they could point to show that it was legally sound, Spena and the ADF declined to comment or be interviewed for this story. Letter to Board of Trustees The ACLU of Michigan formally chimed in on the policy last week, and sent the board a letter drafted by Kaplan warning members that what they were pursuing was an act of censorship, even if the titles are not banned from the library and remain within its walls, located in special sections or with new labels affixed to the covers. 'Doing so impedes the rights of library patrons and runs afoul of the First Amendment,' Kaplan wrote. 'It can also harm marginalized communities who may come to places like public libraries hoping for an inclusive space, and in this particular instance, doing so with regard to LGBTQ+ titles will exacerbate that harm.' Several community members who spoke to Michigan Advance in the course of reporting this story said that Bolin and his allies on the board were moving closer to adopting the policy despite those warnings, in essence inviting a lawsuit to be pioneers on the issue here in Michigan. Among them was Stand Against Extremism LivCo (SAGE) co-founder Julie Ohashi, who has been vocal in her opposition to the board's actions thus far. On Tuesday evening, Spena was expected to speak at the library board's special meeting to discuss the policy in full. No such discussion with Spena or another attorney from the ADF occurred, and it was not clear if Spena or another member of the group were present at the meeting. Bolin mentioned, however, that the legal discourse was changing in America, indicating that courts in the era of Trump might be turning the tide to support measures much like the one being discussed by the Cromaine District Library board. Present at the meeting, however, were several community members, some in support of Bolin and the board's majority on the label issue and plenty of others who said they were disheartened, dismayed and angry that the board would continue moving toward a policy they called discriminatory and clearly illegal. As the board moved through the policy line by line, softening it due to objections from several board members out of fear of being sued and settling more on labeling as the possible avenue, those opposed to the move held signs calling on the board to not mix religion and politics. But those silent protests quickly turned vocal, with shouts and jeers rising above the din of what started as a calm meeting. Board Vice President Jeannine Gogoleski was appointed as the sergeant at arms for the meeting, and she and her husband, Glenn Gogoleski, a member of the Hartland Consolidated Schools Board of Education, began removing disruptive members of the meeting. Tensions rose further when the Livingston County Sheriff's Office was called to ensure there were no further outbursts. The meeting went into recess until deputies could sort out the situation, and they remained there until it ended. Although the board did not adopt a policy on Tuesday, it is expected to do so at its next meeting. During public comment, some read lengthy diatribes laden with Christian scripture, while others lambasted the board inviting what they called a hate group to give the board legal advice and defend them if they are forced into court. One woman held up Alberta's book while speaking, noting that Bolin's name appeared in it multiple times, to which Bolin smiled. Ohashi called ADF a 'hate group' that has described LGBTQ+ rights as a principal threat to religious freedom, and attacking those rights was at the center of their work. 'Their goal is to trigger as many lawsuits that can get to the U.S. Supreme Court as fast as possible,' she said. 'That is precisely the point. They want this to go to court, because ADF's ultimate goal is eliminating LGBTQ+ Americans' status as a protected class of citizens.' Kate Mazzara of Hartland said she feared that the nation was tiptoeing toward religious fascism and that the small district library in her hometown was sliding on the same path. 'Make no mistake about it, that's what this is,' Mazara said. 'It starts with baby steps, and then it's over.'
