California Supreme Court demands State Bar answer questions on AI exam controversy
California's highest court, which oversees the State Bar, disclosed Tuesday that its justices were not informed before the exam that the State Bar had allowed its independent psychometrician to use AI to develop a small subset of questions.
The Court on Thursday upped its public pressure on the State Bar, demanding it explain how it used AI to develop questions — and what actions it took to ensure the reliability of the questions.
The demand comes as the State Bar petitions the court to adjust test scores for hundreds of prospective California lawyers who complained of multiple technical problems and irregularities during the February exams.
Read more: State Bar of California admits it used AI to develop exam questions, triggering new furor
The controversy is about more than the State Bar's use of artificial intelligence per se. It's about how the State Bar used AI to develop questions — and how rigorous its vetting process was — for a high stakes exam that determines whether thousands of aspiring attorneys can practice law in California each year.
It also raises questions about how transparent State Bar officials were as they sought to ditch the National Conference of Bar Examiners' Multistate Bar Examination — a system used by most states — and roll out a new hybrid model of in-person and remote testing in an effort to cut costs.
In a statement Thursday, the Supreme Court said it was seeking answers as to "how and why AI was used to draft, revise, or otherwise develop certain multiple-choice questions, efforts taken to ensure the reliability of the AI-assisted multiple-choice questions before they were administered, the reliability of the AI-assisted multiple-choice questions, whether any multiple-choice questions were removed from scoring because they were determined to be unreliable, and the reliability of the remaining multiple-choice questions used for scoring."
Last year, the Court approved the State Bar's plan to forge an $8.25 million, five-year deal with Kaplan to create 200 test questions for a new exam. The State Bar also hired a separate company, Meazure Learning, to administer the exam.
It was not until this week — nearly two months after the exam — that the State Bar revealed in a news release that it had deviated from its plan to use Kaplan Exam Services to write all the multiple-choice questions.
In a presentation, the State Bar revealed that 100 of the 171 scored multiple-choice questions were made by Kaplan and 48 were drawn from a first-year law students exam. A smaller subset of 23 scored questions were made by ACS Ventures, the State Bar's psychometrician, and developed with artificial intelligence.
'We have confidence in the validity of the [multiple-choice questions] to accurately and fairly assess the legal competence of test-takers,' Leah Wilson, the State Bar's executive director, said in a statement.
Read more: Trump releases new files on RFK assassination
Alex Chan, an attorney who chairs the Committee of Bar Examiners, which exercises oversight over the California Bar Examination, told The Times Tuesday that only a small subset of questions used AI — and not necessarily to create the questions.
Chan also noted that the California Supreme Court urged the State Bar in October to review 'the availability of any new technologies, such as artificial intelligence, that might innovate and improve upon the reliability and cost-effectiveness of such testing.'
'The court has given its guidance to consider the use of AI, and that's exactly what we're going to do,' Chan said.
That process, Chan later explained, would be subject to the Court's review and approval.
On Thursday Chan revealed to The Times that State Bar officials had not told the Committee of Bar Examiners ahead of the exams that it planned to use AI.
'The Committee was never informed about the use of AI before the exam took place, so it could not have considered, much less endorsed, its use,' Chan said.
Katie Moran, an associate professor at the University of San Francisco School of Law who specializes in bar exam preparation, said this begged a series of questions.
'Who at the State Bar directed ACS Ventures, a psychometric company with no background in writing bar exam questions, to author multiple-choice questions that would appear on the bar exam?' she said on LinkedIn. 'What guidelines, if any, did the State Bar provide?'
Mary Basick, assistant dean of academic skills at UC Irvine Law School, said it was a big deal that the changes in how the State Bar drafted its questions were not approved by the Committee of Bar Examiners or the California Supreme Court.
'What they approved was a multiple-choice exam with Kaplan-drafted questions,' she said. 'Kaplan is a bar prep company, so of course, has knowledge about the legal concepts being tested, the bar exam itself, how the questions should be structured. So the thinking was that it wouldn't be a big change.'
Read more: 'It's a shambles': DOGE cuts bring chaos, long waits at Social Security for seniors
Any major change that could impact how test-takers prepare for the exam, she noted, requires a two-year notice under California's Business and Professions Code.
'Typically, these types of questions take years to develop to make sure they're valid and reliable and there's multiple steps of review,' Basick said. 'There was simply not enough time to do that.'
Basick and other professors have also raised concerns that hiring a non-legally trained psychometrist to develop questions with AI, as well as determine whether the questions are valid and reliable, represents a conflict of interest.
The State Bar has disputed that idea: 'The process to validate questions and test for reliability is not a subjective one, and the statistical parameters used by the psychometrician remain the same regardless of the source of the question,' it said in a statement.
On Tuesday, the State Bar told The Times that all questions were reviewed by content validation panels and subject matter experts ahead of the exam for factors including legal accuracy, minimum competence and potential bias.
When measured for reliability, the State Bar said, the combined scored multiple-choice questions from all sources — including AI — performed 'above the psychometric target of 0.80.'
The State Bar has yet to answer questions about why it deviated from its plan for Kaplan to draft all the exam multiple-choice questions. It has also not elaborated on how ACS Ventures used AI to develop its questions.
Sign up for Essential California for news, features and recommendations from the L.A. Times and beyond in your inbox six days a week.
This story originally appeared in Los Angeles Times.
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Yahoo
3 hours ago
- Yahoo
Home Office intervenes as English council bids to stop housing refugees
THE Home Office has asked to intervene in a court case brought by an English council attempting to block asylum seekers from being housed in a local hotel. Epping Forest District Council sought an injunction from the Royal Court of Justice on Tuesday to prevent asylum seekers from being allowed to stay at the former Bell Hotel in the area. If granted, the injunction would mean the hotel's owner, Somani Hotels Limited, must stop housing asylum seekers at the site within 14 days. The Home Office was not represented at a previous hearing in the case on Friday. (Image: Jordan Pettitt/PA Wire) However, at the start of a hearing on Tuesday, at which Justice Eyre is due to hand down his ruling on whether the injunction should be granted, the department asked to be allowed to intervene. Edward Brown KC, for the Home Office, said: 'If the injunction is granted by the court, it will substantially impact on the Home Secretary's statutory duties.' He continued: 'The local authority should in fact have given some consideration to the wider public interest in this application.' Brown added that the injunction bid 'causes particular acute difficulties at the present date'. Piers Riley-Smith, representing Somani Hotels Limited, which owns the Bell Hotel, told the High Court that the company supports the Home Office's request to intervene in Epping Forest District Council's bid for a temporary injunction. But Philip Coppel KC, for the council, said that the Home Office's request was 'a thoroughly unprincipled application made in a thoroughly unprincipled way', and that the department knew of the injunction bid last week but 'sat on their hands'. More to follow…
Yahoo
5 hours ago
- Yahoo
Police warn against allowing Urich to return to work, ‘place where he executed alleged crimes'
A new court hearing examined whether Yonatan Urich, a key suspect in the "Qatargate" case, should face lighter release conditions amid allegations of leaked documents. A tense hearing took place at the Lod District Court on Tuesday morning, a hearing before Judge Amit Michles over whether to lighten the restrictive release conditions of Yonatan Urich, an aide to Prime Minister Benjamin Netanyahu and one of the chief suspects in the 'Qatargate' investigations. His legal defense has made two main arguments. The first is that he wasn't really a public servant, but rather a private citizen. Therefore, there is no danger in allowing him to return to his employment and make contact with anyone connected to the Prime Minister's Office. The second is that there is no justification for not matching his release conditions to those of Eli Feldstein, the former military spokesman for the PMO and the first individual to be interrogated in connection with the case. Feldstein was arrested in November, while Urich was apprehended in March, on suspicion of alleged connections to Qatar and having organized a public relations campaign for the Gulf state, while it serves as a negotiator in the Israel-Hamas ceasefire talks and has close connections to the terrorist organization. Israel Police, in contrast, tried to establish that Urich indeed did function as a public servant, and that his actions undermined the interests of the PMO and the Israeli public at large. Police representative Supt. Aviv Porat said on Tuesday that materials coalesced in the interrogation indicate a 'significant suspicion that after October 7 and as criticism bloomed against Qatar for its connections with and involvement in Hamas, that the Gulf state reached out to Israeli figures to initiate a public-relations campaign to better its image in the eyes of the Israeli public.' Porat added that this alleged plan was approved by Qatari elements and kicked off. The plan allegedly included several courses of action, all aimed at shifting the impression Israelis had of the Gulf state post-October 7 - 'and Urich and other figures played a central role in the execution.' Urich's specific strength allegedly was formulating and pushing the messages out to the public. Porat emphasized that at the core of the issue is the nature of the messaging. 'If it was a different state working with Qatar on public relations for an electrical company, there would be a greater doubt as to whether state security was compromised,' he said. He specified, 'The suspicions that a man who was employed by the Likud and, in practice, worked in diplomacy in the PMO during the war, and, per the suspicions, was bribed by a foreign state to which he provided similar services - all without coming clean about it, not in the legal realm nor in the moral one.' Porat pointed out that several of the individuals who provided testimony in the case attested to Urich being a public servant. He added that there is more work required from investigators. When Michles responded that by that logic, the investigation will never reach a concluding point, Porat responded that the conditions under which Urich will be allowed to return to his place of work 'may enable an obstruction of the investigation.' The police representative added that by current estimates, the investigation against Urich is due to wrap up within the next 45 days. 'This is a man who is under investigation and may legitimately be allowed to return to the very same place from which he carried out the alleged crimes,' said Porat. Noa Milstein, Urich's lawyer, noted that anyone else who was interrogated or provided testimony was allowed to return to their place of work. She added that the list of forbidden contacts was never actually approved by the court, and that the list indicates 'the intentions of the interrogators.' Michles asked the police for an update, specifically on the hypothesis of him being a public servant. Last week, Rishon Lezion Magistrate's Court Judge Menahem Mizrahi released Urich from all restrictions - allowing him to return to work at the PMO and at Perception, the company at the heart of the Qatargate investigations - but gave police the option to appeal the decision. It did, leading up to the Tuesday hearing. 'Leaked documents affair' The crux of the investigation, as it has been released to the public so far, is the 'leaked documents affair,' in which Feldstein allegedly leaked classified military documents to the German tabloid Bild, after permission for their publication was denied by the Israeli military censor. This was allegedly done at the direction of Urich and Israel Einhorn, another aide who was interrogated in connection with the case last month, from where he resides in Serbia. The documents were eventually published, allegedly to sway public opinion on the hostage negotiations. This was around August 2024, when six hostages were killed by their Hamas captors in a tunnel: Hersh Goldberg-Polin, Almog Sarusi, Eden Yerushalmi, Ori Danino, Carmel Gat, and Alex Lobanov.
Business Wire
5 hours ago
- Business Wire
Gold Reserve Provides Update on Court Hearing Regarding Adjournment of Sale Hearing
PEMBROKE, Bermuda--(BUSINESS WIRE)--Gold Reserve Ltd. (TSX.V: GRZ) (BSX: (OTCQX: GDRZF) ('Gold Reserve' or the 'Company') announces that the U.S. District Court for the District of Delaware (the 'Court') held a hearing on August 18, 2025 to determine next steps in the Sale Process and decided inter alia the following: (1) the Sale Hearing would be rescheduled to commence September 15, with a second hearing to potentially take place commencing October 21; (2) the Special Master is to inform the Company whether it is recommending a different bid than the bid submitted by the Company's subsidiary, Dalinar Energy Corporation, by August 25, the Company may match any different bid by August 28, and the Special Master is to make any different, Final Recommendation to the Court by August 29, 2025; and (3) the Company may object to any of the foregoing actions by the Special Master and argument on any such objection will be heard by the Court at the September 15, 2025 hearing. An order by the Court reflecting the above and other related determinations is expected to be issued by the Court and will be posted here. A complete description of the Delaware sale proceedings can be found on the Public Access to Court Electronic Records system in Crystallex International Corporation v. Bolivarian Republic of Venezuela, 1:17-mc-00151-LPS (D. Del.) and its related proceedings. Cautionary Statement Regarding Forward-Looking statements This release contains 'forward-looking statements' within the meaning of applicable U.S. federal securities laws and 'forward-looking information' within the meaning of applicable Canadian provincial and territorial securities laws and state Gold Reserve's and its management's intentions, hopes, beliefs, expectations or predictions for the future. Forward-looking statements are necessarily based upon a number of estimates and assumptions that, while considered reasonable by management at this time, are inherently subject to significant business, economic and competitive uncertainties and contingencies. They are frequently characterized by words such as "anticipates", "plan", "continue", "expect", "project", "intend", "believe", "anticipate", "estimate", "may", "will", "potential", "proposed", "positioned" and other similar words, or statements that certain events or conditions "may" or "will" occur. Forward-looking statements contained in this press release include, but are not limited to, statements relating to any bid submitted by the Company for the purchase of the PDVH shares (the 'Bid'). We caution that such forward-looking statements involve known and unknown risks, uncertainties and other risks that may cause the actual events, outcomes or results of Gold Reserve to be materially different from our estimated outcomes, results, performance, or achievements expressed or implied by those forward-looking statements, including but not limited to: the discretion of the Special Master to consider the Bid, to enter into any discussions or negotiation with respect thereto; the Bid will not be approved by the Court as the 'Final Recommend Bid' under the Bidding Procedures, and if approved by the Court may not close, including as a result of not obtaining necessary regulatory approvals, including but not limited to any necessary approvals from the U.S. Office of Foreign Asset Control ('OFAC'), the U.S. Committee on Foreign Investment in the United States, the U.S. Federal Trade Commission or the TSX Venture Exchange; failure of the Company or any other party to obtain sufficient equity and/or debt financing or any required shareholders approvals for, or satisfy other conditions to effect, any transaction resulting from the Bid; that the Company may forfeit any cash amount deposit made due to failing to complete the Bid or otherwise; that the making of the Bid or any transaction resulting therefrom may involve unexpected costs, liabilities or delays; that, prior to or as a result of the completion of any transaction contemplated by the Bid, the business of the Company may experience significant disruptions due to transaction related uncertainty, industry conditions, tariff wars or other factors; the ability to enforce the writ of attachment granted to the Company; the timing set for various reports and/or other matters with respect to the Sale Process may not be met; the ability of the Company to otherwise participate in the Sale Process (and related costs associated therewith); the amount, if any, of proceeds associated with the Sale Process; the competing claims of other creditors of Venezuela, PDVSA and the Company, including any interest on such creditors' judgements and any priority afforded thereto; uncertainties with respect to possible settlements between Venezuela and other creditors and the impact of any such settlements on the amount of funds that may be available under the Sale Process; and the proceeds from the Sale Process may not be sufficient to satisfy the amounts outstanding under the Company's September 2014 arbitral award and/or corresponding November 15, 2015 U.S. judgement in full; and the ramifications of bankruptcy with respect to the Sale Process and/or the Company's claims, including as a result of the priority of other claims. This list is not exhaustive of the factors that may affect any of the Company's forward-looking statements. For a more detailed discussion of the risk factors affecting the Company's business, see the Company's Management's Discussion & Analysis for the year ended December 31, 2024 and other reports that have been filed on SEDAR+ and are available under the Company's profile at Investors are cautioned not to put undue reliance on forward-looking statements. All subsequent written and oral forward-looking statements attributable to Gold Reserve or persons acting on its behalf are expressly qualified in their entirety by this notice. Gold Reserve disclaims any intent or obligation to update publicly or otherwise revise any forward-looking statements or the foregoing list of assumptions or factors, whether as a result of new information, future events or otherwise, subject to its disclosure obligations under applicable rules promulgated by applicable Canadian provincial and territorial securities laws. For further information regarding Dalinar Energy, visit:
