Students have been called to the office — and even arrested — for AI surveillance false alarms
Lesley Mathis knows what her daughter said was wrong. But she never expected the 13-year-old girl would get arrested for it.
The teenage girl made an offensive joke while chatting online with her classmates, triggering the school's surveillance software.
Before the morning was even over, the Tennessee eighth grader was under arrest. She was interrogated, strip-searched and spent the night in a jail cell, her mother says.
Earlier in the day, her friends had teased the teen about her tanned complexion and called her 'Mexican,' even though she's not. When a friend asked what she was planning for Thursday, she wrote: 'on Thursday we kill all the Mexico's.'
Mathis said the comments were 'wrong' and 'stupid,' but context showed they were not a threat.
'It made me feel like, is this the America we live in?' Mathis said of her daughter's arrest. 'And it was this stupid, stupid technology that is just going through picking up random words and not looking at context.'
Surveillance systems in American schools increasingly monitor everything students write on school accounts and devices. Thousands of school districts across the country use software like Gaggle and Lightspeed Alert to track kids' online activities, looking for signs they might hurt themselves or others. With the help of artificial intelligence, technology can dip into online conversations and immediately notify both school officials and law enforcement.
Educators say the technology has saved lives. But critics warn it can criminalize children for careless words.
"It has routinized law enforcement access and presence in students' lives, including in their home,' said Elizabeth Laird, a director at the Center for Democracy and Technology.
Schools ratchet up vigilance for threats
In a country weary of school shootings, several states have taken a harder line on threats to schools. Among them is Tennessee, which passed a 2023 zero-tolerance law requiring any threat of mass violence against a school to be reported immediately to law enforcement.
The 13-year-old girl arrested in August 2023 had been texting with friends on a chat function tied to her school email at Fairview Middle School, which uses Gaggle to monitor students' accounts. (The Associated Press is withholding the girl's name to protect her privacy. The school district did not respond to a request for comment.)
Taken to jail, the teen was interrogated and strip-searched, and her parents weren't allowed to talk to her until the next day, according to a lawsuit they filed against the school system. She didn't know why her parents weren't there.
'She told me afterwards, 'I thought you hated me.' That kind of haunts you,' said Mathis, the girl's mother.
A court ordered eight weeks of house arrest, a psychological evaluation and 20 days at an alternative school for the girl.
Gaggle's CEO, Jeff Patterson, said in an interview that the school system did not use Gaggle the way it is intended. The purpose is to find early warning signs and intervene before problems escalate to law enforcement, he said.
'I wish that was treated as a teachable moment, not a law enforcement moment,' said Patterson.
Private student chats face unexpected scrutiny
Students who think they are chatting privately among friends often do not realize they are under constant surveillance, said Shahar Pasch, an education lawyer in Florida.
One teenage girl she represented made a joke about school shootings on a private Snapchat story. Snapchat's automated detection software picked up the comment, the company alerted the FBI, and the girl was arrested on school grounds within hours.
Alexa Manganiotis, 16, said she was startled by how quickly monitoring software works. West Palm Beach's Dreyfoos School of the Arts, which she attends, last year piloted Lightspeed Alert, a surveillance program. Interviewing a teacher for her school newspaper, Alexa discovered two students once typed something threatening about that teacher on a school computer, then deleted it. Lightspeed picked it up, and 'they were taken away like five minutes later,' Alexa said.
Teenagers face steeper consequences than adults for what they write online, Alexa said.
'If an adult makes a super racist joke that's threatening on their computer, they can delete it, and they wouldn't be arrested," she said.
Amy Bennett, chief of staff for Lightspeed Systems, said that the software helps understaffed schools 'be proactive rather than punitive' by identifying early warning signs of bullying, self-harm, violence or abuse.
The technology can also involve law enforcement in responses to mental health crises. In Florida's Polk County Schools, a district of more than 100,000 students, the school safety program received nearly 500 Gaggle alerts over four years, officers said in public Board of Education meetings. This led to 72 involuntary hospitalization cases under the Baker Act, a state law that allows authorities to require mental health evaluations for people against their will if they pose a risk to themselves or others.
'A really high number of children who experience involuntary examination remember it as a really traumatic and damaging experience — not something that helps them with their mental health care,' said Sam Boyd, an attorney with the Southern Poverty Law Center. The Polk and West Palm Beach school districts did not provide comments.
An analysis shows a high rate of false alarms
Information that could allow schools to assess the software's effectiveness, such as the rate of false alerts, is closely held by technology companies and unavailable publicly unless schools track the data themselves.
Gaggle alerted more than 1,200 incidents to the Lawrence, Kansas, school district in a recent 10-month period. But almost two-thirds of those alerts were deemed by school officials to be nonissues — including over 200 false alarms from student homework, according to an Associated Press analysis of data received via a public records request.
Students in one photography class were called to the principal's office over concerns Gaggle had detected nudity. The photos had been automatically deleted from the students' Google Drives, but students who had backups of the flagged images on their own devices showed it was a false alarm. District officials said they later adjusted the software's settings to reduce false alerts.
Natasha Torkzaban, who graduated in 2024, said she was flagged for editing a friend's college essay because it had the words 'mental health.'
'I think ideally we wouldn't stick a new and shiny solution of AI on a deep-rooted issue of teenage mental health and the suicide rates in America, but that's where we're at right now,' Torkzaban said. She was among a group of student journalists and artists at Lawrence High School who filed a lawsuit against the school system last week, alleging Gaggle subjected them to unconstitutional surveillance.
School officials have said they take concerns about Gaggle seriously, but also say the technology has detected dozens of imminent threats of suicide or violence.
'Sometimes you have to look at the trade for the greater good,' said Board of Education member Anne Costello in a July 2024 board meeting.
Two years after their ordeal, Mathis said her daughter is doing better, although she's still 'terrified' of running into one of the school officers who arrested her. One bright spot, she said, was the compassion of the teachers at her daughter's alternative school. They took time every day to let the kids share their feelings and frustrations, without judgment.
'It's like we just want kids to be these little soldiers, and they're not,' said Mathis. 'They're just humans.'
___
This reporting reviewed school board meetings posted on YouTube, courtesy of DistrictView, a dataset created by researchers Tyler Simko, Mirya Holman and Rebecca Johnson.
___
The Associated Press' education coverage receives financial support from multiple private foundations. AP is solely responsible for all content. Find AP's standards for working with philanthropies, a list of supporters and funded coverage areas at AP.org.
Hashtags
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
New York Post
9 minutes ago
- New York Post
Intel CEO Lip-Bu Tan to visit White House on Monday after Trump called for his ouster: report
Intel CEO Lip-Bu Tan is set to visit the White House on Monday after President Trump called for his removal last week, the Wall Street Journal reported on Sunday, citing people familiar with the matter. Reuters could not immediately confirm the report. Intel and the White House did not immediately respond to requests for comment. Tan is expected to have a extensive conversation with Trump while looking to explain his personal and professional background, the report said, adding that he could propose ways Intel and the US government could work together. Intel CEO Lip-Bu Tan is reportedly expected to have a extensive conversation with President Trump on Monday. REUTERS Tan hopes to win Trump's approval by showing his commitment to the US and guaranteeing the importance of keeping Intel's manufacturing capabilities as a national security issue, the report added. Last week, Trump demanded the immediate resignation of Tan, calling him 'highly conflicted' due to his ties to Chinese firms and raising doubts about plans to turn around the struggling American chip icon. Tan said he shared the president's commitment to advancing US national and economic security. Trump's intervention marked a rare instance of a US president publicly calling for a CEO's ouster and sparked debate among investors. Reuters reported exclusively in April that Tan invested at least $200 million in hundreds of Chinese advanced manufacturing and chip firms, some of which were linked to the Chinese military. President Trump has demanded Tan resign as Intel CEO, calling him 'highly conflicted' due to his ties to companies in China. AP Tan, a Malaysian-born Chinese American business executive, was also the CEO of Cadence Design from 2008 through December 2021, during which the chip design software maker sold products to a Chinese military university believed to be involved in simulating nuclear explosions. Last month, Cadence agreed to plead guilty and pay more than $140 million to resolve the US charges over the sales, which Reuters first reported.
Boston Globe
3 hours ago
- Boston Globe
These are drug cartels designated as terrorists by the US
Sinaloa Cartel Based in Sinaloa state, in western Mexico Founded more than 30 years ago by Joaquin Guzmán Loera, the drug lord known as El Chapo, and Ismael Zambada García, a boss known as El Mayo, the Sinaloa Cartel has long stood as one of the world's most formidable criminal syndicates. Coordinating an umbrella of criminal cells — not only from Sinaloa state but also from across Mexico, with partners, associates, and operatives around the world — the cartel has become a leading producer of fentanyl, the opioid that has had a devastating effect in the United States. Advertisement But after years evading capture, El Chapo is now serving a life sentence in the United States, and last year, one of his sons abducted Zambada, handing him to US authorities. That set off a war within the cartel — now taking place as the Mexican government, pushed by the Trump administration, is aggressively cracking down on the group. Advertisement Still, the cartel has found ways to adapt and continues to smuggle fentanyl and other drugs into the United States, illustrating how difficult it would be to uproot its network. Jalisco New Generation Cartel Based in Jalisco state, in western Mexico The Jalisco New Generation Cartel, long an archrival to the Sinaloa Cartel, now looks poised to become one of the world's largest drug traffickers through an alliance with a Sinaloa faction controlled by El Chapo's sons. The Jalisco cartel has entrenched itself in large parts of the illegal and legal sectors of Mexico's economy, profiting off not just drugs and weapons but also real estate, avocados, and timeshares. The group has a well-developed network across the Americas, with links into Australia, China, and Southeast Asia, according to the US State Department. Like its competitors, the gang has used brazen acts of violence to maintain control. The group is believed to be linked to the 2020 assassination of the former governor of the state of Jalisco, as well as a training and recruitment camp discovered in the state this year. Cartel del Noreste Based in Nuevo Leon state, in northeastern Mexico The Cartel del Noreste began as the Zetas, violent enforcers of another group. In the early 2000s, the Zetas gained prominence by using violence to send public messages, and by 2012 controlled large swaths of territory. Internal rivalries and the killing of its leader by the Mexican Marines in 2012 appeared to weaken the Zetas. But a splinter of the gang reemerged as Cartel del Noreste, which operated across both sides of the border, trafficking drugs, weapons, and migrants across the border for enormous profits. Advertisement Tren de Aragua Originated in Venezuela From a prison in a northern Venezuelan state, Tren de Aragua's network and influence has spread to other parts of Latin America, and the group has become known for exploiting vulnerable migrants through trafficking and kidnapping. The Biden administration labeled the gang a transnational criminal organization in 2024, and in the United States, people accused of affiliations with it have been charged with crimes such as shootings and human trafficking, mostly targeting members of the Venezuelan community. The Trump administration has made the gang a focus of its deportation efforts and rhetoric, which Venezuelan asylum-seekers say casts an atmosphere of suspicion and stigma over those fleeing violence and political repression. The group has been said to work with another organization, the Cartel de los Soles, that the Trump administration says has ties to Venezuela's president, Nicolás Maduro. However, experts say little is known about such a group, and Venezuela's defense minister dismissed it Friday as an 'invention' by the US government. MS-13 Linked to El Salvador MS-13 emerged on the streets of Los Angeles in the 1980s as a criminal network among immigrants from El Salvador and has shifted its power base to Central America. The group was the first criminal street gang designated as a transnational criminal organization by the US Treasury Department in 2012. It was among the gangs that created chaos and violence in El Salvador for many years, until President Nayib Bukele cracked down on criminal groups through a campaign of mass arrests starting in 2022. The group was a primary target of Trump during his first term, and officials in his second term have continued to prosecute cases. But Bukele has also helped Trump's deportation efforts, for which the United States has paid El Salvador millions of dollars, adding an important sweetener at Bukele's request: the return of key MS-13 leaders in US custody. Advertisement US prosecutors have amassed substantial evidence of a corrupt pact between the Salvadoran government and some high-ranking MS-13 leaders, who they say agreed to drive down violence and bolster Bukele politically. Bukele has denied the existence of such a pact. Gulf Cartel Based in Tamaulipas state, in northern Mexico One of the oldest criminal organizations in Mexico, the gang for years made much of its money and reputation from smuggling cocaine and marijuana across the US border. In the early 2000s, it was one of three main groups behind Mexico's gang wars. Fighting between the Gulf Cartel and their onetime enforcers, the Zetas, turned parts of the region into a battle zone, sending thousands fleeing. In 2012, Mexico celebrated the arrest of the gang's top leader, but his detention, and that of other Gulf Cartel members, created a vacuum that was quickly filled by other leaders. La Nueva Familia Michoacana Based in Michoacán state, in western Mexico La Nueva Familia Michoacana rapidly rose to power in Mexico's crowded drug wars. In their home state of Michoacán, the group made much of its money through kidnapping and extortion. As their control spread, they employed bloodier tactics to enforce their power, sometimes attaching notes to the bodies of beheaded victims. Last year, the US Treasury Department imposed sanctions on the group's leaders as part of an operation to target fentanyl trafficking in the United States. In April, US prosecutors charged senior members of the gang with conspiracy to manufacture and distribute fentanyl, heroin, cocaine, and methamphetamine. Advertisement This article originally appeared in
American Press
3 hours ago
- American Press
CDC shooter believed Covid vaccine made him suicidal
A Georgia man who had blamed the COVID-19 vaccine for making him depressed and suicidal has been identified as the shooter who opened fire late Friday on the U.S. Centers for Disease Control and Prevention headquarters, killing a police officer. The 30-year-old suspect, who died during the incident, had also tried to get into the CDC's headquarters in Atlanta but was stopped by guards before driving to a pharmacy across the street and opening fire, a law enforcement official told The Associated Press on Saturday. The man, identified as Patrick Joseph White, was armed with five guns, including at least one long gun, the official said, speaking on condition of anonymity because they were not authorized to publicly discuss the investigation. A union representing workers at the CDC said the incident was not random and 'compounds months of mistreatment, neglect, and vilification that CDC staff have endured.' It demanded federal officials condemn vaccine misinformation, saying it was putting scientists at risk. Here's what to know about the shooting and the continuing investigation: An attack on a prominent public health institution Police say White opened fire outside the CDC headquarters in Atlanta on Friday, leaving bullet marks in windows across the sprawling campus. At least four CDC buildings were hit, Director Susan Monarez said on X. DeKalb County Police Officer David Rose was mortally wounded while responding. Rose, 33, a former Marine who served in Afghanistan, had graduated from the police academy in March. White was found on the second floor of a building across the street from the CDC campus and died at the scene, Atlanta Police Chief Darin Schierbaum said. He added that 'we do not know at this time whether that was from officers or if it was self-inflicted.' The Georgia Bureau of Investigations said the crime scene was 'complex' and the investigation would take 'an extended period of time.' CDC union calls for condemnation of vaccine misinformation and tighter security The American Federation of Government Employees, Local 2883, said the CDC and leadership of the Department of Health and Human Services must provide a 'clear and unequivocal stance in condemning vaccine disinformation.' Such a public statement by federal officials is needed to help prevent violence against scientists, the union said in a news release. 'Their leadership is critical in reinforcing public trust and ensuring that accurate, science-based information prevails,' the union said. Fired But Fighting, a group of laid-off CDC employees, has said HHS Secretary Robert F. Kennedy Jr. is directly responsible for the villainization of CDC's workforce through 'his continuous lies about science and vaccine safety, which have fueled a climate of hostility and mistrust.' Kennedy reached out to staff on Saturday, saying 'no one should face violence while working to protect the health of others.' Thousands of people who work on critical disease research are employed on the campus. The union said some staff were huddled in various buildings until late at night, including more than 90 young children who were locked down inside the CDC's Clifton School. The union said CDC staff should not be required to immediately return to work after experiencing such a traumatic event. In a statement released Saturday, it said windows and buildings should first be fixed and made 'completely secure.' 'Staff should not be required to work next to bullet holes,' the union said. 'Forcing a return under these conditions risks re-traumatizing staff by exposing them to the reminders of the horrific shooting they endured.' The union also called for 'perimeter security on all campuses' until the investigation is fully completed and shared with staff.
