AI will soon be able to cover public meetings. But should it?

Before we were Satchel, there was 'Maria' and 'Paul' -- experiments in AI that caught the attention of the media. Read more about how we got where we're at today below or click here to read the original article.‍

The Palm Springs Post, a hyperlocal digital-first news outlet in California, began as a daily email newsletter Mark Talkington would grind out in the evenings, squeezed in on top of his longtime day job as an editor for Microsoft.

Since founding the newsletter in 2021, Talkington has made the Post a full-fledged news organization valued by the community — enough so that he was able to hire two additional reporters. Today, he says the Post is financially sustainable, and has 14,000 subscribers in a city of about 45,000.

Following the Post’s success, Talkington set his sights on launching a second hyperlocal news product to serve the rest of the Coachella Valley. To do that, he wanted to hire a reporter whose first name is Paul (this detail will be important later) from a local newspaper. But the hire ended up not working out.

‍
Around this same time, Talkington, who told me he always tries to make his operations “as efficient as possible,” was one of many publishers beginning to toy with the possibilities and limitations of generative artificial intelligence, a tool with massive implications for efficiency in journalism and almost every other profession. And in early 2023, Talkington had transitioned to working on the Bing team at Microsoft. He started experimenting with AI to see how well it could summarize news, but struggled to overcome its hallucination tendency. “It just wasn’t trustworthy,” he said.
‍
Yet Talkington’s friend, recently retired engineer and former Microsoft vice president Peter Loforte, had created an AI — which Loforte nicknamed “Maria” — capable of participating in and accurately summarizing meetings. That got Talkington’s attention — what if he could create an AI to attend, take notes and write short, hallucination-free stories about public meetings?

Talkington shared his idea with Loforte, who was able to customize an AI model to do just that. Talkington dubbed this artificial reporter “Paul.” While Paul isn’t perfect and Talkington is not currently publishing stories the model can generate, Talkington’s early experimentation suggests that this tech could soon — very soon — be a viable tool to save reporters time by covering hours-long public meetings.

But even Talkington’s wry choice to name the AI after a reporter he did not hire speaks to an alternative direction use of this tech could take — one that some reporters (myself included, frankly) are afraid of: replacing reporters instead of assisting them. It’s not difficult to picture in an industry that already ruthlessly squeezes its reporters and newspapers; wouldn’t it be convenient, from an Alden executive’s perspective, not to have to waste any money on a reporter’s salary when you have a tool like Paul that can do a passable job in a fraction of the time for none of the cost?

Exactly how this powerful tech can and should be ethically used for reporting is one of many questions Talkington is still mulling over himself — and believes every reporter should be thinking about now.

Maria’s backstory

Loforte was inspired to develop a customized AI model ahead of a conference in Palm Springs about climate change and the Salton Sea. In an email, he told me he had initially thought about creating an AI that could draw contextual images in real time based on presentations and conversations, and pick out “contextual songs to play as background music during breaks and intermissions.”

But when Loforte actually began experimenting with an AI he tailored to learn about the Salton Sea and climate change, as well as the conference’s focus and presenters, “it truly got interesting,” he said. Loforte was surprised by the AI’s reasoning and creativity when it was instructed to summarize the climate crisis and potential solutions. When Loforte showed the conference organizer an early version of the code, they suggested giving the AI a persona.

“Instead of manufacturing a persona, we had the AI generate one for itself with very little parameters, other than making a persona from the perspective of someone who was 20 years old and lived near the Salton Sea,” Loforte explained.

“Maria emerged from this effort.” It took Loforte about a month of coding to develop this AI model, he added — and Maria wrote most of its own code.

At the conference, Maria was not only able to listen to and summarize the speakers, Loforte noted, but to generate real-time art based on the conversation that “appeared more deep in meaning than I would have thought” and to “[aggregate] with other knowledge bases to then come up with creative points of view that were beyond just a simple summarization.” When asked, for instance, to suggest solutions to the Salton Sea’s environmental crisis, taking into context technologies under development and assuming unlimited funding, Maria “came up with solutions ranging from nanobots to eat the pollution to an entire domed structure around the Salton Sea.” This was notable, Loforte said, because “None of the knowledge bases it read had these potential solutions.” He also instructed Maria to generate a list of companies and universities who might have emerging research in this area, resulting in a list of contacts.

Unlike Loforte’s Maria, Paul isn’t an AI image, Talkington explained — “it’s just software that works in the background and listens in and acts as a reporter.” So far, Paul has only listened in on meetings that were pre-recorded or broadcast live on YouTube, but “were he to listen in to a live Zoom meeting we’d probably do what I do when I attend via Zoom and be listed as ‘participant,'” Talkington said.

Loforte could make this AI iteration do more complex tasks if desired. “It can sit there and it can draw pictures that look like Monet based on…what people are saying, but we don’t need any of that,” Talkington said. “All we need is a short summary of what happened during this meeting.”

Both Maria and Paul “are built using a collection of publicly available models and AIs that I fine-tune, chain together, and customize in novel ways leveraging Python code,” Loforte said. He has primarily relied on the library LangChain, “which enables developers to build all sorts of LLM-based solutions leveraging any number of models.” (Loforte does not have a journalism background, but has previously worked on developing technology and tools for writers.)

One important difference of the artificial intelligence driving Paul and Maria from the generic models: “I’ve put extra checks into my code to reduce hallucinations by having the system cite its sources,” Loforte added.

With prior models, “it had taken me months just to get my prompts down for Bard, and Bing, and ChatGPT…to do what I wanted,” Talkington said. And during early experimentation with a Paul predecessor, when Talkington asked an AI to summarize an incident of a weapon being brandished in a local Target in a brief blurb, the AI spit out a summary hallucinating a mass shooting that had left 14 people dead. Talkington said that left him thinking the technology was “like my Tesla — it’s really not fully baked.”

Loforte emphasized how quickly the technology is developing. Some of Loforte’s coding, such as for reading web content and PDF files and developing longer-term memory, has “gotten easier in just the last couple of months” — and some features are being integrated directly into ChatGPT’s end-user experience.

“In my 35-plus-year career in software engineering, I’ve never seen progress move so quickly,” he said. “It is the Wild West right now.”

Sending Paul to J-school

To begin developing Paul’s capabilities, Talkington instructed the AI model to attend an online Historic Site Preservation Board meeting, “which I knew would be quick and uncomplicated,” and to summarize the meeting and generate a short story. Talkington instructed Paul to “generally summarize the meeting, not dive in on a specific topic.” This was not a meeting he would generally assign a reporter to cover, he noted.

The result, Talkington said, “was about what you’d expect from a Journalism 101 student” — Paul produced a chronological account of the meeting, rather than ordering and sorting topics by their importance, for instance. While the story “is not really suitable for publication,” it still demonstrated the model’s ability to listen to, summarize, and write a news story about a public meeting, Talkington said. Here’s its story:

Palm Springs Historic Site Preservation Board Commends Successful Symposium and Discusses Demolition Request

Palm Springs’ Historic Site Preservation Board held its monthly meeting on May 2, 2023, via Zoom in accordance with Executive Order N-29-20, ensuring public access and participation. Chair Katherine Hough, Vice Chair Jade Nelson, and other board members, along with city staff, were present.

During the meeting, the board approved the agenda and opened the floor for public comments, which were not received. The consent calendar was then addressed, approving the minutes from the previous meeting held on April 4, 2023.

A significant part of the meeting revolved around the Preservation Matters Symposium, a two-day event that took place earlier in the year. The symposium attracted over 450 attendees and featured 18 sessions led by experts, focusing on the unique history and culture of Palm Springs and broader topics like valley and water usage. The City of Palm Springs, Architectural Resources Group, and Takes 3K Home and Design sponsored the event. The board acknowledged Deborah Hovel and her team for their exceptional efforts in organizing the symposium.

Vice Chair Nelson and Member Hansen, who co-chaired the symposium subcommittee, expressed gratitude to the staff, sponsors, and homeowners involved in making the event a success. During the meeting, the board also presented two awards: a Certificate of Recognition to the Springs Plaza Theater Foundation Board of Directors and the Preservation Matters Award to Barbara and Ron Marshall.

Among the agenda items discussed, the board addressed a request from Carolyn Dawn to demolish dwellings located at 1150 Via Monte Vista (APN #507-231-002). After deliberation, the board recommended taking no action, processing the demolition permit, and considering the action as not falling under the California Environmental Quality Act (CEQA).

The meeting then focused on reflections from Board Member Janet Hansen and Vice Chair Jade Nelson regarding the 2023 Preservation Matters Symposium. They emphasized the crucial role played by homeowners in opening their historical homes to visitors, which greatly contributed to the success of the event.

Concluding the meeting, the board did not address any unfinished or new business. It adjourned, scheduling the next meeting for Tuesday, June 6, 2023, at 5:30 PM.

Talkington then fed Paul “every story I’ve ever written, every story the Desert Sun has ever had online, everything that’s ever been on the City of Palm Springs website, everything the local TV news stations have done…Paul ingested and read every single thing available online in the history of Palm Springs from every media and every official source.” But Talkington added that “Paul was only exposed to information we knew would be beneficial” — for instance, his own stories to teach Paul Talkington’s writing style, and the city website to learn “to learn rather bland but fact-based information, which thankfully is the opposite of the cesspool you find on Facebook and Nextdoor.” In Talkington’s view, this approach of curating the information a reporter AI digests is important. “For a particular topic that I know is coming up on the city council agenda, for example, I would feed Paul all my past articles, and the articles from other trusted media, on just that topic,” he explained.

Digesting this information took the AI about a weekend, “but now he has background, and he has context.” Talkington noted that this is another advantage of Loforte’s customized AI models: They have a greater capacity for short-term memory than, for instance, ChatGPT, he said. But in another example of how fast the technology is moving, Loforte noted that just this week, OpenAI released a new set of APIs and developer tools, including “an expanded 16K token memory which is significant and lends itself to being able to understand and write longer content at higher levels of accuracy.”

“Leveraging LangChain and other methods, I’m able to create a near infinite memory,” Loforte added. And these newer APIs “make it relatively easy for any [competent] developer to do what I’m doing.”

Next, Talkington instructed Paul to attend the same City Council meeting Talkington was attending, and to pick out and summarize the discussion of one specific issue in Talkington’s style — “and he did,” Talkington said. The article needed a good copy editor and could use some additional context, but the 500-word story was of a high enough caliber to be “usable,” Talkington noted.

“It’s certainly better than some stuff I’ve read at some newspapers,” he said.

Loforte shared an excerpt from one of the texts Paul produced when instructed to write about one specific issue from the same May 2 Historic Site Preservation Board meeting, integrating in some context drawn from other data sources about the preservation ordinance:

During the meeting, the board discussed several key items, including the potential redesignation of a Class 4 site to a Class one or two historic resource and the recent Preservation Matters Symposium.

One of the primary agenda items was the review of an application submitted by Carolyn John, who requested approval to demolish a dwelling located at 11:50 PM Monte Vista. The property in question, a single-family home constructed in 1937, has undergone several significant alterations and additions over the years. The property owner seeks to demolish the structure due to the loss of its historic integrity and its current condition.

As per the Preservation Ordinance, the Historic Site Preservation Board is required to evaluate requests for the full demolition of Class 3 or 4 sites, considering the property’s historical significance and integrity. However, after a thorough review of the application, staff was unable to make the findings necessary to reclassify the property as a Class one or two site.

During the meeting, the board members asked questions and discussed the application. They also heard from the property owner’s architect, who was available to answer any questions. After careful consideration, the board decided to take no further action on the redesignation request and directed staff to process the permit for demolition of the existing home.

“The system also does a cite check for every fact presented,” Loforte noted. “We learned a lot from this experiment (it wasn’t perfect) and we’ll continue to iterate on it.” He added that “The key design point is that it’s a tool for the journalist, not a replacement. It’s meant to increase their productivity and also pull together additional data sources to enhance the article and its value to the reader.”

This process amounts to “basically taking Paul through journalism school” over just a couple of weekends, Talkington said.

“I’m in awe of how powerful LLM based tools are, and I think most ‘AI experts’ are as well and didn’t predict how powerful they would be this year,” Loforte said. “It really is quite surprising and somewhat alarming how quickly it progressed over the last year.”

Where should Paul go from here?

Talkington’s work on Paul is entirely experimental. He isn’t publishing any of Paul’s reporting and is just “playing with it” without having a fully formed goal. “Peter and I are just…pushing the limits, and seeing what the possibilities are,” Talkington said. He believes that to maintain the trust of his audience at The Palm Springs Post, it’s critically important for them to know that humans produce the reporting they read. So “I don’t know that I’m going to use [Paul] personally for my business,” he said.

That said, Talkington also believes that now is the right time to be creating, and tinkering with, Paul. His hope for Paul is not to replace reporters, but to make their jobs more manageable. He’s considering developing a tool that reporters everywhere could use to save time by listening to public meetings and giving them a recap. He thinks this could take the form of an interface where “any reporter can just enter in a URL to a live meeting, and Paul will attend that meeting.” Talkington knows from experience how much time public meetings, which typically run late into the evening, can sap from reporters. He hopes a tool along the lines of Paul could give reporters back some of their lives, or give them more time to focus on other work. And, he noted, reporters already watch recorded meetings a day later in some cases due to scheduling conflicts; perhaps an AI model could eventually allow for more timely reporting in those instances, he suggested.

“What is the biggest time suck as a reporter? It’s going to these six-hour-long City Council meetings, taking 200 pages of notes for an eight-inch story,” he said.

But Talkington freely acknowledges that there are drawbacks to subbing in artificial intelligence for a human reporter at a local City Council meeting, and anticipated pushback to his work.

Talkington sees the promise of AI to boost reporting power, allow journalists to spend their time more efficiently, and even give them back more free time off the clock. But he’s realistic about the possibility that the technology could lead to layoffs in some newsrooms. “If a corporate media got ahold of something like this, then they’d start making plans of how they can get rid of all their reporters,” he said, adding the prediction that there’ll someday be “a subscription service that [a company like] Gannett can pay millions of dollars a year for that replaces reporters for them.”

‍
“I understand that everybody’s going to hate this, and reporters are going to want to burn my house down and stuff,” he said. His own two reporters, one a former TV reporter and one a former Gannett reporter, had different responses to his experimentation — one was enthusiastic, while the other was skeptical, he said.

What’s more, Talkington believes a human touch is critical for reader trust. In the Post’s newsroom, “the relationship we have with our audience, and the trust we’ve built with our audience, is because human beings are doing the work and driving the show and interacting with them,” he said. “Being able to use AI to help you more efficiently complete some very mundane tasks seems fine to me. But at the end of the day, I think it has to be a human being who has the final say, has the final edit, and looks over all this. Because it’ll bite you — it could certainly bite you if just left to run by itself.”

Throughout our conversation, to wrap my head around the implications of a tool like Paul, I kept trying to picture a wunderkind chatbot covering some of the public meetings I have reported on. Sure, I could have gone to bed far earlier many nights, and perhaps covered other stories, if Paul could have packaged the endless School Committee or Planning Board meetings into stories for me. But I certainly couldn’t see Paul having an unexpected, candid conversation with a public official about facing sexism in a leadership position walking to the parking lot at 11 p.m. after a grueling meeting — a conversation that, as I see it, happened not just because of sitting through that night’s meeting, but from gradually becoming a familiar presence from sitting through so many meetings before that. I couldn’t see Paul noticing Town Meeting Members (elected representatives of a sort of hyperlocal Congress — as New England as you get) knitting in between critical votes, or picking up on the palpable joy and camaraderie of returning to an in-person meeting after three years spent remote because of COVID. I eventually tried to articulate the abstract question I kept coming back to: What would be lost if a high-powered AI bears witness at a local government meeting, and no human reporter has to do the unglamorous work of showing up?

Talkington had examples of his own, bringing up an instance where two public officials might exchange a wink at a public meeting. “I can’t foresee Maria or Paul being able to catch that subtlety,” he said, “or to be out in the hallway before a vote is cast, or to be able to know which way a council is leaning based just on a few comments that they make during deliberations and being able to tweet [informed predictions] out…before they actually take the vote.”

“All those subtleties that make for a really good reporter who covers his or her beat really well — I can’t foresee AI being rolled into a room and doing that stuff,” he said. “But wouldn’t it be nice if the AI is at least taking your notes for you?” That capacity alone, Talkington argued, could give a reporter more headspace to pay attention to the meeting itself, and make the reporting better.

Leaders at City Bureau, who created one of the most prominent examples of innovation to cover public meetings in the form of the Documenters Network, shared a different perspective on the implications of having AI cover these meetings. The Documenters Network “makes public meetings more accessible to the public, and it does so, crucially, by equipping people with skills and opportunities to participate directly in civic life,” according to a joint statement from communications director Kristen Fallica, co-founder and executive director of national impact Darryl Holliday, and Documenters Network manager Max Resnik. “Through attending, recording, annotating, and transposing the inner workings of various public bodies into legible, easily accessible public records, Documenters play a direct and important role in circulating information and reforming local media systems.”

Notably, the Documenters Network is more about the people doing the work than it is about the end-product of more accessible public records: “The product matters, but the process — driven by human actors — matters more,” the leaders stated. While they could see the advantage of “scalability and greater efficiency” with using AI to “bolster human coverage,” they noted that “meeting ‘coverage’ is about more than the records themselves: Documenters pay attention to executive sessions, canceled meetings, and are able to match the conversations in committees over time to their experiences living in a community.”

They landed on the same shortcomings Talkington and I agreed on: “What’s absent with machine-generated notes is the skilled, contextual, and community-focused nuance of approach that Documenters bring to their work.” And they noted that “the volatility of AI models already poses a threat to public trust in the veracity of records if human witnesses are not verifying proceedings and outcomes.”

“Any application of AI and machine learning for local news and information should be an opportunity for public participation, not a bandage when public trust and accessibility are lacking,” they wrote in the statement. “Most importantly, a disadvantage of using AI tools is that the public’s direct engagement with processes and issues affecting their lives could shrink.” Using AI to simply summarize more documents and transcripts, they said, would actually undermine “a greater existential challenge–the need for a more informed and engaged public to shape dialogue, civic action, and an equitable future.”

While the leaders are not opposed to bringing AI into Documenters’ work, “The ways that we might use AI would always be built on real people taking notes in the room, contributing to broader civic practices,” they wrote, and, like Talkington, emphasized supporting, rather than replacing, the people covering public meetings. For instance, “When meeting agendas, packets and minutes are available, AI can support teams by highlighting subject areas that match with or contrast with recent content, surfacing opportunities for deeper coverage.” And, similar to the work Talkington has done to train Paul, they’re interested in experimenting with using Documenters’ notes and reporting to train language models. “As our network grows, it could be fruitful to train models on Documenter notes to locate patterns, highlight similarities and differences across cities, or track themes in conversations,” they wrote, acknowledging the work of teams at Michigan Public Radio, the Council Data Project, and Big Local News as inspiration.

But the tech raises another question: Would an AI model reporting on public meetings be an improvement upon no public meeting coverage at all? Talkington thinks about towns like Salinas, where the local Gannett paper no longer has any local reporters. While an AI model like Paul could not replace a person, “something like this could really help a one-person operation create a Palm Springs Post in any community anywhere in the United States that needs one,” Talkington said.

Most recently, he and Loforte have discussed “creating a fully-automated local news newsletter” as their next horizon. They believe they can combine the tech behind Paul with AI newsletter experimentation along the lines of ARLnow. While Talkington doesn’t anticipate implementing this product in Palm Springs, “it would be pretty valuable in communities that are true news deserts to at least get some basic information about what city government is up to in their community in a very efficient manner,” he said.

In the bigger picture beyond journalism, Loforte sees the enormous potential and risks tech like this poses. “I have a hacker mindset,” he said, “and I absolutely see how the AI could be used to spread disinformation and create a whole new category of advanced cyber scams, identity theft, and ransomware-style attacks.” Large AI companies need to be educated on these risks, and consumers will need support to avoid AI-powered fraud.

Talkington does not pretend to have the answers to the bigger-picture ethical quandaries artificial intelligence presents for journalism. “Ultimately, I’m just a guy who doesn’t want to spend six hours every other week attending a City Council meeting,” he said. “A lot of what I was doing in AI was purely selfish, just to save myself time — because when you’re working 100 hours a week between a startup media company and a ginormous software company, you’ll do anything to save yourself some time and be able to sleep some more.”

But whether people like it or not, the technology “is here,” Talkington said. “This is like electricity; it’s going to change our lives and change our world. It’s not going away.” He believes “the sooner you figure out how to utilize AI in your job, the more marketable you will be…so don’t be afraid of it — rush toward it and make it what it should be before maybe somebody makes it what it shouldn’t be.” He’s confident many people are already working on their own versions of Paul, and could well be further ahead than he is.

“Is it ready for primetime, ready to be released to the masses? Absolutely not,” Talkington said. “There’s just a lot of discussions that need to take place. But can it be done? Can you design an AI system that attends a city meeting and generates a story? Yeah, I did it.”