STORY: Oliners Collaborate with Local Senior Center on Artificial Intelligence (AI) and Augmented Reality (AR) Innovations

May 27, 2025

Andrea Cuadra '13 and her students are working on technological interfaces that integrate the needs and preferences of older adults.

Students working in Olin’s Pl(ai) Lab, run by Andrea Cuadra, assistant professor of computer science, are collaborating with The Center at the Heights, a senior citizen center in Needham, Mass., on pioneering AI technological innovations.

“There is a trend in human-computer interaction that embraces society-centered design, which prioritizes the active involvement and needs of a community in the design process,” says Cuadra, whose expertise lies at the intersection of computing and design, especially with speech-based interfaces. 

Personalized Help with Digital Technology

One of the projects students and residents are working on is called AI Sidekick, which employs AI and AR to help users overcome frustrations with cumbersome digital user interfaces by talking through different issues that require additional technical support and overlaying visual guides on their screens. 

“We created AI Sidekick by combining OpenAI’s natural language processing to interpret spoken questions with a smartphone user manual, transforming it into an interactive guide that provides real-time voice and visual screen overlays,” says Esther Aduamah ’27, a Bioengineering (BioE) major using computer science to advance equitable health policy. “The goal is that when the user gets stuck with a smartphone task they’re trying to complete—such as making a doctor’s appointment or downloading a game app—they would put on a pair of AR glasses that can ‘see’ the screen, which understands where the process stalled and gently guides them through the steps to complete their task.” 

Over time, the AI Sidekick would ideally develop an understanding of its user and know what they know, so it would not try to reexplain things that they already worked on together.

Throughout the semester, students have been visiting the Center to provide tech support on any issues clients may encounter and to collaboratively design the projects they are working on. 

“By developing these relationships, students are also developing a unique sensitivity to the needs of people much older than themselves. Often, they realize how similar their needs are, such as phones running out of storage or allocating the time to learn how to use a new feature on our watches,” says Cuadra.

Supporting Medical Recovery with AI & AR

Another project from Cuadra’s lab is the Pl(ai)ful Movement Interface, which guides patients who are recovering from cancer treatment through a series of therapeutic exercises. This project is part of a collaboration with Memorial Sloan Kettering Cancer Center (MSK) with support from a Healthy Longevity Catalyst Award from the National Academy of Medicine (NAM) received in 2023.

 

“Specifically designed for people recovering from cancer treatment who spend at least half the day sitting in a chair or mostly in bed, Pl(ai)ful Movement is a way to make sure that patients are engaging in their exercises in a playful and safe way,” says Elías López Dalla Nora ’27, a Bioengineering (BioE) major at Olin who is part of the project.

"The initial prototype I created ran on my laptop and used its camera to capture the user’s movement, simulating their motions through visual overlays on the screen,” he added.

For the simulation component, he used an open-source software called MediaPipe to track the position of certain points on the patient’s body, such as joints or fingers, to make sure the exercises are being performed correctly and safely.

Now, after López Dalla Nora’s summer research ended and he brought peers onto the project, the students have transitioned this technology to a smartphone app that can interface with AR glasses so that users can see the composite image of themselves reflected right in front of their eyes.

“The main strength of switching to the AR glasses is that users can talk directly to the interface if they need to adapt exercises or skip them for any reason,” says López Dalla Nora. “We also hope to create software that can change the user’s visual experience in the moment using voice commands—instead of Pac-Man eating strawberries, maybe they want to use the visual of pushing away rocks or picking flowers, for example.”

“We collaborate with health experts at MSK to make sure we are working on pressing needs,” says Cuadra. “A physical therapist curates the list of exercises, and we generate design guidelines based on our research. For example, we now know that the interface should be able to adjust the pace of the exercises and give personalized feedback in real time.”

In May, three Olin students traveled to Stanford University to showcase the Pl(ai)ful Movement Interface at the School of Engineering’s Centennial celebration. Approximately 3,000 people attended.

“Our booth got a lot of traffic, including students, alumni, professors, venture capitalists, student journalists from a local college, and even children and parents,” says López Dalla Nora. “We were almost always engaged in something, whether it was presenting, running the demo, or answering questions.”

Engineering for Everyone

An important element of both projects is inclusive design—a hallmark of Olin’s vision of Engineering for Everyone that is exemplified in other research projects and in the curriculum, such as the Technology, Accessibility, and Design course.

“It’s critical as engineers to look at the emerging technologies with potential to become mainstream and see who they are serving and who they are not,” says Cuadra. “Older adults are often not at the table when designing these technologies, yet they stand to highly benefit or be harmed by them. A big part of my research and something I try to share with my students is investigating ways in which we may be able to integrate the needs and preferences of more people into the design of the technologies that are being built.”

This article was reviewed by Dr. Cuadra, the students quoted, and the Pl(ai) Lab’s collaborators at MSK and the Center.