Ryo Suzuki
University of Colorado Boulder
Saturday, April 26th, 2025
Yokohama, Japan
Augmented Reality (AR) technologies are continuously advancing, both in terms of hardware and software. Smaller form factors, extended battery life, and constant connectivity make it possible to results in a paradigm shift in how we think about AR: while current AR scenarios are focused on specialized applications such as productivity or maintenance, we believe that everyday AR is increasingly feasible. With everyday AR we refer to the making AR always available to users, enabling seamless interactions with the digital world, and potentially replacing or augmenting currently predominant technologies such as smartphone or desktop computing. Users will be able to ubiquitously query and interact with digital information, communicate with other users and virtual agents, and leverage AR for a majority of their interactions.
This rise of everyday AR is fueled not just by improvements in hardware and software infrastructure, but is also heavily tied to recent advances in Artificial Intelligence (AI) and Machine Learning (ML). Generative AI techniques, for example, enable the creation of multi-modal digital content on-the-fly; and Large Language Models lead to advancements in various applications domains and make it feasible for users to interact with virtual agents through text and other inputs in a "natural" way. Leveraging advances in those areas will enable researchers and practitioners to develop new and refine existing interactions for AR. We believe that in order to make everyday AR feasible, we need to adopt an AI-in-the-loop approach, where the digital interactions and content continuously anticipate and adapt to users every-changing needs and context. Thus, we believe the combination of AR and AI will enable us to move beyond monolithic applications in AR towards a truly user-centered and adaptive experience where both scene understanding and content generation becomes dynamic.
The goal of this workshop, which is an evolution of our previous XR and AI Workshop at ACM UIST 2023, is to bridge the fields of AR and AI and enable discussions around the feasibility and requirements for AI-enabled everyday AR. Our previous workshop was broadly centered around the topic of what is possible when AR and AI are combined, the second iteration focuses on everyday AR interactions. We plan to explore usage scenarios of this new paradigm, explore requirements and limitations of current and future AR with AI-in-the-loop, and identify and address common challenges of AR (e.g., accessibility, privacy) and AI (e.g., explainability, sustainability). In that sense, we consider this workshop as a natural evolution, not only we meet but also define a way forwards for the community. This workshop seeks to further refine the thinking of the community towards a shared vision, and should lay the foundation for future workshops at conferences, longer seminars, and work towards a reference paper to consolidate the research efforts. As such we propose a workshop very oriented to shared vision, and make a call for participation in which we ask participants to share how they imagine the future of AR as we move to a world where AR is to AI what screens has been to computers.
Read the Proposal PDF
| Category | Details |
|---|---|
| Application Requirements |
Complete an application form (background, experience, and interests in AR/AI) Optional: Submit a position paper |
| Submission Format | Up to 2 pages, ACM double-column format, without references (for optional position papers) |
| Application Deadline | February 15th, 2025 |
| Participant Selection | Based on relevance of research/background and diversity of participants |
| Attendance Requirements | Participants must attend in person in Yokohama and register for both the workshop and at least one day of the CHI 2025 conference |
This one-day, in-person CHI 2025 workshop invites researchers, practitioners, and students interested in exploring the future of AI-enabled augmented reality (AR) to join us in shaping a vision for "Everyday AR." We aim to create a shared vocabulary and research agenda for AI-driven AR, inspiring new directions and fostering future collaborations. To apply for participation, interested individuals should complete an application form indicating their background, experience, and interests in AR and AI by February 15th. Applicants are encouraged to optionally submit a position paper of up to 2 pages (without references) in the ACM double-column format. Participants will be selected based on their application materials, with a focus on the relevance of their research or disciplinary background to the workshop topic, and a goal of creating a diverse and academically engaging environment.
Note that per CHI 2025 policies, workshop participants (including at least one author of each position paper, for applications that optionally include position papers) must attend the workshop in Yokohama, and all participants must register for both the workshop and for at least one day of the conference.
One of the primary goals is to foster a strong community around this emerging topic, leveraging the networking opportunities provided by face-to-face interactions, such as a group lunch and dinner. To ensure an intimate and engaging workshop experience, we plan to cap attendance at about 30 participants.
This workshop welcomes HCI researchers and practitioners in AR, AI, machine learning, and computational interaction domains to share diverse perspectives and expertise. There are several domains that are not fully explored yet in the literature of XR and AI. We plan to discuss the topics that include but are not limited to the following areas:
Adaptive and Context-Aware AR:
Unlike traditional interfaces on smartphones or computers, AR can embed virtual elements into the physical world. However, poorly designed AR interfaces risk overwhelming or distracting users. Context-aware AR addresses this by adapting virtual elements based on users' needs, context, and environment. Recent research has explored blending AR interfaces with physical objects using computational methods, leveraging everyday objects for virtual interactions, and applying adaptive AR for use cases like improving accessibility for people with low vision. We aim to expand this research into broader everyday applications by using advanced AI capabilities. These can enable AR systems to better understand contextual information, such as room geometry, physical object affordances, and user activities, creating more seamless and adaptive experiences.
Always-on AI Assistant by Integrating LLMs and AR:
Large Language Models (LLMs) can enhance AR by enabling always-on AI assistants that seamlessly support user activities and blend into the environment. Unlike screen-based interactions, LLM-AR integration allows intuitive, multimodal interactions with digital content. Recent work demonstrates augmented object intelligence, gaze-based question answering, and wearable-based assistance. We aim to advance these capabilities by leveraging LLMs' multimodal potential, combining visual, tangible, and spatial modalities unique to AR interfaces.
AI-Assisted Task Guidance in AR:
AI-assisted task guidance offers significant advantages over traditional video-based instruction, which provides static, one-way demonstrations. AI systems can analyze users’ movements in real-time, delivering personalized feedback on form, timing, and technique. This interactive approach has the potential to accelerate learning, prevent bad habits, and improve performance. Beyond full-body activities like fitness or sports, we are also interested in manual tasks such as cooking, repairs, or assembly. Additionally, we aim to explore how AI systems can adapt to user progress, gradually increasing task complexity and offering encouragement for a more engaging and effective learning experience.
AI-in-the-Loop On-Demand AR Content Creation:
Generative AR can revolutionize the creation of interactive AR content by leveraging advances in generative AI. Techniques like GANs and Stable Diffusion enable real-time content generation, 3D animation, and immersive environment design. Beyond static image generation, generative AI can create interactive AR content using LLMs’ code generation capabilities and by making real-world objects interactive. We aim to explore how generative AI can utilize AR’s unique spatial aspects, such as 3D scene understanding and spatial interactions, to generate richer content—text, visuals, motion, video, and 3D objects. We also seek to address the challenges and opportunities of integrating these capabilities into AR environments.
AI for Accessible AR Design:
AI has the potential to enhance accessibility in AR design and experiences, addressing barriers faced by individuals with hand-motor impairments. Traditional input methods, such as bimanual keyboard-mouse setups in tools like Unity or handheld controllers in XR experiences, exclude these users. To overcome this, we propose exploring AI-driven alternative inputs, including voice commands, gaze tracking, facial expression recognition, and subtle body movements interpreted by AI. Additionally, AI can adapt content difficulty, interfaces, and narratives to individual capabilities. This inclusive approach not only expands access to AR design and experiences but also innovates interaction paradigms, transforming how virtual worlds are created and engaged with.
Real-World-Oriented AI Agents:
We are interested in designing AI agents that enhance the physical world by bridging the digital and physical realms. These agents would understand spatial relationships, object affordances, and user activities to assist with navigation, support complex tasks, and provide personalized guidance. Recent advancements, such as AI agents that guide users’ attention through mixed reality avatars, demonstrate promising potential. We aim to explore how AI agents can be designed to be aware of and responsive to real-world contexts, enabling more intuitive and adaptive AR experiences.
Introductions and Lightning Talks:
The organizers will kick off the workshop with introductions and each participant's lighting talks highlighting their related research experience. To start community building, we will facilitate pre-workshop engagement to cultivate and propose an initial set of themes through the creation of a shared slide deck that highlights each participant's background and interests.
Position Paper Presentations:
In this workshop, we will announce a public call for position papers from the CHI community, inviting researchers and practitioners to contribute their insights. The purpose of these talks is to provide an opportunity for attendees to share their opinions and positions on their recent and ongoing work related to AR and AI. We aim to foster a collaborative environment where participants can present radical ideas, exchange feedback, and inspire new directions for research. For each accepted position paper, we plan to allocate 15-20 minutes for presentations. Additionally, with the consent of the participants, we plan to publish these position papers after the workshop to make the insights available to a broader audience and promote further discussion.
Keynote Talks:
We will also invite distinguished speakers to present keynote talks, with the goal of sharing forward-thinking visions and ideas with the audience. These talks are intended to inspire and set future directions for the field, similar to the UIST Vision Talks. Each keynote session will be approximately one hour long, providing ample time for in-depth exploration of ideas. We also plan to publish the title, speaker bio, and abstract on the workshop website to provide attendees with context and background information ahead of the event.
Participant Activities:
Our workshop aims to provide a balanced mix of presentations and interactive participant activities to foster engagement and collaboration among attendees. To achieve this, we have organized several activities designed to encourage hands-on exploration and creative problem-solving. These include: 1) Hackathon and Rapid Prototyping: Teams use low-fidelity materials (paper, cardboard, markers) to mock up an XR interface or device. Teams incorporate how AI might enhance the user experience. Teams present prototypes in a "science fair" style showcase. 2) Design Sprint: We will challenge participants to redesign a common XR experience for users with specific disabilities. Users may integrate AI to propose adaptive interfaces or alternative input methods. 3) XR-AI Mind Mapping: We will create a large, shared mind map of potential XR+AI applications across industries and encourage participants to add ideas throughout the day.
Theme Organization and Discussion:
Participants will collectively extrapolate themes from the presentations and shared readings. The discussed themes will be combined with the initial set of topics developed prior to the workshop to serve as guiding directions for the first round of discussions. Once a preliminary list of discussion topics has been defined, each topic will be assigned a ‘table.’ During the session, participants will rotate between tables to engage in focused discussions of topics of their choice. One participant at each table will be designated as the discussion mediator, whose responsibilities will involve guiding and documenting the discussion. The second session on theme organization will begin with lightning talks by the discussion mediators summarizing earlier conversations. Participants will then collectively revisit the discussion topics, reorganizing accordingly based on the results of the first session and expert perspectives. With the refined list of topics, the remainder of the session will follow the same format as the first round of discussions.
Defining Future Challenges and Research Directions:
In this final discussion session, we will begin by regrouping and refining the list of discussion topics based on the results of the discussion sessions. The final workshop session will focus on summarizing the workshop findings and defining next steps. First, the organizers will provide a recap of the workshop activities, including the defined themes from the pre-workshop activities and a summary of the morning and afternoon discussion outcomes. The floor will then be opened for participants to contribute their reflections on the workshop discussion. A final discussion will be held around potential future directions, such as follow-up workshops and publications.
If you have any questions, feel free to contact us.