Ryo Suzuki
University of Calgary
Sunday, October 29th, 2023
San Francisco, CA, USA
The advent of generative AI, large language models, and new foundation models in computer vision have fostered numerous technological innovations in AI research and its applications. However, interface design and interaction with AI technologies is predominantly confined to graphical user interfaces (GUIs) on computers or smartphones. We believe that the next step in Human-AI Interaction should be oriented towards "real-world" AI interfaces, where the AI technology naturally and seamlessly integrates into our everyday life by blending bits and atoms.
Towards this goal, this workshop aims to unite experts and practitioners in XR and AI to envision the future of AI-enabled virtual, augmented, and mixed reality experiences. Our expansive discussion includes a variety of key topics: Generative XR, Large Language Models (LLMs) for XR, Adaptive and Context-Aware XR, Explainable AI for XR, and harnessing AI to enhance and prototype XR experiences. We aim to identify the opportunities and challenges of how recent advances of AI could bring new XR experiences, which cannot be done before, with a keen focus on the seamless blending of our digital and physical worlds.
Read the Proposal PDF
This workshop welcomes HCI researchers and practitioners in XR (VR/AR/MR), AI, machine learning, and computational interaction domains to share diverse perspectives and expertise. There are several domains that are not fully explored yet in the literature of XR and AI. We plan to discuss the topics that include but are not limited to the following areas:
Generative XR:
We explore the integration of generative AI for XR applications. For example, we are interested in understanding how generative AI can leverage the unique physical and spatial aspects of XR applications, such as 3D scene understanding and spatial and tangible interactions. What are the challenges and possibilities in such integration?
LLMs for XR:
This focuses specifically on the exploration of large language models to augment mixed reality experiences.
How can we leverage the LLMs to naturally augment such interactions, instead of explicitly typing questions on the screen? Beyond text-based interaction, what kind of new applications are possible by integrating the recent advances in multi-modal LLMs to XR environment?
Adaptive and Context-Aware XR:
This focuses on the development of XR interfaces that seamlessly blend into the user’s everyday environment. Current research investigates how computational methods from optimization and machine learning can be leveraged to create XR interfaces that understand and adapt to the context of users and their environment.
Explainable AI for XR:
This addresses the growing need for user-friendly interfaces that help users understand, customize, and interact with mixed reality applications. It involves creating prototyping tools aimed at making XR experiences more accessible and understandable. The challenge is how to enable non-technical users to author adaptive behaviors and how conventional interface prototyping methods can be adapted for the development of adaptive XR.
Prototyping XR with AI:
As XR interfaces become more accessible to end users, lowering the barrier to design and develop XR applications becomes increasingly critical. Current tools primarily focus on developing applications and experiences for static
contexts. Therefore, a major challenge lies in enabling non-technical users to author adaptive behaviors. This is particularly crucial in the development of tangible interfaces, which could unlock significant potential for immersive XR experiences.
Other XR x AI Topics:
We also welcome any possible XR and AI themes, such as intelligent interaction techniques, AI-enabled accessibility in XR, new evaluation methodologies, and real-worldoriented human-AI interaction.
Introductions:
The organizers will kick off the workshop with introductions. To start community building, we will also facilitate pre-workshop engagement to cultivate and propose an initial set of themes.
Participant's Lightning Talks:
Each participant will also give lighting talks highlighting their related research experience to get to know each other.
Keynote Talks:
We also invite a couple of speakers who will deliver presentations to provoke further discussions.
First-Round Theme Discussion:
Participants will collectively extrapolate themes from the presentations and shared readings. The discussed themes will be combined with the initial set of topics developed prior to the workshop to serve as guiding directions for the first round of discussions. Once a preliminary list of discussion topics has been defined, each topic will be assigned a ‘table.’ During the session, participants will rotate between tables to engage in focused discussions of topics of their choice. One participant at each table will be designated as the discussion mediator, whose responsibilities will involve guiding and documenting the discussion.
Second-Round Theme Discussion:
The second session on theme organization will begin with lightning talks by the discussion mediators summarizing earlier conversations. Participants will then collectively revisit the discussion topics, reorganizing accordingly based on the results of the first session and expert perspectives. With the refined list of topics, the remainder of the session will follow the same format as the first round of discussions.
Closing Discussion:
In this final discussion session, we will begin by regrouping and refining the list of discussion topics based on the results of the discussion sessions. The final workshop session will focus on summarizing the workshop findings and defining next steps. First, the organizers will provide a recap of the workshop activities, including the defined themes from the pre-workshop activities and a summary of the morning and afternoon discussion outcomes. The floor will then be opened for participants to contribute their reflections on the workshop discussion. A final discussion will be held around potential future directions, such as follow-up workshops and publications.
UIST 2023 Workshops will take open enrollment. There are limited spaces offered on first-come first-served. You can join the workshop by registering for it when you complete your UIST registration.
If you have any questions, feel free to contact us.