Harmony: Adaptive XR+AI for Human-Centered 3D Interaction
A research initiative to make interfaces perceptive, collaborative, and context-aware—so technology adapts to people, not the other way around.
Why Now
The Problem
Most digital systems assume 2D screens and static interfaces. Even when tasks are spatial, collaborative, and dynamic.
Fixed layouts can't adapt to real-time needs. Context is lost between devices and users.
The Opportunity
XR and AI now enable perceptive environments. Systems can sense attention and interpret context.
Interfaces can adapt complexity in real time. We can move toward human-aware systems that improve learning, creation, and decision-making.
Research Vision
Harmony investigates how AI and XR can co-adapt with people. Systems interpret spatial, behavioral, and social cues to shape interfaces on the fly.
The goal is not to replace human judgment. It's to augment it—with transparency, trust, and control.
Human-Centered First
Design for well-being, accessibility, and meaningful control.
Context as Signal
Use environmental and behavioral cues to inform adaptation.
Collaboration by Design
Support multi-user sensemaking and co-creation.
Edge-Aware Responsiveness
Keep adaptation real-time and privacy-preserving.
Responsible AI
Make adaptation explainable, opt-in, and auditable.
What Is Harmony?
Harmony is a conceptual framework for building adaptive XR experiences. It brings together three interconnected layers that work in concert.
Perception of Context
What's happening?
Signals about space, task, and behavior are summarized into context descriptors. "Just enough" awareness to support adaptation.
Adaptive Mediation
How should the system respond?
Interface complexity and layout tune to moment-to-moment needs. Fewer steps, clearer guidance, better cognitive fit.
Collaboration Orchestration
How do we co-create?
Aligns group attention and shared state across people and devices. Smoother multi-user workflows and human–AI co-creation.
Core Research Axes
Five fundamental questions guide our investigation. Each axis explores a critical dimension of adaptive systems.
1
Context-Aware Interfaces
How can systems interpret task and behavioral cues to adapt layout and feedback?
2
Perceptual Adaptation
When should complexity be expanded or collapsed to match cognitive load?
3
Human–AI Collaboration
How do we support shared agency, transparency, and trust in co-creation?
4
Edge-Aware Responsiveness
What's the right split between on-device adaptation and cloud for privacy and latency?
5
Evaluation & Ethics
Which metrics capture usefulness, trust, and well-being impacts?
Current Explorations
Active Studies
We're scoping pilot studies and prototypes that examine key adaptation mechanisms.
Each exploration focuses on real-world scenarios. Research methods blend HCI evaluation with responsible AI practices.
Adaptive Layout for Spatial Tasks
UI elements scale and relocate based on task phase.
Attention-Safe Guidance
Subtle cues for wayfinding and feedback without overload.
Co-Presence & Shared Focus
Lightweight mechanisms to align group attention in XR.
Explainable Adaptation
Micro-disclosures that show what changed and why.
Where Harmony Helps
Adaptive XR/AI systems can transform domains where context matters. From education to creative work, Harmony principles apply across industries.
Immersive Learning
Adaptive lab simulations and skill training that respond to learner progress.
Accessibility
Context-aware support that meets users where they are, adapting to individual needs.
Assistive Workflows
Guidance for field service, AEC, or healthcare training with real-time adaptation.
Creative Collaboration
Multi-user spatial sketching and review with synchronized attention and state.
Safety-Critical Training
Progressive disclosure to manage cognitive load in high-stakes scenarios.
Roadmap
Harmony development follows a phased approach. Each stage builds on prior learnings while maintaining research rigor.
1
Foundation
Now–6 months
Literature synthesis, study design, low-fidelity prototypes, and IRB planning.
2
Systemization
6–12 months
Iterative prototypes, evaluation methods, and early publications.
3
Ecosystem
12–24 months
Toolkit patterns, reference implementations, and community pilots.
4
Scale
24+ months
Open materials, partnerships, and longitudinal studies where appropriate.
Ethics First: Adaptation must be opt-in, reversible, and user-tunable. Transparency and well-being metrics guide every phase.
Background & Prior Work
Varun Siddaraju
Applied XR/AI practitioner focused on human–computer interaction and immersive systems.
Former Lab Lead at X-Reality Research Lab, Texas State University. Founder/CEO of VeeRuby. Delivered 50+ XR/AI prototypes across education, healthcare, and AEC.
50+
XR/AI Prototypes
Across education, healthcare, and AEC sectors
$33K
Research Grants
Government-funded XR/AI innovation
3
Publications
Including Best Paper Award at ICDT 2018
Selected Publications
- In-Situ Wireless Channel Visualization Using AR & Ray Tracing. Sensors (2020)
- X-Reality: AR meets IoT. IEEE INFOCOM Demos (2018)
- AR Facet Mapping Technique for Ray Tracing. ICDT Best Paper Award (2018)
- Beginning Windows Mixed Reality Programming. Apress (2021)
Get Involved
Collaborate
Interested in HCI, XR, or AI research? Looking to participate in user studies or collaborative pilots?
Harmony thrives at the intersection of context-aware computing, immersive analytics, and human–AI co-creation.
Frequently Asked Questions
Is Harmony a product or research framework?
It's a research-driven framework. Tools released prioritize openness, evaluation, and responsible use.
Will code be open-sourced?
Where feasible and ethical, we share reference designs and patterns. IP-sensitive components follow standard academic channels.
How do you measure adaptive success?
Through mixed-methods HCI evaluation: task outcomes, cognitive load, user trust, and longitudinal engagement.
© 2025 Varun Siddaraju. "Harmony" is a research initiative exploring adaptive XR/AI systems. This website shares concepts and goals only; it omits proprietary methods, datasets, and implementation details. For academic collaboration, please get in touch.