Emerging Interfaces

The Reality-Virtuality Continuum

Milgram's continuum (1994) describes the spectrum from fully real to fully virtual environments.

Real          Augmented       Augmented       Virtual
Environment   Reality (AR)    Virtuality (AV) Environment (VR)
|─────────────|───────────────|───────────────|
              ◄─── Mixed Reality (MR) ──────►

Virtual Reality (VR) Interaction Design

VR immerses users in a fully synthetic 3D environment, replacing the physical world.

Interaction Challenges

VR Design Principles

1. Maintain frame rate: Minimum 72 FPS (90 preferred). Dropped frames cause sickness.
2. Respect the vestibular system: Never move the camera without user initiation. Avoid acceleration.
3. Comfortable interaction zone: Primary UI elements within arm's reach, 0.5-2m from user.
4. Ground the user: Provide a visible floor/ground plane and body representation.
5. Spatial audio: 3D audio cues reinforce direction and presence.
6. Avoid small text: Minimum angular size of ~1.5 degrees. Current headset resolution limits readability.
7. Provide safe boundaries: Guardian/chaperone systems for physical space awareness.

VR Locomotion Techniques

Augmented Reality (AR) Interfaces

AR overlays digital content onto the physical world, maintaining awareness of real surroundings.

AR Display Types

AR Design Principles

1. Anchor to the real world: Digital content should maintain stable spatial relationships with physical objects.
2. Respect occlusion: Virtual objects should be occluded by real objects in front of them.
3. Consider lighting: Match virtual object lighting to the real environment.
4. Minimize clutter: Real-world context already provides visual complexity. AR layers must be restrained.
5. Design for variable environments: AR is used in diverse lighting, spaces, and contexts.
6. Glanceable information: Users should not need to stare at AR overlays. Brief, peripheral-friendly.

AR Interaction Patterns

Placement:     Detect surfaces -> show placement indicator -> tap to place object
Annotation:    Anchor labels or info panels to physical objects or locations
Navigation:    Overlay directional arrows onto real-world walkways
Visualization: Show hidden infrastructure (pipes, wiring) overlaid on walls
Measurement:   Point-to-point distance measurement in physical space
Try-before-buy: Place virtual furniture in a real room

Spatial Computing

Spatial computing treats 3D physical space as the interface canvas. Apple Vision Pro and similar devices exemplify this paradigm.

Spatial Interaction Model

Input:   Eye tracking (look) + Hand pinch (select) + Hand gestures (manipulate)
         Voice commands for text and system control
         Head position for viewport

Output:  Floating windows positioned in 3D space
         Volumetric objects and immersive environments
         Spatial audio anchored to virtual objects

Spatial UI Patterns

• Windows: 2D content panels floating in 3D space, repositionable
• Volumes: 3D content containers bounded by a defined space
• Immersive spaces: Full environment replacement (like VR within an AR device)
• Ornaments: Small UI elements attached to window edges (controls, status indicators)

Design Considerations

• Ergonomic placement: UI at comfortable viewing angles (slightly below eye level, within 1-4m)
• Z-depth hierarchy: Closer elements are more important/interactive
• Passthrough integration: Blend virtual content naturally with physical surroundings
• Shared experiences: Multiple users seeing and interacting with the same spatial content

Wearable Interfaces

Wearable devices are constrained by small screens, limited input, and intermittent attention.

Design Constraints

Smartwatch Interaction Patterns

• Notifications: Short, actionable. Quick replies, dismiss, or deep-link to phone.
• Complications: At-a-glance data widgets on the watch face.
• Short tasks: Set a timer, check weather, start workout.
• Continuous sensing: Heart rate, activity tracking, fall detection.

Tangible Interfaces (TUIs)

Tangible user interfaces use physical objects as representations of and controls for digital information.

Examples

TUI Design Principles (Ishii & Ullmer)

1. Physical-digital coupling: Changes to physical objects are reflected digitally, and vice versa.
2. Spatial interaction: Arrangement of physical objects in space controls digital parameters.
3. Embodied interaction: Leverage existing physical manipulation skills.
4. Peripheral awareness: Tangible displays can convey ambient information through shape, movement, or surface changes.

Ambient Displays

Ambient displays present information at the periphery of human attention, requiring minimal cognitive effort.

Design Principles

• Calm technology (Weiser & Brown): Information transitions smoothly between periphery and center of attention.
• Low attention cost: Glanceable. Users should not need to stop what they are doing.
• Aesthetic integration: Ambient displays should fit naturally into the environment.
• Appropriate abstraction: Represent data through color, movement, sound, or physical form rather than numbers and text.

Examples

Conversational AI

Chatbots

Text-based conversational interfaces for customer support, task completion, and information retrieval.

Design patterns:

• Decision trees: Structured conversation with predefined branches
• Slot filling: Collect required parameters through dialog (origin, destination, date)
• Open-domain: Free-form conversation powered by large language models
• Hybrid: Structured flows with fallback to free-form understanding

Conversational UX principles:

1. Set expectations about capabilities upfront
2. Provide suggested replies / quick actions to reduce typing
3. Handle non-understanding gracefully (do not repeat the same reprompt)
4. Allow human escalation when the bot cannot help
5. Maintain conversation context across turns
6. Confirm critical actions before executing ("Transfer $500 to checking? Yes/No")

Voice Assistants

Voice-first interfaces (Alexa, Google Assistant, Siri) for hands-free interaction.

Design elements:

• Invocation: Wake word ("Hey Siri") or button press
• Intents: What the user wants to do (play music, set alarm)
• Entities/Slots: Parameters for the intent (song name, time)
• Dialog turns: Multi-turn conversations to collect missing information
• Multimodal responses: Voice response + visual card on screen-equipped devices

Generative UI

Interfaces dynamically generated or adapted by AI based on user context, intent, or content.

Approaches

Design Challenges

1. Predictability: Users expect consistent interfaces. Generated UI must maintain structural coherence.
2. User control: Users should be able to override, pin, or reject AI-generated layouts.
3. Transparency: Make it clear when UI is being adapted and why.
4. Graceful failure: When generation fails, fall back to a sensible default layout.
5. Accessibility: Generated interfaces must still meet WCAG standards programmatically.
6. Testing: Non-deterministic outputs make traditional QA difficult. Require constraint-based validation.

The Spectrum of AI-UI Integration

Static UI          Adaptive UI         Conversational UI      Generative UI
(fixed layout,     (layout adjusts     (dialog-driven,        (AI creates UI
 human-designed)    to usage data)      natural language)       on the fly)
|─────────────────|─────────────────────|─────────────────────|
Less AI involvement                              More AI involvement
More predictable                                 Less predictable

Design Ethics for Emerging Interfaces

Emerging interfaces raise new ethical considerations: