Our design process progressed from understanding Nreal Light pain points, to forming HMW statements, conducting technical analysis, and finally iterating on design.

The main design challenges are below, formed as HMW statements, and the designs seen in the above videos.

• New Input Modality: How might we adopt 3rd party peripheral technology (hardware and software) to improve Nreal Light’s usability?

  • Solution: A multi-modal input combining HMD gaze (or head-tracking ray cast) for targeting, and a Samsung Galaxy Watch 4 (incl. Port 6) for object selection and manipulation. Smart Watches are beneficial due to being highly accurate, existing widespread adoption, haptic feedback, battery life, and not requiring line-of-sight

• Innovative UI Components: How might we adopt innovative UI components to improve Nreal Light’s usability?

  • Solutions:

    • 2.5D Window: utilizing useful aspects of 2D and 3D content — allowing for parallax/depth information of 3D models while having legible text in panels .

    • Smart Watch Expanded Wrist UI: expanded Wrist UI overlaid onto the Galaxy Watch 4 (image tracking) as a “hot-key” or “task bar” interface .

    • Radial menu: circular “active state” context menu where selection is dependent on direction. Reduces the fatigue and inaccuracies of “raycast” selection.

    • Hand menu : activates based on a wrist-located button. Hand menu is world-locked, and divides app-specific and platform-level features.

    • “Gather” and “Split view” features: collects all open objects (from anywhere in environment) and allow users to select and maximize four of those objects within their field of view for added focus.

    • Virtual trackpad: similar to a laptop trackpad, that allow users to easily interact with 2D interfaces in an ergonomic way.

The biggest lesson learned during the these projects are that while hand-tracking may solve some of Nreal Light’s tethered phone controller issues, it introduces many others. Ultimately, combining gaze for targeting, and a wrist-based input for selection and manipulation seems like a very promising direction for the future of AR Glasses.

Meta has their CTRL-Labs prototype, Port 6/Doublepoint is making headway, and it wouldn’t surprise me if Apple has plans for their Apple Watch.

Other important lessons learned:

• Ruthlessly Prioritize: Due to the blue-sky nature of the project, the client was excited to keep iterating on more new ideas. However it was vital to take charge, slot “new ideas” as additional backlog items for future projects, and shift from ‘possibility exploration’ to high-fidelity designs in order to meet project timelines.

• Building a culture of resilience: Building psychological safety is important for your teams, especially if your project entails doing something that hasn’t been done before. Creating a culture where teammates can learn fast, iterate, and treat failure as an opportunity for creative improvement will positively effect all aspects of the project.

• Design pillars: By establishing core pillars (like “user comfort”) you have a north star to point to when choosing between different design options with clients. For example, since we were desiging for hand tracking, we had to constantly consider the ergonomic properties of the human body — creating principles for understanding egonomic cost of certain distances from the body, and the likelihood of exhaustion on muscles, joints and ligaments.