The U.S. Patent and Trademark Office has recently granted Apple Inc. a collection of 56 patents, with a focus on the Vision Pro. Notably, one key patent in this series is titled "Light Pattern-based Alignment For Retinal Eye Tracking." This patent primarily addresses the challenges associated with misalignment in retinal imaging-based eye tracking systems used in conjunction with a Head-Mounted Display (HMD).
The granted patent encompasses devices, systems, and methods designed to enhance retinal imaging-based gaze tracking by mitigating misalignments between a content device, such as a display, and a sensor, like a camera, used for retinal eye tracking.
In Apple's patent background, they acknowledge that retinal imaging-based gaze tracking systems typically incorporate an illumination source to light up the retina and an image sensor to capture images containing light reflections from the retina. Gaze tracking involves comparing the current retinal image from a gaze tracking image sensor to a previously obtained map of the retina during enrollment. However, misalignments between the displayed content on an electronic device and the gaze tracking image sensor may occur due to factors like thermal drifts or physical shocks, resulting in gaze direction errors during retinal imaging-based gaze tracking.
The solution proposed by Apple involves integrating light sources, such as point sources with predefined patterns, into the display. These light sources may be embedded in the display, mounted on it, or pass light through it. For instance, infrared (IR) light from IR light sources on the display can be directed onto the retina, producing a detectable pattern visible to the gaze tracking image sensor. Any changes in the relative positioning between the content display and gaze tracking image sensor would cause a corresponding change between the projected IR pattern and the retinal image as seen by the gaze tracking image sensor.
In certain implementations, the retinal image may be generated during device production at a factory by calibrating the illumination pattern with the gaze tracking image sensor, typically in an initial alignment during factory enrollment. Subsequent misalignments between the display and the gaze tracking image sensor can be detected and corrected. This ensures that errors in retinal imaging-based gaze tracking resulting from misalignments are minimized or eliminated.
Another method involves creating a light pattern through a pattern light source with a spatial relationship to a content light source that produces displayed content. Sensor data, including information corresponding to the detected light pattern using retinal imaging, is received at a sensor. An alignment characteristic between the content light source and the sensor is determined based on the spatial relationship and the sensor data. Additionally, an eye characteristic is determined based on the sensor data and the alignment characteristic.
In essence, Apple's innovative patent aims to enhance the accuracy and reliability of retinal imaging-based gaze tracking in HMDs by addressing the potential challenges posed by misalignments between the display and the gaze tracking image sensor. The incorporation of light patterns provides a dynamic solution to detect and correct misalignments, ensuring more precise and effective retinal eye tracking.
For a visual representation, Apple's patent FIG. 2 provides an overview of an HMD, while FIG. 3 is a block diagram illustrating an exemplary content display system used to align a gaze tracking system in an HMD. This patent showcases Apple's ongoing commitment to advancing the capabilities of their devices, particularly in the realm of augmented reality and virtual reality technologies.