H.A.A.M. / Human Augmented Anatomy Mirror

 

This research concerns the design and development of an interactive, multi-user Augmented Reality (AR) display called the Human Augmented Anatomy Mirror (HAAM). As AR becomes more common, standard forms of interaction need to be established. Without standards and best practices AR will be a confusing experience, different on every platform. HAAM serves as an exploration of this issue.

During development, emphasis was put on ease of use and solving issues that arise in a multi-user environment. A large format display is used as a “mirror”, displaying projections of the large organs of the thorax and abdomen onto a live video feed of the user. HAAM utilizes a Microsoft Kinect V2 for tracking of the human body, circumventing the need for physical tracking markers. The Kinect permits freedom of movement and multiple participants while employing a Natural User Interface. Unity3d, a commercial game engine, is used for the development of the application.

 

3D Model Workflow

 A representative sample of raw Visible Human Male dataset, courtesy the National Library of Medicine.

A representative sample of raw Visible Human Male dataset, courtesy the National Library of Medicine.

 Raw 3D model exported from ITK-Snap after masking in 3D slicer.

Raw 3D model exported from ITK-Snap after masking in 3D slicer.

 Segmented data provided by the National Museum of Health and Medicine, Chicago.

Segmented data provided by the National Museum of Health and Medicine, Chicago.

 
  Final Model generated in ZBrush, retopologized and textured.

Final Model generated in ZBrush, retopologized and textured.

 

Tracking System

 

Spheres represent the coordinates for the right and left shoulders and right hip. The cube represents an organ model. The red lines represent three sides of a triangle. Any point along these sides can be calculated. The two white lines represent rays drawn from the shoulders through chosen points (x1, x2) on the opposite side of the triangle. The intersection of the two rays can then be found. This intersection can be adjusted by moving x1 or x2 in real-time. This enables accurate positioning of organs. Joint coordinates were also used to generate a plane. The normal of the plane is calculated, represented here by the yellow ray. This can be used to orient organs.

Distances from shoulder to hip and shoulder to shoudler were calculated and these values were used to scale organs to fit bodies of different proportions. As the user moves closer or further from the sensor the perceived distance between tracking points will change. The same scaling that occurs to fit organs to individual users also scales them based on distance from the sensor.

 

3D View / Cardboard VR

You can interact with the 3d models of the HAAM organs here. Be sure to check out the Google Cardboard option if you have a viewer!

Note that the organs appear reversed. This is because the user sees a mirror image of them selves. Go ahead and touch your right hand side where your liver would be. Everything is in its place, as if you were standing in front of a mirror. One of the aspects I would like explore is the mirroring effect on comprehension. Most people learn anatomy as if they are viewing a body laying on a table or standing in front of them. Is it possible that by providing this mirror experience, people can better relate the locations of organs to their own bodies? Does the kinesthetic sensation of touching the location on their own body and correlating it with an image of the organ allow for better retention? It's something worth exploring, I think!