Building immersive AR experience with React Native

User Engagement:

User engagement plays a key role in user retention and product’s profitability. At Zoomcar we strive to find the best way in which users can engage with our product. We started exploring mobile AR because there are so many unique problems that it can solve and so many delightful experiences it can unlock.

What is AR?

Augmented reality is the virtual expansion of the real world.

AR utilises your existing reality and adds to it utilising a device of some sort. Handheld devices, such as smartphones, are popular at the moment.

As you’ve already noticed, your physical body and mind exist in a mix of space and time that you perceive as reality, and nowadays you can even exist within a computer-generated mix of space and time known as the Virtual Reality. However, in between these two extremes there can exist other types of reality where “normal” Reality and Virtual Reality intertwine to varying degrees, one of which being Augmented Reality.

You must have seen various types of gallery slider around the web, We thought how cool it would be if we can create an AR slider & we did just that!

We built an AR Slider where user can see different cars, interact with them by changing their colour or rotating them.

Here is how it looked finally:

AR Slider showing rotationAR Slider

How we did it?

There are various options to build an AR app. You can use AR-Core for android and AR-kit for iOS devices or you can use React native to build it for both. We chose VIRO, which is a react native based platform for building AR/VR apps.

We are not going to discuss how to get up and running with VIRO React(their doc is really good for this). Instead we are going to understand how building an AR app is different than creating a web page. In order to understand differences and similarities between creating web page and AR app we are going to look into the following points: 1. Positioning 2. Interactions

1. Positioning:

When we are creating web pages the canvas is 2D which means we think along the x and y axis. In case of AR canvas is the real world which introduces another dimension z.

Viro uses a right-handed coordinate system, where the direction of view is along the negative z-axis. The point of view can be modified by changing the Camera. By default, the camera is positioned at [0, 0, 0] and looks in the direction [0, 0, -1]. Objects in the scene are positioned in this 3D coordinate system via the position attribute.5

We are using ViroARPlaneSelector for plane selection, It require minHeight and minWidth attribute which specifies the minimum height and width, in meters, of a plane that the component will “anchor” to. Once the plane is selected ViroARPlaneSelector triggers a callback onPlaneSelected on this callback we are triggering animation to scale car from 0 to 1. 3D Object is placed on position [0,0,0] relative to the selected plane. Other elements in AR view (navigation and colour picker) have been placed using position attribute.

2. Interactions:

Interactions in web is mostly deals with changing elements style(coloraturas, position, shape) based on user input. As web interactions are based on on-screen items user does not need to move physically to interact with the application. In case of AR for user to have immersive experience, it’s very important that user move around the Object in the real world. However when most of the users encounter their first AR experiences, they’ll often forget they can move around. To solve this problem we placed navigation element in AR environment instead of showing 2D buttons. Which encourage them to move around and tap on navigation elements.

AR environment based navigation is not enough until there is not proper feedback when user is interacting with them. We used the concept of real world buttons and when user taps on navigation button it moves slightly in z axis with bounce easing. Here is how tap animation looks like:

AR Slider navigation in action

Here is code for tap animation:

 back: {
   properties: { positionZ: "-=0.025"},
   duration: 50,
   easing: "bounce"
 },
 front: {
   properties: { positionZ: "+=0.025"},
   duration: 50,
   easing: "bounce"
 },
 tapNavigation: [["back","front"]],

To add one move level of interactivity, We added a feature where users can use rotation gesture to rotate the 3D Object. onRotate callback on ViroNode returns rotateState and rotationFactor. We can use rotationFactor to get new rotation values which we can apply back to ViroNode to rotate the object.

Limitations

Although AR on mobile devices has come a long way since its inception, there are still a few limitations. Here are some of them which we faced while building AR Gallery:

  1. Environmental limitations: For now, limitations that may hinder accurate understanding of surfaces include:
    • Flat surfaces without texture, such as a white desk
    • Environments with dim lighting
    • Extremely bright environments
    • Transparent or reflective surfaces like glass
    • Dynamic or moving surfaces, such as blades of grass or ripples in water.
  2. Tracking limitations:
    • When user moves mobile device quickly both AR kit and AR-Core struggle with tracking Object with accuracy.

Web component 101

Web Components, the future of the web

Web components are a set of web platform APIs that allow you to create new custom, reusable, encapsulated HTML tags to use in web pages and web apps.