When I was walking in the shopping mall and walking on the street (with mask ~~) during this COVID pandemic, the silent, the cold and cheerless smashed me for a long time - the businesses like the retail & catering industry are suffering from a huge impact, while the individuals cannot enjoy the ordinary shopping experience.

I was thinking that though we cannot 100% conquer the virus, we can still change our business model to provide those small-middle businesses with promotion and transaction opportunities, and at the same time to offer individuals a better life experience.

So from my professional perspective, I decided to develop a GIS-based B2C Solution, where I used 2D GIS Platform as business side CMS, and Geo Augmented Reality as the client application.

What it does

The solution has the business side and the client side, respectively.

On the business side, the GIS Platform allows the business users to choose and display hierarchy layers including the non-editable layers such as Roads, Suburbs and editable events layer via MyLayers-1 Widget.

After selecting the editable events layer, business users can turn on MyEvents-1 Widget to choose item types, then click on any location of the map to create new points of certain types e.g. FAST_FOOD, ASIAN_FOOD, REFINED_FOOD,....., and input the detailed descriptions for these points, for example: a SUSHI BAR’s owner can create a ASIAN_FOOD type point at a certain location saying “Fresh Salmon for only $3 before 4pm at Miles Food Courts”.

The business user can either manually input the addresses for the items, or can use the auto-filled Esri reverse-geocoding addresses. After the editing, the new data will be submitted and stored in ArcGIS Online Feature Layer via Feature Service. The existing feature points can also be updated or deleted.


On the client side, the client user can run the GeoAR App on any mobile or iPad while they are outdoors and with the permission of using Camera and GPS.

When a client user walks nearby the locations (e.g 20m - 30m, code-side configurable) of those points which were added/updated by the business owners, he/she can use the camera to scan and find out the 3D model(s) of the specific products floating around.

Then the client user can zoom in or out to see clearly about the appearance of the product, and click on the 3D model(s) to see the detailed descriptions of the product, e.g. the message made by the Sushi Bar’s owner “FRESH Salmon for only $3 before 4pm at Miles Food Courts”.

What’s more, the client user can also take the photo of such 3D model items in the augmented reality with natural background, and then share this photo with their friends or social media ~~

            sushi_info             pasta_info_gesture

How I built it

Business Side

The Admin End is based on the framework of ArcGIS Experience Builder Developer Edition, where I developed two core customized widgets in ArcGIS API for Javascript with ReactJS, TypeScript.

The MyLayers widget

  • Register all available ArcGIS Feature Services hosted on ArcGIS Online including the editable event layer in the React State.
  • Construct per single feature layer with functions and classes from ArcGIS API for Javascript, and group all these layers in ReactJS Renderer.
  • Bind Checkbox with React State, push or remove the feature layer to/from ArcGIS Experience Builder Map Components.

For more details about how to develop and deploy the customized widgets of ArcGIS Experience Builder, please watch these two videos on my channel:
Customize Widgets in ArcGIS Experience Builder with ReactJS & ArcGIS API for Javascript (part 1)
Customize Widgets in ArcGIS Experience Builder with ReactJS & ArcGIS API for Javascript (part 2)
How to deploy Customised Widgets of ArcGIS Experience Builder on ArcGIS Portal

The MyEvents widget

  • Get the selected event layer(s) from Map Component when the widget in the initial life cycle.
  • Bind its backbone editable ArcGIS Feature Layer/Service hosted on AGOL with the admin panel front-end form via some functions from ArcGIS API for Javascript.
  • Register the ReactJS State and then listen to map clicking, feature point selecting, form adding, updating and deleting actions.
  • Handling actions with some functions from ArcGIS API for Javascript, for example map point to coordinates, reverse geocoding, feature type filling and so on.
  • Bundle the data and submit to ArcGIS Online Editable ArcGIS Feature Layer.

One thing that deserves to be known is that some of these information can be generated by ESRI Services. For example, I used Esri Reverse-Geocoding Service to automatically generate the address for each geometry point I added, and didn't need manual input.

For more details about these two core customized widgets and how to use it in ArcGIS EXB Developer Edition, please check my codes under my GeoAR-B2C-Admin folder in my github: GeoAR-B2C-Admin

Client Side

The client-side Application is built with pure Javascript HTML5 with the help of open-source libraries such as A-FRAME, it also consumes the data from ArcGIS Online Feature Layers via ArcGIS Feature Service REST API. The Application itself is registered on ArcGIS Online as Mobile Application.

  • Firstly, the App will continuously get the GPS signal from the mobile phone, and get the light from the camera sensor.

  • Secondly, it retrieves the feature points which are input from the business side via ArcGIS Feature Service REST API returned by JSON. As all these points have coordinates, they will be compared with the current GPS coordinates. When the phone gets close to these geolocations, these feature points will be rendered with different 3D textures based on their types and names.

  • There are some Javascript components and functions registered to response to user’s actions, for example clicking on the 3D model to show the detailed information, zooming in or out to see more clearly about the 3D textures, and taking the photo.

To know more details about the client end, please check my codes under GeoAR-B2C-Client folder in my github GeoAR-B2C-Client

Spatial Data Hosting

The initial data definition is done in ArcGIS Desktop, with feature classes in file geodatabase, where we need to initially define the TYPE and its rendered icon represented in the Business Admin End. Then we also need to initially push the raw feature class as feature service in ArcGIS Online as the hosted layer.

On ArcGIS Online, we need to set the hosted feature layer as Public Data Collection by enable editing, so that the hosted feature layer will become the hub shared by both Client End and Admin End.

Challenges I ran into

There are 5 main challenges, 3 technical challenges and 2 business challenges:

  • The GPS Jumping issue: the mobile phone GPS is not a stable data, is not 100% accurate and it might be changed every time it updates, which results in the projected AR 3D Model might jump from one place to another from time to time. Though I have set the mechanism of minimum GPS changing tolerance and GPS changing intervals, it still cannot promise this issue can be solved all the time. I also read some articles about this and there are lots of topics of Oblique Photography that need to be researched. Alternatively, integrating the RKT or CORS net into the solution might also be an option.

  • The Datums and Projection Systems are various and different for different locations around the world. Currently the solution uses WGS_1984_Web_Mercator_Auxiliary_Sphere: EPSG:3857 for everywhere, which is not accurate for GPS real-world anchoring. In future I will research how to automatically allocate proper Datums and Projections Systems to different locations of the world - Esri Toolsets and Location Services can also help about this.

  • Though this web-based GeoAR App does not require the expensive AR Devices, most of the AR Devices’ optical components in the market nowaday are still immature, which can result in some distortions when the circumstances become harsh.

  • In order to really make this prototype to be applicable in the publics, in addition to the technical issues, we will also need to think of how to make it be the commercial closed-loop, for example, we need to make an additional transaction system behind what we have done, to allow the businesses and clients can do and commit the transaction between the GIS backend Admin Panel and GeoAR Frontend App. This is a topic that is the scope out of GIS and needs people with other expertise to help with.

  • How to collaborate the solution with the traditional logistic industry is also a topic that needs social experiments.

Accomplishments that I am proud of

When I showed this work & idea to people around me, what they all said was (sort of) “you deliver a strong positive energy to the world in a geek way, providing the possibilities to the future”. And yeah, within such a short time (just 2 weekends are really available for me to work on this), I came up with this idea and implemented this prototype !!!

What I learned

By participating in this Hackathon, I get more exposure to ESRI’s products, and have a deeper understanding about the correlations among various ESRI products, and the potentials of their combinations. At the same time, I made friends with some participants and ESRI staffs, and enjoyed learning from them from various specialties in GIS and Software.

What's next for GeoAR B2C Solution for Sustainable Future

I hope this prototype is just a start but not an end. There are lots of possibilities about this to be researched, and lots of work to be done. Technically, I will keep researching and solving the issues outlined in the Challenges we ran into section. On the other hand, I would really like to improve the Server Platform and the Client App, and then persuade some local businesses and consumers to try using it, to test if it really helps in their daily businesses and purchasing.

Such a thing is not a one-man job, something like this needs more people to get engaged in, always remember that where there is change, there is hope !! ~~

GeoAR Admin & Client GeoAR Admin & Client GeoAR Admin & Client

Original Video Clip on Youtube

Built With

Share this project: