Buying or leasing a new car is not always as pleasant as one might think. One has to make an appointment with the dealer and if the selection of features such as color, exterior and interior design, as well as of several others is difficult, one has to hope that the dealer is patient. Eventually the process continues with going through the details of the car configuration. Depending on the back-end, this might result in marking the own preferences with boxes on a specification sheet, where sometimes choosing different things can actually lead to infeasible setups, whereupon the process has to be restarted. In the end, one has to discuss financial details, which can lead to reiterations with the selections, and different choices for payment options.

Inspired by the updated back-end by amag we thought about ways to improve the whole experience and out came carly.

What it does

Carly essentially gives the customer the opportunity to design the car in an augmented reality setting, where changes are made via voice commands. This leads to a configuration process, in which the starting point can be for example a car that one sees on the street, from which the configuration using the AR model can be started seamlessly. Once the configuration of all visually relevant parts is done, depending on the preferences regarding non-visual parts, feasible suggestions for configuring for example the engine, the transmission, etc. are made.

How we built it

iOS - Application

We build a native iOS app and used ARKit and CoreML.

Core ML ( is used for the image detection in combination with a cloud back-up if the detection is to slow.

ARKit ( is used to display the 3D car models. The car modelc can be updated in realtime.

Apple Speech Framework ( is used to create speech inputs, which change the looks of the car inside the configurator.


We used the AMAG API ( to get data from from the different vehicles, which are currently avaibale. If a user add a new vehicle we creat a new uquice ID and can than add different configurations for that car. We request the pricing for the current configurations and can thereby calculate the duration it would take to finance that specific car. We extended the functionality of

Credit Suisse API

We created a new user on the (not yet released) Credit Suisse API and added a new customer. We added a KYC and can now request the banking information via the "SearchByName" endpoint. We get the usertype, which in our case is a student. We also get the anual income, anual spending, marital status, total wealth, number of kids and number of car. This data is used inside the app to calculate the money the user has to pay monthly to get the car financed. The user also see's how he does compared to his peers and what impact his financial decision has.

Other / Server

We used Laravel Framework for the communication with the Credit Suisse API. To communicate with the AMAG API we used .NET on Linux. We hosted our server on Microsoft Azure ( We also build a small landing page based on a template. You find the website here:

What we learned

  • Augmented Reality (AR) is going to be huge! and makes a lot of fun
  • AR has a lot more potential than Virtual Reality (VR)
  • Speech could be a great form of input for AR applications
  • The future of the car buying experience is going to change. Very soon.
  • Core ML and Machine Learning in general is already helping people do tasks they weren't able to do before
  • Having an international team is awesome

What's next for Carly

  • Getting feedback from the judges and finding suitable business partner/clients.
  • Getting subscribers for our mailing list
  • Fix bugs and build a fully working product
  • Build a better UI and improve the general experience
  • Work with Credit Suisse to integrate the upcoming (official) API
  • Build a Machine Learning model which detects most of the cars based on a image
  • Integrate our product into the Volkswagen car configurator
  • Integrate bikes and other vehicles into the app
  • Build a platform to add more products into the application
  • Set up a company in Silicon Valley

Built With

Share this project: