Inspiration
User Interface (UI) prototyping is a necessary step in the early stages of application development. Transforming sketches of a Graphical User Interface (UI) into a coded UI application is an uninspired but time-consuming task performed by a UI designer. An automated system that can replace human efforts for straightforward implementation of UI designs will greatly speed up this procedure.
What it does
Our neural network model is a novel approach wherein we employ a Deep Neural Network that is trained on our custom database of UI sketches to detect UI elements in the input sketch. Detection of objects in sketches is a peculiar visual recognition task which requires a specific solution that our deep neural network model attempts to provide. The network transforms the images into a UI representation object which is consumed by our parsing software which will build the final application for different platforms.
How we built it
We used RetinaNet object detection model and trained it on a custom database of images of UI sketches. The result from the network is processed further to create complex elements such as forms which can contain other UI elements (input, buttons, etc.). Now, this results into a JSON representation of UI which is stored against an id in our mongo-db atlas cluster. When we have to share a prototype application, the shareable links embed the id corresponding to the document in mongodb. Now when this link is opened the UI elements in the mongodb document are fetched and the UI parser renders the UI prototype interface.
Challenges we ran into
Preparing the dataset and training a Deep neural network was a big challenge.
Accomplishments that we're proud of
We were able to create the dataset and train it on the object detection model. The stitch-js sdk integration work great to render application.
What we learned
We learned to prepare our custom dataset. We used mongodb Atlas and stitch for the first time with server sent events on atlas for change streams in mongodb.
What's next for DrawCarys
We could use Server sent events and subscribe to them using the stitch-js sdk to re-render the application in real time as we make a UI sketch simultaneously recording and sending the feed of the video.
Log in or sign up for Devpost to join the conversation.