Sketch2Code Block

Hand-drawn sketch
Code generation
Create and Edit sketch
Generate Code from computer sketch
Device preview
Generated Web page

Inspiration

What more could we accomplish if the time to test an idea was zero?

I've been in awe of what computer vision can accomplish and how it can bring about positive change in our business processes. One such area is UI prototyping. Sketch2Code made me rethink the efficiency of existing accepted process by demonstrating the utitlity of AI and computer vision in producing prototypes for user testing. I got the idea that integration with tools such as airtable would make it readily accessible and fit seemlessly in the users' collaboration space while also making them far more efficient.

What it does

Sketch2Code Block allows one to transform a UI sketch into a working web interface within seconds. If you have sketch anywhere, just click a photo and add it to your airtable attachments. Select the attachment and open sketch2code block in "Code" mode.

Sketch2Code runs in 2 modes:

Sketch: You can create sketches and "Save" them to your table. You can edit existing sketches using "Load" feature.
Code: Move to this mode and choose the attachment field containing your sketch to watch your sketch come alive in a web interface embedded inside the block.

Configuration

This block can be used without any configuration. However, configuring the blocks allows for more cool things to be possible. Let's see what configuration possible.

Custom Sketch2Code API: You can train your own Sketch2Code vision model following the documentation provided here. Enable custom API and provide the URL of your hosted Web API and the Blob storage. This will allow you to train on your own images and extend the current microsoft dataset.

Specific fields for Sketches and prototype URLs: When you enable this and pick a specific table and field for sketches and prototype URLs, it will be possible for you to click on a record and the block already knows which fields to look for. Prototype URLs field should be URL/Barcode field type where your Prototype preview URLs will be stored automatically after code generation.

Things to try:

Prototype Preview You can click on "Device preview" after generating your sketch to get a URL to access your web interface from any device. It also shows a QR code which you can scan from your mobile device to immediately open the web interface without any hassle.

Action Buttons You can add Action buttons (which went into public beta recently, yay!) to your airtable. Configure these buttons to open the Sketch2Code block. Combined with configuration in settings, you can start editing your sketch or generate code using buttons.

Barcode field type You can use the field type in your table. This will allow you to use your airtable mobile app as a Barcode/QR code Scanner which you can use to scan the QR Code of your Web interface prototypes. When you scan a QR code using your airtable field, it automatically adds the information in the QR Code to your record

How I built it

Airtable Blocks SDK
Microsoft Azure Cloud and Customvision.ai
React-sketch (runs on Fabric.js)

Challenges I ran into

I really enjoyed learning how people use Airtable and the different processes teams adopt to use airtable effectively according to their needs.
Polishing an Airtable block required thinking about User experience and easy of access, managing user permissions, etc.
Training of Sketch2Code was quite difficult since the repo is written in C# and not actively maintained.

Accomplishments that I'm proud of

Training a custom vision model that performs well for a very small dataset of about 140 images with 10 classification labels.
Building a easy to use extension that integrates with and enhances the design proccesses.