Demo Video :

Here is the link to our Demo Video. It demonstrates the key features of our system.

Inspiration and Motivation :

Imagine a world where you can manage/track who you want to give access to your house using just their picture avoiding the hassle of duplicating keys? A keyless entry system where you will never have to remember carrying your house keys ever again. Sounds too farfetched? Think again. Using the the power of scalable cloud computing such as AWS EC2 and the awesome capabilities of OpenCV libraries, we were able to build a prototype facial recognition system despite the heavy resource constraints imposed by the ATSAMW25 wireless module.

According to the new market research report "Home Automation System Market by Protocol and Technology (Network and Wireless), Product (Lighting, Security and Access Control, HVAC and Entertainment Control), Software and Algorithm (Behavioral and Proactive), and Geography-Global Forecast to 2022", the home automation system market was valued at $39.93 billion in 2016 and is expected to reach $79.57 billion by 2022. With the number of connected devices growing exponentially, we figured this is the perfect opportunity to dive into the smart home automation industry with our product In-Phase.

Hardware Components :

  1. Microchip ATSAMW25 (MCU + Wifi)
  2. ArduCAM-Mini-5MP-Plus OV5642 Camera Module
  3. Grove - LCD RGB Backlight
  4. Grove - Gear Stepper Motor with Driver

Under the Hood :

The entire system consists of 3 PCB's code named Alpha, Mike and Foxtrot that control the camera, motor and lcd display respectively. The Alpha board (Camera controller) is the master and the other two (Mike and Foxtrot) are the slaves which only communicate with the master via MQTT. If a visitor approaches the door, the Alpha board captures the image of the person and streams it out to an HTTP server running on an AWS EC2 instance via a POST request. The HTTP server then invokes a function that performs facial detection + recognition of the incoming image against a set of pre determined authorized users. If the image matches an authorized users, a control signal is sent to the Alpha board to initiate the unlock protocol. If the image does not match any of the authorized users with a high confidence or if it matches multiple people, in the interest of security, it sends a control signal to initiate the deny entry protocol.

After the results of the facial recognition are available, the master sends an MQTT message to both the slaves to initiate the respective unlock/deny protocol. If the unlock protocol is initiated, the LCD displays a welcome message with the name of the visitor and the motor unlocks the door. If the deny entry protocol is initiated, the LCD display a quirky "You shall not pass" message and the motor does not unlock the door.

Key Components :

The following are the major components of the project (in order of development)

1. Internal Non-Volatile Memory

The internal Non-Volatile Memory (NVM) is partitioned as follows : Memory Partition

As a security feature, the application code section of the NVM is protected and only the bootloader is allowed to modify it. However, in order to demonstrate the disastrous effects of allowing the application to write over itself (purely for educational purposes!) a self destruct command is implemented in the Command Line Interface. This command attempts to wipe the application code space. This command never succeeds since the instruction trace gets corrupted due to the overwrite. Though this situation should never occur in a production system (under no circumstance should such function be exposed in production systems!), it is a good idea to foolproof the system! Simply performing a hard reset at this point will invoke the bootloader which detects the corrupt firmware image and initiates a recovery protocol. Refer to the bootloader section for more info on the recovery protocol.

2. External Flash Memory

The External Flash memory is used to store the firmware image's of the application. In the current design, the external flash memory is partitioned as follows :

External Flash Partition

The OTAFU engine of the board always stores the incoming firmware update in the primary boot image. However, using the Boot Image Management controls on the Admin Panel of the Node Red instance, the primary and the secondary boot images can be independently updated. The secondary boot image is useful for running nightly builds of the firmware or for application debugging. For example : In production systems, the primary boot image can be configured to run the regular application binary and the secondary boot image can be configured to run only the Command Line Interface for debugging and remote diagnostics.

The golden image is the fail safe binary which is loaded in case of corrupt/aborted firmware update procedures.

3. Bootloader

The bootloader code always executes on a hard reset of the SAMW25. Before the execution of the core bootloader logic, the following self-tests are run to ensure the drivers are initialized correctly:

  1. External Flash Self Test (Write and Read back data from the external flash self test scratch pad)
  2. Non Volatile Memory Self Test (Write and Read back data from the non-volatile memory self test scratch pad)

If the self tests are successful, the core boot sequence is initiated. The following flow chart explains the boot sequence in more detail : Boot Sequence

As evident from the flow chart, the bootloader explicitly supports the following 4 commands :

  1. No action : Jumps to application start address validity check condition.
  2. Load Primary Image : Loads the primary image from the external flash and writes it to the application code space
  3. Load Secondary Image : Loads the secondary image from the external flash and writes it to the application code space.
  4. Load Golden Image : Loads the golden image from the external flash and writes it to the application code space.

Before writing any of the boot images to the NVM, it performs a clean format of the entire application code space to clear any remnants of the previous firmware binary. After writing the boot image, it performs a final running CRC calculation over the entire application code space to make sure the firmware image it just wrote is correct. If this check passes, the bootloader moves to the final stage which verifies the validity of the application code space start address. If it detects a 0xffffffff, then it treats the application code space as a corrupted image and initiates the recovery mode. The recovery mode is also initiated if the running crc value of the application code space does not match that of the firmware image header.

In the recovery mode, the bootloader simply restores the golden image to the application code space and repeats the running crc calculation check (this time comparing it against the golden image checksum). If this check fails again, then the recovery mode is invoked again. The board is in danger of getting stuck in a bootloop.

4. OTAFU Engine

The following flow chart explains the entire flow of our Over The Air Firmware Update procedure. OTAFU Flowchart

5. Application Firmware Architecture

The application firmware was designed to be extremely modular. Shown below is the folder structure of the entire application firmware : Modular Firmware

Subsystems could be included/excluded from the application code by simply changing the value of the #define MACRO as shown below :

Subsystem Configuration

Such a design made it extremely easy to include/exclude submodules to manage the program footprint. For example, if the camera, lcd and the motor submodules are included in the application build, the program footprint is about 124KB. However, with just the camera submodule included and the other two excluded, the program footprint drops to about 100KB. Apart from the memory footprint management, the goal of the modular firmware design was to ease the system configuration procedure. Different firmware binaries for each of the boards were produced by only enabling the subsystems the board was in charge of controlling. This was done by only changing the app_configuration.h file.

6. Cloud Integration

The facial recognition of the images was performed on an HTTP server running on Amazon Web Service t2.medium EC2 instance using OpenCV python libraries link. The ArduCAM was configured to capture images at a resolution of 640x480. Even at such a low resolution the images produced were ~13KB which was too large to fit into the SAMW25 SRAM. We designed a custom streaming protocol to read the data out of the ArduCAM frame buffer in small chunks and transmit it to the AWS EC2 instance via a multi part POST request. By tuning the size of the send buffer we were able to optimize the image upload time ~2 seconds. The facial detection and recognition took about ~2 seconds. We were able to achieve the end to end working in under 5 seconds!

The HTTP Server provided other functionality such as :

  1. OTAFU Management : Uploading a firmware binary directly to the HTTP server using an upload page.
  2. Firmware Binary Metadata : Webpage to view the metadata of the uploaded firmware binary image.
  3. Authorized User Management : Uploading images of authorized users for the facial recognition program.
  4. Visitor Statistics : Useful statistics such as time of visit, total number of visitors, etc.

Shown below are some of the snippets from the HTTP Server : HTTP Server

We also deployed a PHP server on the same AWS EC2 instance to display a log of all the visitors who tried to access the system. We call this The Phasebook: ![The Phasebook]:

The backend Admin Panel (shown in the image gallery) was deployed using an IBM Bluemix Node Red instance. The admin panel provided the following features :

  1. Log : A log of the important events such as boot up, restart, firmware update, visitor entry grand/deny, etc
  2. OTAFU Update : Initiates the OTAFU procedure.
  3. FW Info : Gets the FW ID of the board.
  4. Boot Image Management : Manages the primary and secondary boot images and sets the image to boot.
  5. Remote Capture : Triggers the In-Phase security protocol (capture image, upload, detect)
  6. Remote Override : Disables the security protocol and locks the door.
  7. Connectivity : Displays the last check in time by the board
  8. System Stats : Shows the connectivity of the Camera, Motor and LCD modules.

Shown below is the snapshot of the Node Red Instance : Node Red Instance

From Idea to Demo Day :

Given our firmware expertise, we decided to focus on a project that was software intensive rather than hardware intensive. Despite this we ran into issues that were beyond our imagination! The following are some of the challenges we faced along the way and how we overcame them : (Please note : the language in this section of the post is informal and relaxed, which works best to convey our feelings as accurately as possible!)

How do we perform facial recognition on the edge?

Unfortunately, despite our exhaustive research, we were not able to find any way of doing facial recognition on the edge given the resource constraint imposed by the course (SAMW25). Even the most stripped down version of OpenCV libraries require the support of a full linux environment. Hence we had to perform the heavy lifting in the cloud as opposed to on the "edge"

Okay, running OpenCV libraries on AWS machines is easy, but how do we get the image from the Camera to the AWS server if we can't fit it into the SRAM of the SAMW21?!

Solving this was perhaps the highlight of the project. Even with an image resolution of 640x480 (which was plenty for the purpose of facial recognition) it was impossible to fit the entire image into the SRAM of the SAMW25. The only way to do this was to stream it out from the frame buffer of the camera and upload it simultaneously to the server. Chunked upload was possible via a multi part POST request. However, the challenge was to synchronize the read buffer of the camera and the transmission buffer of the multi part POST request. We designed a streaming engine with a ping-pong buffer where in the controller would read the data from the camera's frame buffer and fill up the ping buffer. Once the ping buffer was full, the data would be read into the pong buffer. When the pong buffer is being filled, the data in the ping buffer was transmitted. This process repeats until the entire camera image is uploaded.

This streaming engine sounds complicated! Plus we still have to develop custom drivers to communicate with the Camera in the first place! There is no way we can get all this done if we have to wait for PCB's to come back from manufacturing to develop our firmware!!

Luckily we anticipated this problem very early in the design cycle. We decided to start the firmware development on the SAMW25 Xplained Pro board itself and not wait for the PCB to be manufactured. This meant interfacing the camera module through headers and choosing a camera which had standard communication interfaces such as SPI or I2C as opposed to complex camera communication interfaces such as MIPI CSI. The ArduCAM worked out perfectly as we could develop all of its drivers on the SAMW25 Xplained Pro itself. In fact, we focused 99% of our firmware design on the Xplained Pro board in the hope that once the PCB's arrive, we could simple switch out the user_board.h file and the firmware should work out of the box.

We have an end to end working of the entire system on the SAMW25 Xplained Pro board. Time to switch to the PCB's. We just have to modify the user_board.h and we should be good to go. Pretty straight forward process right?

Wrong!!! After going through the board bring up process, I have new found respect to Murphy's law. Literally everything that could go wrong, did go wrong. Our First mistake was a wrong BOM component on the output of the power regulator. We had to desolder a 1.2K Ohm resistor and replace it with a 1.2M Ohm one. Easy fix.

Next, the FTDI chip was not detected as a COM port when connected to a USB cable. After many hours of debugging, turns out it was a faulty chip as using another PCB solved the problem. Strike one PCB.NG. Though this was not a difficult fix, it was definitely a frustrating one!.

Next, eventhough the FTDI chip was detected, there was no activity on the RX and TX lines. This was really baffling since the RX and TX lines from the FTDI chip were connected to the SAMW25 module with just a via in their path. Unless there is a soldering issue for these pins under the SAMW25 housing, there is no other explanation as to why this was happening. Turns out it was exactly that! These pins refused to toggle even after configuring them as GPIO output lines and toggling them. Strike two PCB.NG. That was 10 hours wasted. We had to move the UART interface to other pins and switch to the USB-UART wire connector. Switching the external pins meant new SERCOM pins. The sheer number of SERCOM pins our design required, meant the wiggle room for moving things around was very tiny.

Finally, to add to our SERCOM problems, it turns out not all SERCOM pins can be used as I2C lines. To our bad luck, all of the SERCOM pins we had designated as I2C pins were non compliant. This was a serious problem. The camera needed the I2C interface for its configuration and the LCD screen had an I2C communication interface. Luckily the pins that were configured as motor drivers were I2C compatible. Hence started the process of a full schematic redesign of the SAMW25 pins. The only way to make this work was to use 3 separate boards with each one controlling only one submodule (either the camera, motor or lcd). The modular design of the firmware made it extremely easy to configure the individual build for each of these 3 PCB's using the same code base.

After the UART pins debacle, we decided not to accept anything at face value and decided to implement a GPIO scan feature where in we configure all the headers we have on the board as GPIO pins and toggle them to verify if they are actually connected to the SAMW25 module. It turned out all the header pins on the right side of the board did not toggle meaning they were not connected to the SAMW25 module!. Yes that's right, not one, not two but the entire right side! Strike three PCB.NG!! At this point I had lost my patience to try and check if this was a one off board or if all the other 6 PCB's also had this manufacturing defect. Luckily the three board system design allowed me to get away with using only one side of the board, and with some stroke of luck, the usable I2C lines (which were physically located on the non working side) were actually connected to the SAMW25 on the opposite side.

Okay, now that the SERCOM nightmare is over, we can finally get the system to work right?

Not quite. There was an issue with the power circuitry where powering on the board with just the battery or just the usb would make the code run in a continuous bootloop (constantly restarting). This was a classic brown out condition, where in the voltage would drop when the power consumption exceeded a certain threshold, which would reset the microcontroller. The only way to get around this was to use both the battery and the usb to power the board so that there was enough power to drive the circuit.

Finally, after getting around all this our system was functional! When it came to performing on the demo day, I was glad our system did not let us down! In fact, it performed amazingly well, well enough to fetch us the Judges Choice Award! :)

Built With

Share this project:
×

Updates