Meet Rahul, an immigrant in Dubai doing construction work and remitting his earnings to support his family back home. Without a fixed contract or safety net and on a minimum wage, Rahul is given cash at the end of every day based on the number of hours he’s laboured. Without a bank account or fixed address to report, Rahul is faltering at the poverty line and is in most need of better financial literacy.
What we are trying to address
This is a story echoed by 70 million families headed by blue-collar workers all over the world . Whether it is sole traders selling off a fruit stand, farmhands or a handyman - these individuals are only a couple of sick days away from bankruptcy. And often without any education, they are unable to make sound financial judgements on saving and spending.
The challenges are compounding:
- In the least developed regions, the majority of the adult population is illiterate in nearly one quarter of the countries. 
- 53% of adults feel financially anxious, which creates a cyclic nature of money and mental health problems 
- Further, the pandemic and global recession are expected to aggravate the situation and cause over 1.4% of the world's population to fall into extreme poverty. 
Fortunately, with the increasing affordability of smart devices and mobile data, these individuals can be empowered by technology. And many overcome the language or illiteracy barrier by recognising icons (emojis) and using voice notes to communicate.
By leveraging the behaviour of our target demographic and using voice notes to communicate through a chat system, Gecko is a Telegram bot which interacts by sending and receiving voice notes.
Twice a day, it prompts the user to share if they have had any expenses or received any income. After receiving a text, the user responds with a voice note in their own language simply saying what they have spent or earned. The voice message is converted into text, translated and interpreted to extract the relevant information. Information on earnings, spendings, hours worked is then stored in a database containing their historical financial transaction and the profile of the user. Additionally, sentiment analysis is run on the voice itself to detect whether they are feeling happy or distressed when responding. Depending on the emotions detected, the system evaluates the mental state of the user and signposts the different financial and mental health support services available in that country.
With some of our target users being illiterate or using older technologies, it was imperative to build a solution that could function within their existing ecosystem and with least hassle. Often working extremely long hours, these workers will often be tired by the time they can check their phone again. Hence, using voice notes technology is our primary solution. Additionally, we have mitigated against any incompatibility with different devices by linking to an already used app and running all complex computational features in the cloud.
Try out the bot here!
How we built it
We implemented voice note sentiment analysis and voice to text AI features of Python … Our Sentiment Analysis is based on ensemble learning algorithms, which utilised the wisdom of the crowd theory, moreover, one of the key features is that our AI is trained and improved based on its mistakes.
Some of the APIs we used are:
- Google Cloud Speech API: To perform automatic speech to text in different native languages
- Google Cloud Translation API: To translate text from different languages to command phrases
The following diagram visualise the architecture of our system:
Challenges we ran into
The main challenge we faced was focusing our idea and narrowing it down to a specific implementation that is most impactful for the user and feasible within the given time constraint. We had to think through how each code would communicate with one another and find suitable tools and packages to aid us with easy implementation. As an example, we wanted to use WhatsApp for our messaging interface until we realised its API does not allow access to voice notes, so we had to look for other alternatives, in the end settling on Telegram.
Accomplishments that we're proud of
One of the main achievements is that we worked on a project as a team with the outlook of using technology for the better life of others. Technology in the right hands can change the world. Our project tackles a real issue in the present world and combines the latest and most widely used technology to support those in financial need, addressing not only money problems but also associated mental health issues. Our team worked in a collaborative environment and we are proud of how we managed to work together and help each other in the process.
What we learned
Researching into financial illiteracy and poverty worldwide and its close correlation to mental health made us realise how acute and intertwined these issues are. Having come from different backgrounds, everyone in our team learnt something new: from implementation of API and AI to pitching and branding.
What's next for Gecko
The project can be extended in multiple directions in the future. With the current architecture, the following can be implemented:
- Gecko's messages to be translated to the user's language
- Personalised messaging with the user's name
- Gecko's replies in voice notes in the language of the user
For further development of the project, we are looking at:
- Localising resources and responses
- Including more interactive features of the messaging bot with the user, e.g. advice on remittance, more regular prompts based on user behaviour. For this more profiling of user questions would be needed. After building a financial and psychological profile for each user, we are looking to expand the options of establishments the bot is connecting the user to for support. This establishment can be financial (e.g. help with the loan) or addressing mental health issues (e.g. counselling) depending on the user data collected. This would entail seeking appropriate partners in each of the relevant sectors.
-  LEVITAN, S., & TAGGART, R. (1971). Has the blue-collar worker's position worsened? Monthly Labor Review, 94(9), 23-29
-  Charting the Progress of Population - United Nations Population Division
-  OppU Financial Literacy
-  World Bank: COVID-19 to Add as Many as 150 Million Extreme Poor by 2021