Active Participant in the co:here Challenge PPT: https://docs.google.com/presentation/d/1HuMi3pFE6Ze6WbLasRmqYhfrsBOKPBpr/editusp=sharing&ouid=115905566253954670277&rtpof=true&sd=true GitHub: https://github.com/Munasco/EkoGenie

Inspiration

On average, an adult in the US spends 5.4 hours per day on the phone. Out of these 324 minutes, 265 minutes are spent on non-calling activities. This means that an average American spends 59 minutes per day, or 358 hours per year (almost 15 full 24-hour days) on the phone, either calling or answering calls. From further research, 134 out of the 358 hours are spent Navigating an automated voice system and/or waiting for a human operator, as well as making, amending, and canceling appointments, booking, and reservations. 17 8-hr work days total per year combined.

Furthermore, according to a study by Loyalty360, while 77% of surveyed individuals believe phone calls are the most effective way to get an answer or resolve an issue quickly, 25% of callers give up after being on hold for more than 30 minutes. In addition, according to Microsoft, 30% of customers indicate that not being able to reach a human rep is the most frustrating part of a bad customer service experience.

Consumers are spending hours on the phone to perform menial tasks such as making reservations and flight bookings, and are being put on hold and getting frustrated. Most importantly, they are unable to efficiently resolve a task/issue without a significant time investment.

With the above pain points, we wanted to create a system that will help consumers minimize the time they spend with automated voice message systems, appointment scheduling, and flight/hotel/restaurant bookings.

What Our Solution Does

Introducing EkoGenie, a personal AI Voice Concierge. EkoGenie is capable of converting simple instructions from a user to a real-life appointment/travel booking, without the active need for human intervention/monitoring.

There are two integral parts of EkoGenie’s functionality: 1) taking instructions from the user to convert into an action plan for the task, and 2) working with the counterparty to carry out the desired action. In the first step, EkoGenie would take text or voice instructions from the user and ask any clarifying questions to create a task list. In the second step, EkoGenie calls the corresponding counterparty and performs the required task on behalf of the user.

Here is an example flow:

EkoGenie to Customer:
Customer: “Book a reservation at Blueblood Steakhouse for next Thursday at 6PM, for a party of 5.”
EkoGenie: Researches the Blueblood Steakhouse booking system and required information to create a booking, as well as any non-standard characteristics that may be encountered during the call. EkoGenie compiles the list of required information, and asks clarification questions. For instance, our below response is generated from an example prompt to book an appointment at Blueblood Steakhouse, which has the following requirements:
The time, date, and party size
The location (Patio, Bar, or Indoor Dining Space)
The requirement of child seat(s)
The requirement to reserve parking space for the duration of the sitting
EkoGenie: “Sure. Would you like to book the indoor, the bar, or the patio spaces? Any kids/kids seats required? Any parking space required?”
Customer: “Indoor dining space please, no kids, and yes parking.”
EkoGenie: Compiles the list of information to convey to the counterparty during the call:
Customer name: John Doe
Date: Thursday, March 9, 2023
Time: 6PM EST
Party size: 5
Location: Indoor Dining Space
Child Seats: NO
Parking: YES
EkoGenie: Calls Blueblood Steakhouse.

EkoGenie to Rep:
EkoGenie: “Hello, I would like to book an appointment for John Doe next Thursday at 6PM, for a party of 5.”
Rep: “Sure! Would you like the patio, bar, or a table inside?”
EkoGenie: “Inside please.”
Rep: “Sure thing! Would you require parking during the reserved time slot? What about child seats?”
EkoGenie: “Yes to parking, no to child seats.”
Rep: “Booked! [Repeats details back to EkoGenie].”

EkoGenie to Rep:
EkoGenie: “Your appointment is confirmed for next THursday at 6, Blueblood Steakhouse.”

In the event that a task is impossible to complete due to extenuating circumstances (i.e. no spots/rooms/flights available), EkoGenie will reach back to the customer. For instance, say that Blueblood Stackhouse does not have a table for 5 next Thursday at 6PM:

EkoGenie to Rep:
Rep: “Sorry, but we don’t have a table for 5 at 6PM on Thursday. I have 6:45PM and 7:15PM open on that day, or 6PM on Friday. Would any of these times work for you?”
EkoGenie: “Let me check and get back to you.”

EkoGenie to Customer: “Hi John, sorry but 6PM isn’t available. Could you do 6:45 or 7:15PM? Or perhaps Friday at 6?”
Customer: “Let’s do Thursday at 6:45.”
EkoGenie: “Sure!”

EkoGenie to Rep:
EkoGenie: “We would like to schedule for 6:45PM instead.”
Rep: “Confirmed!”

EkoGenie to Customer:
EkoGenie: “Confirmed for Thursday at 6:45PM, Blueblood Steakhouse, party of 5, inside. A parking space has been reserved.”
Customer: “Great, thanks!”

In essence, EkoGenie eliminates the time that is required for a customer to call, go through an automated voice system, be placed on hold, or speak with a representative to conduct a simple/menial task. As described in our Next Steps section, we will continue to work to improve the complexity and difficulty of tasks that EkoGenie is able to help complete.

How We Built It

First, we developed a way for EkoGenie to interact with the customer using React Native (frontend) and Firebase (for logins and database for the storage of customer information), as well as Flask (allowing communications to React Native, hence enabling communications between EkoGenie and the customer). We chose to use React Native because it demonstrates multi-platform compatibility, allowing us to roll out our apps to both IOS and Android users. In addition, we chose to use Firebase for a smoother login management workflow.

Leveraging co:here, we created a system to classify customer requests/prompts based on provided data. We chose to leverage co:here AI due to its extraordinary NLP capabilities. Next, we queried OpenAI to access information that is required to help complete any tasks and/or deliverables that the customer must provide to carry out a task successfully (in our previous example, the date/time/location/party size/parking/child count etc. of the reserving customer at Blueblood Steakhouse).

In order for EkoGenie to interact with the counterparty (i.e. Blueblood Steakhouse), we leveraged Twilio API infrastructure and Python GTTS Libraries (for Text-To-Speech) and OpenAI Whisper (for Speech to Text) to converse with the customer and the company. We chose to use OpenAI Whisper as it was a mature speech to text platform, and Twilio as we found it convenient to enable voice communications.

While this process involved various platforms and processes, as well as competing priorities and rigorous features selection, our team was able to work together to overcome any challenges that came in the way.

Technologies Used (Our Stack)

In only 24 hours, we were tasked with creating a mobile and ML app to help consumers save time on phone calls.

Stack	Technology Used
Front-End	React Native
Back-End	Python, Firebase, Flask
APIs	Twilio, co:here, OpenAI, OpenAI Whisper

Challenges We Ran Into

One of the central challenges we encountered was validating the customer pain point as well as further analyzing the product-market fit. With extremely limited publicly available data, we had to conduct hours of market research at the beginning of the hackathon to make sure that the team was investing time into solving a real-world challenge. In addition, our research and statistics was also passed through a peer review assurance system to ensure that any data shared within the presentation was accurate and taken from a reputable source. This process was both educational and fascinating, as it provided our team the opportunity to leverage scientific research methodologies to procure data, at the same time learning more about the VoIP industry segments and customer pain points.
In addition, another challenge that we faced was closing the gap between our technical expertise to the level required to fluently leverage co:here AI and OpenAI to create a functional prototype. To mitigate this challenge, we started by conducting online regarding co:here AI’s capabilities, and attended the co:here AI session to ask clarifying questions. In addition, we also sought after networking opportunities with co:here AI representatives, to gain further insights into the amazing capabilities of the platform and how it could be leveraged to create a real-world impact as part of our project. This was an eye-opening experience for the members of our team as we were able to learn both about the functionality, as well as the history and development potential of co:here.
Furthermore, we had a challenging yet intellectually stimulating journey fine-tuning the speech-to-text model to fit various accents, which was one of our major objectives (making the solution accessible to a diverse population, including those in developing countries that may require this solution for their small businesses and personal needs, so that they can be more productive in other areas of life/business activities; We discuss this further in-depth in the final section, Business Viability). For instance, due to the limited data available, our model experienced minor linguistic appropriation and miscategorization of different accents when tested. We are still in the process of mitigating this issue, and look forward to finding an appropriate yet efficient solution.
Lastly, we experienced token constraints when providing prompts, as some of them required a larger number of words to convey (an airline ticket booking would require more information than lunch at Nando’s). To resolve this issue, we used co:here’s Co.Summarize API to summarize all previous prompts to enable us to meet token constraints.
We believe that learning new tools and languages and overcoming technical difficulties will help us further develop our skills in computer programming, software development, and program management, and had an amazing experience working with everyone at NSBEHACKS2023 to create our product.

Accomplishments We're Proud Of

A fully functional Minimum Viable Product (MVP). Within less than 24 hours, our team was able to create a functional proof-of-concept for Ekogenie, to be reviewed and critiqued by our judges and peers. While we would definitely have loved a few more hours for further fine-tuning, we can definitely appreciate the fast-paced work environment over the past 24 hours.
As a team, it has been central to our mission to create a solution that corresponded to our values in Equity, Diversity, and Inclusion (EDI). In particular, we wanted to tailor this solution to fit the needs of the BIPOC community worldwide. To realize this vision, we were able to implement Swahili as one of the languages (besides English) as part of the voice-to-text model, and while it is nowhere near perfect, it is something that we plan to continue to iterate upon and make improvements to in the future.
Networking. Besides creating a fascinating solution to the world being put on hold, we were able to establish lasting connections with like-minded peers, mentors, judges, as well as company representatives. Through visiting various booths set up by CIBC, Block, and co:here, we were able to learn more about the various companies’ cultures, missions, and their commitment to EDI. In addition, by speaking with mentors about our project, we were able to receive integral feedback and advice that would make up a large part of EkoGenie as you see it today.
Andrew had 2 servings of jerk chicken for lunch.

What We Learned

The importance of strategic alignment from hour 1. Due to the fact that some of our teammates are located from other provinces and/or countries, it was challenging to make an in-person team meeting happen. Hence, for those who could not participate in-person, we coordinated communications virtually and facilitated collaborative activities online. However, online communications comes with its own challenges - making alignment a bit more challenging. In our case, we learned that we simply need to spend more time on it, to make sure everyone understands their tasks and responsibilities.
The importance of planning ahead. Over the weekend, Toronto experienced inclement weather conditions which delayed our teammates’ ability to get to Bahen Centre for the opening ceremony. Thankfully, upon consultation with NSBEHACKS2023 executives, there was no penalty for showing up late for the hackathon and registering (close to noon time, for some of us). Next time, we will definitely keep weather conditions into consideration, and plan ahead.
Leveraging expertise of industry representatives to resolve difficult challenges. When working with a set of unfamiliar tools, or struggling to learn new ones, we learned the value of participating in activities such as the co:here AI workshop, and asking questions about the various resources available to help us succeed in meeting our hackathon objectives. We also got to get to know some really cool people and build connections with them in the meantime, which was super helpful for both personal and professional development.
Get dinner early because good butter chicken runs out quickly!

Business Viability

Now… To the most important part of our project presentation - Business Viability!

Market Segments & Revenue Streams

Here is how EkoGenie will make money:

Customer Segment: Individuals and Working Professionals; Small and Medium Businesses Customer segments that we consider targeting involve day-to-day consumers, entre/solo-preneurs, working professionals, as well as small and medium businesses. The reason we target these segments is because we believe that based on their daily activities, these individuals are most influenced by having to call to make appointments/bookings, as well as having to navigate automated voice systems.
In the US, within the consumer/retail segment, there are approximately 298 million active cellphone users that we could potentially target. In Canada, there are approximately 31.38 million for the same. Combined, this market segment makes up one of the largest customer segments in the VoIP industry.
However, as the capabilities of EkoGenie expands over the next months, we plan to expand our customer segments to more commercial, corporate, and institutional clients where we plan to develop an “Non-Retail Solutions Group” within EkoGenie, to capitalize on the opportunities that lie within that customer segment.

Main Revenue Stream: Subscription Based Plan Similar to the business model of ChatGPT, we plan to offer both a free plan as well as a paid plan of EkoGenie, with a tiered service system.
In the free plan, while customers will be able to access all features of EkoGenie, they will be limited to the number of minutes that they use to call to complete tasks, as well as be shown advertisements periodically from our sponsors.
In the paid plan, customers will have full access to EkoGenie, as well as experience unlimited number of monthly minutes and an ad-free experience.
Of course, depending on the level of funding available, we may consider keeping an ad-free version available to the general public, until capacity limits are reached.

Paid Advertising for Free Users: Additional Revenue Stream Similar to Amazon, which leverages an advertising system to attract additional revenue from 3rd party sellers, EkoGenie will also implement a back-end advertising portal for interested individuals to market their products and services periodically to users on the free plan. Advertisers will be provided an opportunity to bid on specific customer segments (individuals who make specific calls to specific numbers, or consent to providing personal demographic/geographical information during their registration processes), with data available to them on who and what to target. This will be the secondary revenue stream for EkoGenie.

Obtaining Users

Here is how EkoGenie will attract and retain users:

Free Trial of the Pro plan
One way that EkoGenie could attract users is by offering them a free trial of the pro plan. On average, 54% of users keep their subscription (either voluntarily or because they forgot to cancel) after a free trial!

Social Media Advertising

Instagram/Tiktok/Facebook Ads
One advertising method that we plan to leverage is social media paid advertisements. This method is appropriate and our rationale for using it is that since most people who use these platforms are accessing it through their mobile platform, we can easily integrate our advertisements and deliver them directly to the target customer.
Partnerships with Influencers to Promote Product
Influencer marketing has become increasingly popular in the past years, and has been a cost-effective method for startups, as well as small and medium businesses to reach their target audience. EkoGenie will work with influencers within the telecommunications space to deliver customized advertisements (such as instagram posts, stories, youtube shorts, facebook group posts, as well as tik tok spark ads) to further the reach of our product and initiative.
Collaborations with Business Partners to offer Discounts
Furthermore, EkoGenie will offer discounts to corporations who plan to offer this service at-scale to their employees and affiliates. The more licenses you purchase, the cheaper it is per license, and the more time (and costs) you save as an organization! For instance, this will be especially popular with professional firms whose employees (consultants, and sales associates) travel on a regular basis, who can leverage this tool to further minimize time wasted on the phone.

Lowering Expenses

Avoiding downtime and costly War Room situations
By participating in recurring maintenance activities, as well as BAT testing prior to the implementation of new features, we can minimize downtime and a voice War Room situations that are both stressful, time consuming, and expensive.

Using EkoGenie to run EkoGenie
In addition, we strive to create a self-sustaining ecosystem where EkoGenie will be able to help solve a lot of operational and economic challenges se,o-independently, with minor human supervision and guidance.

Outsourcing
As a S&M business, EkoGenie will be leveraging legal, accounting, and administrative services to other firms who can perform industry-standard services for lower costs compared to carrying out in-house training, sourcing, and procurement of the same. This is a standard mode of operation across various industry sectors and verticals, where businesses leverage the expertise of others to reduce operational costs and create assurance systems (transfer of risk from the firm to the independent contractor).

Competitive Advantage

Unique Product Offering
Our product offering, as far as our research goes, is a unique solution to an existing market problem. Our only potential competitor, Google Duplex, was terminated a few years ago. This makes it so that our solution, if successfully released and advertised (where the public is made aware of the offering and the potential benefits it could have on their everyday lives), could gain significant market traction in a short period of time.
Good product-market Fit
The problem statement itself is broad, yet our product application is specific. We believe our product is a good fit for the market as it will not be extremely costly (less than 100 dollars per year to maintain an active subscription, with free version available), attracting a large customer base.
Easy to use without extensive training
This is not a product that a working professional would have to receive 5-8 hours of training for prior to being able to use. As we release this app, we will include a simple tutorial after sign up / log in to teach customers how to navigate and take full advantage of the various features of the app, so that they get their money’s worth! (Or completely for free :)
Low Cost to Operate; Strong partnership system with co:here
Our product does not include physical operations and logistics/supply chain management, and leverages our ability to fine-tune existing frameworks, such as co:here API, to bring features to end-users. This SAAS business model will also not include significant work required in the office, minimizing general/administrative and operational costs.

What's next for EkoGenie

As we further develop EkoGenie, we hope to improve on some current implementations and develop additional features:

More complex tasks. We would like EkoGenie to be able to perform more complex, multi-step tasks that may require active correspondence with the user during the call with the counterparty. For instance, calling the bank would require identity verification, which only the user can complete. However, EkoGenie can still be used to complete the rest of the call, and only engage with the user when it is absolutely necessary.
Additional language support. One of our objectives is to ensure this product is accessible to those around the globe. Leveraging co:here’s language detection abilities as well as Google Translate API, we will be able to serve customers in additional geographic regions and linguistic backgrounds.
Stabilize performance. Our functional prototype, despite being functional, is troublesome to operate for extended periods of time. In the long term, we would like to create a version of the product that demonstrates stabilized performance and could self-repair if needed without extensive human interactions.
Minimize downtime. The more frequent and the longer EkoGenie is down, the more potential revenue we forgo as we create negative customer experiences. This will also negatively influence the reputation of EkoGenie as a product. We would like to create an assurance system where downtime is minimized, and property scheduled, so that the majority of our users are never impacted (rollout to APAC during APAC night hours, and same for NA and EMEA, etc).

Timeline:
Month 1-3: further refine product idea and produce higher-level POC
Month 4-6: engage with sponsors to procure funding
Month 7-12: develop and launch V1
Year 1: NA operations, performance stabilization, and continental expansion
Year 2: expand in market share (international territories outside of NA)
Year 3-5: find additional lines of revenue (corporate solutions group)

Thank you for your time in reading our Devpost Project page for EkoGenie.

Built With

co:here
firebase
openai
python
react-native
twilio
whisper

Ekógenie