Voice Assistant

• 

  Solution Landscape Diagram

• 

  Driver details page with the list of assigned deliveries

• 

  Route page wit the list of stops and events

• 

  Route page with the map of all stops

• 

  Mobile app that takes voice commands and respond with the voice

Inspiration

Working in a logistics company, I’ve observed that drivers often need to submit reports during deliveries—whether it’s updating their current location, reporting issues on the road, or sharing the status of a delivery. This process can be time-consuming and disruptive. Inspired to simplify their workflow, I came up with the idea of creating a solution that allows drivers to easily access details about their assigned deliveries and conveniently report various road events or delivery statuses using voice commands.

What it does

The driver accesses a web application and simply presses the 'Start to Talk' button to initiate a seamless verbal interaction with the Einstein agent, which is fully integrated and configured within Salesforce. It allows to retrieve information about assigned deliveries and submit the updates.

How we built it

The core functionality for retrieving and creating data in Salesforce is powered by Agentforce and Apex agent actions. The voice processing component leverages browser capabilities and is built using JavaScript. The front-end logic in the browser handles both user voice recognition and text-to-speech functionality. Once a voice command is captured and translated, Agentforce processes the natural language input and responds appropriately based on predefined instructions.

Challenges we ran into

The access to agent from apex was released only in the end of march, but after 2-3 weeks it stopped working after platform update - after working with salesforce support the following known issue was created https://issues.salesforce.com/issue/a02Ka00000eOn7C We was need to fully rebuild apex logic to invoke agent via API. Other challenges: -Limited documentation on programmatic access from apex to agentforce. -Issues voice recognition and voice synthesis on iPhone.

Accomplishments that we're proud of

• Successfully demonstrated the solution working within the Salesforce Playground mobile app, showcasing that a voice interface can be seamlessly integrated into both existing and new applications built using Mobile Publisher.
• Developed a solution that can be integrated with literally any Agentforce agent, ensuring broad flexibility and adaptability.
• Became early adopters of Agentforce’s programmatic access, which was released just a few weeks ago, positioning us as pioneers in building custom applications on top of Agentforce.
• Created a fully native Salesforce application without relying on any external cloud services, ensuring maximum security, high-preformance and compatibility within the Salesforce ecosystem.
• Delivered a solution capable of addressing real-world business needs, making it highly practical and ready for immediate implementation.

What we learned

Best practices for prompts Agentforce with data cloud

What's next for Driver Voice Assistant

We are going to present the application on Dreamforce We are going to promote the application among Logistics companies Improve User experience with voice command Improve Agentforce configuration to make User Experience much better and seamless

Built With

• agentforce
• apex
• experience
• flow
• javascript