A LegalAgent helping the Supertopia administration

Governments enforce regulations by doing inspections. These return a variety of structured and unstructured data. All is to be evaluated by a jurist. AI solutions work together to help out.

Comment 2

Inspiration

Traditionally legal government administrations are structurally underfunded. Missing big budgets, these administration are obliged to work as efficiently as possible. They always try to find a way so they can do more with less. A legal administration often just contains a couple of resources and are doing a great job trying to cope. They deserve to be supported by systems that allow them to work efficiently.

What it does

It allows the government legal administration to prosecute efficiently against the high number of infractors. They can use big data to spot what cases that have a good chance on reaching a timely result and avoid the cases that are expensive to follow. The jurists will be able to limit the administrative effort and really work using their legal skills.

How we built it

Using an agent to construct logic from three different datasets:

  1. Structured data retrieved from inspections, including legal information such as law articles, historical case decisions, and inspection data.
  2. Available legal data, describing law articles and internal procedures.
  3. Unstructured data, retrieved from various sources and stored in different Azure Data Containers. Azure AI Vision enables data formats not currently supported by Salesforce (which currently only supports PDF and TXT) to be converted from sources such as Excel files, JPG, GIF, etc., into text files in real-time. This information is subsequently loaded into Data Cloud DMOs, where it can be indexed and accessed either directly by the agent or through prompt templates.

Additionally, we have developed preliminary predictive models that provide jurists with statistical insights regarding the likelihood of successful prosecution, enabling informed decisions before proceeding with cases.

Challenges we ran into

The agent sometimes has a mind of its own, honoring the instructions... but not always.

The agent grounding via Knowledge does not yet support related files. This made it difficult to categorise the grouding documents, as there is only one data library allowed accross all topics. I partly solved this by going via unstructured data and retrievers going against their indexes.

The big data set that comes in the Data Cloud via Azure needs to be categorised more, even within Azure so that different categories of data can be build and separately queried. We could use more connections going against different Azure paths.

The test data set to construct the predictions was difficult to fine-tune with an acceptable R coefficient. I left things as they were, considering that the real-life exercise could fetch that data from a migration. For now, I used ChatGPT to get me a data set with some variance.

Accomplishments that we're proud of

Supertopia is a multi lingual country. Information sets are available, however mostly accross structures and sometimes in Dutch, sometimes in French and sometimes in English. The data set that I have structured therefore was kept Dutch and French on the Infractions. The Data library contains multi-lingual documents. The instructions of the agent are in English. I was expecting a worse result shuffling around with languages like that, however things seemed to be OK. I expect things to be better if we put the real implementation in a country language like french, dutch or both.

The Azure AI integration towards Data Cloud using the Semantic Index and Retrievers gives a good result. The excel files and other "structured" input worked perfectly like you see in the demo. The inspection report was a PDF and works fine out of Data Cloud. Even a PDF Visial AI txt version worked well. I have been playing around a lot with hand written text. This was dependent on handwriting. Mine was difficult, however my daughters' handwriting worked like a charm. Cool stuff, so sky is the limit on data integration. It even makes me wonder whether in the future I will really integrate having Data Cloud or leave things unstructured and reachable by an Agent.

Even after all the work, setbacks and successes, I actually still believe that my project will be executed...once upon a time.

What we learned

It is still difficult to work with the instructions and keep on getting a consistent result. I am not sure if long term usage with the thumbs up and down improves this. The semantic indexing is working great. The agent indeed is a great tool and is not replaceable by a prompt builder filling a field. The agent memory is needed to handle more complex use-cases.

What's next for LegalAgent

I have done already a demo to the legal team. They were impressed about how far this technology already was. I will keep on fine-tuning and following up on the improvement that are needed to have a more stable and granular outcome possible. Its a continuous build until its really used.

Built With

  • azure
  • azure-ai-vision
  • azure-containers
  • azure-functions
  • chatgpt
  • python
  • salesforce
  • salesforce-agentforce
  • salesforce-datacloud
  • salesforce-knowledge
  • salesforce-servicecloud
Share this project:

Updates