Sherlock & Claude

Relationships
Documents
Events
Entities
Web search
Locations
Q&A for recognised documents

Inspiration

Investigative journalism shines a light on the dark corners of our society by uncovering stories of corruption, abuse, inequality, and more. However, conducting in-depth investigative journalism requires immense time and effort. Providing journalists with cutting-edge tech tools to automate parts of the investigative process could help expose stories that would otherwise be buried.

What it does

Sherlock & Claude is a document mining tool tailored for the needs of investigative journalists. Journalists can upload large sets of documents and use the LLM's features to quickly surface key information about people, organisations, locations, and events. This helps them connect the dots to reveal untold stories.

How we built it

The core document mining capabilities are built using the state-of-the-art Anthropic Claude model.

We leverage the following frameworks:

Anthropic Claude API
Google OCR API
Haystack
Brave Search API
Cohere Rerank API
Streamlit

Challenges we ran into

Despite the support of the long input context Anthropic Claude model struggles to retrieve accurately all entities from the collection of multipage financial documents.

Accomplishments that we're proud of

We implement two stage extraction procedure which drastically improves extraction accuracy. At first stage the individual documents processed separately and then at the second stage we merge multiple extractions with the same prompt base Anthropic Claude framework.

What we learned

This project is a good demonstration how single prompt-base framework can solve multiple NLP task:

Named Entity Extraction
Coreference Resolution
Information Extraction
Reasoning
and many others

We tested these technologies at high limit with real task and we are quite impressed with result.

What's next for Sherlock & Claude

No doubt, there are few points of improvements. But we hope we successfully demonstrate how these technologies could be used to benefit society. We believe journalists would use similar tools for new projects.