BookLang

Inspiration

• Reader opinions are fragmented across languages, platforms, and editions. Star ratings don’t explain why.
• Historical reviews miss fast-moving shifts driven by adaptations, price changes, and social buzz.
• Publishers, retailers, and authors need trustworthy, multilingual insights—grounded in evidence, not guesswork.

What it does

• Compares sentiment across languages for any book, series, or edition, with calibrated, apples-to-apples metrics.
• Breaks down sentiment by themes like pacing, characters, prose, tropes, translation quality, and audiobook narration.
• Tracks real-time shifts in posts/comments vs. historical baselines; flags emerging topics and anomalies.
• Surfaces representative multilingual review quotes with translation toggles and confidence scores.
• Powers discovery tools (read-alikes, motif explorer) based on what readers actually say, not just metadata.

How we built it

• Ingested historical reviews and real-time streams from social, forums, and retailer comments; auto-detected language and deduplicated near-duplicates.
• Used multilingual embeddings and classifiers (sentiment + multi-label themes) fine-tuned for book-review context.
• Calibrated models per language to ensure fair comparisons; maintained rolling aggregates (24h/7d/30d) against baselines.
• Implemented hybrid retrieval (semantic + keyword/entity boosts) and grounded generation with source citations.
• Delivered dashboards and alerts via an OLAP-backed API; stored raw text, embeddings, and audit logs for transparency.

Challenges we ran into

• Ensuring sentiment parity across languages with different slang, tone, and code-switching.
• Detecting spoilers and toxicity consistently while preserving useful signal for analysis.
• Controlling for platform, edition, and region mix so comparisons weren’t biased.
• Handling real-time data quality: deduplication, brigading spikes, and late-arriving events.
• Building trust: providing clear confidence intervals, significance tests, and easy access to source evidence.

Accomplishments that we're proud of

• A language-calibrated sentiment dashboard that reveals true cross-language differences, not model bias.
• Theme-level divergence matrix that surfaces nuanced insights (e.g., translation humor drift, narrator reception).
• Real-time delta monitor that caught shifts within hours, with evidence-backed alerts.
• Spoiler-aware, citation-first summaries that stakeholders can confidently share.
• Read-alike and motif explorer that meaningfully improves discovery across markets.

What we learned

• Calibration and stratification are critical for fair multilingual comparisons.
• Minimal translation for analysis, with translation for display, strikes the best balance of accuracy and performance.
• Domain-specific taxonomies (tropes, narration, translation quality) unlock far richer insights than generic sentiment alone.
• Transparency—citations, confidence, and per-language performance—drives adoption and trust.

What's next for BookLang

• Expand language coverage and improve slang/code-switching handling with targeted fine-tuning.
• Add edition-aware comparisons for new translations and audiobook re-releases, including narrator-specific analytics.
• Launch proactive alerts for adaptation news and market events with causal evidence trails.
• Integrate with publisher/retailer tooling (BI dashboards, CMS, CRM) and add role-specific reporting.
• Open a benchmarking hub: public multilingual retrieval/sentiment leaderboard for book reviews, plus sample datasets.

Built With

• airia
• clickhouse
• deepl
• linkup
• openai