TubeSaku Ranking Sentinel

• 

  TubeSaku Ranking Sentinel

Inspiration

At TubeSaku, our website, we analyze Japanese YouTube gaming videos on a per-game basis, compiling weekly game rankings and tracking creator milestones.

Maintaining this system required a massive amount of manual work. Each week, we had to review videos that the automated classifier failed to identify, pinpoint the games being played, add new titles or aliases, correct ambiguous keywords, and then recalculate the rankings.

While popular titles are easy to spot and manage manually, challenges arise with new releases, indie games, niche communities, sequels, abbreviations, inconsistent naming conventions, and titles that overlap with existing game names.

This created a serious issue that went beyond mere data quality.

If TubeSaku failed to identify a game, the corresponding video might not be accurately reflected in the rankings. This meant that a creator achieving remarkable success within a niche community could see their accomplishments go unrecognized by the system.

This is a particularly critical issue for smaller-scale creators.

TubeSaku operates a "Creator Record System" designed to track and archive achievements, such as ranking highly for a specific game. You do not necessarily need millions of subscribers to achieve meaningful success; a creator with only a few hundred or thousand subscribers can still become the top creator for a specific game during a given week.

However, such achievements can only be discovered if the underlying data is classified correctly.

To ensure that "long-tail" games do not slip through the cracks of our classification system, we built "TubeSaku Ranking Sentinel."

Instead of waiting for humans to notice unregistered games, the production system can now detect and investigate its own classification errors, modify and verify the classification taxonomy, and measure whether those corrections actually improved ranking accuracy.

What it does

"TubeSaku Ranking Sentinel" is an agent that autonomously modifies and maintains TubeSaku's YouTube game classification taxonomy.

It monitors actual classification results in the production environment to identify videos that the current game dictionary fails to recognize.

Rather than immediately sending every unrecognized video to an LLM (Large Language Model), it prioritizes unresolved cases with significant impact. It checks YouTube's proprietary game metadata where available and invokes Gemini with Google Grounding only when further inference or web searches are required.

For unresolved cases, the agent performs the following operations:

Detects unclassified or questionable YouTube game videos. Extracts game candidates (identifying information). Groups multiple videos that appear to refer to the same game. Cross-references candidates against the existing game classification taxonomy. Uses Gemini and Google Search Grounding to verify whether the title corresponds to a real game. Determines whether it is a new release, an alias, a sequel, a remaster, a variation, or an incorrect match (false positive). Generates safe classification keywords. Detects conflicts (duplication or misidentification) with existing games. Updates the classification taxonomy only when strict safety conditions are met. Executes verification and regression testing. Rolls back unsafe changes. Reclassifies the production dataset. Measures the improvement achieved by the changes.

Cases that are extremely difficult to judge are not forced into the classification taxonomy but are instead escalated to a human review process.

The result is not merely an AI classifier; it is an agent capable of executing the following cycle on its own production system:

Perceive (Failure Detection) → Ground & Reason (Gemini + Search) → Safeguard & Plan (Collision & Variant Analysis) → Act (Taxonomy Patching) → Reflect & Validate (Regression Testing & Rollback)

Ranking Sentinel also features the ability to proactively handle upcoming games before they are released. You can test new titles discovered from external game sources—such as Steam—against existing matching systems before they trigger ranking errors.

Consider, for example, the following title:

ELDEN RING NIGHTREIGN

This title could be mistakenly subsumed under the existing keyword:

ELDEN RING

Traditional anomaly detectors miss these 'silent false positives' because the string technically matches. Sentinel detects this semantic collision, prevents ranking pollution, and prioritizes the new entity.

Ranking Sentinel proactively detects such conflicts; it verifies that the new title represents a distinct game and then generates a more specific, higher-priority rule.

How we built it

TubeSaku had already established a production-grade ranking generation pipeline based on Python, CloudSQL, YouTube data, and a rule-based game classification system (taxonomy).

For this hackathon, we built "Ranking Sentinel"—an autonomous decision-making layer (agent layer)—designed to wrap around that existing production system.

The core workflow is composed of the following elements:

• Core Agent Intelligence: Gemini (with native Google Search Grounding for live entity resolution)
• Agent Framework & Safety: Deterministic heuristic filter + Multi-stage validation guardrails + Automated regression harness
• Production Infrastructure: Google Compute Engine (GCE), CloudSQL, MySQL for test, SQLite State/Cache Store, Python

Key architectural principles include:

Prioritizing deterministic (certain) evidence and using AI only when inference is truly necessary.

For instance, if YouTube itself provides reliable game identification data that already exists within TubeSaku, there is no need to invoke Gemini at all.

If metadata is missing, inconsistent, or points to an unknown title, Gemini investigates candidates using live web information.

We also learned not to place blind faith in a single source.

In one real-world example, a video titled "MACHINE PARTY" was linked to Monster Hunter Wilds based on YouTube metadata. Ranking Sentinel detected this discrepancy by cross-referencing the platform's metadata against the title, other videos, and external evidence.

Furthermore, the system distinguishes between distinct game identities and mere spelling variations.

For example:

「ほの暮しの庭」 と 「ほの暮らしの庭」

...can be safely treated as aliases for the same game. However:

"EA SPORTS FC 24" and "EA SPORTS FC 26"

must never be normalized (merged) into the same title.

Similar safety logic applies to sequels, remasters, DLC, generic keywords, and substring matches (substring collisions).

The publicly available TubeSaku service runs on Google Compute Engine. Since the full ranking dataset for the production environment is managed within a relatively large MySQL database, the hackathon demo utilizes a compact, one-day dataset on GCE; this allows for a demonstration of the entire Sentinel workflow without copying the full production database.

This enables the demo to showcase the following sequence of steps:

Data closely resembling the production environment → Autonomous investigation → Classification system modification → Verification → Re-classification → Published results on Google Cloud

Challenges we ran into

The challenge hardest was not getting Gemini to identify game names.

The hard part was deciding when an AI-generated answer was safe enough to modify a production taxonomy.

Real-world game names contain many dangerous edge cases.

A short keyword such as Stack, for example, may refer to a legitimate game, but using the word stack as a substring matcher can create large numbers of false positives.

Similarly, a title may be a real product but still should not become a separate game entry. Remasters, complete editions, DLCs, demos, and rereleases all require different treatment.

We also found that platform metadata itself can be wrong.

That meant the architecture could not be:

YouTube says X → trust X

or:

Gemini says X → trust X

Instead, we needed multiple independent checks.

Another difficult problem was preserving precision.

Our goal is not to classify 100% of unknown videos. Some videos contain several games, omit the game name entirely, or cannot be matched safely.

Forcing those videos into a category would make the ranking less trustworthy.

So Ranking Sentinel intentionally prefers:

precision over coverage

and only self-repairs cases where the evidence is strong enough.

Caching was another important challenge. A production agent that repeatedly researches the same videos would be unnecessarily expensive and slow. We added caches for YouTube metadata, first-stage Gemini audits, and second-stage verification so repeat runs can reuse previous findings.

Finally, the existing production ranking system uses a large local MySQL dataset. For the hackathon demo, we needed to preserve the realism of the workflow while making it reproducible and practical to run on Google Cloud. We therefore isolated the minimum data required for a one-day end-to-end demonstration.

Accomplishments that we're proud of

The most important accomplishment is that Ranking Sentinel has already improved real TubeSaku production data.

📊 Production Impact (Single Autonomous Run on Top 200 Unmatched Videos):

• Classified Videos: 20,247 → 20,653 (+406 videos recovered autonomously)
• Unmatched Queue: 5,952 → 5,546 (-6.8% reduction)
• Candidate Resolution: 45 candidates processed
  • 41 verified as new standalone game entities
  • 3 safely mapped as game variants/aliases
  • 1 escalated to human-in-the-loop (0% hallucinated bad commits)
• Safety & Reliability: 201/201 classifier regression tests passed with 0 taxonomy collisions.

We treat graceful escalation as a first-class success metric, not a failure. A production-grade autonomous agent must have the metacognitive capability to know when not to act.

The Safe Apply stage successfully tested 41 new game entries while passing: 201 classifier regression cases and validating the updated game taxonomy without duplicate game IDs.

We also reduced repeated Gemini calls dramatically through caching. On repeated runs, most previously researched cases are resolved from cached evidence instead of calling the model again.

This demonstrated something important to us:

Ranking Sentinel is not just an LLM wrapped around a script.

It is a production-oriented agent that combines reasoning, tools, memory, deterministic safeguards, validation, rollback, and measurable feedback.

What we learned

The biggest lesson was that autonomy is not about giving an AI permission to change everything.

Useful autonomy requires knowing exactly where AI reasoning is valuable and where deterministic systems are safer.

We started with the idea of asking Gemini to identify missing games.

But the project evolved into something more interesting.

The system now asks several different questions:

Can existing deterministic metadata resolve this first? Do multiple videos provide consistent evidence? Is this actually a new game? Is it only an alias? Is it a sequel or a remaster? Could this keyword collide with another title? Is the platform metadata itself wrong? Is the proposed repair safe? Did the repair improve real production data? Should the agent act, or should it escalate?

We also learned that an agent can improve fairness in a data product indirectly.

Ranking Sentinel does not promote creators or manipulate rankings.

Instead, it improves the underlying observation system.

If a niche game is missing from the taxonomy, creators around that game are harder to measure. By making the taxonomy continuously adapt to new and long-tail games, the system gives those communities a better chance to appear accurately in rankings and historical records.

That distinction became central to the project:

We are not changing who wins. We are improving our ability to notice who already did.

By combining Google's frontier reasoning with self-healing deterministic safeguards, Ranking Sentinel demonstrates how agentic systems can reliably sustain real-world data infrastructure without human operational fatigue.

What's next for TubeSaku Ranking Sentinel

The immediate next step is to make the entire Sentinel workflow a continuously running production agent.

Instead of manually chaining individual stages, one execution will handle:

detect → verify → repair → validate → reclassify → measure

with safe cases applied automatically and uncertain cases routed to human review.

We also plan to expand proactive discovery beyond Steam to sources such as:

Nintendo PlayStation Xbox game publisher announcements other release feeds

The same architecture can eventually extend beyond games.

TubeSaku is also developing rankings for areas such as cover songs, where the underlying problem is remarkably similar:

new entities appear continuously, names change, aliases emerge, and a static taxonomy gradually becomes outdated.

That leads to a broader concept:

TubeSaku Entity Discovery Agent

A continuously learning layer that watches real creator activity, discovers entities that the current system does not understand, verifies them against the live web, and safely updates the taxonomy behind creator analytics.

Our long-term goal is not simply to maintain better rankings.

It is to make niche success easier to discover.

A small creator may never appear on a global trending page, but they may still be the #1 creator for a particular game, community, or format during a particular week.

Those achievements can become useful proof of expertise, audience fit, and creator-market relevance.

By making long-tail games and communities visible, we hope TubeSaku can create more opportunities for discovery, collaboration, sponsorship, and monetization for both emerging creators and indie game developers.

Built With

• gce
• gemini
• googlegenai