Fallout(ish)

What if we used an LLM as game engine.

Comment

Inspiration

I love New Vegas, and was curious how much "freedom" would this give. I wanted to offer both an immersive experience, and a funny non-sense on for those who wanted. I think I managed to offer both haha.

What it does

Fallout "parody" using an LLM as game-engine. Everything is AI-gen, from the scenarios, to the visuals. You can create your character, define your stats, and jump into the world. There is not "defined" objectif, it's up to you to act as you feel like.

How we built it

  • Backend: FastAPI orchestrates turns, tracks state, and serves audio/images.
  • OpenAI gpt‑4.1‑mini drives narrative and tool calls.
  • Using FAL for model calling
  • FLUX.1 SRPO for the first image generation
  • FLUX.1 Kontext [dev] for most of visual generation (identity‑preserving, portrait edits, scenes).
  • ElevenLabs powers TTS and STT.
  • Frontend: React + Vite + Tailwind.
  • PSQL: to save game states

Challenges we ran into

  • Latency: If the visuals are amazing but have to wait 5mins per turns, it's not fun to play. It's a balance (that's also where Kontext [dev] shines)
  • Keeping portrait identity consistent across wildly different scenes while still feeling reactive.
  • Designing prompts and tool‑calling loops that are robust (NPC creation, visual edits) yet predictable.
  • Making the UI “feel” like a game: timing audio/image swaps, adding loading overlays, and micro‑feedback.
  • Time: I managed to lock my 7am to 8am everyday(ish) to work on it, 1h is not a lot of time ^^'

Accomplishments that we're proud of

  • Cohesive loop: talk -> world reacts -> you hear it + see changes.
  • Convincing identity: thanks Kontext.
  • Clean, hackable : Code is OSS and pretty easy to read, would love to see people contribute
  • Overall results: Honeslty, I'm quite happy about the outcome; that wasn't an easy one.

What we learned

  • SRPO is sooooo fast :D
  • You can actually use an LLM to prompt your AI-gen
  • Kontext can take way more context than I though (even more knowing it's the "[dev]" version)
  • Small UX touches can make the different

What’s next for Fallout(ish)

  • Streamed audio for near‑instant narration; tighter visual timing.
  • Character creation via image import/webcam; NPC faces generated then edited in‑scene.
  • Progression: XP, level‑ups, health, inventory, and competencies.
  • Longer contextual memory for consistent NPCs and richer situations.
  • Multiplier support (imagine if other players' action impact your own world)

Built With

Share this project:

Updates