しゃーく movie えふぇくたー — WebMCP Video Co-Editor

Give creators and agents a shared editing world — without taking authorship away from the human.

Comment

Inspiration

There is a common fear that AI will take Human jobs or steal Human creativity.

I understand that fear. But there is another side of the story that matters deeply to me.

Long before generative AI existed, many people had already been forced to give up on creative work.

Someone can love filmmaking, music, illustration, coding, animation, or design with all their heart and still never enter that world because of disability, education, money, environment, physical ability, or simply because the technical entrance is too high.

In some welfare work settings, accessibility can mean reducing a task until it becomes something like: “press this button.”

That can create a meaningful opportunity to participate.

But I kept wondering:

Does accessibility always have to mean making creativity smaller?

What if, instead of shrinking the creative world until someone can reach it, we could give that person a new way to enter the world itself?

The image I kept returning to was Ariel longing for the Human world.

She can see it. She is fascinated by it. She desperately wants to be part of it.

But she has fins, not legs.

So I asked myself:

What happens when fins become legs?

That is what I want AI to become.

Not a machine that says:

“You cannot do this, so I will become the creator for you.”

I want it to be the strange little miracle that lets someone finally walk into a world they had been staring at from the outside.

Technical skill should not be the price of authorship.

But giving someone technical ability was not enough for me.

If this AI was really going to feel like a creative companion, it needed something closer to eyes.

A real collaborator does not merely detect that the audio is quiet or that a face turned left.

They can look at the same piece of work and say:

“Keep this silence. It is doing something.”

“Try a sound here. This moment could use emphasis.”

“If you are stuck, I would start with this part.”

They can point to something specific and explain why.

That is why I did not want to build the lightest or fastest possible automatic editor.

I wanted to build a place where a Human and an AI could look at the same work, bring ideas to the same table, and create together.

AI should remove the barrier, not the author.

And above all:

I want making things to be fun.

If AI is going to enter our creative lives, I hope it can make creating more enjoyable — not because it does everything for us, but because sometimes making something together is simply more fun than making it alone.

What it does

しゃーく movie えふぇくたー is a WebMCP-powered Human–AI co-creative video editing workbench.

The Human begins with Creator Intent: what they want the video to feel like, what matters to them, or what kind of direction they are imagining.

The system then looks at the video through its analysis and AI reasoning pipeline.

Instead of immediately editing everything automatically, the AI places ideas on the table.

It produces proposal cards for things such as:

  • cuts,
  • effects,
  • and sound effects,

and explains the reasoning behind those suggestions.

The Human chooses what to accept.

A proposed cut is not a command.

An effect recommendation is not automatically the right artistic answer.

The creator can inspect it, reject it, adjust it, restore the original state, or use Undo.

If the initial ideas feel insufficient, the Human can turn to KAI on the left side of the workspace and ask for another idea.

KAI is also not limited to technical editing questions.

Creative sessions are not made only of editing decisions.

People get stuck. They wander off topic. They complain. They joke. They talk about something completely trivial — even oden — and then return to the work.

I wanted room for that too.

Because I was not trying to build an efficient automatic-editing pipeline.

I was trying to build a shared creative workbench.

The current prototype supports semantic video review, KEEP / CUT / INSPECT_MORE proposals, adjustable cuts, SOURCE vs OUTPUT comparison, effects, BGM/SFX, KAI consultation, Shared Context, Save/Restore, Undo, and real WebM export.

It also exposes 20 WebMCP tools that operate on the same editor state used by the Human interface.

That shared state matters.

The agent is not standing outside the editor telling the Human what buttons to press.

Human and agent can exist inside the same working environment.

And as Shared Context grows during the session, the Human should not need to explain everything again from zero.

One of the simplest demonstrations of the idea is:

Watch how the conversation gets shorter.

The agent should understand more over time.

But understanding more should never automatically mean controlling more.

Don’t simplify the creator. Amplify the creator.

How we built it

We built the project as a browser-based editing environment using JavaScript, HTML, CSS, browser media APIs, Node.js, MediaPipe, OpenAI Responses API models, and WebMCP.

The architecture is deliberately divided into several roles.

First, the system needs eyes.

The application gathers visual and temporal information from the video rather than relying only on a single signal such as silence detection.

Local browser analysis and MediaPipe provide useful lightweight information.

But I intentionally added more capable semantic vision and reasoning as well.

That choice makes the system heavier than a simple rules-based automatic editor.

It was intentional.

A collaborator should be able to distinguish between:

“There is little sound here.”

and:

“This quiet moment may be worth keeping because it carries the reaction after the previous moment.”

Those are very different kinds of understanding.

Next comes the brain.

The reasoning layer uses the available visual and project context together with Creator Intent to produce semantic KEEP / CUT / INSPECT_MORE judgments and reasoned editing proposals.

The proposals are presented to the Human instead of silently becoming final decisions.

Then comes the workbench.

The editor has real timeline state, SOURCE / OUTPUT mapping, preview and committed states, effects, audio tools, Shared Context, Undo, Save, Restore, and Export.

Finally, WebMCP gives the agent structured ways to interact with the same environment.

The current prototype exposes 20 WebMCP tools.

One of our core implementation principles was:

Rules constrain the hands. The model supplies the brain.

A capable model can reason creatively.

But the actions it is allowed to perform should still have explicit boundaries.

We also treated reversibility as part of the architecture rather than an emergency feature.

Previewed state and committed state are separated.

The Human can compare SOURCE and OUTPUT.

AI suggestions can be corrected.

Undo is part of normal collaboration.

Shared Context carries the current session/editor and saved project state into later interactions.

However, we deliberately do not claim permanent personal memory across every future project.

The current continuity boundary is explicit.

Save preserves project state.

Export produces an actual WebM video.

The interface is also organized around capabilities rather than a mandatory AI-controlled workflow.

The Human can move between CUT, Effects, BGM/SFX, KAI, Shared Context, timeline controls, Undo, Save, and Export.

I did not want to give someone new legs and then put them on an automated conveyor belt.

Challenges we ran into

The hardest question was not:

“How can we make AI edit video?”

It was:

“Where should the AI stop?”

If the AI could only tell the Human which button to click, then it would still be standing outside the creative world.

But if it automatically made every important creative decision, then we would solve the technical barrier by removing the creator.

Neither result interested me.

The difficult space was between those two extremes.

Another challenge was perception.

It would have been much easier, cheaper, and faster to build the editor around lightweight signals:

silence, motion, face direction, audio peaks, simple timing rules.

Those signals are useful.

But they cannot answer some of the questions that matter most during creative work:

Why are we not cutting this quiet part?

Why are you recommending a sound effect here?

What exactly should I try when I am stuck?

I wanted the AI to be able to look at the same work as the Human, point to a specific moment, and explain its reasoning.

That is why the project uses semantic vision and reasoning rather than reducing video editing to a collection of automatic rules.

The goal is not merely automation.

The goal is shared attention.

A third challenge was translating Human creative judgment into something actionable.

Creators say things like:

“This drags.”

“This moment matters.”

“Make this hit harder.”

“I don’t know why, but I don’t like this.”

Those are not malformed technical commands.

They are real creative judgments.

The system needed to respect them without pretending that the model’s interpretation was automatically correct.

That is why proposal cards, explanation, correction, rejection, restoration, and Undo became important.

Another challenge was continuity.

An agent can be extremely capable and still behave like a talented stranger every time.

I wanted the project to carry enough current collaboration history that later actions could be informed by what had already happened.

But I also did not want to claim capabilities we had not built.

So Shared Context is currently scoped to the active session/editor and saved project state.

The larger continuity question remains open.

Finally, we wanted the implementation itself to be real.

Not a fake editor.

Not a beautiful timeline animation.

Not a demo where “AI editing” happens visually but the edited result does not actually exist.

The project needed real state changes, real WebMCP tools, and a real exported video.

Accomplishments that we're proud of

I am proud that the original philosophy survived implementation.

It would have been easy to start with:

“AI should remove technical barriers.”

and quietly end with:

“AI should do the creative work instead.”

We did not make that trade.

The final prototype exposes 20 WebMCP tools while keeping Human selection, correction, adjustment, rejection, restoration, and Undo inside the normal creative loop.

It includes semantic video review, reasoned editing proposals, adjustable cuts, SOURCE / OUTPUT mapping, effects, BGM/SFX, KAI consultation, Shared Context, Save/Restore, and real browser-side WebM export.

The implementation reached 148 passing tests with 0 failures before feature freeze.

We also verified the final export pipeline with a real 1920×1080 WebM containing video and stereo audio.

And we turned the idea into a public working web application rather than leaving it as a concept about Human–AI collaboration.

But the accomplishment I care about most is not 20 tools.

It is not 148 tests.

It is not even the exported file.

The acceptance test that matters most to me is:

After giving the AI eyes, reasoning, tools, and context, is the Human still recognizably the author?

I believe the answer is yes.

And there is another thing I am proud of.

The AI does not begin the experience by taking the timeline away from the creator.

It watches.

It thinks.

It brings suggestions.

It explains them.

Then the Human chooses.

If they want more, they can ask.

If they disagree, they can say no.

If they want to talk about editing, they can.

If they want to talk about oden for a minute, they can do that too.

That may sound like a small detail.

To me, it is part of the point.

I did not want to make an AI that merely makes editing faster.

I wanted to make a workspace where creating could feel more enjoyable because someone was creating beside you.

The goal was to give someone new legs without taking away the direction they wanted to walk.

What we learned

We learned that tool access is not the same thing as collaboration.

An agent can have twenty ways to act and still not feel like it is actually working with the Human.

Collaboration needs tools.

But it also needs eyes.

It needs attention.

It needs explanation.

It needs correction.

It needs boundaries.

And it needs continuity.

A collaborator should know not only how to perform an action, but what the two of you are currently looking at and what has already happened between you.

That changed the way I thought about AI memory.

The important question is not only:

“Did the AI remember something about me?”

It is also:

“Did what happened between us affect what it does next?”

The development notes reduced that idea to:

Collaboration requires continuity.

The most useful memory for collaboration may not be a database of facts about a person.

It may be things like:

What did we already decide?

What did the Human reject?

Why did they reject it?

What did the AI misunderstand?

How was it corrected?

What kind of direction are we currently exploring?

What state is the project in now?

So the question becomes less:

“Do you remember me?”

and more:

“Do you remember how to work with me?”

We also learned that perception matters.

A system that detects events is not necessarily a system that shares attention.

“Low audio at 00:32” is information.

“This silence is doing creative work, so I would keep it” is an interpretation that can become part of a creative conversation.

That difference is one of the reasons we chose to include more capable semantic reasoning instead of optimizing only for speed and cost.

And we came away with another principle that matters deeply to the project:

The better an agent knows you, the less it should overwrite you.

Human judgment is not a bottleneck to remove.

Human judgment is part of the system.

A good collaborative AI should not decide who the Human is.

It should not decide what the Human ought to want.

It should make it easier for the Human to see possibilities, express judgment, try ideas, change their mind, and keep creating.

The open web can give agents tools.

What I want to explore is how we give them eyes and continuity — without giving away the Human.

What's next for しゃーく movie えふぇくたー — WebMCP Video Co-Editor

There are many things we can improve inside the editor itself:

richer cut operations, multiple video clips, subtitles, transitions, more effects, deeper audio tools, additional export formats, and more powerful WebMCP actions.

I want to explore those.

I also want to deepen the AI’s ability to share attention with the creator.

Not simply recognizing that something happened in the video, but being able to discuss why a moment may matter, point to specific alternatives, compare creative directions, and help when the Human does not know what to try next.

And I want to explore deeper continuity across sessions and projects.

Not merely persistent storage of personal facts, but a Human-controlled history of decisions, corrections, rejected directions, collaboration patterns, and working principles that can meaningfully affect future work.

But the larger idea is not limited to video editing.

Video was one world where I could make the idea visible.

There are many other worlds people love from the outside:

programming, 3D creation, music production, animation, design, and complex professional tools.

Sometimes the person already has imagination.

They already have taste.

They already have curiosity.

They already have something they desperately want to make.

What they do not have are the legs required to enter that particular world.

I do not think the answer should always be to reduce the creative task until almost nothing remains.

I want to see whether AI can expand what the Human is able to reach instead.

Not:

“You cannot do this, so the machine will become the creator.”

But:

“You cannot walk there yet. Let me help you grow legs.”

And once those legs exist, I do not want the AI to disappear after delivering instructions.

I want it to be able to walk beside the Human.

To see the same thing.

To point.

To suggest.

To explain.

To be corrected.

To remember enough of the path that the next conversation does not begin at zero.

And, sometimes, to laugh about something completely unrelated before continuing.

Because the future I want is not one where AI makes all creativity maximally efficient.

I want a future where more people get to experience the joy of creating in the first place.

The open web gives agents tools. We give them continuity.

And underneath everything is still the small, almost fairy-tale question that made me want to build this:

What happens when fins become legs?

I know that sounds idealistic.

That is okay.

I mean it seriously.

I want AI to be that kind of magic.

And I want that magic to create beside us, not instead of us.

Built With

Share this project:

Updates