Geometric Linguistic Transformations

Discover and test reusable geometric directions for negation, tense, questions, and other linguistic transformations inside transformer embedding spaces.

Comment

Inspiration

Hi, I’m Anna.

I’m thirty-six, I have worked in IT for about fourteen years, and this project began because one day I decided to apply for a master’s program in machine learning.

It is one of those strong programs where you can receive a scholarship, but first you have to show that you are capable of doing something real.

And then I remembered something.

A very long time ago, in the distant year of 2012, I graduated from university and defended a thesis connected with Lie algebras.

After that came fourteen years of ordinary IT life: systems, databases, architecture, documentation, code, teams, deadlines, production incidents, and approximately one million things that had very little to do with academic mathematics.

I had forgotten a lot.

Some ideas that I once understood now looked completely unfamiliar. I had almost no recent academic practice. I did not have a laboratory, a university position, a scientific team, or fifteen free years to study everything again from the beginning.

So I asked GPT-5.6 a simple question:

Can ideas from Lie algebra be useful for understanding transformers?

I did not ask it to write a paper for me.

At first, I only wanted to understand whether the question made any sense.

That conversation became an experiment. The experiment became a repository. The repository became a growing independent research program.

The original question was this:

When a transformer changes a statement into a negation, a question, another tense, or another linguistic form, does that change leave a repeatable geometric trace inside the model?

Not only a word such as “not”.

Not only a question mark.

Not only the final form of the sentence.

But the transformation itself.

The project was inspired by Lie algebra, but the current results do not prove that transformers contain a Lie algebra. That is still an open and much more difficult question.

What it does

The basic idea is simple.

I take two related sentences:

She is happy.
She is not happy.

A transformer converts each sentence into a numerical representation called an embedding.

I then calculate the difference between the two embeddings:

delta = embedding(transformed sentence) - embedding(original sentence)

This delta is a vector: a direction and a distance inside the model’s embedding space.

Then the research asks:

  • Do examples of the same transformation move in similar directions?
  • Can we distinguish negation from a question, a tense change, uncertainty, passive voice, or formalization?
  • Does the pattern remain when the test sentences discuss completely different topics?
  • Does the delta contain information that is not already present in the transformed sentence?
  • Do different transformer architectures organize these transformations in similar ways?
  • Can transformations be combined?
  • Are the results real, or are we simply detecting surface clues such as particular words and punctuation?

The repository is a working and reproducible research workflow.

It contains:

  • experiment scripts;
  • controlled linguistic datasets;
  • embedding extraction and analysis pipelines;
  • source-only, target-only, concatenation, and delta baselines;
  • semantic and harder holdout experiments;
  • statistical controls;
  • result tables and figures;
  • reproducible report builders;
  • draft research papers;
  • reviewer-response notes;
  • a dated research diary;
  • positive, negative, and mixed results.

The current evidence suggests that the relationship between two sentence embeddings can contain useful information about a linguistic transformation beyond the information found in either endpoint alone.

But this is not presented as a finished theorem.

Some results are strong. Some are mixed. Some beautiful early results turned out to be misleading.

The repository preserves all of them.

How I built it

I want to be unusually honest about this part:

I wrote very little of the code by hand. Codex wrote most of the implementation.

I did not suddenly remember every mathematical concept, research method, Python library, and experimental technique that I had not used for fourteen years.

Instead, I worked with Codex as a coding and research partner.

I brought the question.

I decided what I wanted to test.

I chose which results looked important and which looked suspicious.

I decided when an experiment was not strong enough, when another baseline was needed, and when a beautiful conclusion had to be removed.

Codex turned those decisions into working code.

It helped me:

  • implement experiment pipelines;
  • refactor research scripts;
  • generate and process datasets;
  • run model comparisons;
  • add harder controls and ablations;
  • calculate statistical results;
  • build tables, figures, and reports;
  • organize the repository;
  • document the experiments;
  • preserve a traceable history of the work.

GPT-5.6 helped in a different way.

It helped me return to mathematical ideas I had forgotten. It explained unfamiliar concepts in plain language. It helped me read papers paragraph by paragraph, compare my results with related work, formulate objections, and turn unclear questions into experiments that could actually be tested.

But AI did not decide what the evidence meant.

When a result looked almost perfect, I did not ask Codex to make a better chart.

I asked it to try to break the result.

Those additional experiments showed that some perfect syntax scores could be explained by surface clues in the transformed sentence.

That changed the conclusion of the research.

So the real workflow was not:

AI generates code, and I accept the result.

It was:

I ask a question. Codex implements an experiment. We inspect the result. I challenge it. Codex implements another test. The claim becomes smaller, more careful, and hopefully more true.

I remain responsible for the questions, decisions, interpretations, and any mistakes that are still present.

The research foundation existed before OpenAI Build Week.

During Build Week, I used Codex and GPT-5.6 to make the project easier to inspect, reproduce, test, and review, and to document clearly which parts existed before the event and which work was completed during the submission period.

Challenges I ran into

The first challenge was returning to research after fourteen years away from academic mathematics.

I often did not understand a term in a paper.

Sometimes I remembered the name of a concept but not how it worked.

Sometimes I did not know whether my question was interesting, obvious, impossible, or simply wrong.

AI made it possible to cross that first wall quickly, but understanding an explanation is not the same as proving that the explanation is correct.

The second challenge was separating real geometric information from easy shortcuts.

A model may appear to recognize negation because the transformed sentence contains the word “not”.

It may recognize a question because of punctuation or sentence order.

It may appear to discover a transformation while actually learning only the final form of the sentence.

That is why the project compares delta vectors with the original sentence, the transformed sentence, and both endpoints together.

The third challenge was resisting the desire to make the result sound larger than it really was.

The project began with Lie algebra.

That is an exciting idea.

But an exciting origin does not mean that the final evidence proves algebraic structure.

Some checks that initially looked impressive turned out to be mathematical or implementation identities rather than scientific evidence. Some hard-holdout experiments weakened the original claim. Some transformations behaved better than others.

I kept these results instead of hiding them.

The final challenge was working with AI itself.

Codex can write code very quickly. GPT-5.6 can give a clear and convincing explanation.

Neither speed nor confidence guarantees correctness.

The generated code still has to be run. The results still have to be checked. References still have to be verified. A convincing interpretation still has to survive alternative explanations.

Accomplishments that I am proud of

I am proud that this became a real, open, reproducible research project instead of remaining one interesting conversation with GPT.

The repository now contains experiments across multiple transformer architectures, progressively harder evaluation settings, statistical controls, reports, figures, and separate research directions.

But I am most proud of the parts where the project proved itself wrong.

One early experiment produced perfect classification results.

It would have been very easy to place that number at the top of the README and call it a breakthrough.

Instead, I asked for stronger baselines.

The new tests showed that the endpoint sentence already contained enough surface information to solve much of the task.

The perfect result was not evidence of deep geometric generalization.

Finding that out made the project better.

I am also proud that the repository reports boundary conditions where delta vectors do not win, and that the Lie-inspired experiments are described as diagnostics rather than as proof of a Lie algebra.

A negative result is not a failed experiment when it prevents a false conclusion.

And, personally, I am proud that at thirty-six, after fourteen years in industry, I returned to a type of work that I had almost decided no longer belonged to me.

What I learned

I learned that AI does not have to replace human thought.

It can remove the wall that prevents a person from reaching the work in the first place.

Without AI, returning to this field would have required months just to remember the vocabulary, find the right papers, understand unfamiliar methods, and learn how to implement every experiment.

With AI, every unknown concept became a question I could ask immediately.

A paper could be discussed one paragraph at a time.

A vague idea could become a small experiment.

A suspicious result could be attacked from several directions in one evening.

This does not make universities, professors, experienced researchers, or peer review unnecessary.

It does something else.

It makes the entrance wider.

My research may be imperfect.

Some of its conclusions may change.

The main hypothesis may eventually turn out to be unimportant, incorrect, or completely useless.

That possibility does not make the attempt worthless.

There are more than eight billion people on Earth.

Imagine that one billion of them chose one scientific question they genuinely cared about and spent only two to four hours a week exploring it with the help of AI.

Most of those experiments would not become famous papers.

Many would fail.

Some would repeat things that were already known.

Some would be badly designed.

Some, including mine, might lead nowhere.

But a small part of them would find a new connection, notice an overlooked problem, test an unusual idea, improve an existing method, or solve something that makes life easier for millions of other people.

One billion people contributing even a tiny amount of honest curiosity would create an amount of scientific attention that humanity has never had before.

That is the most important idea behind this project.

Not that everyone is suddenly a professor.

Not that AI always knows the answer.

But that many more people can now begin asking serious questions and making small, verifiable contributions.

Science does not become weaker when more people are allowed to approach it.

It becomes larger.

What was completed during Build Week

During OpenAI Build Week, the project did not become a separate demo app. Instead, Codex helped turn the existing research repository into a clearer, more testable research workflow and added a new behavior-level track called GLT-STEER.

The main completed Build Week work was:

  • a focused GPT-2 question activation-steering experiment;
  • a no-steering base-rate control showing that GPT-2 does not naturally add question marks under the tested declarative prompts;
  • an out-of-template control using freeform declarative sentences that were not part of the training templates;
  • prompt-robustness checks across multiple prompt wrappers;
  • a content-preservation audit asking whether steering only adds punctuation or also preserves source content;
  • a copy-prompt follow-up showing that copy-like prompts preserve source content much better while still adding question form;
  • a no-steering copy-prompt baseline showing that copy-like prompts alone do not produce question marks;
  • a DistilGPT-2 replication showing that the effect transfers qualitatively but becomes much weaker;
  • a first negation-steering attempt showing that the question-steering recipe does not automatically transfer to other transformations;
  • a harder out-of-template question audit showing that the question effect survives structurally diverse sentences, although content preservation becomes harder;
  • a delta-coherence diagnostic showing that GPT-2 question deltas are much more internally coherent than negation deltas, which helps explain why question steering works more cleanly under the current method;
  • a full GPT-2 layer sweep showing that negation still does not become a clean steering target just by changing the intervention layer;
  • an exclamation-mark control showing that final punctuation markers are much easier to steer than sentence-internal negation;
  • a GPT-2 vs DistilGPT-2 question-delta norm diagnostic showing that DistilGPT-2 compresses later-layer question directions;
  • a direct DistilGPT-2 layer/gain sweep showing that the earlier weak replication was parameter-sensitive rather than a hard model failure;
  • public result summaries and CSV artifacts in the repository;
  • cleanup of public repository structure so the visible repo remains a research artifact rather than a pile of submission notes.

The most recent result is important because it narrows the claim in a useful way.

The steering vector is not only causing GPT-2 to emit random question marks. Under copy-like prompts, it often preserves the source content and adds question form. But this works much better for prompt formats that already encourage copying or repeating the source sentence, and it remains weak for bare or quoted prompts.

The follow-up no-steering audit is important: copy-like prompts by themselves produced 0.0000 question marks across 960 no-steering rows from GPT-2 and DistilGPT-2. This means the prompt explains source retention, but it does not explain the question-form effect.

The DistilGPT-2 replication is useful but humbling. It preserves the qualitative target-versus-control separation, but the best question-and-preserved rate drops to 0.4625. That means the effect is not simply universal across GPT-family models; model dependence is now part of the honest story.

The first negation-steering attempt is also important. It does not reproduce the question result. Under the same copy-like prompt design, negation target rows do not clearly beat matched controls. This means the project now has a sharper boundary: question steering works best so far, but other transformations need different prompts, metrics, or steering methods.

The harder out-of-template audit is encouraging. It uses passive constructions, subordinate clauses, proper names, and numeric/time expressions. The question vector still produces question marks at rates between 0.7125 and 0.7500, while matched controls peak at 0.0375. The content-preserved rate is lower than in simpler sentences, which is exactly the honest boundary we should report.

The newest diagnostic helps explain the question-versus-negation split. In GPT-2 hidden states, question transformation deltas are much more clustered than negation deltas across all layers. At the layers used for steering, question mean pairwise cosine is about 0.94-0.97, while negation is about 0.56. This suggests that the question vector is a cleaner intervention direction, while negation is geometrically more heterogeneous and probably harder for an autoregressive model to express by a single simple steering vector.

The layer sweep strengthens that boundary: trying every GPT-2 layer did not reveal a clean negation intervention site. The best negation target-and-preserved rate reached only 0.1729, with controls still nontrivial.

The exclamation control is a useful sanity check. A trivial final-marker transformation, statement -> statement!, steers very cleanly: up to 1.0000 exclamation-and-preserved in-template and 0.8000 on hard out-of-template sentences. This supports the idea that the question result is partly helped by the fact that ? is a final surface marker, while negation requires changing sentence-internal structure.

The GPT-2 vs DistilGPT-2 diagnostic also explains part of the weaker DistilGPT-2 replication. DistilGPT-2 does not have smaller question deltas everywhere, but its later-layer question directions are much smaller than GPT-2's. At the final relative layer, the mean-norm ratio is 0.3341 and the centroid-norm ratio is 0.2923.

The direct DistilGPT-2 layer/gain sweep revises that story. The earlier aggregate replication looked weak, but tuning the intervention site and gain recovers a strong setting: at gain=1.0, layer 2, and same_sentence prompts, DistilGPT-2 reaches question-and-preserved rate 0.8250, with matched controls at or below 0.0500. The same sweep also shows that gain matters: gain=0.5 is usable but weaker, while gain=1.5 over-steers and damages preservation.

The current careful claim is:

A GPT-2 question-transformation activation vector can steer output toward question form, and under copy-like prompts it can often preserve recognizable source content. Copy-like prompts alone do not create the question marks in the tested setup. DistilGPT-2 is also steerable, but it is much more sensitive to layer and gain: the best direct sweep setting reaches 0.8250 question-and-preserved, while over-strong gain collapses preservation. A first negation attempt and a full-layer negation sweep fail to produce a clean negation editor. A follow-up diagnostic suggests that negation deltas are much less geometrically coherent than question deltas, and an exclamation control shows that final punctuation markers are much easier to steer than sentence-internal transformations. A harder out-of-template audit supports generalization for question form but shows that content preservation gets harder as sentences become more complex. This is not yet robust general-purpose semantic editing.

This is exactly the kind of result I want the project to preserve: positive, useful, but bounded.

What is already done

The current public repository contains:

  • GLT-DV: delta-vector diagnostics with endpoint controls;
  • GLT-SPOT: signed-permutation tests for ordered linguistic composition;
  • GLT-MOLT: learned matrix/operator diagnostics and null controls;
  • GLT-XFER: cross-model transfer and alignment stress tests;
  • GLT-AFFECT: early affective-scale geometry controls;
  • GLT-STEER: activation-steering experiments for GPT-2 and DistilGPT-2 question formation;
  • requirements and reproducibility notes;
  • result CSVs and experiment summaries;
  • research drafts and roadmap files;
  • Zenodo/citation metadata for archived research snapshots.

What’s next

The immediate next step is criticism.

I invite the judges, researchers, and developers who see this project to examine the methods, code, controls, conclusions, and negative results as critically as possible.

I would rather discover a serious problem now than build a beautiful theory on top of a weak assumption.

The technical research will continue with:

  • stronger multi-seed validation;
  • more difficult semantic controls;
  • naturally written text instead of only controlled examples;
  • additional modern transformer architectures;
  • more languages;
  • cross-model alignment;
  • better tests of transformation composition;
  • stricter semantic-preservation metrics for GLT-STEER;
  • additional replication of question steering on another model family or another seed;
  • redesigned steering tests for transformations beyond question formation, especially negation, tense, and modality;
  • experiments that test whether geometric directions can change model behavior, not only classify transformations.

The longer-term question is whether similar methods can describe transformations connected not only with grammar, but also with uncertainty, attitude, emotion, intensity, and subjective experience.

Perhaps one day models could be trained and controlled using richer representations of how meaning changes, rather than only through words, labels, and examples.

Perhaps this research will contribute to that.

Perhaps it will only show that one particular approach does not work.

Both outcomes are useful when they are tested honestly and shared openly.

This project began with one slightly unreasonable question.

I hope it encourages someone else to ask theirs.

Anna, thirty-six, with love for everything new and not yet explained.

Built With

Share this project:

Updates