MiniMind

Inspiration

Large language models have become powerful tools for programming, but they require huge datasets and computing resources. We wanted to understand how these models actually work by building a smaller one from scratch. MiniMind was created to explore how much coding ability a compact language model can learn when it focuses on a specific domain instead of trying to be a general-purpose AI.

What it does

MiniMind is a small Transformer-based language model specialized in programming code. It learns to predict the next token in a code sequence and can generate code based on a given prompt. Generated code can also be tested with functional test cases, helping us measure whether the model produces code that actually works rather than simply looking correct.

How we built it

We created an end-to-end pipeline: Code Dataset → Data Cleaning → Tokenizer → Transformer → Training → Evaluation → Code Generation → Testing The model is trained from scratch and designed to stay within the hackathon's 50M trainable-parameter limit. We evaluate the model using metrics such as validation loss, perplexity, code-generation performance, and functional correctness.

Challenges we ran into

The biggest challenges were learning and implementing concepts such as tokenization, embeddings, self-attention, backpropagation, and Transformer training. We also had to balance model size, dataset quality, training time, and available computing resources while staying within the parameter limit.

Accomplishments that we're proud of

We built a complete language-model pipeline instead of relying on a pretrained commercial model. MiniMind covers the entire process from preparing code data to training a Transformer, generating code, and evaluating the generated programs. We are especially proud of turning complex ML concepts into a working project that we can experiment with and understand.

What we learned

We learned how data, tokenization, model architecture, training, and evaluation work together to create a language model. We also learned that good-looking generated code is not necessarily correct, which made functional testing an important part of our project.

What's next for MiniMind

Future versions could support multiple programming languages, improve code-generation quality, add instruction-following capabilities, and use more advanced evaluation methods. We also want to explore model optimization and efficient on-device inference so that a small coding model can run with limited computing resources.

Built With

• artificialintelligence
• byte-pair-encoding
• code-generation
• code-llm
• cuda
• decoder-only-transformer
• deeplearning
• generative-ai
• git
• github
• gpu
• jupyternotebook
• largelanguagemodels
• llm
• machine-learning
• mixed-precision
• natural-language-processing
• naturallanguageprocessing
• neuralnetworks
• python
• pytorch
• self-attention
• tokenization
• tokenizer
• transformers