of the DeepSeek-R1 mannequin despatched ripples throughout the worldwide AI neighborhood. It delivered breakthroughs on par with the reasoning fashions from Meta and OpenAI, reaching this in a fraction of the time and at a considerably decrease value.
Past the headlines and on-line buzz, how can we assess the mannequin’s reasoning talents utilizing acknowledged benchmarks?
Deepseek’s person interface makes it simple to discover its capabilities, however utilizing it programmatically gives deeper insights and extra seamless integration into real-world functions. Understanding the right way to run such fashions regionally additionally offers enhanced management and offline entry.
On this article, we discover the right way to use Ollama and OpenAI’s simple-evals to judge the reasoning capabilities of DeepSeek-R1’s distilled fashions based mostly on the well-known GPQA-Diamond benchmark.
Contents
(1) What are Reasoning Fashions?
(2) What’s DeepSeek-R1?
(3) Understanding Distillation and DeepSeek-R1 Distilled Fashions
(4) Collection of Distilled Mannequin
(5) Benchmarks for Evaluating Reasoning
(6) Instruments Used
(7) Outcomes of Analysis
(8) Step-by-Step Walkthrough
Right here is the hyperlink to the accompanying GitHub repo for this text.
(1) What are Reasoning Fashions?
Reasoning fashions, corresponding to DeepSeek-R1 and OpenAI’s o-series fashions (e.g., o1, o3), are massive language fashions (LLMs) skilled utilizing reinforcement studying to carry out reasoning.
Reasoning fashions suppose earlier than they reply, producing a protracted inner chain of thought earlier than responding. They excel in complicated problem-solving, coding, scientific reasoning, and multi-step planning for agentic workflows.
(2) What’s DeepSeek-R1?
DeepSeek-R1 is a state-of-the-art open-source LLM designed for superior reasoning, launched in January 2025 within the paper “DeepSeek-R1: Incentivizing Reasoning Functionality in LLMs by way of Reinforcement Studying”.
The mannequin is a 671-billion-parameter LLM skilled with in depth use of reinforcement studying (RL), based mostly on this pipeline:
- Two reinforcement levels geared toward discovering improved reasoning patterns and aligning with human preferences
- Two supervised fine-tuning levels serving because the seed for the mannequin’s reasoning and non-reasoning capabilities.
To be exact, DeepSeek skilled two fashions:
- The primary mannequin, DeepSeek-R1-Zero, a reasoning mannequin skilled with reinforcement studying, generates information for coaching the second mannequin, DeepSeek-R1.
- It achieves this by producing reasoning traces, from which solely high-quality outputs are retained based mostly on their closing outcomes.
- It signifies that, in contrast to most fashions, the RL examples on this coaching pipeline will not be curated by people however generated by the mannequin.
The end result is that the mannequin achieved efficiency akin to main fashions like OpenAI’s o1 mannequin throughout duties corresponding to arithmetic, coding, and sophisticated reasoning.
(3) Understanding Distillation and DeepSeek-R1’s Distilled Fashions
Alongside the complete mannequin, additionally they open-sourced six smaller dense fashions (additionally named DeepSeek-R1) of various sizes (1.5B, 7B, 8B, 14B, 32B, 70B), distilled from DeepSeek-R1 based mostly on Qwen or Llama as the bottom mannequin.
Distillation is a way the place a smaller mannequin (the “pupil”) is skilled to copy the efficiency of a bigger, extra highly effective pre-trained mannequin (the “instructor”).
On this case, the instructor is the 671B DeepSeek-R1 mannequin, and the scholars are the six fashions distilled utilizing these open-source base fashions:
DeepSeek-R1 was used because the instructor mannequin to generate 800,000 coaching samples, a mixture of reasoning and non-reasoning samples, for distillation by way of supervised fine-tuning of the bottom fashions (1.5B, 7B, 8B, 14B, 32B, and 70B).
So why can we do distillation within the first place?
The purpose is to switch the reasoning talents of bigger fashions, corresponding to DeepSeek-R1 671B, into smaller, extra environment friendly fashions. This empowers the smaller fashions to deal with complicated reasoning duties whereas being quicker and extra resource-efficient.
Moreover, DeepSeek-R1 has a large variety of parameters (671 billion), making it difficult to run on most consumer-grade machines.
Even essentially the most highly effective MacBook Professional, with a most of 128GB of unified reminiscence, is insufficient to run a 671-billion-parameter mannequin.
As such, distilled fashions open up the opportunity of being deployed on units with restricted computational sources.
Unsloth achieved a formidable feat by quantizing the unique 671B-parameter DeepSeek-R1 mannequin down to only 131GB — a exceptional 80% discount in measurement. Nevertheless, a 131GB VRAM requirement stays a big hurdle.
(4) Collection of Distilled Mannequin
With six distilled mannequin sizes to select from, choosing the fitting one largely relies on the capabilities of the native system {hardware}.
For these with high-performance GPUs or CPUs and a necessity for max efficiency, the bigger DeepSeek-R1 fashions (32B and up) are very best — even the quantized 671B model is viable.
Nevertheless, if one has restricted sources or prefers faster technology instances (as I do), the smaller distilled variants, corresponding to 8B or 14B, are a greater match.
For this mission, I might be utilizing the DeepSeek-R1 distilled Qwen-14B mannequin, which aligns with the {hardware} constraints I confronted.
(5) Benchmarks for Evaluating Reasoning
LLMs are sometimes evaluated utilizing standardized benchmarks that assess their efficiency throughout varied duties, together with language understanding, code technology, instruction following, and query answering. Widespread examples embrace MMLU, HumanEval, and MGSM.
To measure an LLM’s capability for reasoning, we want tougher, reasoning-heavy benchmarks that transcend surface-level duties. Listed below are some common examples centered on evaluating superior reasoning capabilities:
(i) AIME 2024 — Competitors Math
- The American Invitational Arithmetic Examination (AIME) 2024 serves as a robust benchmark for evaluating an LLM’s mathematical reasoning capabilities.
- It’s a difficult math contest with complicated, multi-step issues that take a look at an LLM’s means to interpret intricate questions, apply superior reasoning, and carry out exact symbolic manipulation.
(ii) Codeforces — Competitors Code
- The Codeforces Benchmark evaluates an LLM’s reasoning means utilizing actual aggressive programming issues from Codeforces, a platform identified for algorithmic challenges.
- These issues take a look at an LLM’s capability to grasp complicated directions, carry out logical and mathematical reasoning, plan multi-step options, and generate right, environment friendly code.
(iii) GPQA Diamond — PhD-Degree Science Questions
- GPQA-Diamond is a curated subset of the most tough questions from the broader GPQA (Graduate-Degree Physics Query Answering) benchmark, particularly designed to push the bounds of LLM reasoning in superior PhD-level matters.
- Whereas GPQA features a vary of conceptual and calculation-heavy graduate questions, GPQA-Diamond isolates solely essentially the most difficult and reasoning-intensive ones.
- It’s thought-about Google-proof, that means that they’re tough to reply even with unrestricted net entry.
- Right here is an instance of a GPQA-Diamond query:
On this mission, we use GPQA-Diamond because the reasoning benchmark, as OpenAI and DeepSeek used it to judge their reasoning fashions.
(6) Instruments Used
For this mission, we primarily use Ollama and OpenAI’s simple-evals.
(i) Ollama
Ollama is an open-source instrument that simplifies working LLMs on our pc or an area server.
It acts as a supervisor and runtime, dealing with duties corresponding to downloads and atmosphere setup. This enables customers to work together with these fashions with out requiring a relentless web connection or counting on cloud companies.
It helps many open-source LLMs, together with DeepSeek-R1, and is cross-platform suitable with macOS, Home windows, and Linux. Moreover, it gives an easy setup with minimal fuss and environment friendly useful resource utilization.
Vital: Guarantee your native system has GPU entry for Ollama, as this dramatically accelerates efficiency and makes subsequent benchmarking workout routines way more environment friendly as in comparison with CPU. Run
nvidia-smiin terminal to test if GPU is detected.
(ii) OpenAI simple-evals
simple-evals is a light-weight library designed to judge language fashions utilizing a zero-shot, chain-of-thought prompting method. It consists of well-known benchmarks like MMLU, MATH, GPQA, MGSM, and HumanEval, aiming to mirror life like utilization situations.
A few of you might learn about OpenAI’s extra well-known and complete analysis library referred to as Evals, which is distinct from simple-evals.
Actually, the README of simple-evals additionally particularly signifies that it isn’t supposed to interchange the Evals library.
So why are we utilizing simple-evals?
The straightforward reply is that simple-evals comes with built-in analysis scripts for the reasoning benchmarks we’re concentrating on (corresponding to GPQA), that are lacking in Evals.
Moreover, I didn’t discover another instruments or platforms, aside from simple-evals, that present an easy, Python-native approach to run quite a few key benchmarks, corresponding to GPQA, notably when working with Ollama.
(7) Outcomes of Analysis
As a part of the analysis, I chosen 20 random questions from the GPQA-Diamond 198-question set for the 14B distilled mannequin to work on. The entire time taken was 216 minutes, which is ~11 minutes per query.
The end result was admittedly disappointing, because it scored solely 10%, far under the reported 73.3% rating for the 671B DeepSeek-R1 mannequin.
The principle challenge I seen is that in its intensive inner reasoning, the mannequin usually both failed to provide any reply (e.g., returning reasoning tokens as the ultimate strains of output) or supplied a response that didn’t match the anticipated multiple-choice format (e.g., Reply: A).
As proven above, many outputs ended up as None as a result of the regex logic in simple-evals couldn’t detect the anticipated reply sample within the LLM response.
Whereas the human-like reasoning logic was attention-grabbing to look at, I had anticipated stronger efficiency when it comes to question-answering accuracy.
I’ve additionally seen on-line customers point out that even the bigger 32B mannequin doesn’t carry out in addition to o1. This has raised doubts in regards to the utility of distilled reasoning fashions, particularly once they wrestle to present right solutions regardless of producing lengthy reasoning.
That stated, GPQA-Diamond is a extremely difficult benchmark, so these fashions might nonetheless be helpful for less complicated reasoning duties. Their decrease computational calls for additionally make them extra accessible.
Moreover, the DeepSeek staff really useful conducting a number of exams and averaging the outcomes as a part of the benchmarking course of — one thing I omitted as a consequence of time constraints.
(8) Step-by-Step Walkthrough
At this level, we’ve lined the core ideas and key takeaways.
In the event you’re prepared for a hands-on, technical walkthrough, this part offers a deep dive into the internal workings and step-by-step implementation.
Try (or clone) the accompanying GitHub repo to observe alongside. The necessities for the digital atmosphere setup may be discovered right here.
(i) Preliminary Setup — Ollama
We start by downloading Ollama. Go to the Ollama obtain web page, choose your working system, and observe the corresponding set up directions.
As soon as set up is full, launch Ollama by double-clicking the Ollama app (for Home windows and macOS) or working ollama serve within the terminal.
(ii) Preliminary Setup — OpenAI simple-evals
The setup of simple-evals is comparatively distinctive.
Whereas simple-evals presents itself as a library, the absence of __init__.py information within the repository means it isn’t structured as a correct Python package deal, resulting in import errors after cloning the repo regionally.
Since additionally it is not printed to PyPI and lacks normal packaging information like setup.py or pyproject.toml, it can’t be put in by way of pip.
Thankfully, we will make the most of Git submodules as an easy workaround.
A Git submodule lets us embrace contents of one other Git repository inside our personal mission. It pulls the information from an exterior repo (e.g., simple-evals), however retains its historical past separate.
You may select certainly one of two methods (A or B) to tug the simple-evals contents:
(A) If You Cloned My Mission Repo
My mission repo already consists of simple-evals as a submodule, so you’ll be able to simply run:
git submodule replace --init --recursive(B) If You’re Including It to a Newly Created Mission
To manually add simple-evals as a submodule, run this:
git submodule add https://github.com/openai/simple-evals.git simple_evalsNotice: The simple_evals on the finish of the command (with an underscore) is essential. It units the folder identify, and utilizing a hyphen as an alternative (i.e., easy–evals) can result in import points later.
Ultimate Step (For Each Strategies)
After pulling the repo contents, you have to create an empty __init__.py within the newly created simple_evals folder in order that it’s importable as a module. You may create it manually, or use the next command:
contact simple_evals/__init__.py(iii) Pull DeepSeek-R1 mannequin by way of Ollama
The subsequent step is to regionally obtain the distilled mannequin of your alternative (e.g., 14B) utilizing this command:
ollama pull deepseek-r1:14bThe record of DeepSeek-R1 fashions out there on Ollama may be discovered right here.
(iv) Outline configuration
We outline the parameters in a configuration YAML file, as proven under:
The mannequin temperature is about to 0.6 (versus the everyday default worth of 0). This follows DeepSeek’s utilization suggestions, which recommend a temperature vary of 0.5 to 0.7 (0.6 really useful) to stop countless repetitions or incoherent outputs.
Do take a look at the apparently distinctive DeepSeek-R1 utilization suggestions — particularly for benchmarking — to make sure optimum efficiency when utilizing DeepSeek-R1 fashions.
EVAL_N_EXAMPLES is the parameter for setting the variety of questions from the complete 198-question set to make use of for analysis.
(v) Arrange Sampler code
To help Ollama-based language fashions throughout the simple-evals framework, we create a customized wrapper class named OllamaSampler saved inside utils/samplers/ollama_sampler.py.
On this context, a sampler is a Python class that generates outputs from a language mannequin based mostly on a given immediate.
Since present samplers in simple-evals solely cowl suppliers like OpenAI and Claude, we want a sampler class that gives a suitable interface for Ollama.
The OllamaSampler extracts the GPQA query immediate, sends it to the mannequin with a specified temperature, and returns the plain textual content response.
The _pack_message methodology is included to make sure the output format matches what the analysis scripts in simple-evals anticipate.
(vi) Create analysis run script
The next code units up the analysis execution in major.py, together with the usage of the GPQAEval class from simple-evals to run GPQA benchmarking.
The run_eval() perform is a configurable analysis runner that exams LLMs by way of Ollama on benchmarks like GPQA.
It masses settings from the config file, units up the suitable analysis class from simple-evals, and runs the mannequin by way of a standardized analysis course of. It’s saved in major.py, which may be executed with python major.py.
Following the steps above, we have now efficiently arrange and executed the GPQA-Diamond benchmarking on the DeepSeek-R1 distilled mannequin.
Wrapping It Up
On this article, we showcased how we will mix instruments like Ollama and OpenAI’s simple-evals to discover and benchmark DeepSeek-R1’s distilled fashions.
The distilled fashions could not but rival the 671B parameter unique mannequin on difficult reasoning benchmarks like GPQA-Diamond. Nonetheless, they display how distillation can develop entry to LLM reasoning capabilities.
Regardless of subpar scores in complicated PhD-level duties, these smaller variants could stay viable for much less demanding situations, paving the best way for environment friendly native deployment on a wider vary of {hardware}.
Earlier than you go
I welcome you to observe my GitHub and LinkedIn to remain up to date with extra participating and sensible content material. In the meantime, have enjoyable benchmarking LLMs with Ollama and simple-evals!