Why should the model read the same files again?
Suppose the first request contains 50,000 tokens of repository context. The agent writes a patch, runs a test, and gets a short error message. Most of the next request is still the same 50,000 tokens. The server can save the results of processing that opening text and reuse them when the next request starts the same way. It then only needs to process the new part. This is prefix caching, and it is already an important way to reduce the cost of running agents. Caching solves the repeated work, but the agent keeps discovering new information. It opens another file or gets a different test failure, and the model has to read that text for the first time. DeepSeek’s first change reduces the cost of this new input.Can reading the input take less work?
Reading a long file and writing a short patch put different demands on the model. Reading requires it to process all the input. Writing requires it to generate new tokens one at a time. These stages are called prefill and decode. In a conventional decoder model, every input token passes through every layer. V4.1 Flash has 40 layers, but it separates them into two groups of 20. The first group reads the prompt. Its results are used to prepare the information the second group needs about the full conversation. Most of the prompt therefore only has to pass through the first 20 layers. The second group also processes the last 128 prompt tokens to prepare its attention over recent text. Once the model starts writing the answer, each new token passes through both groups. DeepSeek calls the first group the encoder and the second the decoder. The design is Causal Encoder-Decoder, or CED. It uses about 8 billion active parameters when reading an input token and 16 billion when generating an output token. For long inputs, DeepSeek calculates that prompt processing approaches half the work of running every token through the full model. The time saved on a complete request will also depend on generating the answer and the other work the server performs. Technical report, section 2.2 The encoder also keeps its own attention information; the diagram follows how the decoder gets ready to generate an answer. This is a new architecture trained from scratch for V4.1 Flash, unlike July’s V4 Flash 0731 update, which kept the earlier design and size. Official model card Our agent can now reuse work for files it has already read and spend less computation on a new file. But saving work introduces another cost: the server has to keep the saved information somewhere.What happens when thousands of agents keep their history?
For each conversation, the server stores representations of earlier tokens that the model can refer to while generating an answer. Attention reads these stored values to find relevant information. This storage is called the KV cache, after attention’s keys and values. A longer conversation needs a larger cache. Thousands of active conversations need thousands of caches. Even if reading the input becomes cheaper, memory can still limit how many agents a server handles at once. V4.1 Flash reduces this memory use by sharing information between layers. Several layers can use the same stored representation of the long history, and some can also reuse an earlier layer’s selection of relevant entries. Each layer still handles attention over recent tokens separately. DeepSeek also stores the main global cache values in a four-bit format and trains the model to account for the numerical errors that introduces. Sharing, compression, and lower precision together bring the reported global KV cache down to 890 bytes per token, about one quarter of the corresponding cache in V4 Flash. Technical report, section 2.3 At one million tokens, this part of the cache uses about 890 megabytes. The server still needs memory for weights, recent-token attention, intermediate calculations, and working buffers. The saving helps it keep more conversation history without that particular part of memory use growing as quickly.What should the server keep while an agent is waiting?
Our agent may stop generating text for a while because a test is running or someone is reviewing its patch. Keeping every paused conversation in GPU memory would leave less room for agents that are actively working. Servers can save reusable cache information in system memory or on SSDs and bring it back when the conversation continues. DeepSeek found that attention state for recent tokens occupied nearly half of the saved cache in its previous system. V4.1 Flash keeps recent copies of this state in system memory for a short time, but stops retaining it in long-term storage. If the state is missing when the agent returns, the model processes the latest 128 tokens again to rebuild it. That rebuilt state is approximate because the original also depended on older text. DeepSeek reports very little effect on quality in its evaluations and trains with this replay behavior. It identifies resumed sessions and retrieval of scattered details from a long history as areas that still need more testing. The method is called SWA Bounded Replay. Technical report, sections 3.2.2 and 6 Combined with the smaller global cache, this gives DeepSeek a saved cache that uses about one eighth of the previous space under the same workloads. A service can retain more reusable history for a given amount of storage. Technical report, section 3.2.1 For a coding agent, a useful test is to pause a task, let its cache expire, and then resume work that depends on something it read much earlier. That tests the trade-off where your application will actually encounter it.Can the model also spend less work writing the patch?
So far, the changes help the model read code and keep track of it. The agent still needs to generate reasoning, tool calls, and the patch itself. Generating every token in a separate step can leave the user waiting. V4.1 Flash drafts several possible next tokens in one pass, then lets the main model check them. When the draft is accepted, the server makes progress on several tokens together. A rejected draft uses work without making the same progress, so the amount of drafting matters. This approach is called speculative decoding. DeepSeek’s version, DSpark, chooses how much draft text to check based on the estimated chance of acceptance and the serving engine’s performance under load. This makes its benefit depend on the tasks and the number of requests being served. The report explains the method but does not provide a universal tokens-per-second result for sizing a cloud server. Technical report, section 2.4.3What other work does DeepSeek remove inside the model?
What other work does DeepSeek remove inside the model?
The model also has to move intermediate values between its internal operations. Single-Pass mHC lets DeepSeek combine steps for mixing the streams of information that pass between layers into one GPU operation called Mega-mHC. DeepSeek reports roughly half the memory traffic for intermediate values in that operation compared with its previous implementation. This is a saving for that operation, rather than a twofold speedup for the whole model. Technical report, section 2.4.1
How do these changes lead to such a cheap API?
Follow the same agent through the system again. When it sends old code, the server can reuse saved work. When it reads a new file, the model does less work to process it. While the agent is active, its global attention cache takes less memory. When it pauses, saving its history takes less storage. Drafting can also reduce the number of separate steps needed to generate its next response. These savings apply to different parts of the same request, which is why the combination matters. A provider serving many users can also keep its GPUs busy and spread their cost across those requests. As of September 10, DeepSeek’s off-peak API prices are:
All three prices double during peak hours. Cached input costs 98% less than uncached input, so the amount of conversation history the server can reuse has a large effect on an agent’s bill. Official prices and billing hours
The architecture gives us reasons why serving can cost less. The API prices are still a business decision, and DeepSeek has not published enough operating data to calculate its profit per request. We work through the customer’s bill, including cache hits and repeated agent turns, in the deployment and cost guide.
Does the agent still finish the job correctly?
A cheap attempt helps only if the agent can complete the task. If it needs more retries or leaves a patch that a developer must repair, those costs belong in the comparison too. DeepSeek reports improvements over V4 Flash on several agent tests:
These are DeepSeek’s results at maximum reasoning effort. The software running the agent matters too: the same model scores 74.2% on DeepSWE through mini-SWE and 65.6% through Codex. On the harder Terminal-Bench 4.0 test, V4.1 Flash scores 31.2%, below the leading result of 51.8%. Technical report, section 5.3
This leaves a practical choice for a coding team. More reasoning generally means more output tokens to pay for, but it does not improve every task. DeepSeek’s coding results sometimes level off or fall at intermediate effort settings. Try the public API’s low, high, and max settings on your own tasks and compare the cost of a successful result. The report maps those settings to 50, 75, and 100 on the model’s learned effort scale. Technical report, pages 30 and 48–49
How was the model trained and how were these tests run?
How was the model trained and how were these tests run?
DeepSeek trained V4.1 Flash from scratch on 45 trillion tokens of text and image data. Images were included during the initial training, allowing an agent to inspect screenshots, charts, and rendered interfaces. The model accepts text and images and generates text, with support for up to one million tokens of context.Its later training kept supervised fine-tuning, reinforcement learning, and on-policy distillation. These methods use example answers, rewards, and guidance on responses the model generates itself. DeepSeek attributes the main improvements in these stages to better tasks and environments for training agents. Technical report, sections 4 and 5The coding evaluations used a one-million-token context, temperature 1.0, and top-p 0.95. Terminal-Bench used DeepSeek Harness Minimal, DeepSWE used mini-SWE, and AutomationBench used its official setup. We have not independently reproduced these results at Tensorfuse.
Does cheaper inference mean smaller or different hardware?
The model can use less computation per token while still needing a large amount of memory for its weights. V4.1 Flash has 552 billion parameters in its main language model and another 196 billion in a component called Engram. Those add up to 748 billion parameters, even though only about 8 billion or 16 billion are active for a particular token. Engram addresses a different kind of repeated work. The model encounters familiar token sequences across many requests, and Engram lets it retrieve learned information about those sequences from lookup tables. These tables come with the trained model. They are separate from the KV cache, which is built from a particular conversation. Because the input tokens determine which table entries are needed, the server can begin fetching them before the GPU reaches the operation that uses them. DeepSeek describes fetching entries from system memory while the GPU continues working. V4.1 Flash has two Engram modules whose tables and projections use eight-bit floating point, or FP8. Technical report, section 2.4.2 This gives system RAM useful work alongside GPUs. It does not show that running the whole model on CPUs would be fast enough for an interactive coding agent. That needs a supported implementation and measurements of actual response times. There is already an AMD GPU option. SGLang’s V4.1 preview lists four MI350X GPUs as verified, alongside configurations with four NVIDIA B300 or four GB300 GPUs. This gives providers specific alternatives they can evaluate today. SGLang’s hardware listDoes the four-bit cache require a particular chip?
Does the four-bit cache require a particular chip?
The main global KV cache is stored in a four-bit format, but the values are converted to higher precision before the attention calculation. Using this cache therefore does not require native matrix multiplication in that same four-bit format. The component that selects relevant context uses the standardized MXFP4 format.This removes one possible hardware requirement. Efficient serving still needs supported implementations for the model’s attention, experts, memory lookups, and communication between chips. Support for one AMD GPU also needs to be checked separately from support for another GPU, a TPU, or an AWS Trainium instance. Technical report, section 2.4.4