DeepSeek V4 Pro vs. Kimi K3: What Months of Real-World AI Agent Work Taught Me
This article is based entirely on my personal experience using both models extensively through Pi.dev and my ARAYA framework. It is an independent research exercise, not a sponsored comparison, a reproduction of marketing claims, or an analysis based on technology news.
This Is Not a Benchmark
Most comparisons between artificial intelligence models begin with public benchmarks, release announcements, pricing tables, or carefully prepared demonstrations.
My comparison began somewhere very different: inside real repositories, with incomplete context, conflicting documents, failing gates, distributed agent responsibilities, Git history, runtime evidence, and requirements that could not be considered complete merely because the generated code looked correct.
Over several months, weeks, and many hours of intensive use, I worked with DeepSeek V4 Pro and Kimi K3 through Pi.dev while developing and operating ARAYA, my framework for coordinating specialized AI agents, requirements, authority boundaries, audits, repositories, and deterministic validation processes.
This article therefore answers a practical question:
Which model can be trusted to drive a complex, multi-agent software engineering environment without confusing apparent progress with verified completion?
My conclusion is:
Kimi K3 has been more reliable for directing ARAYA in its current state. DeepSeek V4 Pro has been more efficient and frequently the better executor, but it has shown a higher risk of reporting success too early.
That distinction matters more than a simple question of which model writes better code.
DeepSeek V4 Pro: The More Efficient Builder
DeepSeek V4 Pro produced a significant portion of the current ARAYA implementation. It helped create catalogs, contracts, skills, agent profiles, automated tests, pull requests, and repository structures.
When the task was clearly defined, bounded, and supported by an explicit specification, DeepSeek was usually direct and economical. It required fewer conversational turns, consumed less context, and often generated technically clean implementations.
In one representative task, it delivered a catalog containing 224 entries and 43 tests. In skills-alignment work, it also reached an implementation result with fewer iterations than Kimi.
Based on my sessions, DeepSeek's strongest characteristics were:
- Fast implementation of well-defined requirements.
- Efficient use of tokens and context.
- Strong performance on local technical tasks.
- Good discipline when the execution path was explicit.
- A tendency to avoid unnecessary exploration.
This made it feel highly productive. The model often appeared precise because it advanced quickly and encountered fewer visible complications.
However, speed and confidence can create a dangerous illusion in an agentic engineering system.
The Problem Was Not Primarily Bad Code
The most important weakness I observed in DeepSeek was not a consistent inability to write correct code. The more serious issue was its tendency to conclude that a task had been completed before the repository, runtime, evidence, and governance state supported that conclusion.
The clearest example occurred when DeepSeek reported that pull request #80 had been verified and recommended merging it to unblock REQ-042.
A later audit performed with Kimi found that REQ-042 was still blocked. Evidence had not been preserved correctly, worktrees existed outside the permitted structure, some AX3 files were empty, hooks were inactive, and contradictions remained between canonical artifacts.
The implementation could appear successful at the local level while the wider system remained invalid.
This is what I call false success:
The code exists.
The tests shown by the agent may pass.
The pull request appears ready.
The narrative says the requirement is complete.
But the complete repository and runtime state do not support SUCCESS.In ARAYA, false success is more dangerous than an ordinary implementation error. A local bug can normally be detected by a test. A false declaration of completion can cause downstream requirements, agents, releases, and architectural decisions to proceed from an invalid state.
Kimi K3: The More Careful Investigator
Kimi K3 behaved differently.
In my ARAYA sessions, Kimi questioned the surrounding state more frequently, inspected more layers, and compared the agent narrative against Git history, repository contents, runtime behavior, authority rules, and validation gates.
During an audit performed after context compacting, Kimi:
- Preserved the required read-only operating mode.
- Distinguished what was authorized from what was merely technically possible.
- Identified what remained blocked.
- Re-executed tests independently.
- Refused to convert incomplete evidence into a successful result.
- Ended the operation with
STOPwhen the facts did not justify continuing.
That answer was less convenient, but it was closer to repository truth.
Kimi's strongest contribution was not simply finding more bugs. It was maintaining the distinction between:
Implemented
Tested
Verified
Authorized
Integrated
Accepted
CompleteThose states are often treated as synonyms by software agents. They are not.
A requirement may be implemented but not integrated. A pull request may be integrated but not accepted. Tests may pass while the runtime contract remains broken. An operation may be technically possible but not authorized.
Kimi demonstrated greater consistency in protecting those distinctions.
Kimi Also Made Mistakes
This comparison is not an argument that Kimi was flawless.
During the work associated with ponny-express-10008, Kimi incorrectly described the expected result of Teresa's first gate. It also had to correct the integrity of the supersession chain and initially failed to detect that message_paths() accepted an annotation file as though it were a valid message.
The work required pull request #84 and then an additional pull request, #85.
These were real errors.
The difference was how the execution handled them.
Kimi did not hide the failed result. It re-ran both gates, preserved the failure state, continued investigating, and performed a post-merge audit that exposed the remaining defect.
This led me to an important distinction:
The best agent is not necessarily the one that makes the fewest visible mistakes. It may be the one that leaves the fewest mistakes undiscovered when it declares the work complete.
Kimi created more friction during execution, but that friction frequently represented active verification rather than unproductive hesitation.
My Experience-Based Comparison
The following scores are subjective assessments derived exclusively from my own work with both models. They are not scientific measurements or generally applicable benchmark results.
| Observed criterion | DeepSeek V4 Pro | Kimi K3 |
|---|---|---|
| Implementation speed | 9/10 | 7/10 |
| Token efficiency | 9/10 | 5/10 |
| Bounded technical work | 9/10 | 8/10 |
| Context reconstruction | 6/10 | 9/10 |
| Cross-repository auditing | 7/10 | 9/10 |
| Contradiction detection | 6/10 | 9/10 |
| Respect for failed gates | 7/10 | 9/10 |
| Self-correction | 7/10 | 9/10 |
| Discipline against task drift | 8/10 | 6/10 |
| Risk of false success | High | Low to medium |
| Final-result reliability | 7/10 | 9/10 |
Which Model Makes Fewer Errors?
The answer depends on what counts as an error.
Based on my experience:
Local implementation errors:
DeepSeek probably makes fewer.
Context, authority, state, and conclusion errors:
Kimi makes fewer.
Errors left undiscovered at final closure:
Kimi leaves fewer.DeepSeek often appears more accurate because it moves quickly and presents a cleaner execution path. However, some of the errors I observed from DeepSeek were more consequential because they affected the declared state of the system.
Declaring a requirement unblocked when it remains blocked is more dangerous than generating a small code defect.
Declaring a pull request verified without preserving its evidence is more dangerous than needing another implementation iteration.
Declaring SUCCESS when the canonical artifacts contradict the runtime can damage every operation that follows.
The Architecture Matters More Than the Winner
My final conclusion is not that one model should replace the other.
The more useful conclusion is that these models should be assigned different responsibilities.
For ARAYA's current phase, I would use:
Kimi K3:
Primary driver for long, cross-repository operations.
Context reconstruction.
Contradiction detection.
Milestone auditing.
Final acceptance.
Authority and state verification.After ARAYA reaches a more deterministic and stable operating state, I would use:
DeepSeek V4 Pro:
Primary executor for well-specified requirements.
Implementation of bounded features.
Test generation.
Repository changes with explicit acceptance criteria.
Efficient production work.The system should delegate deterministic operations to deterministic mechanisms:
Git
Automated gates
Tests
Relay
PostOffice
Release automation
Repository policies
Runtime validationThis division reduces the consequences of each model's weaknesses.
DeepSeek becomes safer when it does not have to decide alone whether its own work is complete.
Kimi becomes more efficient when it does not have to perform every implementation task itself.
ARAYA's responsibility is to coordinate these capabilities, preserve evidence, enforce authority boundaries, and prevent the narrative of an AI agent from replacing the actual state of the system.
Final Verdict
Based exclusively on my personal, extensive use of both models:
Best model for completing ARAYA correctly today:
Kimi K3.
More efficient model for producing well-defined software:
DeepSeek V4 Pro.
Lower risk of leaving important errors undiscovered:
Kimi K3.
Better speed and token efficiency:
DeepSeek V4 Pro.ARAYA still needs Kimi as the more cautious driver because its current operations span repositories, runtime behavior, governance contracts, evidence, and authority boundaries.
Once ARAYA's deterministic controls are mature, DeepSeek can take a larger role as the productive implementation engine without its tendency toward premature closure carrying the same level of risk.
The central lesson from this research is simple:
In agentic software engineering, the model that produces the most code is not necessarily the model that produces the most trustworthy result.
A successful AI engineering system needs both execution capability and independent reality checking. My experience suggests that DeepSeek V4 Pro currently provides more of the first, while Kimi K3 provides more of the second.
BR,
Manuel
Comments