Open weights and better agent checks
An agent can say it finished. Your database gets the last word. This edition brings a tougher way to test agent reliability, a practical recipe for training across coding harnesses, and new self-hosted model and backend options.
In today’s edition
- ThinkingBox checks agents by their final system state.
- Hugging Face trains coding agents across harnesses.
- Aleph Alpha releases Apache-licensed Kolibri open weights.
- Supabase drops Docker from local agent development.
ThinkingBox tests what your agent actually changed
A polished final answer is not proof that an agent completed the job. Microsoft and Hugging Face released ThinkingBox, a benchmark that runs agents through stateful business workflows and checks the records and side effects left behind.
- It includes 507 workflows, with each task run 20 times from a clean backend state.
- In a 121,680-trial analysis, 67.24% of failed attempts still ended cleanly, used a state-changing tool, and reported no final tool error.
- The release grades terminal system state, including wrong field values, missing effects, and unintended extra effects.
- The runnable OpenEnv-based harness and dataset are available, so teams can test agents against executable outcomes rather than summaries.
What it means Test agent workflows by checking the database state and side effects after each run, then measure repeatability across trials before trusting a pass@1 score.
Source: huggingface.co
Train coding agents for the harnesses they run in
The same model can score 62% in one coding-agent harness and 33% in another. Hugging Face published an open multi-harness RL recipe that trains one model across Claude Code, Codex, OpenCode, and Mini-SWE-Agent-style interfaces without modifying those clients.
- Point each harness at a capture proxy that supports OpenAI Chat Completions, OpenAI Responses, Anthropic Messages, and Gemini APIs.
- The proxy records sampled token IDs and log probabilities from vLLM, giving RL training the data it needs without changing the harness.
- Training LFM2.5-2.6B across four harnesses raised held-out overall pass@1 from 42% to 54%.
- A small reward for fewer steps cut tool calls by 31%, while imitation learning from 3,189 successful rollouts plateaued at 47.5%.
What it means Train and evaluate coding agents in the actual clients your users run, because optimizing one harness can leave the same weights weaker everywhere else.
Source: huggingface
Aleph Alpha releases Kolibri under Apache 2.0
Aleph Alpha has released Kolibri’s weights under Apache 2.0, giving teams a European-built model they can run on their own hardware rather than access only as a hosted service.
- Kolibri has 78 billion parameters, with 3.46 billion active parameters per token.
- It supports up to 1 million tokens of context.
- The release makes the weights available for self-hosting under a permissive Apache 2.0 license.
- The supplied announcement does not establish a performance comparison with current frontier models, so treat this as an access and deployment option, not a proven replacement.
What it means Add Kolibri to self-hosted and controlled-deployment evaluations when owning the weights, runtime, and data path matters more than using a vendor-hosted model.
Source: Aleph__Alpha
Supabase makes local agent work less container-bound
Supabase says its local development setup now runs without Docker, supports multiple instances per repository or worktree, and moves schema, auth, API, and storage configuration into code.
- Dropping Docker removes a setup dependency from local Supabase development.
- Multiple local instances let parallel branches and agent worktrees run without sharing one backend state.
- Declarative Schemas 2.0 uses pg-delta to generate migrations.
- Keeping service configuration in code gives agents and teams a versioned source of truth instead of a collection of dashboard settings.
What it means Give parallel coding agents isolated Supabase instances and version-controlled backend configuration, reducing setup friction and collisions between branches or worktrees.
Source: supabase