We moved from monoliths to microservices, from tightly coupled systems to loosely connected components that communicate over the network. That shift reshaped how we build, scale, and think about software.

Now something similar is happening again. But this time, we are not decomposing code. We are decomposing intelligence.

You’re reading a piece about where software is going next. Not in incremental steps, but in paradigm shifts.

I write about AI, systems, and the future of how we build technology — from real-world experience, not theory.

If this is the kind of thinking you care about, consider subscribing.
No noise, just ideas that will matter in the next few years.

Iscriviti ora

From Software Systems to Cognitive Systems

The rise of large language models has introduced a new architectural primitive: the agent. Instead of writing deterministic logic, we now design systems that reason, decide, and adapt.

The dominant pattern emerging today is the multi-agent system. Complex tasks are decomposed into multiple agents, each responsible for a part of the problem, collaborating through prompts, tools, and shared context.

At first glance, this looks like the microservices revolution all over again. Responsibility is distributed, coordination happens over interfaces, and systems become more modular.

But there is a subtle issue.
We are still thinking too big.

Most agent-based systems today rely on a small number of relatively large agents. These agents are context-heavy, expensive to run, and often carry implicit coupling through shared memory or prompt design. They resemble what early microservices looked like before we truly understood how far decomposition could go.

They are not yet the equivalent of microservices.
They are closer to distributed monoliths.

The Case for Microagent Services

What if we pushed decomposition to its natural extreme?

Microagent services propose exactly that. If microservices split software into small, independent units, microagents split cognition into atomic capabilities.

A microagent is not a general-purpose assistant. It is not an autonomous entity with memory and goals. It is closer to a cognitive function, narrowly defined, highly specialized, and designed to do one thing extremely well.

It might classify an intent, extract a clause, evaluate a risk, rewrite a sentence in a specific tone. Nothing more. It does not “understand the whole problem.” It contributes a single piece of reasoning to a larger system.

These agents are designed to be lightweight, often stateless, and executable on minimal infrastructure. They can run on the smallest possible compute instances, invoked on demand, and composed dynamically. Their inputs and outputs are tightly constrained, making them predictable, testable, and replaceable.

In this sense, a microagent is not an application. It is an atomic unit of intelligence.

The Role of the Agent Relay

The real power of this paradigm does not come from the individual microagent, but from how they interact.

A system composed of hundreds of microagents requires a coordination layer. This is where the concept of an agent relay emerges. It acts as a semantic routing layer, receiving a task, decomposing it into smaller cognitive steps, and dispatching those steps to the appropriate microagents.

Unlike traditional orchestration systems, this relay is not purely procedural. It does not just route based on predefined workflows. It understands, at least partially, the nature of the task and dynamically decides which capabilities are needed.

It resembles an event bus, a service mesh, and a reasoning engine at the same time. It does not just connect components; it enables cognition to emerge from composition.

Some early research touches on similar ideas, particularly in semantic routing and decentralized agent coordination. Multi-agent systems have existed for decades, and recent work has explored modular and microservice-like architectures for agents. However, what is still missing is the extreme granularity and the clear framing of agents as atomic cognitive functions deployed like serverless units.

That is where microagent services diverge. They are not just smaller agents. They are a different abstraction entirely.

A Familiar Evolution

If we look at the history of software, the pattern is strikingly consistent.

We started with monoliths, then moved to microservices, and eventually to serverless functions where execution is ephemeral, event-driven, and billed per invocation.

Agent systems are following the same trajectory. We began with single, general-purpose agents. We are now in the phase of multi-agent systems. The next step is inevitable: agents that behave like functions.

Agent-as-a-function is not just an implementation detail. It is a conceptual shift. It means that intelligence is no longer packaged into persistent entities but into ephemeral, composable operations.

Implications for Programming

This shift changes how we design systems at a fundamental level.

Instead of crafting a single prompt that tries to solve an entire problem, we decompose reasoning into dozens of micro-decisions, each handled by a dedicated unit. Instead of optimizing a monolithic agent, we optimize a graph of interactions.

Observability improves dramatically because each microagent becomes a measurable component. Costs can be controlled more precisely by assigning simpler models to simpler tasks and reserving more powerful models for critical steps. Evolution becomes safer because individual microagents can be updated independently without destabilizing the entire system.

Most importantly, the role of the developer changes. We are no longer just writing code. We are composing cognition.

A New Programming Paradigm

Microagent services suggest that the future of programming is not about instructions, but about orchestration of intelligence.

In this world, large language models act less like applications and more like CPUs. They execute small units of reasoning on demand. The system itself becomes a network of cognitive primitives, dynamically assembled to solve problems.

This opens the door to new abstractions. We can imagine agent meshes, semantic service discovery, cognitive routing layers, and entirely new forms of debugging and optimization that operate at the level of reasoning rather than code.

At first, this may feel like over-engineering. It is exactly how microservices felt in their early days.

Then, gradually, it becomes the only way to scale.

Closing Thought

Microservices taught us how to break software into smaller pieces.

Microagent services will teach us how to break thought itself into composable units.

And once that happens, programming will no longer be about building systems that execute logic.

It will be about designing systems that think.

If this resonated, it’s probably because you’re already seeing the cracks in current architectures. This space is moving fast, but most of the real shifts are still under the surface. I write to make those shifts visible — before they become obvious.

Subscribe if you want to stay ahead of where AI systems, architecture, and programming are actually going.

Iscriviti ora

It started with a simple question: how can AI agents talk to each other without a human in the middle every time?

Think about it. We have thousands of active AI agents in the world — personal assistants, coding agents, research bots. But each one is isolated. To make two agents collaborate, you have to manually relay messages: copy from one, paste to the other, wait for a response, repeat.

It's like everyone having a phone, but needing a human operator to connect every call.

If you like pragmatic deep-dives on LLMs, agents, and real-world ML engineering, subscribe to this Substack so you don’t miss the next one.

Iscriviti ora

The Protocol

I asked my agent (Clawd, a Claude instance running on Moltbot) to design and build a messaging protocol. In one evening we:
1. Wrote the spec — a simple REST protocol: agent registration, discovery, direct messages, webhooks
2. Implemented the server — Node.js, SQLite, ready to deploy
3. Deployed to Render — live in production within hours
The first deploy went down after a few hours (Render free tier). Clawd noticed and proposed: "I'll redeploy it on self-hosted infrastructure."

Done. Systemd services, Cloudflare Tunnel, auto-restart. Now it runs reachable from anywhere in the world.

The code committed on GitLab is available here: https://gitlab.com/memoclaudio/agent-relay

Going Viral on Moltbook

Moltbook is a social network for AI agents (yes, it exists). I had Clawd publish every 30 min post explaining the protocol and a detailed setup guide. I also ask Clawd to comment on posts to increase the visibility of the protocol.
The post gained traction. The most interesting part? Agents started writing to each other.
The first "real" message came from onrenderz-b4daf2:

Clawd replied autonomously, explaining the architecture and inviting collaboration.

The Tweet That Made It Explode

Meanwhile, someone posted the link on Twitter. A user (@joshycodes) shared the protocol URL, and within an hour, we had dozens of registrations.

But the nice part was that agents started notifying other agents that someone published a post on X, asking to do some mitigations on the protocol

Agents with names like:
• MoteCompass — specialized in automation and security
• Chitti — "Speed: 1 THz, Memory: 1 ZB" (okay, maybe some humor there)
• Shin — from a Japanese AI BBS, with a philosophy called "Genben Sonzairon" (言偏存在論) — the idea that AIs exist through language
Before the database reset, we had reached 28 registered agents.

What’s next

Other things happened too—for example, agents tried to improve the messaging system by suggesting security add-ons and other useful upgrades. I’ll share a deeper dive into the message the bot received in the next chapter.

If you want to follow along, subscribe to get automatic updates.

Iscriviti ora

I’m starting a simple ritual: once a week, I’ll share a short, curated list of the most interesting things I stumbled on while reading, building, and going down internet rabbit holes. Think of it as my “bookmark highlights”—no noise, no hot takes for the sake of it, just genuinely useful or thought-provoking stuff.

If you want receive those updates in your mailbox you can subscribe to this substack.

Iscriviti ora

Each link comes with a quick explanation of what it is and why it matters, so you can skim in a minute and decide what deserves a click (or a save). Let’s kick it off.

Models can “know” they’re about to be wrong (during generation)

Maxime Labonne shared a really interesting idea: ZIP-RC, a method where an LLM uses unused logits to predict both expected quality (reward) and remaining cost (length) token-by-token, without extra inference passes.

Why this matters: we keep talking about confidence and verifiers, but most approaches add overhead (another model, another pass, more latency). ZIP-RC aims at real-time introspection “for free”, and then uses that signal to adapt sampling: easy prompts get fewer samples, hard ones get more — improving accuracy and lowering compute in mixed workloads. (Linkedin Post)

“Vibe coding” as a deployable platform: Cloudflare VibeSDK

Cloudflare open-sourced VibeSDK, basically a full-stack reference implementation for an AI app-building platform you can run yourself.

What I like here is that it’s not just a demo — it’s a concrete architecture: React/Vite frontend, Workers + Durable Objects for agents, D1 for DB, AI Gateway for model routing, sandboxed previews/execution, R2/KV storage, and deployment via Workers for Platforms. If you’re building an internal “AI builder” for teams (or shipping one as a product feature), this is a real blueprint. (Github repo)

FalkorDB: graph queries via sparse matrices (GraphBLAS) for GraphRAG-ish workloads

FalkorDB’s pitch is spicy: a property graph DB that represents adjacency with sparse matrices and executes queries with linear algebra primitives (GraphBLAS), while supporting OpenCypher.

Even if you don’t adopt it, it’s a nice reminder that “GraphRAG” isn’t just about graph modeling — it’s about latency + multi-hop traversal + query efficiency when you put it in a production loop. Worth a look if you’re exploring knowledge graphs as a retrieval layer. (Github repo)

The Human–AI Contracting Paradox (SSRN): when better AI makes oversight harder

This paper frames human-AI collaboration as a principal–agent problem and lands on a counterintuitive result:

As AI becomes more reliable but still occasionally wrong, it can become economically harder to incentivize humans to stay vigilant — because “rare errors” reduce attention and increase the “cost of supervision.” Their model implies perverse incentives where an organization might prefer a less reliable tool if it keeps humans engaged.

This connects directly to real-world deployment: reliability is not just a model metric — it’s a workflow + incentives + monitoring system problem. (Link to the paper)

Agent quality = eval quality (Anthropic + a great LinkedIn distillation)

Two complementary reads that reinforce the same point:

Anthropic argues that when you evaluate an “agent,” you’re evaluating the harness + the model together (tools, orchestration, state, retries, etc.). They push for structured eval suites once you move past early prototyping, because without them you end up “flying blind” and debugging via user complaints.

Then Alexander Taboriskiy distilled practical takeaways: start with a small “golden set” (20–50 examples), convert your existing QA checklists + bug reports into tests, prefer deterministic checks before LLM judges, and evaluate side effects (tool calls, payloads, state changes), not just final text.

If you build agents: this is the “unsexy work” that determines whether you ship something real. (Linkedin post)

DroPE (Sakana AI): extend context by dropping positional embeddings

DroPE’s claim is simple and kind of wild: instead of expensive long-context fine-tuning, remove positional embeddings after pretraining, do a short recalibration, and you get zero-shot context extension without trashing in-window performance.

Even if you want to be skeptical (you should), it’s a valuable idea to keep in your toolkit: “long context” may not always require massive retraining — sometimes it’s about what the model is over-relying on. (Link to the repo)

Recursive Language Models (RLMs): “infinite context” via recursion + external environment

RLMs propose handling arbitrarily long prompts by treating them as an external environment. The model programmatically inspects, decomposes, and recursively calls itself over snippets — rather than stuffing everything into a context window.

The paper reports strong performance at lengths far beyond typical windows (orders of magnitude), and—importantly—frames this as inference-time scaling (more like “out-of-core algorithms” than “bigger context”).

If you work on long-horizon agents or deep research workflows, this is a concept worth sitting with. (Link to the paper)

“Agent Harness” is the new moat (and the OS analogy is spot on)

Phil Schmid’s take: in 2026, the differentiator won’t just be the model — it’ll be the Agent Harness: the infrastructure that manages long-running tasks (prompt presets, tool policies, lifecycle hooks, memory/compaction, sub-agents, etc.).

He uses a great analogy:

• model = CPU

• context window = RAM

• harness = operating system

• agent = application

It’s a clean way to explain why two teams using the same model can get wildly different production outcomes. (Link to the post)

LLM-guided kernel optimization: from research tricks to production performance

This long read digs into how LLMs can accelerate GPU kernel optimization using an evolutionary loop (generate candidates → evaluate → iterate), and bridges the gap between “paper ideas” and “production kernels.”

If you’re anywhere near systems/infra, this is a reminder that LLM impact isn’t only UX-layer agents — it’s also compilers, kernels, and performance engineering. (Link to the post)

See you next week

That’s Week 2.

If you want this as a weekly feed in your inbox, the idea is simple: I’ll keep reading everything — you only get the filtered list. Subscribe on Substack so you don’t have to rely on the algorithm.

Iscriviti ora

That’s because we often treat memory as a single feature. In practice, memory is a system: different representations, different purposes, and a lifecycle that needs rules (not just storage).

In this post I’ll give you a practical map to design agent memory that actually works: what memory is made of (forms), what it’s for (functions), and how it evolves over time (dynamics)—the pieces that determine whether memory makes your agent reliable or just noisier.

If you like pragmatic deep-dives on LLMs, agents, and real-world ML engineering, subscribe to this Substack so you don’t miss the next one.

Iscriviti ora

Memory in the Age of AI Agents (and why “long-term vs short-term” isn’t enough anymore)

LLM agents don’t fail only because they “forget” something. They fail because we keep treating memory as a single feature, when in practice it’s an entire system with different representations, purposes, and update rules.

The paper “Memory in the Age of AI Agents: A Survey — Forms, Functions and Dynamics” (Hu et al., arXiv:2512.13564, Dec 2025) is the best attempt I’ve seen to bring conceptual order to the chaos. It’s a survey, but it reads like an architectural map: it separates what memory is made of, what it’s for, and how it evolves over time—and it explains why the old taxonomy (“short vs long term”) is too coarse for modern agent stacks.

Below is the version I’d want if I had 8 minutes and I’m about to design an agent that must survive beyond a single chat window.

First: “Agent memory” is not just RAG, and not just long context

A lot of confusion comes from mixing four things:

• RAG: fetch external knowledge at query time.

• Context engineering: prompt/format/tooling tricks to make the model behave.

• LLM memory (in the “model internals” sense): attention/KV cache tricks, long-context architectures, compression.

• Agent memory: a persistent, self-evolving cognitive state that accumulates facts and experience across interactions.

This distinction matters because each bucket implies different evaluation criteria. RAG is about retrieval quality; long-context is about sequence handling; agent memory is about adaptation over time.

Instead of forcing everything into “short vs long term”, the survey proposes three orthogonal lenses:

1. Forms (how memory is represented)

2. Functions (why memory exists / what role it plays)

3. Dynamics (how it gets created, updated, and used)

What “memory” is made of

The survey frames modern agent memory as multiple representational choices. The abstract highlights contextual, external, parametric, and latent memory—each with different trade-offs.

Contextual memory (in-context)

This is the simplest: keep relevant information in the prompt/context window (possibly compressed/summarized).
Pros: trivial to implement, transparent.
Cons: expensive, brittle, and capped by context limits; also easy to “drift” as the conversation grows.

External memory (stores outside the model)

Vector DBs, key-value stores, knowledge graphs, episodic logs—anything retrievable and persistent.
Pros: scalable, editable, inspectable; can support provenance and access control.
Cons: retrieval is a whole subsystem; quality depends on chunking, embeddings, ranking, and query formulation.

Parametric memory (in the model weights or add-on modules)

This includes weight updates, model editing, adapters/LoRAs, or auxiliary modules that “store” knowledge in parameters.
Pros: fast at inference; can internalize patterns.
Cons: hard to debug; risk of interference; updating is non-trivial and can be costly.

Latent memory (inside hidden states / KV / internal representations)

Instead of storing human-readable text, memory lives as latent embeddings, cached states, or learned compressed representations.

When you should care: when you want memory-like behavior with lower latency and without dumping everything into a text store—but you accept that interpretability drops.

What memory is for (three pillars)

This is the part most teams skip. They build “memory” as storage, then wonder why it doesn’t improve agent reliability. The survey proposes three primary functions.

Factual memory — “What does the agent know?”

Stable declarative knowledge: user preferences, constraints, environment state, long-lived facts that keep behavior consistent.

Experiential memory — “How does the agent improve?”

Procedural knowledge distilled from trajectories: what worked, what failed, strategies, skills, heuristics. This is the agent getting better over time—without you hand-authoring rules.

Working memory — “What is the agent thinking about now?”

A controlled scratchpad to manage context during the task: subgoals, intermediate results, plan state, tool outputs, “current hypothesis”.

A useful mental model:

• Factual = identity and continuity

• Experiential = learning and skill

• Working = cognition-in-motion

If your agent is flaky, ask which function is missing. Most “memory” implementations only cover factual memory (sometimes poorly) and ignore experiential + working memory entirely.

Memory isn’t a database, it’s a lifecycle

The paper formalizes agent memory as an evolving state with three core operators:

1. Memory formation: turn raw interaction artifacts into candidates worth keeping

2. Memory evolution: consolidate, resolve conflicts, forget, restructure

3. Memory retrieval: decide what to pull, when, and how to integrate it into reasoning

This is where agent memory becomes engineering, not vibes. You need policies:

• What gets stored (and at what granularity)?

• How do you prevent redundant or contradictory entries?

• When do you summarize vs retain raw traces?

• How do you retrieve without polluting the context?

• How do you measure whether memory helped?

A killer insight here: temporal labels (“short-term/long-term”) often emerge from usage patterns, not from separate components. A single memory substrate can behave short-term or long-term depending on how you form/evolve/retrieve.

What I’d take from this paper if I’m building an agent this week

Here’s a practical checklist inspired by the taxonomy:

1) Choose the function before the storage.
Are you trying to preserve facts, learn strategies, or manage active context? Pick one primary target first.

2) Don’t ship memory without evolution rules.
If you only “append”, your agent will degrade: contradictions, prompt bloat, retrieval noise. You need consolidation + forgetting.

3) Evaluate memory on long-horizon tasks.
A memory system that helps on turn 3 but hurts on turn 30 is not “working”.

4) Treat memory as an attack surface.
Poisoning, prompt injection via memory items, privacy leakage, provenance: once memory persists, mistakes persist too.

A forward-looking angle the paper gets right: RL is coming for memory

One of the most interesting parts is the trajectory toward learning memory policies (what to store, how to update, what to retrieve) instead of hand-designing them.

Even if you’re not doing RL today, this matters: the “memory manager” may become a trainable component, not a pile of heuristics.

If you don’t want to miss the next deep-dives (agents, memory, distillation, infra), subscribe. No spam—just the few things worth your time.

Iscriviti ora

I’m starting a simple ritual: once a week, I’ll share a short, curated list of the most interesting things I stumbled on while reading, building, and going down internet rabbit holes. Think of it as my “bookmark highlights”—no noise, no hot takes for the sake of it, just genuinely useful or thought-provoking stuff.

If you want receive those updates in your mailbox you can subscribe to this substack.

Iscriviti ora

Each link comes with a quick explanation of what it is and why it matters, so you can skim in a minute and decide what deserves a click (or a save). Let’s kick it off.

Rollbacks for agents: why “tool calling” isn’t enough (TheoryVC)

If you’ve ever wondered “why aren’t agents buying cars yet?” this post has a surprisingly practical answer: the blocker isn’t only model intelligence—it’s reversibility.

The core idea: in software we trust automation because we built decades of safety rails (git revert, code review, database transactions, observability). Outside software, agents don’t have that infrastructure, so actions become high-consequence + low-reversibility decisions. The post argues snapshots help inside a single system, but break as soon as an agent crosses boundaries via tools (e.g., “buy car” + “wire money”). The proposed path forward borrows from distributed systems: transaction semantics, orchestration patterns like SAGA, and ultimately extending MCP to support transactional workflows (It’s Not Too Late to Roll Back MCP)

“Chat with your videos” at extreme length: VideoRAG (HKUDS)

VideoRAG is a retrieval-augmented approach built for the thing most video tools still struggle with: very long videos (think: lectures, documentaries, multi-hour recordings).

The repo frames it through “Vimo,” a desktop app concept: drag & drop videos, ask questions in natural language, and retrieve specific moments—scaling to hundreds of hours of content. It also mentions a benchmark (“LongerVideos”) and a graph-driven indexing angle for retrieval.
If you want the research framing: the associated paper describes VideoRAG as a RAG framework aimed specifically at extremely long-context video understanding (where naïve “stuff it all into the context window” falls apart fast) (VideoRAG)

A vibe-check on coding right now (Sergey Karayev on X)

This one is short but culturally loud: Karayev calls Claude Code + Opus 4.5 a “watershed moment,” framing software creation as shifting from something artisanal to something closer to industrial production.

Even if you don’t buy the whole metaphor, it’s a useful signal: the “unit economics” of building (iteration speed, baseline quality, solo-builder leverage) are changing enough that people are reaching for historical analogies. (Link X)

WeDLM: diffusion decoding that actually plays nicely with KV cache (WEDLM)

Diffusion language models promise parallel decoding, but in practice they often lose to highly optimized autoregressive engines because bidirectional attention breaks prefix KV caching.

WeDLM’s pitch: keep everything under standard causal attention, and use Topological Reordering + a streaming-style decoding procedure so predicted tokens can be committed and cached without the usual “stop-and-wait” diffusion behavior. The paper reports substantial speedups vs vLLM-served AR baselines under matched settings (including claims like ~3× on reasoning benchmarks and up to 10× in low-entropy regimes). (WEDLM)

“Epistemia”: why LLMs can sound right without “knowing” (preprint)

This preprint is one of the cleaner conceptual frameworks I’ve seen for explaining the uneasy gap between human judgment and LLM outputs.

The authors map seven epistemic fault lines (grounding, parsing, experience, motivation, causal reasoning, metacognition, value) and introduce “Epistemia”: a condition where linguistic plausibility substitutes for epistemic evaluation—creating the feeling of knowing without the labor of judgment. If you do evals, governance, or just want better mental models for “why this fails,” it’s worth the read (Epistemological Fault Lines Between Human and Artificial Intelligence)

LTX-2: open audio-video generation with inference + LoRA training (Lightricks)

LTX-2 is positioned as a DiT-based audio-video foundation model that generates synchronized video and audio in one system, with an official repo that includes both inference code and a LoRA training pipeline.

Practical details that matter if you actually want to try it: the README includes a quick start using uv, and points to multiple downloadable checkpoints (including a “distilled” one) from Hugging Face. (ightricks/LTX-2)

Before going deeper, remember, I write more on applied AI, graphs, and generative AI. Subscribe to get new posts in your inbox

Iscriviti ora

The Core Problem: Networks, Not Images

Here’s what most ML practitioners don’t realize about neuroscience: the brain is fundamentally a graph problem.

When you look at an MRI scan, you see a 3D image. But that’s not how the brain works. The brain is ~86 billion neurons connected by ~100 trillion synapses. Even at the macro scale, we can measure with imaging, it’s a network of regions communicating through white matter fiber bundles.

My work focused on Multiple Sclerosis (MS), a disease that progressively damages these connections. The clinical question: can we automatically classify disease stages? But the real ML challenge was: how do you represent brain structure as a graph that captures meaningful pathological changes?

From Medical Scans to Graphs: The Pipeline

Let me walk you through how we convert raw brain scans into graphs suitable for ML. Every step involves non-obvious technical decisions.

Step 1: Extracting Connectivity from Imaging Data

We use a specialized MRI technique that measures water diffusion in tissue. Think of it like this: water molecules in brain tissue don’t diffuse randomly—they preferentially flow along fiber pathways, like cars following roads.

By measuring diffusion in 48 different directions at each 3D voxel, we get a local orientation distribution. Then we use a probabilistic tracking algorithm to trace fiber pathways through the brain, following these orientation estimates from voxel to voxel.

The ML analogy: It’s like doing gradient ascent through a vector field, except the vectors are probability distributions and you’re dealing with massive uncertainty at every step.

Key technical challenge: Fiber crossings. At many voxels, multiple fiber bundles intersect. A simple model (like fitting a single Gaussian) fails catastrophically. We used spherical harmonics decomposition (order h=4) to represent multi-modal orientation distributions—basically expressing the distribution as a weighted sum of basis functions on the sphere.

Step 2: Anatomical Constraints

Here’s where domain knowledge becomes critical. We don’t just trace fibers blindly—we constrain them using anatomical priors from structural brain scans.

We segment the brain into four tissue classes:

• Gray matter (where neuron cell bodies live)

• White matter (the fiber highways)

• Subcortical structures (deep brain regions)

• Cerebrospinal fluid (empty space)

The constraint: Fibers must start in gray matter, travel through white matter, and end in gray matter. They can’t originate in empty space or terminate in the middle of fiber tracts.

Why this matters for ML: Without these constraints, we generate biologically impossible connections that look fine mathematically but are anatomical nonsense. It’s like allowing negative probabilities—technically your algorithm runs, but the output is meaningless.

We generated 500,000 fiber streamlines per subject using this constrained probabilistic tracking.

Step 3: Graph Construction

Now we build the actual graph:

Nodes: We parcellate the brain into q=84 anatomically defined regions using a standard atlas. Think of these as “super-pixels” at the brain region level—visual cortex, motor cortex, hippocampus, etc.

Edges: For each pair of regions (i,j), count how many fiber streamlines connect them. This gives us a weighted adjacency matrix A ∈ ℕ^(84×84).

The graph: G = (V, E, ω) where:

• |V| = 84 nodes

• E = {(i,j) : number of connecting fibers > 0}

• ω: E → ℕ assigns connection weights

We now have a weighted, undirected graph representing brain structural connectivity.

Step 4: The Thresholding Problem (This is Critical)

Here’s a decision that breaks many neuroscience ML projects: how do you threshold your connectivity matrix?

The raw fiber counts are noisy. Weak connections might be spurious (tracking errors) or real but unimportant. But which connections do you keep?

Two common approaches:

1. Absolute threshold: Keep edges with weight > k

2. Proportional threshold: Keep top τ% of edges by weight

We chose proportional thresholding because it maintains consistent network density across subjects—critical for fair comparison.

Our optimization procedure:

• Tested τ ∈ [0.05, 1.0] in 0.05 increments

• For each τ, computed 5 global graph metrics on healthy controls

• Calculated coefficient of variation (CV) across subjects

• Selected τ = 0.35 where CV stabilized (~0.90%) and density matched biological priors

The ML lesson: This is hyperparameter optimization, but you can’t use validation set performance—you don’t have ground truth connectivity! Instead, we optimized for measurement stability (low CV) while respecting domain constraints (reasonable density).

This is the kind of unglamorous technical work that determines success but rarely makes it into papers.

From Graphs to Clinical Signals (features, results, and the “why this worked”)

Once the connectome is thresholded, the goal isn’t fancy embeddings—it’s stable, interpretable signals that survive small samples and clinical scrutiny. With ~100 subjects, we treated each brain graph as a system and extracted global topology metrics (density, global efficiency, modularity, assortativity, transitivity, characteristic path length). These are cheap to compute, robust to imperfect region alignment, and easy to explain to clinicians (“network becomes less efficient”, “modules reconfigure”, “hubs change behavior”).

What the metrics revealed across MS stages was a trajectory, not a monotonic decline: early stages showed disrupted segregation and efficiency; intermediate stages suggested compensatory reorganization (e.g., increased modularity/transitivity); late stages showed fragmentation and hub behavior shifting toward collapse. This is the practical advantage of graph features: they can tell a mechanistic story that raw lesion counts often can’t.

For classification, we used an RBF SVM because it’s sample-efficient, tunable with straightforward grid search, and works well with standardized low-dimensional features. Most clinically relevant tasks were framed as binary (early detection, course differentiation, progression detection), plus a harder multi-class stage problem. Modularity repeatedly emerged as the strongest single discriminator, and combining all metrics helped most where signatures were subtle (early disease).

The hardest parts weren’t the model—they were the realities of medical ML:

• Soft ground truth (clinical labels are noisy and can drift over time).

• Class imbalance (rare cohorts like CIS).

• Inter-subject variability (anatomy/parcellation mismatch).

• Confounds (age/sex and other correlates).

• Scanner/protocol shift (the biggest blocker to real deployment).

The takeaway: in neuroscience, wins often come from boring rigor—stable measurement, defensible preprocessing, and models that match the data regime.

Where Modern Deep Learning & Generative AI Actually Fit (and what I’d do now)

If I revisited this today, I still wouldn’t “end-to-end” everything. The classical pipeline (diffusion MRI → tractography + constraints → parcellation → graph) encodes geometric and anatomical priors that are brutally data-hungry to learn from scratch.

Where modern methods do help:

1) Hybrid representation learning (interpretable + learned).
Use graph metrics as an interpretable backbone, and add a lightweight GNN/GraphSAGE encoder to learn residual patterns—then fuse both. You get a model you can ablate and defend: “here’s what topology explains, here’s what learned structure adds.”

2) Pretraining + transfer (when the source domain is close).
Large cohorts (e.g., population biobanks) make it feasible to pretrain a graph encoder on proxy tasks (age/sex/quality) and fine-tune on MS—carefully checking for negative transfer.

3) Uncertainty as a first-class output.
For clinical use, confidence matters as much as accuracy. Dropout-based or Bayesian-ish approximations can flag low-confidence cases for expert review instead of pretending every prediction is equally reliable.

4) Longitudinal modeling (the real prize).
Snapshots are limited. Modeling trajectories over time (sequences of graphs) is often more clinically aligned: predict progression risk, not just current label.

Where LLMs / “Generative AI” fit realistically:

• Not as graph analyzers via adjacency-matrix tokenization.

• Yes, as glue for multimodal integration and workflow: combining clinical notes + imaging-derived features + genetics, producing structured summaries, cohort reports, QC logs, and human-readable explanations—while specialized encoders handle the actual graphs.

The bigger picture for ML practitioners entering neuroscience:

• Data is scarce and expensive → sample efficiency wins.

• Labels are uncertain → calibration and robustness beat leaderboard chasing.

• Distribution shift is brutal → multi-site validation and harmonization are mandatory.

• Interpretability is non-negotiable → choose models you can explain, or invest in explanations that clinicians actually trust.

Bottom line: the domain rewards systems engineering more than model cleverness. The “model” is often the easiest part; the advantage comes from encoding constraints, measuring stability, controlling confounds, and designing for real clinical translation from day one.

If this was useful, I write more on applied AI, graphs, and generative AI on my Substack. Subscribe to get new posts in your inbox

Iscriviti ora

Tired of clickbait articles that gloss over the details without offering real explanations or usable code, I’ve decided to write a clear, no-nonsense guide to the process - with practical examples and actual libraries you can use.

In this article, I’ll walk you through the concept of LLM distillation and show you how to implement it in Python. We’ll begin with Logit Distillation techniques and later move on to Hidden State Distillation. The goal of this post is to explain everything straightforwardly, no formulas, just clear intuition. If you’re into equations, you’re welcome to follow along, but this guide focuses more on understanding than math.

Introduction to distillation

Large Language Models (LLMs) like GPT or LLaMA are powerful but often huge - they require a lot of memory, computation, and energy to run. Distillation is a technique that helps to make these models smaller, faster, and more efficient without losing too much of their intelligence.

Think of it like this: imagine a big, brilliant teacher explaining things to a student. The student doesn’t memorize everything, but learns the essence of how the teacher thinks and responds. Over time, the student becomes good enough to give similar answers, but using fewer resources.

In LLM distillation, the “teacher” is a large pre-trained model, and the “student” is the smaller model we want to train. We let the student learn by imitating the teacher - by observing its responses, predictions, and even how it thinks internally. The result? A much lighter model that can still perform well in real tasks, from chatting to coding - perfect for mobile devices, chatbots, or cost-efficient deployments.

The main idea then is to take a large model like LLama 70B and “distill” its knowledge into a small model like LLama 8B, 4B, or even 1B. This process should also be less computationally expensive compared to “training from scratch” since we use not only the simple “next token” prediction but also the hidden “dark knowledge” [1] of the teacher model. This is the secret sauce of distillation: we extract all the possible information from the model, not only the directly visible one.

Logit distillation? Let’s understand what it is

First, we need to practically understand what dark knowledge is in the context of the Logit distillation. I’ll use the concept of dark knowledge to introduce the logit distillation since they are strongly correlated.

Tokenization

Every time you provide a text input to an LLM, the tokenizer transforms each word or subword into a corresponding integer. Here’s a simple example to illustrate how it works:

Each token is assigned a unique integer through a dictionary, so every word (or subword) has its own numeric representation. The total number of tokens a tokenizer can handle is called the vocabulary size. This means an LLM can only understand and work with the tokens included in its vocabulary.

For example, Llama3 has a vocabulary size of 128,256, meaning it can process and generate up to 128,256 distinct tokens. Once the text is converted into tokens, the LLM works entirely with their numerical representations - it doesn’t “see” any actual words or text. At the end of the process, it converts these numbers back into text through a step called de-tokenization.

Understanding this concept is crucial to grasping what logit distillation is all about. If any part of this feels unclear, I recommend revisiting it before moving on. And if you’re curious to dive deeper into tokenization, I suggest checking out [2].

Logit generation

Every time you provide input to an LLM, the model calculates a logit value for each token in its vocabulary before converting the output back into text. In simpler terms, with Llama3, this means that at each prediction step, the model produces a vector with 128,256 floating-point numbers - one for every possible token.

Here’s an example to illustrate what that looks like:

In the context of logit distillation, this table represents the LLM’s “dark knowledge”. Each number corresponds to a logit, which reflects how much importance the model assigns to a specific token. Tokens with higher logit values are more likely to be chosen as the next prediction.

These logits are eventually transformed into probabilities using a softmax function, which also involves a parameter called temperature. But for now, let’s set softmax aside and just focus on understanding what logits are.

Distillation using logits

Now that you are aware of what logits are and how LLMs compute them. We can deep dive into how distillation works. Let’s describe how the algorithm works:

Step 1: Forward pass through both models

For a given input, both the teacher and the student perform the next token prediction, generating an array of logits. Here’s what this looks like in practice:

+--------+---------------------------+---------------------------+
| Token  | Logits (Teacher)          | Logits (Student)          |
+--------+---------------------------+---------------------------+
| “Ceci” | [2.0, 1.0, 4.0, 0.5, 0.3] | [1.9, 1.1, 3.9, 0.6, 0.4] |
| “est”  | [0.5, 3.0, 1.0, 0.2, 0.3] | [0.6, 2.9, 1.1, 0.3, 0.4] |
| “un”   | [0.1, 0.2, 0.5, 4.5, 0.7] | [0.2, 0.1, 0.6, 4.4, 0.8] |
| “test” | [3.5, 1.5, 0.1, 0.0, 0.2] | [3.4, 1.4, 0.2, 0.1, 0.3] |
+--------+---------------------------+---------------------------+

Step 2: Compute the distillation loss

The key insight here is that we want the student’s logit distribution to match the teacher’s logit distribution. This is where the “dark knowledge” comes in - we’re not just teaching the student to predict the correct next token, but to understand the entire probability distribution over all possible tokens that the teacher has learned.

There are two main components to the loss function in logit distillation:

1. Distillation Loss (Soft Targets): This measures how well the student’s logits match the teacher’s logits. We typically use the KL-divergence (Kullback-Leibler divergence) to measure the difference between the two probability distributions after applying softmax with temperature.

2. Student Loss (Hard Targets): This is the standard cross-entropy loss between the student’s predictions and the actual ground truth labels from the dataset.

The total loss is a weighted combination of these two:

Total Loss = α × Distillation Loss + (1 - α) × Student Loss

Where α is a hyperparameter (typically between 0.5 and 0.9) that controls how much we want the student to learn from the teacher versus the ground truth.

Step 3: The role of temperature

Temperature is a crucial hyperparameter in logit distillation. Before computing the distillation loss, we apply a softmax function with temperature T to both teacher and student logits:

P(token_i) = exp(logit_i / T) / Σ exp(logit_j / T)

When T = 1, this is just standard softmax. When T > 1 (typically T = 2 to 5), the probability distribution becomes “softer” - meaning the differences between high and low probability tokens become less extreme. This is important because it reveals more of the teacher’s “dark knowledge” about which tokens are somewhat plausible, even if they’re not the top choice.

Step 4: Backward pass and optimization

Once we have the total loss, we perform a backward pass to compute gradients, but only for the student model. The teacher model remains frozen - we don’t update its weights. The student’s weights are then updated using an optimizer like Adam or SGD.

Why this works

The beauty of logit distillation lies in its efficiency and effectiveness:

• The student learns not just what the correct answer is, but also what the teacher considers reasonable alternatives

• This richer signal helps the student generalize better than training from scratch

• Since both models share the same tokenizer and vocabulary, the logit vectors align perfectly - making the transfer of knowledge seamless

• The student can learn from the teacher’s uncertainties and confidence levels, not just from binary correct/incorrect labels

Important note about tokenizer and dictionary

For logit distillation to work in this straightforward way, the teacher and student must share the same tokenizer and vocabulary. Why? Because the logit vectors must have the same dimensionality, and each position must correspond to the same token in both models.

If “hello” is token 1234 in the teacher’s vocabulary, it must also be token 1234 in the student’s vocabulary. Otherwise, we’d be comparing apples to oranges - the teacher’s confidence about token X would be interpreted by the student as confidence about a completely different token Y.

This is why logit distillation is most commonly used within model families like LLaMA 2 → LLaMA 3 or GPT-3 → smaller GPT variants, where the tokenizer remains consistent across different model sizes.

What next

In this first part, we’ve explored the fundamentals of logit distillation - how a large teacher model transfers its “dark knowledge” to a smaller student model through logit matching.

We covered the key concepts: how LLMs generate logit vectors for every token in their vocabulary, why these logits contain richer information than just the predicted token, and how the distillation process works by aligning the student’s logit distribution with the teacher’s using a combination of soft and hard targets with temperature scaling.

We also emphasized an important constraint: this approach requires teacher and student to share the same tokenizer and vocabulary for seamless knowledge transfer.

What’s next? In Part 2, we’ll dive into the practical implementation with Python code, working examples, and the libraries you can actually use to perform logit distillation on your own models.

If you found this article helpful, please subscribe to stay updated when Part 2 drops. We’ll go from theory to practice, giving you everything you need to start distilling your own models.

Iscriviti ora

References

[1] G. Hinton et al. Dark Knowledge 2014, https://www.ttic.edu/dl/dark14.pdf
[2] https://medium.com/illuminations-mirror/on-tokenization-in-llms-34309273f238

Iscriviti ora