ZSvirt: engine architecture — async, stateless, workflow and plugins

It took me a while to bring the second article of the series. Work has been nuts at the datacenter — studies, LAB for some side projects, and my 3D FARM, which I am actually taking time to professionalize. It is already making decent money, but professionalizing takes presence. In the middle of all that, there was also the TrueNAS article that had to go out hot, with an expiry date right on top.

Anyway. Now for real: in the first article we looked at ZSvirt from the outside — instant snapshots via ZFS, automatic provisioning, unified API. This time it is the opposite. We are going to open the engine. Because how it was built explains why it works the way it works.

And it is serious engineering: ZSphere, ZStack’s commercial platform, runs in production in more than 1,000 global environments. ZSvirt is that heir engine, and the inheritance shows up in five pillars you can recognize in every commit of the public repository.

1. Async framework: does not block, does not wait

On first read of the code, what jumps out is the deliberate absence of long-running synchronous operations. If you create a VM, migrate a disk, take a snapshot — anything that takes more than a few milliseconds — the request does not sit there waiting. That is intentional, and it already removes one of the biggest frictions I see in client environments.

Instead, the system uses async messages. You call an endpoint (e.g. /vm), it returns immediately with a job ID. Internally, the request became a message that entered a queue and is being processed by workers. In ZSvirt, you can poll that job ID to track progress, and when it finishes, the result is there.

# Example (API pseudocode):
POST /api/v1/vm
{
  "name": "web-01",
  "cpuNum": 4,
  "memory": 8589934592
}

# Returns immediately:
{
  "uuid": "vm-123",
  "jobUuid": "job-456",
  "status": "Pending"
}

# Then you poll the job:
GET /api/v1/jobs/job-456
# Returns real-time status, progress, final result

Why this matters: on a traditional vCenter, if you fire 100 create-VM requests in parallel, the server queues them all, but you stay blocked waiting for responses. With async, the 100 requests arrive, get queued, and return in milliseconds. The real processing happens in background. System throughput scales without the client having to wait.

This is especially critical in automation environments (Terraform, Ansible, SDKs) and mass operations — which is exactly where Proxmox today suffers with timeouts and degradation. I have seen a Terraform pipeline hang for half an hour because a pvesh returned 500 in the middle of creating 40 VMs. With async, that stops being a project problem.

2. Stateless services: horizontal scale without synchronization

Each service inside the ZSvirt architecture (API, workflow, compute, storage, network) is stateless. That means:

• No request depends on state kept from a previous request
• No service needs to sync with another over shared in-memory state
• If a service fails, another can pick up the work from scratch

In practice: you spin up 10 API instances in parallel, put a load balancer in front, and requests distribute automatically. If one dies, the others keep going. No “session stickiness”, no sync quorum, no cache going stale.

That contrasts with architectures that keep state in memory (Proxmox, for example, where the pve-daemon process is responsible for syncing with all nodes). If the daemon goes down, you lose visibility. I once lost a whole afternoon debugging a Proxmox cluster with the daemon out of sync — log shows one thing, the UI shows another, and the real state only comes out of corosync. In ZSvirt, visibility comes from a central database (part of the management node): everything that happened is recorded, and any service can read and continue from where it left off.

3. Workflow engine: orchestration with automatic rollback

When you create a VM in ZSvirt, internally that is not an isolated command. It is a workflow — a sequence of steps with dependencies, validations, and automatic rollback.

Think of that Swedish furniture assembly. If on step 4 the screw does not go in, you do not leave the three previous half-finished steps in the living room — you take it apart and go back to the safe point. That is literally the workflow engine contract: all or nothing.

Example (simplified):

1. Validate VM parameters
2. Reserve resources on the host (CPU, memory)
3. Create primary volume on storage
4. Create network interface
5. Generate boot configuration
6. Start VM

If any step fails, the system automatically reverts the previous steps. If it fails on step 4 (network), it:

• Undoes the network creation (step 4)
• Undoes the volume creation (step 3)
• Releases the reserved resources (step 2)
• Returns an error to the user

All of that without manual intervention. No orphaned disks, no hanging configuration, no reserved resource you forgot to release.

This is especially critical in complex operations like VM migration, disk resize, or production config updates. You do not want partial failure — you want all or nothing.

4. Message bus: component decoupling

Internally, services do not call each other directly (no RPC or REST between workers). Instead, everything goes through a centralized message bus.

The workflow manager publishes a “create volume” message: Event(type: VolumeCreate, targetId: vol-123, params: {...}). Any storage worker listening picks up the message, processes it, and publishes Event(type: VolumeCreated, success: true) back.

Benefits:

• Decoupling: storage does not need to know about compute, compute does not need to know about network
• Scalability: you bring up new storage workers without changing the system
• Resilience: if a storage worker dies, the message stays in the queue and another worker picks it up
• Traceability: every message is audited, you see everything that happened

This is quite different from architectures using direct calls. In Proxmox, if one node’s daemon dies, the orchestrator cannot coordinate with the others. In ZSvirt, the message bus is independent of the workers — if one worker dies, others take over.

5. Plugin architecture: horizontal extensibility

The last pillar is that almost everything is a plugin. Storage backend? Plugin. Network service? Plugin. Compute hypervisor? Plugin (KVM is the default, but the architecture supports others).

New resource types, new capabilities, new integrations — everything enters as a plugin that registers in the system and exposes its endpoints via API.

That explains how TrueNAS can be a native storage backend — it is a plugin that implements the storage interface and integrates with the workflow engine. No code fork, no upstream merge, no recompile. Pure plugin.

How these five things work together

Imagine a complex operation: you want to migrate a VM between two hosts and take a snapshot at the same time (for an immediate pre-migration backup).

1. Async: you POST /vm/migrate + /vm/snapshot in parallel. Both return immediately with job IDs.
2. Stateless: those requests can hit different APIs (you have 3 instances running).
3. Workflow: internally, each operation is orchestrated as a workflow — migrate has 8 steps, snapshot has 4. If any fails, automatic rollback.
4. Message bus: migrate publishes “VM migration started” on the bus. Snapshot worker listens, can take a pre-migration backup. If the network worker sees the host is busy, it publishes an event the workflow engine respects.
5. Plugins: the snapshot worker is a storage plugin. Migration uses a compute plugin. Neither knows about the other — they only talk via the message bus.

Result: complex operation, auditable, resilient, and ready to grow — all without global lock, single point of failure, or shared in-memory state.

What you do not see (but matters for engineering)

There are a few code details worth mentioning:

Idempotency: every operation is idempotent by default. If you send the same request twice, the second waits for the first to finish and returns the same result. Never two VMs with the same ID, never two disks with duplicate UUIDs.

Eventual consistency: the system favors availability over immediate consistency. After an operation, it may take a few seconds for all nodes to see the updated state — but they eventually converge. That is acceptable for most VM workloads.

Circuit breaker patterns: if a service starts failing (e.g. storage responding slowly), the message bus stops sending requests to it, logs alert, and you can intervene. No cascading failures.

In my read

The architecture you see in ZSvirt is not new in concept — it is intelligent application of principles you see in mature distributed systems (Kafka, Cassandra, Kubernetes). The difference is that it is aimed at the specific problem of virtualization — where you need reliability, scalability, and auditability.

Comparing with Proxmox here is inevitable — and before any PVE expert raises a hand, let me be clear: Proxmox was designed to be lean, direct, and simple to operate, and it delivers very well on that. ZSvirt started from a different project premise: dedicated management plane, isolated from compute, with ZSphere history running in more than 1,000 enterprise environments. Different philosophies for different audiences, not a “right and wrong”.

But do not confuse scale with simplicity. ZSvirt is harder to debug when something goes wrong, because state is distributed and needs to be reconstructed from events. Proxmox you see everything in one daemon. The trade-off is clear: more scalability, more operational complexity.

Being honest about the limit of this analysis: I have not installed the engine at scale yet — my environment is the live demo and the public repository. This is engineering reading, not a production report. The real lab, with install and first VM, is the article coming in the series. Anyone running a large cluster, tell me in the comments — I want to validate this read against practice from people running 50+ hosts.

Next in the series: the honest technical comparison with Proxmox and Nutanix — where each one wins, where each one loses, and where it fits in your real infrastructure.

Sources used in this article: ZSvirt repository on GitHub · README — Architecture · Roadmap 2026H2 · Official Glossary · ZSvirt Docs