Vidocq already does most of its work at build time: CDI wiring, JSON binding, REST routing and data access are generated as bytecode before the application ever starts. What is left at startup is the JVM itself: loading classes, running the boot, and warming up the JIT.
CRaC (Coordinated Restore at Checkpoint) takes care of that part. You start the application, let it warm up, and take a snapshot of the whole process. Later, in a fresh container, you restore the snapshot instead of starting from scratch, and the application answers with its classes loaded and its code already compiled.
Vidocq now supports it. This post explains the design, how to try it, and what you need to know before putting a snapshot in production.
The one rule of a checkpoint
A snapshot cannot carry what belongs to the operating system: a listening socket, a database connection, an open file. If any of these is still open when the checkpoint is requested, the JVM refuses to take it. So everything that holds such a resource has to let go of it before the checkpoint and take it back after the restore.
In Vidocq, that means several bricks: Chappe holds the HTTP listeners, Mansart holds the connection pools and the transaction recovery journal, Humboldt holds the connections of its telemetry exporters, and Vauban’s class loader holds the application jar open. And the application holds resources of its own.
One module knows CRaC, the bricks do not
The usual way to support CRaC is the org.crac library, a facade that finds the JDK’s CRaC API at runtime by
reflection, so that the same jar runs on any JDK. Vidocq avoids runtime reflection: we generate code at build time
instead. So we took the other path and split the work in two.
The bricks implement a neutral interface from the runtime’s SPI, Suspendable. It knows nothing about CRaC:
public interface Suspendable {
/** Releases what a snapshot cannot carry. Called before the snapshot, in reverse start order. */
void suspend();
/** Takes back what suspend() released. Called after the restore, in start order. */
void resume();
}The runtime hands the extensions that implement it to whoever asks, in start order, from the extension list it
already builds with ServiceLoader. No reflection, no scanning.
Then one optional module, vidocq-runtime-crac-extension, depends on the JDK’s own jdk.crac module. It registers a
single CRaC resource that drives the whole chain:
SuspendableChain chain = new SuspendableChain(context.suspendables());
this.resource = new Resource() {
@Override
public void beforeCheckpoint(Context<? extends Resource> ctx) {
chain.suspend(); // reverse start order, like a shutdown
}
@Override
public void afterRestore(Context<? extends Resource> ctx) {
chain.resume(); // start order, like a boot
}
};
Core.getGlobalContext().register(resource);This split has a cost, and we chose to pay it: because the module requires jdk.crac, building it, and running an
application that declares it, needs a JDK with CRaC. Our CI builds it in a dedicated Maven profile, on Azul Zulu CRaC,
so the rest of the build stays on a standard JDK. An application without the extension sees none of this.
What lets go, and what comes back
-
Chappe, the HTTP server, stops its listeners before the checkpoint and binds them again after the restore.
-
Mansart's connection pools close their connections, then reopen them after the restore. On the way back they read their configuration again, so a restored container can be pointed at another database through its environment.
-
Mansart's transaction recovery journal closes its file and opens it again by its path.
-
Humboldt, MicroProfile Telemetry, waits for the export in progress, then closes the HTTP clients of its OTLP exporters, and opens them again after the restore.
-
Vauban's class loader releases the application jar and reopens it on the next class it has to read.
The last one was not on our list. The end-to-end test found it: the first checkpoint of an application started with an application layer failed, because the class loader still held the jar open. The fix belongs in Vauban, so it went there.
Your own resources: two CDI events
Vidocq knows only its own sockets, connections and files. A client the application opened itself (a JDK
HttpClient, a gRPC channel, a JDBC connection outside the Mansart pool), a file it keeps open, data it should load
before the snapshot: those belong to the application. It takes part through two CDI events of the runtime’s SPI,
io.vidocq.runtime.spi.event.Checkpoint and Restore, which it observes like any other event (Vidocq/vidocq#497).
The two events have a fixed place in the chain. Before the checkpoint, the HTTP server stops first, so no request
reaches the application while it lets go. Then the Checkpoint observers run, while the pool, the transactions and
the telemetry are still open: they can still query the database. Those let go after them. After the restore, the
order is reversed: the Restore observers run once the pool is back, and before the server takes requests again.
The case that started it: an application needs a 50 MB reference file, which it downloads from another service. Loaded into memory before the checkpoint, it lands in the image, and no restored process downloads it again.
@ApplicationScoped
public class Catalogue {
private static final URI SOURCE = URI.create("https://reference.example.com/catalogue.csv");
private volatile HttpClient client = HttpClient.newHttpClient();
private volatile Map<String, Part> parts;
private volatile Instant loadedAt;
void beforeCheckpoint(@Observes @Priority(100) Checkpoint checkpoint) throws IOException, InterruptedException {
if (parts == null) {
load(); // (1)
}
client.close(); // (2)
}
void afterRestore(@Observes Restore restore) throws IOException, InterruptedException {
client = HttpClient.newHttpClient(); // (3)
if (loadedAt.isBefore(Instant.now().minus(Duration.ofHours(1)))) {
load(); // (4)
}
}
private void load() throws IOException, InterruptedException {
var response = client.send(HttpRequest.newBuilder(SOURCE).build(), BodyHandlers.ofInputStream());
try (InputStream body = response.body()) {
parts = Parts.read(body);
}
loadedAt = Instant.now();
}
}-
Loaded into memory before the snapshot: the 50 MB land in the image.
-
An open connection makes the checkpoint fail. Close what the snapshot cannot hold.
-
Opened again after the restore.
-
The image can be days old: check that the data is still fresh enough.
What to know about these observers:
-
How old is the image?
Restore.checkpointAt()is the instant of the checkpoint the process comes from. WithInstant.now(), it tells how long the image waited before this restore. -
Order. Observers run in the order of their
@Priority, the lowest first, one after the other, on the thread that takes the checkpoint. -
Failures. An observer that throws makes the checkpoint fail. The JDK then runs the restore side all the same, so the
Restoreobservers are called: write them so that they cope with aCheckpointobserver that did not finish. -
No
jdk.cracin the application. Its module readsio.vidocq.runtime.spi, notjdk.crac. On a JDK without CRaC, the events are simply never fired, and the same code runs unchanged.
The Checkpoint and restore page of the documentation has the rest: the order of every step, what an image keeps, and how to read a refused checkpoint.
Trying it
vidocq create knows the extension. Add it with the others:
vidocq create --name demo -x cassini-rest -x cracNext to the project, you get a Dockerfile.crac that builds and runs the application on Azul Zulu CRaC, with the
commands to use it in its header. Take a checkpoint of a running container:
docker build -f Dockerfile.crac -t demo-crac .
docker run -d --init --name ckpt -v crac-img:/crac \
-e JAVA_OPTS="-XX:CRaCCheckpointTo=/crac -XX:+StartAttachListener" demo-crac
# warm it up, then:
docker exec ckpt jcmd io.vidocq.runtime.core.Vidocq JDK.checkpointThen restore it in a fresh container, from the same volume:
docker run --rm --init -p 8080:8080 -v crac-img:/crac \
-e JAVA_OPTS="-XX:CRaCRestoreFrom=/crac" demo-cracTwo details there come from experience. --init matters: a JVM running as process 1 does not accept the attach
that jcmd needs. And behind --init, the process is named exe, which is why jcmd is given the main class
rather than a process name.
The rest of the tooling follows. vidocq doctor fails when the project has the crac extension but the JDK has no
jdk.crac module, and warns on macOS or Windows, where you can build but not take a checkpoint. The jlink goal adds
jdk.crac to the image by itself as soon as a module of the application requires it.
How we test it
A script starts a packaged application on a CRaC JDK, creates data over HTTP, takes a checkpoint (the process exits once its image is written), restores it, then checks that the data is still there and that the restored process accepts new requests. It runs every night in our CI, on Azul Zulu CRaC.
What a snapshot does not carry, and what it carries too well
-
Linux only. Checkpoints are a Linux feature. You can develop on a Mac, but take the snapshot in a Linux container.
-
A CRaC JDK. We test with Azul Zulu CRaC 25 and its default engine, which needs no extra Linux capabilities in the container.
-
Configuration after a restore. Environment variables and system properties are those of the restore command, and Mansart’s pools read theirs again. But values read once at startup from a configuration file stay as they were at checkpoint time, and so does the HTTP port: read again, in a
Restoreobserver, what may differ. -
Secrets. A snapshot is a copy of the heap. Whatever was in memory at checkpoint time, a database password, a signing key, is in the image files. Store and ship them as you would store the secrets themselves.
Numbers
We measured a restore against a regular start on an application built for the purpose, Nomenclature. It reads a bill of materials of 600,000 parts from a CSV file, indexes it, evaluates validation rules on every part, computes a checksum and makes 20,000 warm-up calls before it reports ready. The number of rules sets three start-up tiers: about 10 seconds (s), about one minute (m) and about eight minutes (l).
For each tier and number of cores, the harness started the application cold several times, took one checkpoint once
it was ready, and restored that checkpoint as many times. It ran on one machine (Intel Core i7-8700B, Linux), with
Azul Zulu 25.0.4.1 CRaC and its default engine, the server pinned to 1 or 4 cores. There were 7 runs of each mode for
tiers s and m, and 3 for tier l, on 4 cores only. Vidocq was built from main at f7625f13, before the application
events of Vidocq/vidocq#497, and has not been measured again since. Medians, with the range of the runs:
| Tier, cores | Cold start | Restore | Cold / restore |
|---|---|---|---|
s, 1 |
10.8 s (10.7–10.9) |
402 ms (391–435) |
27x |
s, 4 |
7.9 s (7.8–8.0) |
392 ms (390–410) |
20x |
m, 1 |
58.0 s (57.5–58.8) |
406 ms (390–448) |
143x |
m, 4 |
54.7 s (54.4–55.8) |
414 ms (388–548) |
132x |
l, 4 |
467 s (411–476) |
359 ms (358–360) |
1,300x |
What the runs say:
-
A restore takes about 400 ms, whatever the start-up took. The medians go from 359 to 414 ms in every tier and number of cores. The gain is therefore set by the application: the longer it takes to start, the more a restore saves. Of a cold start, Vidocq’s own part, outside the application’s start-up work, is about one second on 4 cores (1.9 s on 1 core). A restore skips both.
-
The state survives. The 5 checkpoints all succeeded, and every cold start, checkpoint and restore of a tier reports the same checksum.
-
A restored process is small. Five seconds after it is ready, a restored process holds 53 to 68 MB of resident memory, against 569 to 655 MB for the same process started cold: the garbage of the start-up work is not in the image. The image itself takes 301 to 315 MB on disk, 88 to 90 MB with
gzip -1, and the checkpoint takes 2.1 to 3.2 seconds. -
It costs almost no CPU. From the restore to five seconds after ready, a restored process uses 0.20 to 0.29 s of CPU, against 10.7 to 470 s for a cold start.
-
The first request is served at once. Restored on 4 cores, the first request is answered in 5 to 7 ms (median), against 24 to 32 ms cold.
And what they do not say:
-
Tier l is noisy. One of its three cold starts was about a minute shorter than the two others (411 s, against 467 and 476 s). Its restores did not vary.
-
One machine, one warm cache. The image was read from a local Docker volume, still in the page cache after the checkpoint, not from a cold disk or a remote store. The restore ran on the same JVM build, processor and host as the checkpoint, as CRaC requires. And the start-up work is synthetic, chosen to make long start-ups.
The method, the commands and every run are in the entry BENCH-20261105-01 of
the benchmarks' BENCH.md, next to the
suite itself, bench-crac.
CRaC support, with the application events, ships in Vidocq 0.4.0-RC2. As always, we would like to hear what you find when you try it.
Transparency: this article was written with the help of an AI, Claude, which also helped build the feature it describes.
