When a serious CVE drops, the vulnerability gets all the fear. It is the thing with a score and a name, the thing your security team is already emailing about. So it feels like the vulnerability is the dangerous part.

Ask anyone who has run a fleet for a few years which one actually took production down, though, and a lot of them will name the patch. The update that was supposed to close the hole pulled in an incompatible library, or bumped a version something else quietly depended on, and the thing you did to get safe is the thing that woke you up at 2 AM. The bug was a risk. The change you made to fix it was a different risk, and it was the one you owned.

What stings about it is that the conflict was usually preventable. The information existed before the change ever reached a host. You just didn't get to look until it was live, when the only moves left are the bad ones.

CIQ Enterprise Linux Manager (ELM) is built to give you that look earlier. It is the fleet lifecycle management tool that comes with RLC Pro, and it does a lot across a fleet: it mirrors your content, curates it, provisions bare metal, and controls what every host is running. This post is about one thing it does especially well, which is making a change safe before you roll it out.

See the conflict before it ships

In ELM, what your hosts install is a content view: a named, versioned snapshot of exactly which packages and versions ship together. When you fork a new view to bring in an update, ELM runs a forward-only dependency check across it. The change maps forward through the view and the conflicts surface right there, on a draft you have not shipped, without re-walking the whole dependency tree every time.

So the incompatible library shows up while it is still cheap to deal with. You hold that package back, and ELM keeps the outstanding Common Vulnerabilities and Exposures (CVEs) visible on it, filtered by severity and Common Vulnerability Scoring System (CVSS) score. You are not trading a clean rollout for a blind spot. You can see the conflict you avoided and the exposure you are carrying on the version you held, and you make the call with both in front of you.

Prove it on a few hosts before the whole fleet

Seeing the conflict is half of it. The other half is not betting the fleet on being right.

ELM promotes a content view through stages you define, the familiar development to test to production path. You put the change on a small test group first, watch how it behaves on machines that look like production without being production, and only then elevate the exact same view up the chain. Nothing reaches the fleet on faith. It reaches the fleet after it has already run somewhere real and held.

That's the part that changes how the work feels. Rolling out stops being one held-breath moment that becomes a few controlled steps, each one reversible, each one telling you something before the next.

Ship exactly what you signed off on

When the view is right, ELM signs it at curation. The snapshot is immutable and the signature travels with it, so a host installs the content you approved and nothing that drifted in behind it. Every promotion lands in the audit log.

This is what turns a patching story into an answer for your security office. Not "we think the fleet is current," but a record of which signed, immutable version each host group runs, when it shipped, and who moved it there.

Get back to safe in one move

You will still get one wrong now and then. Everyone does. A change looks clean, clears the test group, and misbehaves in production for a reason nobody caught.

Rollback in ELM is a re-point, not a rebuild. Because every version is an immutable snapshot, getting back to a known-good state means pointing the host group at the prior one. It is immediate, and production is not waiting on a repository to rebuild while you sweat.

Where to take it next

Everything above is what the product does. The only test that matters is what it does on your fleet. No two content sets or change calendars look alike, so run your own through it.

The best version of that is not a canned demo. Instead, we’d love to have a conversation directly with you: about the content set you actually run, the change you are most nervous about, the migration you are weighing. That is the kind of thing our sales engineers can dig into with you. Start that conversation here.

Rather poke at it yourself first? Go deep on what ELM does.

In 2001, Wolfgang Resch was studying HIV drug resistance when a biological question pulled him toward computation.

Researchers could measure a viral isolate’s clinically relevant characteristics through laboratory assays. But genomic sequencing was becoming faster and less expensive, and a growing collection of viral sequences had already been linked to experimentally determined results. Wolfgang and a colleague saw an opportunity: Could they train an algorithm to predict a virus’s phenotype from its sequence?

The resulting neural network became the subject of Wolfgang’s first computational biology paper. It also foreshadowed the direction his career would take, from experimental biochemistry and virology to bioinformatics, high-performance computing, and eventually Fuzzball at CIQ.

It was not a decision to leave science behind for technology. The science itself kept leading him toward increasingly complex computational problems.

“My career journey, in one way or another, has revolved around growing scientific knowledge,” Wolfgang says. “First by doing research myself, then helping others to accomplish their research goals.”

Following the science into computation

Wolfgang earned his undergraduate equivalent degree in biochemistry from the University of Tübingen in Germany before completing his doctorate in biochemistry at the University of North Carolina at Chapel Hill. His doctoral research focused on HIV-1 drug resistance, evolution, and fitness, with most of his time spent in the laboratory.

The neural network project began as something of a side investigation. Wolfgang had a problem that called for computational analysis, a colleague interested in similar questions, and a doctoral adviser who gave them the freedom to pursue it.

“Curiosity and an excellent work environment,” Wolfgang says, summarizing the combination that made the project possible.

After graduate school, he conducted postdoctoral research on pox viruses at the National Institute of Allergy and Infectious Diseases, part of the National Institutes of Health. His work remained primarily laboratory based, although computational analysis continued to play a role.

He then moved fully into bioinformatics, first at the National Center for Biotechnology Information and later at the National Institute of Arthritis and Musculoskeletal and Skin Diseases. The timing was significant. High throughput sequencing was generating scientific data at unprecedented rates, and the datasets Wolfgang analyzed required increasingly large computational resources.

As the data grew, his work moved closer to the infrastructure behind the research.

From conducting research to enabling it

Wolfgang eventually joined the NIH High Performance Computing facility as a support scientist. Over the next decade, he helped researchers select and run scientific applications, adapted open source tools for the NIH cluster, taught classes and workshops, and worked across storage, scheduling, system administration, and cloud resources.

The role gave him a view of research computing from both sides. He understood the questions scientists wanted to answer, but he also understood the systems, operational constraints, and specialized knowledge required to answer them at scale.

“HPC systems are large, complex beasts that require specific technical knowledge to avoid many pitfalls,” Wolfgang says. “Specific technical knowledge that researchers should not have to worry about, in an ideal world.”

Researchers were not only confronting the complexity of the cluster itself. They also had to manage growing volumes of data, preserve reproducibility, and accommodate emerging ways of working that traditional HPC systems had not necessarily been designed to support.

Interactive computing, persistent and shareable services, and containerized applications could be difficult to add to environments centered on conventional batch jobs. At the same time, administrators could not predict every type of research that would eventually arrive on their systems.

“As designers and administrators of HPC systems, we can never anticipate all the needs researchers will have,” Wolfgang states. “Systems need to be flexible enough to accommodate or adapt to unanticipated usage patterns, and administrators need to put the mission first and help the systems adapt whenever possible.”

That experience left Wolfgang with two principles that can be difficult to reconcile: research computing should be accessible to people with less infrastructure expertise, and it must give administrators enough control to protect a complex shared environment.

It also reinforced the value of having scientists, administrators, and engineers involved in system design. Each sees constraints and opportunities the others might miss. Wolfgang carries all three perspectives into his current role.

Bringing a researcher’s perspective to Fuzzball

As a Research Computing Engineer at CIQ, Wolfgang helps customers and partners deploy Fuzzball, gathers feedback about how it could better serve them, adds applications, and builds integrations with external tools. He also draws on his own experience as both a user and administrator of HPC systems to anticipate what researchers may need.

“There is a fine line,” he says. “Feedback from external users is very important, and we should avoid planning without enough feedback.”

Fuzzball addresses many of the challenges Wolfgang encountered at NIH. It combines familiar HPC concepts with useful ideas from container-based environments, allowing teams to define portable workflows that include their jobs, services, resources, code, and containers. Those workflows can run across local infrastructure, traditional schedulers, and cloud resources.

For researchers, that can make analyses easier to reproduce, interactive applications easier to deploy, and workflows easier to relocate when local hardware cannot accommodate a project. It can also reduce how much infrastructure knowledge a researcher needs before putting advanced computing resources to work.

Fuzzball 4.2 carries that goal forward by allowing AI agents to interact with the platform within permissions set by the organization.

“Allowing AI agents to interact safely with the Fuzzball platform will lower the barrier to entry for new users and significantly speed up the development of new Fuzzball workflows,” Wolfgang explains.

He sees opportunities to incorporate Fuzzball into existing agentic workflows and help administrators investigate cluster issues. The release also allows workflows to submit and manage other workflows, enabling more complex applications and simplifying integrations with workflow engines such as Nextflow.

CIQ’s work with Arcee AI shows how these capabilities can support emerging AI and scientific workloads. Arcee develops the open weight Trinity model family and is working to advance open weight models for science. The company uses Fuzzball to coordinate selected workloads across dispersed computing resources while retaining its existing training and inference stack.

For Wolfgang, the relationship is exciting both for the scientific work it could enable and for the feedback it brings to Fuzzball.

“Arcee has developed the open weight Trinity models and will push forward open weight science models, which will be crucial for accelerating scientific discovery,” he says. “It is exciting to see Fuzzball involved in this endeavor.”

Feedback from Arcee’s team has also helped CIQ develop a new way to incorporate dispersed and otherwise disconnected resources into Fuzzball clusters. It is the kind of exchange Wolfgang values: users bring real requirements, and those requirements help the platform adapt to work its designers could not fully anticipate on their own.

Keeping the mission first

While the tools have changed dramatically since Wolfgang trained a neural network to study HIV phenotypes in 2001, the underlying pattern has not.

A scientific question led him to computation. Growing datasets led him to bioinformatics and HPC. Supporting NIH researchers showed him how often infrastructure complexity stands between people and the work they want to accomplish. Now, at CIQ, he helps develop a platform intended to move more of that complexity away from the researcher.

His move into research computing has not ended his connection to biomedical science. In 2025 and 2026, Wolfgang had the opportunity to contribute to publications with a former postdoctoral adviser and another researcher studying components of the poxvirus entry fusion complex.

But he does not see infrastructure work as separate from scientific contribution.

“Supporting research computing does contribute to scientific research directly,” he asserts. “That is good enough for me.”

For Wolfgang, moving from the laboratory to the computing systems behind modern research has only expanded the ways he can contribute. With Fuzzball, he is helping build an environment in which scientists can spend less time mastering infrastructure and more time pursuing the questions that brought them to it.

Guest post by Zach Deane-Mayer @ AI Insight Solutions

The frontier labs sell turnkey AI, priced by the token. The lab sets that price, picks the deprecation schedule, and can revoke your access at any time. If your business depends on AI, you want to own the stack your AI runs on. That’s sovereign AI.

Building a sovereign AI stack yourself is daunting: you need GPUs, drivers, system software, an inference engine, and a frontend. Even the strongest engineering teams can struggle to deliver a stack this complicated.

However, you don’t have to go it alone. RLC Pro AI (Rocky Linux from CIQ) now ships as a batteries-included container image: the drivers, the system dependencies, and the software for an inference backend. A neo cloud (a serverless GPU cloud) ships the GPUs. You grab the pre-baked image and run it wherever you choose. There is no stack to assemble and nothing to install.

Combine them and you have a sovereign AI stack in three steps:

- Pick a model.

- Bake the weights into an image built from the RLC Pro AI base image.

- Deploy the image to a neo cloud of your choice: this is your inference backend.

For a model, I typically start with the Gemma family: it’s small enough to run on a single GPU and smart enough to provide real value in production. Ollama serves the weights from inside the image, so inference is fully offline, and you’re off to the races with an OpenAI-compatible endpoint (e.g., you could connect it directly to your coding agents).

The whole backend for this setup is about 40 lines of code, and the GPU scales to zero when idle. If you want a chat interface, deploy Open WebUI as a second container and point it at your backend container:

This stack runs on rented hardware. The neo cloud supplies compute; everything else is yours:

- You own the model: the weights live in an image you built.

- You own the runtime: no vendor other than the neo cloud sits in the request path.

- You own the data: chat history lives in a data store you run.

- You are not locked in: move your containers to a different cloud whenever you want.

If you want to try this yourself, you can follow the step-by-step tutorial to deploy the backend or read the detailed design document. Sovereign inference used to be a whole build process. Now it's a simple pull and deploy.

On CIQ Enterprise Linux Manager (ELM), you resolve entitlement once, at your mirror. Your subscription authenticates against CIQ Depot at sync time, and every host afterward installs from that mirror. That is true at 200 hosts and at 20,000, because entitlement resolves upstream of your fleet.

Every content view you build inherits entitlement from one Depot sync, and so does every host that installs from the view.

Give entitlement one place to resolve

Depot, the CIQ service your RLC Pro subscription authenticates against, governs which repositories you can mirror. You point ELM at Depot with your credentials, sync those repositories onto your own mirror, and curate content views out of them. Your host groups install from a published view.

Enterprise Linux entitlement covers the Depot share of that mirror. Your own internal repositories and open repositories such as Extra Packages for Enterprise Linux (EPEL) sync on the same schedule, and a content view draws from all three sources as one catalog. Mirroring accepts any RPM-based distribution, while curation, dependency resolution, and errata target Enterprise Linux content only.

What does a host check before it installs a package from a content view? Its repository configuration. That configuration reaches the host through the optional lightweight ELM agent or through a repository file you hand out with Ansible. Either way, the host resolves the content view version its host group points at, then installs through standard package management. See what ELM covers, from mirror through rollback.

Take the subscription-enforcement service out of the runbook

An entitlement model that reaches the host creates operational work, and that work scales with the fleet. Your operations team stands the service up, keeps it reachable from every segment that installs packages, registers each host against it, and reconciles that record as machines join, move between groups, and retire. Teams run this model successfully at every fleet size. That work lands in the runbook, and it stays there for the life of the fleet.

ELM asks for a mirror, the storage under it, a sync schedule, and the curation work that moves each view into production. None of that scales with host count. Point another 5,000 machines at a content view, and you change the host count while the entitlement path stays where it was.

What happens to a rollout if Depot is unreachable? It finishes. The content the rollout needs already sits on your mirror, and the ELM control plane makes no license callback and sends no telemetry. An interruption at Depot delays your next mirror refresh, and the promotion you scheduled for Friday still runs.

ELM governs its users on a separate track, with its own accounts and role-based access control, wired to a directory over LDAP or SAML. You decide who can curate a view, who can promote one into production, and who can only read what shipped.

Entitlement decides which content your subscription covers. Role-based access control decides who on your team moves it.

See entitlement resolve at the mirror Request a demo: ELM syncs entitled content, curates a view, and promotes it into production

Keep entitlement out of the install path

Four decisions determine what a host runs, and each resolves in a different place.

The first two decisions are separate, but an enforcement model binds them together by design, which is why it needs a service in the middle. Keep them apart, and your rollout path runs on repository configuration alone.

Disconnected segments resolve entitlement before the content ever leaves a connected system. Lifecycle control that runs on-premises covers that boundary in full.

Evaluating ELM for your own fleet? Request a demo of a content view curated, promoted, and rolled back

When you migrate to RLC Pro, your lifecycle workflow moves onto CIQ Enterprise Linux Manager (ELM). ELM gives migrated hosts a local mirror, curated content views, rollback to earlier versions, and network provisioning for the machines you build next.

With ELM, the tooling side of your migration is predictable: your admins keep the content-view model they already use, one control plane covers every step from mirror to new machine, and the same setup serves every source distribution.

Moving the operating system and moving the lifecycle workflow are separate workstreams, and ELM gives the second one a destination before the first one starts.

Plan the tooling as its own workstream, next to the OS conversion

For RHEL, Oracle Linux, and CentOS-family sources, CIQ's migrate2rlc script converts each host to RLC Pro within the same major version. That conversion changes what runs on the host. Three questions sit outside it: where your hosts pull packages from afterward, which content each group of hosts may install, and how your team builds a replacement machine.

Those three questions belong to the lifecycle layer, and ELM answers each of them for an RLC Pro fleet. Put that layer in your migration plan from the first week.

Rebuild the lifecycle layer on your infrastructure in three steps

ELM runs in containers in your environment. Standing it up follows the same order the content travels: mirror it, curate it, then deliver it to hosts.

Step 1: Mirror your content into one local store. ELM pulls RLC Pro content from CIQ Depot with your Portal credentials, and adds EPEL, your custom repositories, an rsync mirror, or an uploaded ISO alongside it. Each sync verifies packages against the repository metadata and stores every file once.

Step 2: Curate content views and decide what reaches production. A content view pins or excludes packages and versions, accepts your custom RPMs, and records an immutable version every time it composes. Because ELM hard-links packages instead of copying them, a view composes in seconds. ELM checks dependencies when a view composes and flags conflicts as warnings before rollout, and rollback re-points a view to a version it already published. To promote, you compose a production view from a development view your team has already approved.

A pinned content view keeps production exactly where you left it until your team promotes the next version.

Step 3: Deliver content to hosts and build new machines. Hosts register through a lightweight agent and join nested host groups. Each group receives the content views assigned to it, and every host consumes its view as a standard dnf repository. New machines install over the network through iPXE, DHCP, and TFTP, with templated kickstarts, an encrypted credentials vault, and Secure Boot support.

Keep the workflow your admins already run

If your admins work in content views today, they keep that vocabulary on ELM: content views, immutable versions, pins, promotion, and rollback. The migration goes into rebuilding definitions on a model your team already knows.

Four things to know about how ELM runs:

- Access control connects to your directory. Role-based access control works with local accounts, LDAP, or SAML, and ELM keeps an audit log.

- Lifecycle operations are scriptable. ELM exposes a REST API for your scripts, and it launches Ascender automation against the hosts you select.

- Entitlement resolves at the mirror. Your RLC Pro subscription authenticates against CIQ Depot when ELM syncs, and hosts install from the local mirror. ELM runs no subscription-enforcement service, sends no telemetry, and makes no license callback.

- Disconnected sites get the same workflow. ELM runs air-gapped when your environment requires it, with content brought in from a local ISO image.

Your team recreates repositories, content views, and host groups in ELM, content doesn't migrate on its own, and both platforms can run side by side until the last host group moves. The four-phase plan for moving a content-view workflow to ELM shows how to sequence that overlap.

Get the same tooling answer from any starting distribution

Whichever distribution you start from, your fleet lands on RLC Pro, and ELM manages content and provisioning for the hosts that arrive.

RHEL, Oracle Linux, and CentOS-family environments share Enterprise Linux binary compatibility with RLC Pro, so many applications carry over with little to no change and migrate2rlc handles the host conversion. SUSE and Amazon Linux 2 come from a different lineage, so the move involves more than a host conversion, and CIQ's migration engineers scope that work with you. Find your starting point for a move to RLC Pro walks through each path.

Start your Enterprise Linux Manager migration with one host group

ELM is available with all RLC Pro subscriptions at no additional cost.

Start small. Stand ELM up next to your current tooling, mirror your content, and build the content views for a single non-production stage. Then move one host group, run a full cycle of compose, promote, and rollback, and expand group by group on your calendar.

Ready to scope your migration? Plan it with CIQ's migration team.

This week, Sylabs announced that they will end-of-life SingularityPRO, Singularity Enterprise, and SingularityCE on October 25, 2026, coinciding with the end of their engineering in the SingularityCE project and Singularity ecosystem.

We at CIQ want to recognize and thank Sylabs for concluding their work on SingularityPRO and Singularity Enterprise by releasing as open source functionality previously reserved for their commercial offerings. This is the best way to ensure that these features live on, and I wouldn't be surprised to see them picked up and maintained by the Apptainer community.

Singularity began, as many things have, as one of Gregory Kurtzer's projects at Lawrence Berkeley National Laboratory. At the time (the long, long ago year of 2015), containerization was the hot new thing, and everyone wanted to run Docker on their HPC cluster. But Docker's security model was built for hosting system services, not for multiple users on a shared cluster. Singularity was built for that multi-user use case, allowing anyone to benefit from the growing container ecosystem without needing special privileges. Greg originally founded Sylabs in 2017 to develop Singularity and the growing ecosystem around it.

Singularity was my first experience with containers, too, running an HPC cluster at the University of Colorado Boulder. I had a user who wanted to run an Ubuntu-only application on our Enterprise Linux cluster. I had vaguely heard of this containerization thing before, and I thought it might be a way to solve this researcher's need. I was pleasantly surprised when, only a few hours later, I had a working Ubuntu container with their application built and ready to go as a single SIF image that they could run anywhere in our environment.

Today, containerization has only grown in popularity, underpinning the entire "cloud native" movement, particularly via the Kubernetes ecosystem and the Open Container Initiative, as well as CIQ's own Fuzzball. Containers' pervasive use in multi-tenant environments has made them a frequent target for security research, particularly in this era of AI code analysis. (Interestingly, much of this research has rediscovered the same issue that motivated building Singularity in the first place: that containers aren't really a sufficient privilege boundary.)

In 2021, the Singularity open source project joined the Linux Foundation under its new name, Apptainer. Since then, the Apptainer open source project and community have been led by Dave Dykstra ("Dr. Dave") from Fermi National Accelerator Laboratory, with support from a number of committed contributors in the community (including CIQ). Meanwhile, Sylabs continued their own development of Singularity as SingularityCE, along with their commercial offerings.

While Sylabs has announced the end of their suite of Singularity products, the Apptainer community and project continues on from the original open source foundation. Current Singularity users should feel right at home: Apptainer still provides the singularity command for backwards-compatibility, and it runs the same SIF images. Apptainer is also available from CIQ, including as part of the Warewulf Pro cluster management offering. And for teams that need more than the runtime itself, we've announced Apptainer Pro, which will add long-term support releases, preflight compatibility checks, execution records and cluster image distribution on top of open source Apptainer. We're taking early customers now, with priority for organizations moving from Singularity, with wider availability coming soon.

The Apptainer project has a long life ahead of it, and I'm thankful to Dr. Dave and the rest of the community for all the work they continue to put into developing and maintaining such an important part of our software ecosystem.

If you're interested in getting involved with, or just learning more about, the Apptainer project, you can join the Slack community or mailing list. Or, if you're already a Singularity or Apptainer user and are looking for the assurances of a commercially supported product, take a look at our Apptainer Pro early customer program.

And to everyone at Sylabs: I hope for nothing short of the very best for you in your next endeavor, and we look forward to the next incredible thing you’ll build.

~ Jonathon Anderson

Running a large language model on your own GPUs should be easy. Without Fuzzball, assembling all the pieces and wiring it all together and making it accessible is complex. Now Fuzzball puts a large language model on your GPUs, served and ready for traffic. It also ships with the serving stack already assembled: the catalog entry, the replica pool, and with an OpenAI-compatible endpoint.

Fuzzball v4.3 makes it easy to self-host AI models on your infrastructure and to connect those models to AI agents running inside the cluster or locally.

Fuzzball v4.3 also continues our support for environments that don't fit the traditional HPC structure, with expanded support for node-local storage in both local and disaggregated environments.

Turn-key models and agents

There's a new vLLM catalog entry that serves a HuggingFace model from a pool of replicas on NVIDIA or AMD GPUs:

fuzzball workflow catalog start vLLM --values Model=hf://openai/gpt-oss-20b

That's all you need to start hosting a model, ready for inferencing. Point any OpenAI-compatible client or agent at it and start incorporating locally-hosted AI into your development or process.

The vLLM catalog entry has a number of parameters to tune the generated workflow for the model you want to host; but there are eleven preset catalog entries that have already had the GPU count, context size, and vLLM arguments worked out and validated per model:

- GPT-OSS 20B and 120B

- Ministral 3 8B and 14B

- Qwen3-Coder 30B and Next 80B

- Nemotron 3 Nano and Super

- Mistral Medium 3.5

- Llama 4 Scout

- Gemma 4 31B

Starting one of those is just its name; the model, the GPU fit, and the serving arguments are already decided:

fuzzball workflow catalog start "Nemotron 3 Nano"

Invoked bare like that, the CLI walks you through the entry's settings with the preset's value filled in for each, so pressing Enter through the list gets you the validated configuration. Pass --values instead and it skips the prompts, with anything you don't name left at its preset:

fuzzball workflow catalog start "Nemotron 3 Nano" --values Volume=my-models,MaxReplicas=8

That keeps the downloaded checkpoint on a persistent volume rather than pulling it again on every start, and lets the pool grow to eight replicas under load.

Each model is packed with a built-in LiteLLM Model Gateway that puts a single OpenAI-compatible endpoint in front of the workflow. This allows the workflow to scale up and down based on demand, keeping a single endpoint for clients and agents. Or disable the workflow's gateway and run a single gateway for all your models: this lets you have a single endpoint that can discover every model-serving workflow you have access to.

Fuzzball also comes with two coding agents in the catalog, OpenCode and hermes-agent, which run in the cluster and can detect your model gateways automatically.

Finally, a rag catalog entry runs a document retrieval service built on haiku.rag. It can ingest PDF, DOCX, PPTX, HTML or plain text documents and expose hybrid vector and full-text search exposed as MCP tools on a workflow endpoint.

Autoscaled pools and pool endpoints

To support the catalog work above, service endpoints now work against an autoscaled pool, not just a single-instance service.

A pool's endpoint keeps one URL for the life of the workflow and forwards each request to whichever replica is ready at that moment. Scale up and new replicas join the rotation as they pass readiness; scale down and one leaves when its drain period starts, after in-flight requests finish. Callers never learn how many replicas exist.

Pools can also idle at zero. A pool with replicas.min: 0 keeps its endpoint while nothing is running, and a request that arrives gets a 503 with Retry-After: 120 and starts a replica. Retry after the cold start and you're served. The request has to be authenticated, so a scope: public endpoint will never wake its pool, which is deliberate, because otherwise anyone on the internet could turn your GPUs on. A burst of requests starts one replica, not one per request, and every wake shows up in fuzzball workflow events.

If you'd rather do your own load balancing, per-replica: true gives every live replica its own endpoint alongside the pool's.

Endpoints can now carry arbitrary key-value annotations that make endpoints discoverable to clients. (This is how the LiteLLM gateway finds the pools it should front.) And a service can combine multinode with autoscaler.replicas, so each replica is a gang-scheduled group of ranks serving from rank 0.

Expanded support for node-local volumes

Fuzzball can be configured to make node-local storage available as local volumes. In the past, this has only supported ephemeral volumes; but Fuzzball v4.3 adds support for persistent local volumes, and improves handling of ephemeral local volumes as well.

Every volume on a local provisioner (declaring local: true, only supported with the hostpath provisioner today) has exactly one holding node, which is recorded during fuzzball volume create, or specified with fuzzball volume create --node. Fuzzball places every stage that mounts the volume on the volume's holding node.

An ephemeral local volume is created at the start of the workflow, and later workflow stages that mount the volume are placed on the volume's holding node as well.

Placement is capacity-aware: nodes without room for the requested size are skipped, and targeting a node by hand that has insufficient space fails.

Local volumes do not currently support concurrent multi-node workflow stages. This means multinode (e.g., MPI) jobs, concurrent task arrays, and multi-node scale-up services need a shared volume to work today.

That last case is worth noticing if the pool endpoints above just caught your eye: autoscaled serving and node-local volumes are mutually exclusive by construction. If you point the vLLM entry's Volume at a persistent volume, it wants a shared provisioner -- the entry's MaxReplicas defaults to 4.

Before you upgrade

Three changes need attention:

- Catalog repository URIs are now restricted to HTTPS and publicly routable hosts by default. Repositories that don't qualify will stop syncing and need re-adding over HTTPS. Operator-managed deployments can relax this with spec.fuzzball.workflowCatalog.restrictions.allowPrivateAddresses.

- Catalog entry IDs change on the first sync after upgrade, because the ID now derives from the catalog source as well as the entry's own id. Saved links and scripts referring to entries by UUID need updating.

- Ingress routes on the kong and nginx classes now require TLS and redirect plaintext callers with a 308. Other classes are unchanged.

v4.3.0 also includes all fixes already released through v4.2.3.

Full release notes, including the complete fix list and the security fixes in this release, are at docs.ciq.com/fuzzball.

Updating Linux is a solved problem. Right up until you have a hundred servers.

Update everything, all the time

If you run Linux on your own machine, updates are easy. Whether it's apt in the Debian family or pacman on Arch, the loop is the same. Run the update and get what's new, often within days of it shipping upstream.

It's the same model your laptop runs on. The vendor pushes, you get it. For one person on one machine, it's great. If something breaks, you're the only one who notices... and you're the one who fixes it.

But fast has a cost. In July 2025, three packages in the Arch User Repository, the community-run corner of Arch, posed as patched browser builds and installed a remote access trojan. The community caught them and they were gone in about two days. Two days is a quick response for a hobbyist desktop. It's a very long time if those packages landed on production servers.

That's the trade enterprises make when they pick Enterprise Linux. Less new, much less risk.

The legacy Enterprise Linux playbook

Legacy Enterprise Linux vendors have traditionally gone the other way. They update less often, on purpose. Someone at the vendor spends a lot of time deciding which RPM package versions go into each release, and which ones sit out because they're unproven or might not play nice with the rest of the system.

That curation is a big part of what the subscription pays for, and the vendor's repository becomes the pillar everything gets built around. Fair enough. Stability is the whole point, and it's the same reason Rocky Linux from CIQ exists.

But the way the legacy vendors deliver that stability hasn't changed much in a long time, and at fleet scale it starts to creak.

Where it starts to creak

- The vendor curates for everyone, which means nobody curated for you. Their testing, their priorities, their calendar. Who do you actually want deciding what lands on your production servers?

- Every host goes and asks for itself. A hundred servers means a hundred machines checking the same off-site list, over and over. A caching proxy cuts the traffic, but it doesn't decide anything.

- Staging is on you. Want dev to update first, get tested, and then let production follow on its own schedule? That's manual work, and you redo it every cycle.

- Your own packages become orphans. Most shops have a few custom RPMs. They live outside the curated repository, so someone babysits their installs and updates by hand.

A layer between the repository and the fleet

That's why we built CIQ Enterprise Linux Manager (ELM). It sits between where packages come from and the hosts that install them, and it gives the decision back to your team.

It starts with a source of truth you own. ELM mirrors upstream repositories onto your storage, and hosts pull from you and not constantly over the internet. That runs on infrastructure you already operate, connected or not.

Then you decide what each host installs. A content view is a named, versioned snapshot of exactly what a group of hosts should get. One view can combine upstream Enterprise Linux content, RLC Pro content from CIQ, and your own custom RPMs. The internal packages get managed like everything else instead of on the side. And nothing gets built until you hit compose.

Then you split the fleet. Dev tracks the latest snapshot. Production is pinned to one you've tested and doesn't move until someone moves it. Promoting is re-pointing production at a newer snapshot. Rolling back is re-pointing it at an older one. Every compose is kept, so you have a history of what ran where. (For more info, I wrote about that in The CVE didn't bring the fleet down.)

Say you need to pull a package out of the fleet. You exclude it from the dev content view and recompose. Your dev hosts stop seeing it. Your production hosts still have it, because production is pinned to the snapshot you already tested. Same fleet, two different answers, and both are the ones you chose. Once dev checks out, you promote, and production follows.

None of that asks you to give up what you picked Enterprise Linux for in the first place. You still get a stable, curated core. What changes is who decides when it reaches each host. The mirror lives on your storage, your custom packages sit right alongside everyone else's, and production moves when your team moves it.

And it's not another tool to buy and budget for. CIQ provides ELM at no additional charge to all RLC Pro subscriptions.

Production should change when you decide

It should be dictated when a package changes and also not on the vendor's calendar.

For the bigger picture, read Own your Enterprise Linux lifecycle, or head to the CIQ Enterprise Linux Manager page for the details.

At first glance, neuroscience, Linux containers, solutions architecture, and technical product writing might look like separate careers. For David Godlove, they are different expressions of the same goal: helping people solve important problems.

“The constant thread that runs through all of my career is the desire to make an impact,” David states. “I want my work to have a positive impact on the world.”

“That impact was clear as a neuroscientist,” he explains. “I was working as an individual contributor to advance human knowledge. When I became a staff scientist, I started enabling other researchers to advance knowledge at a faster pace by removing roadblocks preventing them from using HPC. And now I’m doing much the same by helping to develop and document a product that is making HPC the easiest and most powerful it has ever been.”

That product is Fuzzball. After nearly five years at CIQ, David now serves as its technical product writer and as a member of CIQ’s leadership team, a role that draws on every stage of his career, from using HPC for science to helping build and explain the tools behind it.

From conducting research to enabling it

David trained as a neuroscientist at Vanderbilt University and completed his postdoctoral work at the National Institutes of Health. As a researcher, he studied how brain circuits interact to produce experience, developing real-time experimental systems and new ways to analyze large datasets.

He also saw the practical constraints researchers face. Deadlines, grants, and funding pressures often require scientists “to figure things out quickly and move to the next step. So it is very difficult to convince a scientist to go back and refactor some code or rerun an analysis.”

“They don’t have time to make their code more robust or to become expert programmers or understand computer hardware,” David continues. “Fuzzball aims to solve that problem. Researchers can focus on their area of expertise and forget about becoming computer scientists too.”

David began addressing that challenge directly when he joined the team supporting Biowulf, the NIH supercomputer, as a staff scientist. Rather than conducting one line of research himself, he helped many scientists overcome the technical obstacles that stood between them and their discoveries.

That experience gave him an intimate view of HPC’s complexity. As David puts it, “There is an old joke that every supercomputer is serial number 1, and it’s funny because it’s true. Almost every cluster is unique at the hardware level. And then more so when it comes to software.”

At NIH, custom Bash and Perl scripts, cron jobs, hardware heterogeneity, and years of institutional knowledge all helped keep the cluster running. They also created details that administrators had to support and researchers had to understand. The experience showed David how much scientific progress can depend on making infrastructure easier to operate, access, and reproduce.

Learning Linux by breaking things

David’s path into container technology began with a support ticket. A group of NIH neuroscientists asked for Singularity, the container platform founded by CIQ CEO Gregory Kurtzer, to be installed on Biowulf. Because David regularly worked with those researchers, he took the request.

Singularity was still a young project with rough edges. David began experimenting with it, then fixing bugs and adding features. He was drawn to containers because they gave him safe, isolated environments in which to test ideas, make mistakes, and learn more about Linux without damaging the underlying system.

That initial ticket led David from the Singularity open source community into professional container development. He worked on the platform as a software engineer and later as a release manager and community advocate, collaborating with engineers while engaging directly with users to debug problems and guide new features. Singularity eventually became Apptainer, and the lessons David learned from its development continue to inform his work at CIQ.

“I’m still breaking things in my current role,” he says. “I test new features and try different things in Fuzzball.”

Bringing the user’s perspective into the product

David joined CIQ as a solutions architect and later became technical product writer for Fuzzball. The title might suggest that his work begins after a feature is finished, but documentation is only one part of the role.

David follows new engineering developments closely and tests features as they emerge, sometimes uncovering bugs in the process. He gives demonstrations, speaks with customers and prospective customers, and brings their feedback to the engineering team as it sets development priorities. His experience as a scientist, HPC user, software developer, educator, and solutions architect helps him understand a capability at both a technical level and a practical, user-facing one.

“Not only have I worked with both users and engineers, but I’ve actually been in both of those roles myself,” he says. “It is really important to keep the big picture in mind when you are trying to understand and explain technical bits.”

That perspective has been particularly useful as Fuzzball has evolved. The platform was designed from the beginning to run both on premises and in the cloud, but its early capabilities reflected the cloud computing experience of its engineering team. Over time, Fuzzball added capabilities geared more directly toward on premises environments, including easier volume management and intelligent job scheduling. It has also expanded to support a growing range of AI model training and inference workloads.

Making private AI inference easier with Fuzzball 4.3

Fuzzball 4.3 marks the next stage of that evolution. Its most significant addition is a turnkey workflow template that lets users launch large language models for inference with only a few clicks.

Users can select from nearly a dozen preconfigured LLMs in the workflow catalog, deploy the model on an organization’s HPC cluster or in a cloud environment, and interact with it through a browser or command line client. Behind the interface, Fuzzball can scale model services up or down according to demand and provide a durable endpoint through which users can access them.

Before Fuzzball 4.3, users who wanted to run an LLM that was not already in the workflow catalog needed to create a Fuzzfile, determine the model’s compute requirements, sometimes through trial and error, and work out how to reach an autoscaled pool of nodes without a single endpoint. Although Fuzzball’s workflow editor made creating a Fuzzfile relatively approachable, the process still added time and required infrastructure knowledge.

Fuzzball 4.3 removes much of that work. Users can move from choosing a model to serving it without first becoming experts in resource requirements, autoscaling, or service connectivity.

For David, the value goes beyond convenience. Organizations increasingly want the benefits of generative AI without sending proprietary code or other sensitive information to an external model provider. Fuzzball 4.3 enables them to host models within infrastructure they control while still giving users a straightforward way to interact with those models.

“This is a requirement that we have heard from lots of organizations,” David says. “v4.3 makes it easier than ever to use AI while maintaining data sovereignty.”

Making complexity approachable

David has spent much of his career teaching people how to use HPC and container technologies. That educator’s mindset influences both how he explains Fuzzball and how he thinks its user experience should work.

When he teaches, David begins with broad concepts before moving into greater technical depth. Fuzzball’s interface follows a similar progression. A new user can start with a template from the workflow catalog and run it with a few clicks. Once the user understands the template, they can open it in the graphical workflow editor for more granular control. When they are ready to go deeper, they can view and edit the underlying Fuzzfile in YAML.

This layered approach gives newcomers an accessible starting point without limiting experienced users. It also reflects the principle that has connected every stage of David’s career: technology should remove roadblocks for people doing consequential work.

“Researchers need to expect easy access, ease of use, reproducibility, and portability from their HPC infrastructure,” David says. “These are the design goals of Fuzzball.” He emphasizes: “Domain experts need to be able to focus on their science. They already have a full-time job. They should not be forced to start a second career as computer scientists.”

Whether he is helping a scientist use a supercomputer, testing a new Fuzzball capability, or explaining an unfamiliar workflow, David’s aim remains the same: make sophisticated technology more accessible so that other people can use it to make an impact of their own.

Recognition highlights YMX’s leadership in sustainable fleet operations and in deploying EV yard trucks to support decarbonized yard logistics across the U.S.

HENDERSON, NV — May 21, 2026 — YMX Logistics, a leading national third-party logistics provider specializing in integrated yard logistics for enterprise shippers, has received the 2026 Green Fleet Award from Heavy Duty Trucking. The annual recognition honors fleet operators demonstrating measurable progress, operational excellence, and leadership in sustainable transportation.

YMX Logistics deploys high-capacity electric yard trucks in demanding 24/7 environments across major logistics hubs and manufacturing facilities nationwide. At the center of YMX Logistics’ sustainability strategy is its autonomous Yard Operating System (YOS), which connects physical operations with digital intelligence to improve throughput, reduce bottlenecks and dwell time, optimize asset utilization, lower operating costs, and support emissions reduction initiatives across complex yard environments.

The recognition follows YMX Logistics' receipt of the 2026 Leading Off-Road Fleet Award at ACT Expo earlier this month, reinforcing the company’s continued investment in fleet electrification and yard optimization. Over the last twelve months, YMX Logistics doubled its zero-emission yard truck footprint across customer operations nationwide.

“We see many companies investing in electric yard trucks without fully realizing the operational value they expected,” says Matt Yearling. “You can easily add zero-emission vehicles to your yard operations, but that alone does not guarantee operational outcomes. One of the main reasons YMX operates one of the most successful EV yard truck programs in the country is its Yard Operating System (YOS), which integrates electrification into a broader operational framework that continuously optimizes execution, asset utilization, and operational flow to maximize performance, improve uptime, and successfully scale EV operations across complex enterprise environments.”

“Electric yard operations introduce an entirely different level of operational complexity,” says Erin Mitchell, COO of YMX Logistics. “Success depends on how well organizations coordinate execution across people, equipment, infrastructure, and workflows. Our focus has been building an operational model that allows customers to scale sustainability initiatives without sacrificing reliability, throughput, or service performance.”

For the full list of winners, visit https://www.truckinginfo.com/articles/top-green-fleets-2026-how-fleets-are-reducing-emissions-in-the-real-world