GPU cluster operations and embedded engineers
Engineers on the cluster you already run
The cluster is delivered, the researchers are on it, and the team that runs it is three people. OPSaaS puts engineers on your existing GPU, AI or HPC cluster: embedded in your team, or running day-2 operations and on-call for you. The hardware, the datacenter and the contracts stay yours.
Our engineers are based in France and work across the whole stack, from BMC and firmware to the fabric, the parallel filesystem, Slurm or Kubernetes, and the serving layer. That matters because a GPU cluster rarely fails where the symptom shows: a slow training job is often an optic, a firmware mismatch or a filesystem under checkpoint load. See how we diagnose a fault across layers.
Need a new cluster rather than hands on an existing one? See turnkey GPU clusters.
When teams call us
- GPUs sit idle while the queue is full. Job fragmentation, wrong partitions, no topology-aware scheduling, quotas nobody trusts. GPU utilization is the first number we look at.
- Jobs slow down or fail with no error. NCCL all-reduce bandwidth below what the fabric should deliver, Xid errors, thermal throttling, a dataloader starved by storage.
- Nodes stay out of service for weeks. Nobody owns the path from fault detection to triage, RMA, post-repair validation and return to service.
- The upgrade backlog grows. NVIDIA drivers, CUDA, firmware and BMC baselines, the OS, Slurm, Kubernetes: every upgrade is postponed because the last one broke something.
- On-call rests on one or two people. No runbooks, no post-incident reviews, and monitoring that catches faults after the users do.
- New racks are arriving. A new GPU generation next to the old one, to bring up, burn in and fold into the same scheduler without stopping production.
- Hiring takes months. HPC engineers, GPU infrastructure engineers and AI infrastructure SREs are among the hardest profiles to recruit, and the cluster cannot wait.
Two ways to work with us
| Embedded engineers | Managed operations and on-call | |
|---|---|---|
| What it is | OPSaaS engineers join your team and work from your backlog | OPSaaS runs a defined scope of the cluster for you |
| Basis | Time and materials, from a few days a week to full time | Monthly fee for an agreed scope and coverage |
| Who steers | Your team lead | OPSaaS, against objectives agreed with you |
| Hours | EU hours, daily overlap into US hours | EU and US hours, 24/7 on-call on demand |
| Good fit when | You have a team and need senior hands or a missing skill | You want users served without building an operations team |
Embedded engineers
Senior engineers who work as part of your infrastructure or platform team: your tickets, your stand-ups, your incident channel, your repositories. Typical work: a Slurm or Kubernetes migration, an InfiniBand or RoCEv2 fabric to retune, a storage tier to add, a monitoring stack to rebuild, an expansion to bring up, or simply the daily run next to your engineers. You get the skill you are missing now, and your team learns it along the way. Short reinforcements of a few weeks are possible too.
Managed operations and on-call
We take responsibility for the day-2 operations of a scope agreed with you: the scheduler, the fabric, storage, the whole platform, or a single layer. Monitoring and alerting, incident response, node lifecycle and RMA follow-up with your vendors, upgrades, capacity and utilization reporting, user support for your researchers. Handover happens in your channel, not in a ticket queue.
Both can be combined, and both can start small. Many clients start with one embedded engineer and add on-call later.
What we work on, layer by layer
| Layer | Day-2 work | Tools |
|---|---|---|
| Hardware and firmware | Node health, firmware and BMC baselines, Xid triage, RMA, burn-in and post-repair validation | NVIDIA HGX and DGX, GB200 and GB300 NVL72, BMC, IPMI, Redfish, DCGM, nvidia-smi |
| Fabric | Link and optics health, subnet manager, RDMA tuning, congestion and PFC, rail-optimized topology checks | InfiniBand, NVIDIA UFM, RoCEv2, Spectrum-X, NCCL, GPUDirect RDMA |
| Storage | Throughput under checkpoint and dataloader load, quotas, tiering, upgrades | Lustre, GPFS (IBM Storage Scale), BeeGFS, WEKA, VAST, DDN, Ceph |
| Scheduling | Partitions, priorities, fair share, topology-aware and gang scheduling, accounting, user quotas | Slurm, Kubernetes, NVIDIA GPU Operator, Kueue, Volcano, Run:ai, Base Command Manager, Slinky |
| Platform and serving | Drivers, CUDA and container runtimes, user environments, notebooks, inference endpoints | NVIDIA drivers, CUDA, MPI, Apptainer, Spack, Ray, Kubeflow, vLLM, Triton |
| Observability | Dashboards, alerting, SLIs and SLOs, capacity and utilization reports | DCGM exporter, Prometheus, Grafana, Loki, OpenTelemetry |
| Automation | Provisioning, configuration and upgrades as code, GitOps | Ansible, Terraform, Python, Bash, Argo CD |
| Security | Tenant isolation, access, patching, audit evidence | See security |
What we measure
Operations are judged on numbers your users feel. We agree on them at the start and report them every month.
- GPU utilization: allocated versus actually used, per team and per partition.
- Queue wait time: median and p95, per job size.
- Job failure rate caused by infrastructure, as opposed to user code.
- Nodes out of service: how many, for how long, and where each one stands between fault and return to service.
- Time to detect and time to restore for incidents, with a post-incident review for each one that matters.
- Upgrade lag: how far drivers, firmware, the scheduler and Kubernetes are from the versions you target.
How an engagement starts
- A 30-minute call with an engineer. Your cluster, your team, what hurts today.
- Access and read-through. We read the cluster as it is: hardware inventory, fabric and storage health, scheduler configuration, monitoring, the last incidents. Findings are written down and ranked by impact.
- A backlog you agree to. What to fix first, what to automate, what to leave alone.
- Steady work. Embedded in your team or running the agreed scope, with a monthly report on the numbers above.
Handover: what you keep
Runbooks, architecture decision records, configuration baselines, dashboards and the change history all live in your Git repositories and your tools. If you hire the team that replaces us, we train it and hand over. Nothing depends on software or accounts that belong to OPSaaS.
From operations to the next cluster
When the existing cluster is full, we size the expansion from the utilization data we already have, and can deliver it as a turnkey GPU cluster. For clusters in Europe, see also our guide to deploying a GPU cluster in Europe and sovereign AI infrastructure.
Frequently asked questions
Can OPSaaS work on a GPU cluster we did not buy from you?
Yes. Most of the clusters OPSaaS engineers work on were built by someone else: a system vendor, an integrator or the client’s own team. We start by reading the cluster as it is (hardware, fabric, storage, scheduler, monitoring, incident history), then work on it with your team and your vendors.
Do you provide managed Slurm or managed Kubernetes on our hardware?
Yes. OPSaaS operates Slurm and GPU-aware Kubernetes on clusters that stay yours, in your datacenter or your colocation: upgrades, scheduling policy, quotas, accounting, monitoring and incident response. The configuration lives as code in your Git repository.
Do you offer 24/7 on-call for a GPU cluster?
Yes, on demand. OPSaaS engineers work from France in EU hours with daily overlap into US hours. 24/7 on-call is added when the cluster needs it, with escalation paths and runbooks agreed beforehand.
Do your engineers work remotely or on site?
Mostly remotely, from France, inside your tools and your channels. We come on site when the work calls for it, such as an expansion, a bring-up or a migration. Day-to-day hands-on work in the hall is done with your datacenter team or your colocation’s remote hands.
How does an engagement with embedded engineers work?
One or more OPSaaS engineers join your team on a time-and-materials basis, from a few days a week to full time, for a few months or longer. They take tickets and projects from your backlog, join your stand-ups and your incident channel, and document what they do in your repositories.
Can you audit our GPU cluster before we commit?
Yes, on request. A short health check covers GPU and node health, fabric and storage performance against what the hardware should deliver, scheduler configuration and utilization, and the monitoring and upgrade state. You get a written list of findings ranked by impact. Most engagements skip the formal audit: the same read-through happens in the first weeks of the work.
Which hardware and software do you operate?
NVIDIA HGX H100, H200, B200 and B300 systems, DGX, GB200 and GB300 NVL72 racks, and AMD Instinct. InfiniBand and RoCEv2 fabrics with UFM. Lustre, GPFS, BeeGFS, WEKA, VAST and Ceph. Slurm, Kubernetes with the NVIDIA GPU Operator, Run:ai and Base Command Manager. DCGM, Prometheus and Grafana. Ansible and Terraform.
What happens when the contract ends?
Handover is planned from the start. Runbooks, configuration, dashboards and the change history are in your Git repositories and your tools. If you hire the team that replaces us, OPSaaS trains it and hands operations over.
Do you only work on clusters in Europe?
No. Europe is our home market, and OPSaaS engineers also operate clusters for clients in the US and elsewhere, in EU and US time zones.
Need this in production?
Tell us what you are building. An engineer replies.