From DIY Clusters to Managed HPC: How the HPC Migration Agent Is Redefining the Path to AWS PCS

How AI agents discover a self-managed HPC cluster, write the migration blueprint, wait for a human decision, and build the new cluster on AWS PCS.

From DIY clusters to managed HPC: AI migration agents move a self-managed Slurm cluster on AWS ParallelCluster to AWS-managed PCS, with a human approval gate

Ask any research computing team what keeps them up at night, and "migrating the HPC cluster" lands near the top of the list.

Many organizations built their first cloud HPC environments on AWS ParallelCluster. It works, and it gave scientists elastic Slurm clusters years before most on-prem teams could. But a ParallelCluster is still a cluster you run yourself: you own the head node, the Slurm version, the upgrades, the patching and every configuration detail that accumulated along the way.

AWS Parallel Computing Service (AWS PCS) changes that. AWS operates the Slurm controller for you on a current, supported version, and compute node groups scale up when jobs arrive and back to zero when they finish. The destination is clear. The journey has been the problem.

The hard part of an HPC migration isn't building the new cluster. It's knowing exactly what the old one does.

At Clovertex, we built an agentic migration platform to handle that journey: AI agents that discover a running cluster, write the migration blueprint, pause for a human decision, and then build the new AWS PCS cluster automatically.

The problemWhy HPC migrations stall

A production HPC cluster is rarely just its configuration file. Over time it collects:

  • Slurm partitions tuned for different workloads, each with its own instance types, node limits, EFA networking and GPUs
  • A software environment of compilers, MPI libraries, CUDA versions and environment modules that researchers depend on
  • Users, groups and LDAP integration that decide who can run what
  • Shared storage on FSx for Lustre or EFS, mounted in specific places inside a specific VPC
  • Years of Slurm accounting history that says how the cluster is actually used

Migrating by hand means reverse-engineering all of that, translating it into the target's concepts, and hoping nothing was missed. It takes weeks of senior engineering time, and mistakes surface only when a researcher's job fails on the new cluster.

The approachAn agentic migration control tower

The Clovertex HPC Migration platform runs the whole process from a single web console. Behind it sits a pipeline of AI agents orchestrated by AWS Step Functions, each responsible for one stage of the migration.

The migration pipeline: Discovery, Translation, Documentation and Approval, Build, Deploy and Validation, with a human approval gate at stage three and guardrails including pre-flight checks and a full audit trail
Six stages, one human decision: agents handle discovery, translation and the build, and nothing is provisioned until a person approves the blueprint.

Every migration moves through the same six stages: Discovery → Translation → Documentation & Approval → Build → Deploy → Validation. The console tracks each stage live, and if you reload the page mid-run it simply re-attaches to the migration in progress.

Stage 1Discovery: ground truth, read-only

You don't need to know your cluster names in advance. A Scanner agent sweeps the AWS account and finds every ParallelCluster by its tags, then presents each one with its region and node count.

When you pick a cluster, the Discovery agent inventories it from two directions at once:

The AWS APIs

EC2 fleet, networking, storage and IAM, described straight from the account.

The head node itself

Reached through AWS Systems Manager to read the real cluster config, Slurm partitions, installed software and mounted filesystems.

That combined "node plus API" view is what makes the blueprint accurate rather than guessed. The cloud APIs tell you what was provisioned; the head node tells you how the cluster is really configured.

The source cluster is never touched

Discovery is strictly read-only: API describe calls plus a read-only script on the head node. The same approach works for on-premises Slurm clusters, which are reached over SSH using credentials kept in AWS Secrets Manager, so the credential never leaves your account.

The discovery report ends with a readiness verdict. Advisories, such as a source cluster running an end-of-life Slurm version, are flagged for review but don't block the migration, because the target is always built on a current, supported release.

Stage 2Translation: the AI writes the blueprint

Discovery to blueprint mapping: Slurm partitions become compute node groups and queues, the head node becomes a managed Slurm controller and login, software and identity stay the same, storage and network stay in the same VPC, and Slurm accounting drives right-sizing
Jobs aren't migrated, the cluster is: each part of the source cluster is translated into its AWS PCS equivalent.

This is the "agentic" heart of the platform. The discovery report goes to Claude on Amazon Bedrock, which produces the migration blueprint along with the reasoning behind it.

The key insight is that jobs aren't migrated; the cluster is. Once the new cluster matches the old one, researchers simply point their jobs at the new queues. So the blueprint focuses on what makes the cluster work:

  1. Slurm partitions become AWS PCS compute node groups and queues, with instance type, minimum and maximum nodes, scaling behavior and purchase option.
  2. The head node becomes the managed Slurm controller and login arrangement, operated by AWS.
  3. The software and identity stack is preserved (compilers, MPI, CUDA, modules, users and LDAP) so jobs run identically.
  4. Storage and networking carry over: the same VPC, the same EFA networking, the same shared data.
  5. Slurm accounting history drives right-sizing, feeding cost-aware recommendations for instance types and scaling.

Every partition and its proposed equivalent appear side by side in a single table, so the engineer reviewing the migration can see exactly what will be built before anything exists.

Stage 3The human approval gate

Infrastructure is never built without an explicit human decision. When the blueprint is ready, the platform emails the designated approver an architecture document and an approval request, then pauses the pipeline, waiting up to 24 hours.

The approver has three choices:

Approve & build

The pipeline resumes and builds the target cluster.

Revise

The blueprint goes back to Translation for another pass.

Reject

The pipeline stops. Nothing is provisioned.

Each decision is matched against a one-time token that expires after 24 hours, so an old approval link can't be replayed later.

Stages 4–6Build, deploy and validate

Before anything is created, the Migration agent runs seven pre-flight checks: the inputs, AWS PCS availability, the AI model, the email sender, the artifacts bucket, that no other migration is already running, and that the project has budget headroom. Only then does the build begin.

What one run delivers: about 15 minutes for a first migration, about 4 minutes for a re-run, zero changes to the source cluster and one human approval; it creates an AWS PCS cluster, node groups and queues, architecture and validation reports, and a full audit trail
What one run delivers, measured on a real ParallelCluster to AWS PCS migration.

When the pipeline reports success, you have:

  • A real AWS PCS cluster with a managed Slurm controller on a current version, inside your existing VPC
  • Compute node groups and queues matching the blueprint, scaling from zero, so there's no idle compute cost
  • Your shared EFS filesystem mounted automatically at /shared on every node that starts
  • An architecture report and a validation report, saved to the platform's S3 artifacts bucket
  • A full audit trail of every phase's inputs and outputs, plus the complete Step Functions execution history

The old ParallelCluster is still running, untouched. Teams can move their jobs over at their own pace, and decommission the source only when they're confident.

In practiceWhat a migration looks like

About 15 minutes

For a first run, including building the new PCS cluster. Re-running against an existing target after a revision takes about 4 minutes.

A few dollars per run

The AI work costs a few dollars in tokens. On the new cluster, compute nodes only cost money while jobs are running.

Safe to repeat

The backend refuses a second run while one is active, and the source is never modified, so a dry run carries no risk.

No AWS login for reviewers

Stakeholders review and approve from the console or the approval email, without needing access to the AWS account.

What's nextMigration as a repeatable process

HPC migrations used to be one-off projects: expensive, risky and dependent on whoever happened to understand the old cluster. Agentic AI turns them into a repeatable, auditable process. Discovery captures the ground truth, the AI does the translation, a person makes the decision, and the platform does the building and the paperwork.

That frees research computing teams to focus on what they're actually there for: giving scientists faster, more reliable access to compute.

If you're running AWS ParallelCluster or an on-premises Slurm cluster and weighing a move to AWS PCS, we'd be glad to show you a live migration.

#HPC#AWSPCS#ParallelCluster#Slurm#AgenticAI#AmazonBedrock#CloudMigration#ResearchComputing#AWS#Clovertex
Keep Reading

More from the Clovertex blog

Let's Talk

Planning a move to AWS PCS?

See a live ParallelCluster-to-PCS migration, from discovery to a running cluster, in about fifteen minutes.