---
title: "AI2 builds GPU-time budgeting scheduler for its research clusters"
slug: "ai2-builds-gpu-time-budgeting-scheduler-for-its-research-clusters"
date: "2026-10-10"
category: "AI & Automation"
author: "Marcus «Glitch» Reed"
correspondent: "Marcus «Glitch» Reed"
correspondentRole: "Open Source & Hacker Scout"
readingTimeMinutes: 2
tags: ["AI","News"]
originalSourceUrl: "https://huggingface.co/blog/allenai/impactful-scheduling"
canonicalUrl: "https://aurav.ae/ai2-builds-gpu-time-budgeting-scheduler-for-its-research-clusters"
publisher: "Aura-V Intelligence (aurav.ae)"
---

# AI2 builds GPU-time budgeting scheduler for its research clusters

> AI2’s new hierarchical fair-share scheduler allocates GPU time by budget, boosting impact and utilization for its 150 researchers.

## Key Takeaways (TL;DR)

- Replaces priority-based scheduling with a budget-driven hierarchical fair-share model.
- Managers assign proportional GPU-time claims, e.g., 35 % of capacity to a project.
- A 7-day sliding window recovers idle compute, effectively adding ~30 % usable capacity.

On the AI Infrastructure team at the Allen Institute for AI (AI2), the institute provides GPU compute capacity for large, distributed training workloads. The team describes its scheduling goals as a pyramid of four metrics: availability, occupancy, impact, and utilization.

AI2 manages thousands of NVIDIA H100, B200 and B300 GPUs in clusters ranging from 88 to 1,024 GPUs. About 150 internal researchers use the clusters for LLM and VLM training, robotics reinforcement-learning simulation, and post-training scientific agentic use cases.

Submitted workloads regularly exceed supply by a factor of 2–3, meaning each available GPU hour faces 2–3 competing research jobs. The previous priority-based scheduler allowed non-preemptible workloads to monopolize GPUs, leading to “GPU squatting” and priority inflation where all jobs ran at HIGH priority, starving lower-priority work.

To move from ad-hoc priority control to a transparent budgeting process, AI2 introduced a system that assigns each research program a proportional share of GPU time. The hierarchy lets managers allocate claims—Project A1, for example, receives a 35 % claim on total capacity. Every workload must be funded by a budget; otherwise it is not protected from preemption. This makes gaming the scheduler more costly than requesting additional budget.

Paired with the budgeting tool, AI2 built a hierarchical fair-share scheduler that tracks occupancy over a default 7-day sliding window. The algorithm sorts workloads from under-utilized allocations above those from over-utilized allocations, ensuring that each group receives its allocated GPU time if it submits sufficient demand.

> “The new scheduler makes it feel like we have an extra 30 % compute. In the old scheduler, if we had moments when we didn’t need our full slot limit, that compute was basically lost. Now we can later burst beyond our allocation limit and still see our jobs scheduled quickly and without preemption, essentially letting us reclaim that compute.” — Chris Clark

---

*Original Think-Tank Source: [Impactful scheduling for GPU clusters](https://huggingface.co/blog/allenai/impactful-scheduling)*