Storage Spaces Direct Performance Tuning: Best Practices for Production Clusters

📊 S2D Architecture Primer

Storage Spaces Direct (S2D) uses the locally attached NVMe, SSD, and HDD drives in each cluster node to create a shared storage pool. Unlike traditional SAN, there is no shared storage fabric — data resides on the nodes themselves and is replicated across nodes for fault tolerance.

The S2D I/O path for a write operation: VM write → SMB Redirector → RDMA network → destination node → cache tier (NVMe/SSD) → capacity tier (NVMe/SSD/HDD). The key performance levers are the cache tier design, resiliency mode, and RDMA network configuration.

📺 Cache Tier Design

The S2D cache tier absorbs write I/O and serves as a read cache for hot data. In All-NVMe configurations, the fastest drives serve as cache for the slower capacity drives. The cache tier dramatically influences both sequential throughput and random IOPS.

Cache tier rules in 23H2:

S2D automatically identifies the fastest drives as cache (NVMe > SSD > HDD)
The default cache-to-capacity ratio in S2D is 1:3 — one cache drive per three capacity drives
Cache drives should be no more than 10% of total capacity for effective caching
Persistent write-back cache (enabled by default on NVMe) requires UPS or battery-backed storage for data integrity

⚠️

All-NVMe cache configuration: When all drives are NVMe, S2D still applies a software cache in the form of a write buffer. The CacheMetadataReserveBytes setting controls cache metadata overhead. For high-performance workloads, consider pinning critical volume data to NVMe tiers using storage QoS policies.

🛡️ Resiliency Settings

Resiliency type has a major impact on both storage efficiency and write performance. The tradeoffs:

Resiliency Type	Min Nodes	Storage Efficiency	Write Overhead	Best For
Two-way mirror	2	50%	2× writes	2-node clusters, non-critical data
Three-way mirror	3	33%	3× writes	Databases, VMs, production workloads
Mirror-accelerated parity	4	50–60%	Mixed	Capacity-optimized (cold/warm data)

PowerShell — Create a volume with specific resiliency settings

# Create a three-way mirror volume for production VMs
New-Volume -FriendlyName "VMs-Prod" `
  -StoragePoolFriendlyName "S2D on ClusterName" `
  -FileSystem CSVFS_ReFS `
  -ResiliencySettingName Mirror `
  -NumberOfDataCopies 3 `
  -ProvisioningType Fixed `
  -Size 4TB

# Verify the volume resiliency and health
Get-VirtualDisk -FriendlyName "VMs-Prod" | 
  Select FriendlyName, ResiliencySettingName, NumberOfDataCopies, 
    OperationalStatus, HealthStatus, Size

🌐 Network Configuration for Maximum S2D Throughput

The S2D storage network is the most critical performance component. Incorrect RDMA or network configuration is responsible for 80% of S2D performance issues in production:

RDMA enabled on storage NICs: Verify with Get-NetAdapterRdma — all storage NICs must show Enabled: True
SMB Direct enabled: Get-SmbClientConfiguration | Select EnableBandwidthThrottling, EnableLargeFileSharing
Jumbo frames (MTU 9014) on storage NICs and switches: Standard 1500 MTU dramatically reduces S2D throughput
Priority Flow Control (PFC) for RoCE v2 deployments: PFC must be enabled on the switch ports connected to storage NICs, with Priority 3 mapped to storage traffic

PowerShell — Validate RDMA and jumbo frames on storage NICs

# Check RDMA status on all adapters
Get-NetAdapterRdma | Format-Table Name, InterfaceDescription, Enabled

# Check MTU settings
Get-NetAdapterAdvancedProperty -Name "StorageNIC1","StorageNIC2" `
  -RegistryKeyword "*JumboPacket" | 
  Select Name, DisplayValue

# Run SMB Bandwidth test between nodes
Test-SmbBandwidth -DestinationIPAddress "192.168.20.2" `
  -BufferSize 4096 `
  -TestType Write

💻 Workload Placement and Storage QoS

Storage QoS policies prevent noisy-neighbor scenarios where a single VM monopolizes storage IOPS and impacts other workloads. For production clusters running mixed workloads:

PowerShell — Create Storage QoS policies for tiered workloads

# Create a policy for tier-1 workloads (high IOPS guarantee)
New-StorageQosPolicy -Name "Tier1-SQL" `
  -PolicyType Dedicated `
  -MinimumIops 5000 `
  -MaximumIops 20000

# Create a policy for tier-2 workloads (capped to prevent noisy neighbor)
New-StorageQosPolicy -Name "Tier2-General" `
  -PolicyType Dedicated `
  -MaximumIops 3000

# Apply policy to a VM virtual disk
Get-VM "SQL-Prod-01" | 
  Get-VMHardDiskDrive | 
  Set-VMHardDiskDrive -QoSPolicyID (Get-StorageQosPolicy -Name "Tier1-SQL").PolicyId

📖 Official Microsoft Documentation

The primary authoritative reference for Storage Spaces Direct is the Microsoft Learn documentation: Storage Spaces Direct overview — learn.microsoft.com. This covers the full architecture, hardware requirements, fault tolerance models, and deployment guidance maintained by the Windows Server team.

Key sections in the official documentation relevant to production deployments:

Hardware requirements — validated NIC types, drive firmware versions, and HBA support matrix
Understanding the cache — how S2D determines cache vs capacity drives, cache binding behaviour, and write-back persistence
Fault tolerance and storage efficiency — mirror and parity resiliency types, nested resiliency for 2-node clusters, and the relationship between fault tolerance and usable capacity
Choosing drives — NVMe, SSD, and HDD placement guidance and the cache-to-capacity ratio recommendations
Performance history — built-in time-series performance data collected by S2D, queryable via PowerShell

💡

Performance history PowerShell example: S2D collects built-in performance counters without any additional agents. Query them with Get-ClusterPerf or through the dedicated cmdlets documented on Microsoft Learn — no third-party monitoring tool is required for basic S2D performance tracking.

📊 Benchmarking Tools

Before and after any S2D performance tuning, establish baselines with the right tools:

DiskSpd: Microsoft's official storage benchmarking tool for Windows — the correct tool for S2D baseline testing. Use the -L flag for latency percentiles.
VMFleet: Workload simulation tool specifically designed for S2D clusters — simulates multiple VMs running simultaneous storage workloads
Performance Monitor (PerfMon): For ongoing monitoring, capture Cluster CSVFS and Storage Spaces counters

DiskSpd — Baseline test (4K random read/write from a VM)

diskspd.exe -c50G -d60 -r -w30 -t8 -o32 -b4K -L -Z1M -Sh C:	estfile.dat

# Flags:
# -d60    : 60 second test duration
# -r      : random I/O
# -w30    : 30% write, 70% read
# -t8     : 8 threads per file
# -o32    : 32 outstanding I/O per thread
# -b4K    : 4KB block size (OLTP simulation)
# -L      : measure latency

✅ S2D Performance Tuning Checklist

✅ All drives are on the validated hardware compatibility list (HCL)
✅ RDMA enabled on all storage NICs — verified with Get-NetAdapterRdma
✅ Jumbo frames (MTU 9014) configured on storage NICs and all switches in the storage path
✅ SMB Direct enabled and SMB Multichannel active across all node pairs
✅ PFC configured on switches (for RoCE v2 deployments)
✅ Network ATC intents in Provisioned state on all nodes
✅ S2D cache drives are the fastest drive type in the server
✅ Volume resiliency matches workload criticality (3-way mirror for production)
✅ Storage QoS policies applied to prevent noisy-neighbor scenarios
✅ DiskSpd baseline recorded before any changes — kept for comparison

Storage Spaces Direct S2D RDMA Performance NVMe SMB Direct Resiliency Benchmarking

Continue reading

Windows Server Windows Server 2025 + Azure Arc: Managing Hybrid Servers at Scale Azure Local Azure Local 23H2: Complete Architecture Guide

🔗 Continue the conversation on LinkedIn

Follow Mohamed Dyabi for insights on Azure hybrid infrastructure, adaptive cloud, and Microsoft certifications across EMEA.

Follow on LinkedIn