Proxmox cluster (primair en failover) met redundante datastores Full flash storage aangeboden

Ons cluster is opgedeeld geweest op twee locaties zodoende de eerste een primaire locatie was en de secundaire de failover.

Het primaire Proxmox cluster heeft 7 online compute nodes, 2 SSD/full-flash storage nodes, 448 CPU(s), 3.44 TiB RAM, and 20.07 TiB SSD opslag op een Jovian DSS oplossing met 2 storage nodes (Intel Xeon gebaseerd).

• 7 active compute nodes
• 1 powered-off spare compute node
• 2 SSD / full-flash storage nodes
• 2 x Netgear ProSafe M4300-48X 10GbE switches
• 2 x Netgear SFP+ DAC stacking cables

De failover heeft 5 compute nodes en 1 spinning-disk based storage node (ook weer Jovian DSS maar als replicatie).

• 5 compute nodes
• 1 spinning-disk based DR storage node
• 1 x Netgear ProSafe M4300-8X8F 10GbE / SFP+ switch

De totale hoeveelheid van beide clusters is 1024 cores, 6 TB RAM and 48 TB storage (SSD primair + Spinningdisk als replica)

Opsomming specs Primaire cluster:

Compute/storage layout	7 active compute nodes, 1 powered-off compute node (cold spare), and 2 SSD/full-flash storage nodes
Logical CPU(s) shown by Proxmox	448 CPU(s) (AMD Epyc Rome)
Estimated physical CPU cores for active nodes	224 cores
Estimated CPU threads for active nodes	448 threads
Memory	3.44 TiB total
Storage	20.07 TiB total / 15.78 TiB used / 79% used

Node	CPU(s)	RAM	Root disk	CPU model
pve-ww-2	64	503.55 GiB	67.73 GiB	AMD EPYC 7502P 32-Core
pve-ww-3	64	503.55 GiB	67.73 GiB	AMD EPYC 7502P 32-Core
pve-ww-4	64	503.55 GiB	67.73 GiB	AMD EPYC 7502P 32-Core
pve-ww-5	64	503.55 GiB	67.73 GiB	AMD EPYC 7502P 32-Core
pve-ww-6	64	503.52 GiB	64.05 GiB	AMD EPYC 7532 32-Core
pve-ww-7	64	503.52 GiB	64.05 GiB	AMD EPYC 7532 32-Core
pve-ww-8	64	503.52 GiB	64.05 GiB	AMD EPYC 7532 32-Core

Storage nodes (Jovian-DSS): (2 units dus)
2x Intel® Xeon® Scalable Silver 4215R Processor - 8 Core - 16 Threads - 3.2GHz - 4GHz Turbo - 130W -
11MB Cache
4x Samsung 64GB DDR4 3200Mhz Registered ECC, Supermicro certified
2 x 920W Redundant High-efficiency Platinum Level Power Supplies
1x Supermicro AOC-S3008L-L8E, Broadcom 3008
1x Supermicro AOC-S25G-B2S-O 25GBE SFP+, Broadcom BCM57414
2x Supermicro AOC-STG-I4T Network Card, 4 x 10Gbit/sec RJ45 Ethernet, Intel X710 + X557 chip
2x Supermicro AOC-SLG3-2M2-O
2 x Intel® Optane™ P4801X - 200GB NVME M.2 – Enterprise
8x Samsung SSD PM883, SATA3, Enterprise 1.92TB
1x Supermicro MCP-220-82616-0N Rear 2 X 2.5"
2x QSFP28 - QSFP28 DAC 1meter - Passive (Melanox Coding)


Replicatie cluster spec:

Node	CPU(s)	RAM	Root disk	CPU model
pve-st-1	64	503.55 GiB	67.73 GiB	AMD EPYC 7502P 32-Core
pve-st-2	64	503.55 GiB	67.73 GiB	AMD EPYC 7502P 32-Core
pve-st-3	128	503.55 GiB	67.73 GiB	2x AMD EPYC 7502P 32-Core
pve-st-4	128	503.55 GiB	67.73 GiB	2x AMD EPYC 7502P 32-Core
pve-st-5	128	503.52 GiB	64.05 GiB	2x AMD EPYC 7532P 32-Core

Spinning disk Offsite DR server Z2 - +- 35TB netto
2x Intel® Xeon® Scalable Bronze 3206R Processor - 8 Core - 8 Threads - 1.9GHz - 85W - 11MB Cache
8x Samsung 16GB DDR4 3200Mhz Registered ECC, Supermicro certified
1x Supermicro, 920W Redundant Power Supplies Platinum Level
2x Supermicro AOC-SLG3-2M2-O
2x Intel® Optane™ P4801X - 100GB NVME M.2 – Enterprise
1x Supermicro AOC-S3008L-L8E, Broadcom 3008
6x WD Ultrastar DC HC330 10TB SAS 256MB 512e
1x Supermicro MCP-220-82616-0N Rear 2 X 2.5" HDD kit
2x Samsung SSD PM883, SATA3, Enterprise, 240GB

Component	Quantity / basis	Estimated resale value
Primaire compute nodes	7 active nodes + 1 spare, EPYC 32-core class, 512 GB RAM class per active node	EUR 24,000 – 40,000
Failover DR compute nodes	5 DR compute nodes, EPYC 32-core class, lower RAM configuration	EUR 10,000 – 20,000
Primaire SSD / full-flash storage nodes	2 storage units with Xeon Silver CPUs, ECC RAM, Optane and Samsung PM883 enterprise SSDs	EUR 7,500 – 15,000
Failover (replicatie) spinning-disk DR storage node	1 DR storage server with 10TB SAS disks and Optane devices	EUR 2,500 – 6,000
10GbE switching	2 x Netgear M4300-48X, 1 x Netgear M4300-8X8F, DAC cables	EUR 2,500 – 6,000
Miscellaneous server value	Chassis, redundant PSUs, Supermicro boards, HBAs, NICs, rails and spare parts	EUR 5,000 – 10,000

Hieronder een lijst met componenten van de eerste aankoop (daarna is het cluster nog uitgebreid met extra opslag en geheugen en nodes).

Quantity	Description - Out-of-the-box cluster, full installation & delivery
Network	2 x NETGEAR ProSafe M4300-48X switch 2 x NETGEAR SFP+ DAC 1 Meter
5 Nodes	1 x AMD EPYC Rome 7502P 4 x Samsung 64GB DDR4 3200Mhz Registered ECC, Supermicro certified 2 x Supermicro AOC-STG-I4T Network Card, 4 x 10Gbit/sec RJ45 Ethernet, Intel X710 + X557 chip 1 x Samsung PM981a - 256GB NVME M.2
2 Storage units	2 x Intel Xeon Scalable Silver 4215R 4 x Samsung 64GB DDR4 3200Mhz Registered ECC, Supermicro certified 2 x 920W Redundant High-efficiency Platinum 1 x Supermicro AOC-S3008L-L8E, Broadcom 3008 1 x Supermicro AOC-S25G-B2S-O 25GBE SFP+, Broadcom BCM57414 2 x Supermicro AOC-STG-I4T Network Card, 4 x 10Gbit/sec RJ45 Ethernet, Intel X710 + X557 chip 2 x Supermicro AOC-SLG3-2M2-O 2 x Intel Optane P4801X - 200GB NVME M.2 8 x Samsung SSD PM883, SATA3, Enterprise 1.92TB 1 x Supermicro MCP-220-82616-0N Rear 2 X 2.5" 2 x QSFP28 - QSFP28 DAC 1meter - Passive (Mellanox Coding)
1 Offsite DR server Z2	2 x Intel Xeon Scalable Bronze 3206R Processor 8 x Samsung 16GB DDR4 3200Mhz Registered ECC, Supermicro certified 1 x Supermicro, 920W Redundant 2 x Supermicro AOC-SLG3-2M2-O 2 x Intel Optane P4801X - 100GB NVME M.2 1 x Supermicro AOC-S3008L-L8E, Broadcom 3008 6 x WD Ultrastar DC HC330 10TB SAS 256MB 512e 1 x Supermicro MCP-220-82616-0N Rear 2 x Samsung SSD PM883, SATA3, Enterprise, 240GB
Software	1 x Open-E JovianDSS - JovianDSS Advanced Metro HA Cluster Feature Pack HA 2 x Open-E JovianDSS - Open-E JovianDSS Storage extension 2 x Open-E JovianDSS - Open-E JovianDSS Storage Extension 16TB License Key 2 x Open-E JovianDSS - Standard Support or Support Renewal 5 years 4TB to 16TB
Software for Offsite DR server Z2	1 x Open-E JovianDSS - Open-E JovianDSS 1 x Open-E JovianDSS - Open-E JovianDSS Storage Extension 64TB 1 x Open-E JovianDSS - Standard Support 5 years 20TB to 128TB