Overview

Sandisk delivers storage solutions built on Flash and advanced memory technologies. The Hive platform is a swarm of hundreds of identical storage nodes, each combining a Marvell/XSight DPU with SSDs and running a full Ceph stack: OSD, Monitor, Manager, MDS, SeaStore, and an NFS front end. Standing up, reconfiguring, and recovering a cluster of this size by hand does not scale.

This role owns the automated hardware discovery and Ceph role-assignment pipeline: the flow that inventories every node and its hardware, decides which daemons each node should run, and drives the cluster from bare metal to a serving state. The pipeline must run fully autonomously by default, with a clean manual-override path for lab, bring-up, and failure-injection scenarios.

This is a specialized discipline on the critical path for every test cluster, every silicon bring-up, and every customer-shaped deployment.

What You Will Own

• •Automated hardware discovery: detect nodes as they power on and enumerate their hardware (DPU/SoC model, SSD media, DRAM, RNIC/network ports, BMC) using out-of-band and in-band inventory via BMC/Redfish/IPMI, PXE/DHCP boot, and cloud-init/first-boot agents; produce a single authoritative machine inventory consumed by the rest of the pipeline.
• •Role assignment and cluster composition: given the discovered inventory, decide and apply which Ceph roles each node runs (OSD, MON, MGR, MDS, and NFS gateway) encoded as declarative placement specifications; own MON quorum sizing and placement, MGR redundancy, MDS/gateway placement, and CRUSH map/failure-domain layout.
• •Bare-metal to serving bring-up: drive the end-to-end sequence including node provisioning, OS/image deploy, cluster bootstrap, daemon deployment via the Ceph orchestrator (cephadm-style), ceph-volume-style OSD provisioning on SSD media, and health convergence to HEALTH_OK.
• •Autonomous and manual modes: make the default path zero-touch so a rack powers on and self-assembles into a healthy cluster, while exposing deterministic manual controls to pin roles, hold a node out, force a specific topology, or reproduce a customer/lab configuration for testing.
• •Lifecycle and recovery automation: node add/remove, drain and rebalance, daemon replacement, MON re-quorum after loss, MDS/OSD failover validation, and re-discovery after re-imaging, integrated with Hive's Fast Recovery and BMC work.
• •Reconciliation and drift control: continuously compare declared desired state against observed cluster state and converge, with clear reporting when reality diverges from intent.
• •CI/lab integration: wire the pipeline into automated test so any commit can spin up a correctly-composed multi-node cluster on real hardware and tear it down cleanly.

Core Requirements

• •Deep experience in lab hardware discovery, inventory, and automated provisioning at fleet scale.
• •Bare-metal automation: PXE/DHCP boot, BMC out-of-band management (Redfish/IPMI), image/OS deployment, cloud-init/first-boot.
• •Infrastructure-as-code and config automation (Ansible, Terraform, or equivalent tooling) with a declarative, reconcile-to-desired-state mindset.
• •Strong scripting and automation skills in Python and shell, plus experience with CI systems such as Jenkins or GitLab CI.
• •Comfort designing systems that are autonomous by default with reliable manual override.

Strongly Preferred

• •Ceph operational knowledge: OSD/MON/MGR/MDS roles, the orchestrator/cephadm model, placement specs, ceph-volume, CRUSH maps and failure domains, MON quorum, and cluster health/lifecycle.
• •Distributed-systems fluency: quorum/Paxos intuition, rebalance/recovery behavior, failure-domain reasoning.
• •Networking bring-up for storage fabrics including RoCEv2 and Ethernet port discovery.
• •Familiarity with DPU/SoC-based nodes and constrained-node environments.