Senior/Principal - Artificial Intelligence Infrastructure, NM/CA- Hybrid

Sandia National Laboratories
$114,000 - $227,500
401(k), relocation assistance
United States, New Mexico, Albuquerque
1515 Eubank Boulevard Southeast (Show on map)
Oct 29, 2025
Apply for Job Job ID 696235 Location Albuquerque, NM Full/Part Time Full-Time Regular/Temporary Temporary Add to Favorite Jobs Email this Job About Sandia Sandia National Laboratories is the nation's premier science and engineering lab for national security and technology innovation, with teams of specialists focused on cutting-edge work in a broad array of areas. Some of the main reasons we love our jobs: Challenging work with amazing impact that contributes to security, peace, and freedom worldwide Extraordinary co-workers Some of the best tools, equipment, and research facilities in the world Career advancement and enrichment opportunities Flexible work arrangements for many positions include 9/80 (work 80 hours every two weeks, with every other Friday off) and 4/10 (work 4 ten-hour days each week) compressed workweeks, part-time work, and telecommuting (a mix of onsite work and working from home) Generous vacation, strong medical and other benefits, competitive 401k, learning opportunities, relocation assistance and amenities aimed at creating a solid work/life balance* World-changing technologies. Life-changing careers. Learn more about Sandia at: http://www.sandia.gov These benefits vary by job classification. What Your Job Will Be Like Sandia's artificial intelligence (AI) team is building the U.S. Department of Energy's (DOE) next-generation AI Platform, an integrated scientific AI capability that delivers rapid, high-impact solutions for national security, science, and applied energy missions. The Platform is based on three pillars: Models, Infrastructure, and Data. You will join the Infrastructure Pillar team to design, deploy, and operate the unified compute-and-data fabric that underpins all mission workflows from AI model training and simulation steering to real-time inference at experimental and production facilities. We anticipate multiple hires for the Infrastructure Pillar that collectively span the set of responsibilities and skills described below. Likewise, new hires will be expected to work in conjunction with existing Sandia staff and teams from other DOE laboratories to deliver on this ambitious, fast-paced project. Importantly, we anticipate that while AI Platform development will leverage existing AI and data science tools extensively, success will also require considerable innovation and problem solving to address the unique needs of DOE applications. If this sounds like an exciting challenge to you, we look forward to reading your application! Key Responsibilities: Architect and implement the hybrid compute fabric Integrate exascale HPC systems with elastic cloud resources and specialized AI accelerator clusters (on-prem and in-cloud) Deploy ruggedized edge servers and digital-twin infrastructure for sub-millisecond inference and real-time physics simulations Develop infrastructure services and orchestration Build federated Kubernetes clusters, container registry services, resource registry, and job scheduling abstractions Implement self-configuring distributed clusters with intelligent network overlays, AI-driven traffic steering, and sensor-driven control loops\ Design secure networking and enclaves Configure ESnet-backed, multi-tier WAN overlays with low-latency, geo-diverse routing, failover, and encryption protocols Provide software-defined, dynamic security enclaves for CUI/Restricted Data with attested runtime and curated egress Enable observability, provenance & monitoring Deploy unified logging, metrics, dashboards, and trace-analysis across cloud and on-prem environments using OpenTelemetry, Prometheus, ELK, or equivalent Automate provenance capture for compute jobs, data movements, and AI workflows Support federated identity and access control Integrate multiple identity providers, attribute-based access controls, and allocation models for risk-shared governance Manage enterprise licensing, token agreements, and software audits for AI and HPC frameworks Manage the full lifecycle of the AI platform's infrastructure, including capacity planning, upgrades, documentation, and performance monitoring Implement and enforce security best practices within container environments, including Role-Based Access Control (RBAC), secrets management, network policies, and vulnerability scanning. On any given day, you may be called upon to Stand up a new GPU-accelerated cluster, configure Slurm/Kubernetes, and validate performance benchmarks Troubleshoot cross-site data transfers over ESnet and optimize WAN throughput for a petabyte-scale lakehouse Deploy a hardened enclave for a classified ML training job with differential-privacy egress controls Script an IaC workflow (Terraform/Ansible) to provision edge compute nodes Collaborate with the Models team to tune network and storage parameters for distributed training jobs Present real-time infrastructure status and forecasts to stakeholder Present prototype demos and research results to stakeholders across DOE, DoD, IC, and industry The selected applicant can work a combination of onsite and offsite work. The selected applicant must live within a reasonable distance for commuting to the assigned work location when necessary. distance for commuting to the assigned work location when necessary. Salary Range $114,000 - $227,500 Salary range is estimated, and actual salary will be determined after consideration of the selected candidate's experience and qualifications, and application of any approved geographic salary differential. Qualifications We Require Bachelor's degree in Computer Science, Electrical Engineering, Mathematics, or a related STEM field plus five (5) years of directly relevant experience, or an equivalent combination of education and experience Ability to acquire and maintain a DOE Q clearance Qualifications We Desire The ideal R&D S&E Artificial Intelligence candidate for Sandia National Laboratories will in addition possess the following: Graduate degree in a relevant computationally-intensive discipline where an independent research project was a graduation requirement (e.g., independent project, thesis, or dissertation). Experience in developing software and AI systems for enterprise and national security applications. Demonstrated software development skills and familiarity with modern software development practices. Proven ability to work and communicate effectively in a collaborative and interdisciplinary team environment mentoring junior engineers Also, for this posting we are seeking individuals with the following experience: Ph.D. in a STEM field with focus on high-performance or distributed computing (Data Science, Data and Computing Systems, Informatics or a related STEM field with a significant data systems research component Experience with HPC systems administration (Slurm, PBS, Flux) and cloud platforms (AWS, Azure, GCP) Proficiency in container orchestration (Kubernetes, Docker) and infrastructure as code (Terraform, Ansible) Networking background: ESnet, VLANs, WAN overlays, encryption, and failover design Hands-on experience with storage architectures: Lustre, GPFS, object stores, multi-tier caching Experience implementing DevSecOps principles and security best practices in containerized infrastructure, including network policies, classification management, and vulnerability scanning. Experience with federated Kubernetes and large-scale container platforms across classification domains Familiarity with secure enclave technologies and zero-trust security models Background in digital-twin or real-time simulation steering architectures Experience with SIEM environments, such as Splunk, and operational management of application infrastructure services Proficiency in observability toolchains (OpenTelemetry, Prometheus, Grafana, ELK) and automated log analytics Knowledge of DOE/NNSA compute and networking environments (Frontier, Aurora, Perlmutter, ESnet) Integrating experimental facilities, robotics, or 3D-printing systems into automated AI workflows Deploying large-scale secure enclaves for CUI and RD applications Coordinating publicprivate partnerships on HPC and AI infrastructure deployments Working in cross-lab federated teams with shared governance and risk models Ability to obtain and maintain a SCI clearance, which may require a polygraph test. About Our Team The Center for Computing Research (CCR) at Sandia creates technology and solutions for many of our nation's most demanding national security challenges. The Center's portfolio spans the spectrum from fundamental research to state-of-the-art applications. Our work includes computer system architecture (both hardware and software); enabling technology for modeling physical and engineering systems; and research in discrete mathematics, data analytics, cognitive modeling, and decision support materials. You will be part of a multi-disciplinary, mission-focused team delivering the compute and data backbone for transformative AI systems. Occasional travel may be required. If you're passionate about building the software infrastructure that powers cutting edge AI systems, we want to hear from you. Posting Duration This posting will be open for application submissions for a minimum of seven (7) calendar days, including the 'posting date'. Sandia reserves the right to extend the posting date at any time. Security Clearance Sandia is required by DOE to conduct a pre-employment drug test and background review that includes checks of personal references, credit, law enforcement records, and employment/education verification. Applicants for employment need to be able to obtain and maintain a DOE Q-level security clearance, which requires U.S. citizenship. If you hold more than one citizenship (i.e., of the U.S. and another country), your ability to obtain a security clearance may be impacted. Applicants offered employment with Sandia are subject to a federal background investigation to meet the requirements for access to classified information or matter if the duties of the position require a DOE security clearance. Substance abuse or illegal drug use, falsification of information, criminal activity, serious misconduct or other indicators of untrustworthiness can cause a clearance to be denied or terminated by DOE, resulting in the inability to perform the duties assigned and subsequent termination of employment. EEO All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, or veteran status and any other protected class under state or federal law. NNSA Requirements for MedPEDs If you have a Medical Portable Electronic Device (MedPED), such as a pacemaker, defibrillator, drug-releasing pump, hearing aids, or diagnostic equipment and other equipment for measuring, monitoring, and recording body functions such as heartbeat and brain waves, if employed by Sandia National Laboratories you may be required to comply with NNSA security requirements for MedPEDs. If you have a MedPED and you are selected for an on-site interview at Sandia National Laboratories, there may be additional steps necessary to ensure compliance with NNSA security requirements prior to the interview date.