Hiring and Building an Internal AI Team: Roles, Skills, and Compensation Bands

The market for AI talent is bifurcated. Senior research-grade ML engineers at frontier labs make seven figures. Applied AI engineers shipping production features in regulated enterprises make a fraction of that, do most of the actual work, and are far harder to retain than to hire. If you are building an internal AI team in 2026, you need to know which game you are playing.

This is the hiring playbook we use with enterprise clients standing up an AI capability for the first time. Six core roles, what they actually do, where to find them, what to pay, and how to avoid the interview traps that cost you six months and a Director of AI.

The six core roles

1. ML Engineer

What they actually do. Build, fine-tune, and deploy models. Own the model lifecycle: data prep, training (or evaluation if foundation-model-only), evaluation, deployment, monitoring. The hands-on technical core of your team.

Must-have skills. Python proficiency at staff-engineer level, deep familiarity with at least one foundation model API (Anthropic, OpenAI, Bedrock, Vertex), evaluation harness experience (Promptfoo, Inspect, LangSmith, internal frameworks), familiarity with vector databases, experience shipping at least one production AI system.

Nice-to-have. Fine-tuning experience (LoRA, full fine-tune), distributed training, model serving optimization, on-call experience.

2026 US compensation band. Base $165K to $245K. Annual bonus 15% to 25%. Equity $60K to $200K annualized. Total comp $220K to $420K. Bay Area and NYC add 15% to 20%.

Where to source. Senior backend engineers at SaaS companies who have shipped AI features in the last 18 months. Avoid pure researchers transitioning out of academia unless they have a year of production shipping. Avoid Kaggle competitors; competition skills do not transfer.

Interview pitfalls. Whiteboard ML theory is a waste of time for this role. Test on a real take-home: given this dataset and this API budget, build an evaluation harness for a sentiment classifier and explain your tradeoffs. The ones who can ship will produce something working in 4 hours. The ones who cannot will produce a research memo.

2. AI Product Manager

What they actually do. Translate business asks into technical scope. Own the use case backlog, prioritization, success metrics, and stakeholder management. The scarcest hire in this list.

Must-have skills. 5+ years product management, at least 2 years shipping data or ML products, ability to read and reason about evaluation results without help from an engineer, comfortable with probabilistic outcomes (not every input gets a deterministic output).

Nice-to-have. Direct industry experience in your vertical, hands-on prompt engineering, prior experience at an AI-native company.

2026 US compensation band. Base $175K to $255K. Annual bonus 20% to 30%. Equity $80K to $220K. Total comp $240K to $450K.

Where to source. Senior PMs at data infrastructure companies, observability companies, or B2B SaaS with ML features. The bar is whether they can argue with an ML engineer and not lose; not whether they know the math.

Interview pitfalls. Do not ask them to design Uber for AI. Give them a real internal use case and watch them tear it apart. The good ones will identify three reasons the use case is the wrong starting point and propose a better one. The bad ones will draw a roadmap.

3. MLOps / Platform Engineer

What they actually do. Own the platform: model gateway, eval harness, observability, vector stores, fine-tuning pipelines, cost monitoring. The plumbing nobody notices when it works and everyone notices when it does not.

Must-have skills. Strong DevOps or platform engineering background, Kubernetes, Terraform or equivalent IaC, hands-on experience with at least one inference platform (Bedrock, Vertex, Azure AI Foundry, self-hosted), comfort with high-throughput async systems.

Nice-to-have. GPU operations experience, prior MLOps at scale, security clearance (for regulated and federal contexts).

2026 US compensation band. Base $170K to $240K. Annual bonus 15% to 20%. Equity $50K to $180K. Total comp $210K to $380K.

Where to source. Senior platform engineers at SaaS companies, especially those who built developer platforms. Avoid generalist DevOps who have never operated a high-cost, high-throughput system; cost discipline is hard to teach.

Interview pitfalls. Beware of candidates whose entire experience is one cloud vendor. The platform stack is heterogeneous and getting worse. Ask: how would you handle a vendor going down for 4 hours in the middle of a production workload? The good answers involve failover, the bad answers involve hope.

4. AI Governance / Risk Lead

What they actually do. Own the risk register, AI policy, regulatory engagement, model cards, vendor due diligence. The role that does not look critical until your first incident.

Must-have skills. Background in compliance, audit, legal, or risk; ability to read NIST AI RMF, EU AI Act, and emerging US state-level regulations and translate to internal policy; comfort working across legal, security, and engineering.

Nice-to-have. Direct AI policy experience (rare), industry-specific compliance background (HIPAA, PCI, FedRAMP, GLBA), former regulator or auditor.

2026 US compensation band. Base $160K to $235K. Annual bonus 15% to 25%. Equity $40K to $150K. Total comp $195K to $340K.

Where to source. Recovering compliance or audit professionals who have spent 3+ years on data or model risk. Former Big Four risk advisory consultants. Avoid pure lawyers; they tend to overpolice and slow programs.

Interview pitfalls. Ask them to design a policy exception process. Bad answers involve adding committees. Good answers involve clear thresholds, named accountable parties, and a 5-business-day SLA.

5. Applied AI / Prompt Engineer

What they actually do. The unsung middle layer. They make models actually work on the specific use case. Prompt engineering, retrieval design, agent flow design, evaluation authoring, edge case handling. The role most enterprises underinvest in.

Must-have skills. Strong written reasoning, experimental discipline, comfort with iterative debugging of non-deterministic systems, basic Python or TypeScript proficiency, evaluation harness fluency.

Nice-to-have. Domain expertise in your business, prior content design or technical writing experience (more relevant than you think), experience with agent frameworks (LangGraph, CrewAI, internal frameworks).

2026 US compensation band. Base $145K to $210K. Annual bonus 10% to 20%. Equity $30K to $120K. Total comp $175K to $320K.

Where to source. Strong individual contributors from solutions engineering, technical product, content design, or developer relations. Often non-traditional backgrounds. The best ones I have hired came from journalism, philosophy, and law school dropouts.

Interview pitfalls. Whiteboard coding kills this role. Give them a failing prompt and 90 minutes to fix it on a real model. Watch how they iterate, what they measure, when they ask for help.

6. Data Engineer for AI

What they actually do. Build and operate the pipelines that feed RAG and fine-tuning workloads. Document ingest, chunking strategy, metadata extraction, retrieval evaluation, freshness monitoring.

Must-have skills. Strong data engineering background (Airflow, dbt, Spark or equivalent), comfort with unstructured data (PDFs, transcripts, code, images), vector database experience, evaluation discipline for retrieval quality.

Nice-to-have. Prior search engineering experience, prior knowledge graph experience, experience with document AI services (Textract, Document AI).

2026 US compensation band. Base $160K to $225K. Annual bonus 15% to 20%. Equity $45K to $160K. Total comp $200K to $360K.

Where to source. Senior data engineers at content-heavy companies (media, legal tech, healthcare). Search engineers transitioning from keyword to semantic retrieval.

Interview pitfalls. Bad chunking destroys most RAG systems. Ask them to evaluate three chunking strategies on a real document set. The good ones will refuse to answer in the abstract and ask for the corpus.

Build vs buy vs borrow

For each role, you have three options. The honest answer is most enterprises should use all three in different ratios.

Build (FTE). Best for: ML Engineer, AI PM, MLOps, Governance Lead. Roles where institutional knowledge compounds and continuity matters. Worst for: niche skills you need for 6 months.

Buy (contractor). Best for: Applied AI Engineer (especially for spike capacity), specific fine-tuning projects, one-time evaluation harness builds. Worst for: governance and platform roles where context loss creates risk.

Borrow (consulting partner). Best for: program architecture, capability buildout coaching, regulated industry expertise, FedRAMP or HIPAA expertise you cannot hire fast enough. Worst for: ongoing production operations. If your consulting partner is operating your platform 18 months in, something is wrong.

A typical 12-person year-one AI team at a 10,000-person enterprise:

| Role | FTE | Contractor | Consulting | |------|-----|-----------|------------| | Head of AI / CoE Director | 1 | - | Advisory | | AI Product Manager | 2 | - | - | | ML Engineer | 3 | 1 | - | | MLOps / Platform | 2 | - | Architecture | | Governance / Risk Lead | 1 | - | Policy support | | Applied AI Engineer | 2 | 2 | - | | Data Engineer for AI | 1 | 1 | - |

Total FTE comp at midpoint: roughly $2.6M annualized. Contractor and consulting add $1.2M to $2M depending on intensity. Plan accordingly.

Common hiring failure patterns

1. Hiring a researcher to ship product. Different skill set entirely.
2. One unicorn instead of three specialists. The "AI engineer who does everything" does nothing especially well.
3. Comp band too low. You get what you pay for. If your band is 30% below market, you hire people who could not get hired at market.
4. No technical hiring manager. HR cannot screen AI engineers. Get a senior IC involved in every loop.
5. Skipping the take-home. Whiteboard interviews do not predict shipping. Take-homes do.

Retention is harder than hiring

The market for AI talent is a poaching market. Your offer letter is a 12-month option, not a permanent commitment. Plan for it.

Retention levers that actually work:

1. Mission and problem quality. AI engineers leave because the problems are boring. Make sure your top three hires are working on the most consequential use cases.
2. Compute and tool access. Cheap to fix, high signal. Engineers leave companies that gate API access through three layers of approval.
3. Conference and publication budget. $5K to $10K per engineer per year. Encourages external writing and speaking. Costs less than one bad replacement hire.
4. Internal mobility. The platform engineer who wants to move into AI product management should be able to. The applied AI engineer who wants to spin up a research week should have it.
5. Compensation refresh every 9 months. Not 18. The market moves too fast. Build the budget for it.

What does not work: ping-pong tables, snack budgets, AI-themed swag. Engineers in 2026 see through these. Substantive levers only.

The Director of AI hire is the keystone

If you make a mistake on this hire, every other hire is contaminated. The Director sets the bar for who else gets hired, sets the culture, and is the face of the program to the executive team.

The most common mistake: hiring a brilliant ML researcher because they have a PhD and a strong publication record. The second most common: hiring a McKinsey alum who has built an AI strategy deck but has never shipped a model.

What you want: someone who has shipped production AI in an enterprise context, run a team of 8 or more, and can speak credibly to both engineers and executives. These people exist, but they are scarce and expensive. Expect a 4 to 6 month search and a comp package in the $450K to $700K total range.

Three interview signals that matter more than the resume:

1. Can they describe in concrete detail one AI project they shipped that failed, and what they learned?
2. Can they explain a technical tradeoff in language a CFO would understand?
3. When you describe your messiest internal use case, do they ask sharper questions than your steering committee did?

If yes to all three, make the offer fast.

A note on internal mobility

Some of your best AI hires are already on payroll. Strong backend engineers with curiosity, data analysts who have been quietly using LLMs for two years, product managers who shipped data products. Internal mobility is faster, cheaper, and retains better. Build a 90-day applied-AI residency program and run it twice a year.

Next steps

The first three hires set the culture of your AI team for the next five years. Get them right and the next twelve hire themselves. Get them wrong and you spend a year unwinding the damage. If you want a calibration call on your hiring plan, your interview loops, or your build-vs-buy mix, that is the kind of engagement One Frequency runs with clients in the first 90 days of CoE buildout. The ai-implementation-roadmap-enterprise piece pairs naturally with this if you are sequencing the team against the program.