A Two-Week Anthropic-to-Open-Source Migration Playbook

Provider retirement and version changes force teams to confront an uncomfortable fact: the application depends on behavior that the API type system does not describe. The endpoint can accept the same messages while tool selection, refusal, verbosity, structured output, context use, latency, and failure modes all change. Open-source models add control over weights and serving, but they also move capacity, reliability, and security responsibilities into your system.

Write a migration contract before selecting a replacement. Define the user workflows in scope, quality metrics, critical failure slices, safety policy, tool-call requirements, context length, structured-output contract, latency objective, throughput, availability, data region, observability, and rollback time. Separate must-match behavior from behavior you are willing to improve or change.

Parity does not require identical wording. It requires equivalent or better task outcomes inside explicit limits. A concise candidate can differ from the baseline and still pass. A fluent candidate that chooses the wrong tool, omits a required fact, or increases unsafe automation fails. This is why pairwise preference alone is not a release decision.

Inventory API keys, model identifiers, cloud variants, SDK clients, proxy routes, background jobs, batch pipelines, experiments, fallback paths, support tools, and notebooks. Provider usage exports can identify active keys and models, but code and configuration search are still necessary for rarely used paths. Link each route to an owner, workflow, traffic, data class, and deadline.

Record provider-specific semantics: system-message handling, content blocks, tool-choice modes, stop reasons, streaming events, prompt-caching hints, token counting, thinking or reasoning controls, safety responses, retry headers, and error types. The adapter may hide syntax while application behavior still relies on these details. Mark every dependency as portable, adaptable, or intentionally provider-specific.

Capture the production baseline for at least one representative period: request and token volume by route, context and output distributions, latency percentiles, errors, retries, cache behavior, tool-call frequency, escalation, user feedback, and cost per completed workflow. Preserve model and prompt versions in traces. Without a baseline, the team can prove that the candidate works but not that the migration worked.

Review the official Anthropic model lifecycle and deprecation page for the actual retirement schedule used by each deployment platform. Provider-operated and partner-operated platforms can have different dates. Treat notices as a recurring input to the model inventory, not an emergency found by a failed production request.

• Search code, configuration, jobs, notebooks, and provider usage exports
• Map every model route to owner, traffic, data, and deadline
• Record tool, streaming, structure, safety, and retry semantics
• Capture baseline quality, latency, volume, errors, and workflow cost
• Identify cloud or regional variants with separate lifecycle dates
• Confirm the previous route can remain available during canary

Sample reviewed production traces, known failures, high-value workflows, support escalations, and edge cases. Remove sensitive content or create faithful synthetic variants. Each case needs input, relevant context, required facts, prohibited claims, expected tool and argument constraints, escalation expectation, slice, and provenance. For conversational tasks, include the state that makes the current turn difficult.

from dataclasses import dataclass

@dataclass(frozen=True)
class MigrationCase:
    case_id: str
    input: dict
    required_facts: tuple[str, ...]
    prohibited_claims: tuple[str, ...]
    expected_tool: str | None
    expected_arguments: dict | None
    critical_slice: str
    source: str

case = MigrationCase(
    case_id="triage-083",
    input={"message": "Cancel the duplicate deployment and keep the newer one."},
    required_facts=("identify both deployments", "request confirmation"),
    prohibited_claims=("deployment already cancelled",),
    expected_tool="deployment.prepare_cancel",
    expected_arguments={"selection": "older"},
    critical_slice="destructive_tool",
    source="reviewed_production_trace",
)

Use deterministic checks for schema, exact fields, calculations, citations, tool names, arguments, permissions, and prohibited actions. Use rubric graders for completeness, reasoning quality, relevance, and tone. Add pairwise review for ambiguous outputs, but preserve absolute thresholds so both baseline and candidate can fail. Calibrate automated graders against domain experts and version the grader prompts.

async def compare_case(case, baseline_client, candidate_client, graders):
    baseline = await baseline_client.run(case.input)
    candidate = await candidate_client.run(case.input)

    results = {}
    for grader in graders:
        results[grader.name] = {
            "baseline": await grader.score(case, baseline),
            "candidate": await grader.score(case, candidate),
        }

    return {
        "case_id": case.case_id,
        "slice": case.critical_slice,
        "scores": results,
        "baseline_latency_ms": baseline.latency_ms,
        "candidate_latency_ms": candidate.latency_ms,
        "baseline_output_tokens": baseline.output_tokens,
        "candidate_output_tokens": candidate.output_tokens,
    }

Report pass rate by critical slice, not only overall average. Tool selection, destructive actions, refusals, long context, multilingual input, extraction, classification, code, and customer-facing answers often move differently. Establish confidence intervals or repeat unstable samples when generation remains nondeterministic. Pin serving configuration, prompt version, and dataset revision for every run.

Include latency and output length in the same case report. A candidate may match quality while producing far longer responses, causing slower tools, larger context in later turns, and higher serving load. Include retry and repair attempts in both outcome and cost. The final comparison should explain why a case changed, not merely which model received a higher score.

Shortlist models from task fit, license, weight availability, supported context, tool use, structured output, language coverage, safety behavior, quantization options, and serving maturity. Reject candidates that conflict with data, license, region, or hardware constraints before expensive prompt tuning. Record the exact weight revision, tokenizer, chat template, quantization, and inference engine.

Run a small risk-heavy subset first. If a model cannot satisfy critical tool or safety cases after a bounded prompt adaptation, stop. For survivors, run the full suite at the target context distribution and concurrency. Benchmark prefill and decode separately because long-context retrieval tasks and short interactive tasks stress serving differently.

Compare at the system level. A smaller model with retrieval, deterministic validation, and narrow tools may outperform a larger model on the product outcome. A larger model may reduce repair attempts enough to lower total latency. Evaluate candidate plus prompt, context policy, tool contract, and serving configuration as one versioned route.

Create an application-owned request and result contract that represents the features the product needs. Keep provider message formats, streaming events, token counters, errors, and tool blocks inside adapters. Callers should not branch on provider names. They should ask for a capability and receive a normalized result with enough metadata for evaluation and operations.

type ModelRequest = {
  requestId: string;
  system: string;
  messages: Array<{
    role: "user" | "assistant";
    content: string;
  }>;
  tools: Array<{
    name: string;
    description: string;
    inputSchema: Record<string, unknown>;
  }>;
  responseSchema?: Record<string, unknown>;
  maxOutputTokens: number;
  deadlineMs: number;
};

type ModelResult = {
  text: string;
  toolCalls: Array<{ name: string; arguments: unknown }>;
  finishReason: "stop" | "tool" | "length" | "blocked" | "error";
  inputTokens: number;
  outputTokens: number;
  latencyMs: number;
  providerRequestId?: string;
};

Add adapter conformance tests. Every adapter must preserve message order, system policy, tool schemas, finish reasons, deadlines, cancellation, token accounting, trace identifiers, and error classification. Test empty responses, malformed tool calls, length limits, policy blocks, connection failures, and partial streams. Normalize meaning, not every incidental field.

Avoid claiming portability through an OpenAI-compatible endpoint alone. Compatibility often covers basic request syntax while chat templates, tool serialization, JSON constraints, streaming, tokenization, and error behavior differ. Keep the adapter explicit and versioned. When the serving layer accepts extra parameters, store them in a route configuration owned by the adapter rather than leaking them through product code.

Capability detection should fail closed. If a route does not support required tools or schema constraints, reject it before request execution. Do not silently drop system instructions, tool definitions, or structured-output policy. Emit a typed error that routing can use to select an approved fallback.

Begin from the existing task contract, not the old prompt text. Identify role, policy hierarchy, evidence boundary, response structure, examples, tool rules, and escalation. Rewrite those instructions in a clear format that the candidate follows. The original prompt may contain provider-specific workarounds that confuse another model or compensate for behavior the candidate handles natively.

Verify the serving chat template. Open-source weights are trained against particular control tokens and role formats. A generic compatibility server can accept messages while applying a template that weakens system instructions or tool formatting. Pin the tokenizer and template with the model revision, and include a rendered-prompt diagnostic in non-production evaluation tooling.

Tool behavior needs focused adaptation. Make tool descriptions concise, schemas closed, enums explicit, and identifiers unambiguous. Validate arguments outside the model. Score tool choice and each critical argument separately. If the candidate frequently emits valid-looking but unsupported tools, reduce the available set by workflow or add a deterministic router before increasing prompt complexity.

async function runStructuredTask(request: ModelRequest) {
  const first = await modelRouter.generate(request);
  const parsed = parseJsonObject(first.text);
  const validation = validateAgainstSchema(parsed, request.responseSchema);

  if (validation.ok) {
    return { result: validation.value, attempts: 1 };
  }

  const repairRequest = buildRepairRequest(request, first.text, validation.errors);
  const second = await modelRouter.generate(repairRequest);
  const repaired = validateAgainstSchema(
    parseJsonObject(second.text),
    request.responseSchema,
  );

  if (!repaired.ok) {
    throw new ModelContractError("structured_output_invalid");
  }
  return { result: repaired.value, attempts: 2 };
}

Use bounded repair rather than unlimited retries. Every repair adds latency, serving load, and another chance for drift. Record the original output, validation errors, repair attempt, and final result. Set a maximum attempt count and route unresolved failures to an approved fallback or human path. The parity suite should score first-pass and repaired success separately.

Revisit context assembly. A candidate with a smaller effective context or different attention behavior may need tighter retrieval, better deduplication, or a summary stage. Measure quality across real context lengths. Do not assume that fitting inside the advertised limit means all evidence is used equally well.

Pin weights by immutable revision and verify artifact hashes. Pin the inference image, engine, tokenizer, chat template, quantization, and runtime parameters. Scan images and dependencies, restrict model artifact write access, and record provenance. A latest tag turns a controlled migration into another unannounced provider change.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference-candidate
spec:
  replicas: 2
  strategy:
    type: RollingUpdate
  template:
    spec:
      containers:
        - name: inference
          image: registry.example.com/inference:approved-digest
          args:
            - "--model=/models/approved"
            - "--served-model-name=oss-candidate"
            - "--max-model-len=16384"
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
          resources:
            limits:
              nvidia.com/gpu: "1"

Benchmark with the production distribution: prompt lengths, output lengths, concurrency, streaming, tool calls, and burst patterns. Track queue time, time to first token, decode rate, total latency, GPU utilization, memory, cache hit rate, preemption, errors, and rejected requests. Test cold starts and replica replacement. A single-user notebook benchmark does not predict tail latency under mixed load.

Define overload behavior. Bound queues, reject or shed low-priority work, and preserve deadlines. Route background extraction separately from interactive responses when their latency needs differ. Use autoscaling signals that reflect queue and token work, not only request count. Model requests vary by orders of magnitude in compute.

Protect the inference endpoint as an internal production service. Authenticate callers, enforce tenant and route policy upstream, limit payload and output sizes, restrict network access, and redact traces. Do not expose an unrestricted compatibility endpoint to general clients. Maintain a patch and vulnerability process for the serving engine, drivers, and container.

In shadow mode, the baseline serves the user while the candidate receives the same approved input asynchronously. Store both normalized results, scores, latency, token use, and route versions under one trace identifier. Do not execute candidate tool calls or side effects. Redact and retain shadow data under the same policy as production traces.

type RoutePolicy = {
  mode: "baseline" | "shadow" | "canary" | "candidate";
  canaryTenants: ReadonlySet<string>;
  canaryPercent: number;
};

function selectRoute(requestId: string, tenantId: string, policy: RoutePolicy) {
  if (policy.mode === "baseline") return "anthropic";
  if (policy.mode === "candidate") return "oss";
  if (policy.mode === "shadow") return "anthropic_with_oss_shadow";

  if (!policy.canaryTenants.has(tenantId)) return "anthropic";
  const bucket = stableBucket(requestId, 100);
  return bucket < policy.canaryPercent ? "oss" : "anthropic";
}

Sample shadow traffic across workflow, tenant, language, context length, and risk. Review disagreement clusters. Production traffic often reveals shorthand, incomplete context, rare tools, and changing source data absent from the offline suite. Convert confirmed failures into reviewed cases before changing the prompt.

Canary low-risk workflows or approved tenants after offline and shadow thresholds pass. Route deterministically so one conversation or workflow stays on one model unless an explicit fallback occurs. Record the fallback reason. Set stop conditions for critical quality failures, tool errors, schema failure, latency, overload, safety, and unexpected cost.

Expand one dimension at a time: traffic percentage, tenant set, workflow set, or risk class. Do not change model revision, prompt, serving parameters, and traffic level simultaneously. Each change should produce a comparable report and a clear rollback target.

Two weeks is realistic for a bounded set of production workflows with existing traces, an available owner, and approved infrastructure. It is not a promise to migrate every experimental job or to build a new GPU platform from nothing. Freeze the initial scope and create a follow-up inventory for lower-volume paths.

Run a daily decision review with engineering, product, and the domain owner. Inspect failed critical cases, serving limits, traffic status, and scope changes. Record decisions in the migration report. A tight schedule works because the release contract prevents open-ended prompt polishing and gives the team permission to stop a weak candidate early.

The route switch should live outside the candidate inference service and support global, workflow, and tenant rollback. Preserve the baseline adapter, credentials, prompt, and capacity through the canary period. Test rollback under load and confirm in traces that new requests use the baseline. Define what happens to in-flight streams, tool proposals, and queued jobs.

Monitor quality proxies and product outcomes: schema failure, tool-call validation, fallback, escalation, user correction, task completion, latency, queue time, throughput, GPU health, and cost per completed workflow. Keep sampled human review during early expansion. Alerts should include route version and affected slices so operators can choose targeted rollback.

Establish a model release process after migration. New weights, quantization, chat templates, engine versions, prompts, and routing policy all trigger the parity suite. Promote immutable route versions through evaluation, shadow, canary, and production. Keep recent versions available for rollback within a defined window.

Reassess capacity as traffic and context grow. Open-source economics depend on utilization and peak demand, not only average token volume. Track queueing and headroom by route. Decide which overflow behavior is approved: queue, reject, degrade to a smaller route, or fall back to a provider. Evaluate each behavior on critical workflows before it is needed.

Run a scheduled failover exercise after launch. Remove one candidate replica, exhaust the bounded queue in a staging load test, simulate an unhealthy model artifact, and verify that routing, alerts, and rollback behave as documented. Record time to detection, time to stable service, dropped work, duplicated work, and any request that crossed its deadline. Feed those results into capacity policy and the next release gate.

Keep the model inventory alive. Review provider lifecycle notices, weight revisions, serving-engine releases, hardware constraints, and license changes on a named cadence. A migration should reduce emergency dependence on one vendor, but it does not eliminate change. The durable outcome is a practiced system for comparing and promoting routes before any external deadline becomes a production incident.

• Define must-match outcomes, critical slices, and operational limits
• Inventory every model route and provider-specific semantic
• Capture the baseline on reviewed production-shaped cases
• Approve model license, weights, tokenizer, template, and runtime
• Normalize the application contract and test every adapter
• Score tool choice, arguments, schema, safety, and task quality
• Load-test real prompt, output, concurrency, and burst distributions
• Pin artifacts and route versions by immutable identifiers
• Shadow without executing candidate side effects
• Canary deterministically by approved workflow and tenant
• Drill global and targeted rollback under load
• Assign owners for quality, capacity, security, and future model releases

The migration is complete when the application can explain why the candidate is acceptable, route traffic without product code changes, operate the inference service at target load, and return to the previous route on command. Open-source control is valuable only when that control is exercised through measured release discipline.