> ## Documentation Index > Fetch the complete documentation index at: https://tensorfuse.io/docs/llms.txt > Use this file to discover all available pages before exploring further. # AMD GPU inference: understand ROCm and vLLM > Evaluate AMD Instinct GPUs for LLM inference with ROCm and vLLM. Understand kernel compatibility, quantization, GPU memory, and multi-GPU scaling. Moving a model from an NVIDIA GPU to an AMD GPU can preserve the application API while changing the entire execution stack underneath it. For AMD inference, choose a compatible ROCm driver, runtime, serving image, and model implementation together. Then compare correctness and latency on the exact GPU generation you plan to use. ## What changes below the API? AMD Instinct devices are GPUs. ROCm supplies the software stack used to run supported machine-learning workloads on them. AMD documents vLLM serving with prebuilt ROCm images and installation paths for selected hardware in its [vLLM setup guide](https://rocm.docs.amd.com/projects/ai-ecosystem/en/latest/inference/vllm.html). Your client can continue sending chat requests, but a CUDA container does not become an AMD container by changing a device label. Compiled extensions, attention kernels, collective communication, and quantization implementations need compatible AMD paths. Check the coordinated firmware, driver, OS, and runtime requirements in the [ROCm compatibility matrix](https://rocm.docs.amd.com/en/latest/compatibility/compatibility-matrix.html). ## Why doesn't a larger memory number settle the comparison? More device memory can hold larger weights or more request state. The application still depends on memory bandwidth, effective compute utilization, and kernel efficiency. A model that fits on one large-memory GPU may avoid some communication, while a multi-GPU layout may improve throughput for another workload. Treat an AMD GPU generation as a specific target. AMD's [inference performance recipes](https://rocm.docs.amd.com/en/docs-7.1.1/how-to/rocm-for-ai/inference/benchmark-docker/vllm.html) associate model configurations with particular images and devices. A result for MI300X or MI350X does not establish performance on a Radeon desktop GPU, even when both use ROCm. ## Trace a migration in four steps 1. **Confirm the model path.** Check the exact checkpoint, quantization format, context length, attention backend, and any custom operators. Test tool calls and structured output before benchmarking. 2. **Use a matched software stack.** Start from an upstream recipe for the GPU and record its image digest. Avoid mixing unrelated host and container versions. 3. **Measure one configuration.** Separate startup, prefill, and decode. Replay realistic prompts, including cache hits and misses, at increasing request rates. 4. **Scale deliberately.** Record device connectivity and communication settings. Tensor parallelism, expert parallelism, and extra replicas solve different capacity problems. AMD's [vLLM optimization guide](https://rocm.docs.amd.com/en/docs-7.2.4/how-to/rocm-for-ai/inference-optimization/vllm-optimization.html) describes AITER kernels and ROCm-specific tuning. Apply such settings to the versions and hardware they document; benchmark each change instead of copying a collection of flags. ## When is AMD worth evaluating? An evaluation is useful when you have access to AMD capacity and a supported model path, especially if allocation, memory, or measured cost limits your current deployment. Compare at the same latency and quality target using the [benchmarking guide](/docs/guides/inference/benchmarking), with migration effort included in the decision. This explainer does not add an AMD `gpu_type` to Tensorfuse. The [configuration reference](/docs/concepts/configuration) remains the source for documented deployment options. ## Related guides Evaluate software compatibility alongside hardware. Place AMD GPUs alongside cloud ASICs and specialized systems. ## Put this into practice Use the [AMD vLLM how-to](/docs/guides/inference/how-to/amd-vllm) for host checks, a ROCm container, a local chat endpoint, and troubleshooting.