InferenceSamagra Sharma
DeepSeek V4.1 Flash: why it matters and why inference is cheaper
A coding agent can read the same files many times before fixing one bug. Follow how DeepSeek reduces that work and the memory needed to keep the conversation going.InferenceTutorialSamagra Sharma
Self-host DeepSeek V4.1 Flash: hardware, cloud setup, and costs
A rented GPU keeps costing money while your agent waits for tests. Start with one request, build up to a working cloud deployment, and compare the full bill with the API.How-to guidesInference
How to make LLM inference cheaper
Start with a measured baseline, then follow practical setup guides for the runtime or hardware that fits your workload:- Quantized inference with vLLM
- AMD GPUs with ROCm and vLLM
- AWS Inferentia and Trainium with Neuron
- Google TPU inference with vLLM
- Apple silicon inference with MLX
- CPU inference with llama.cpp
- Compare Groq, Cerebras, and SambaNova APIs
InferenceSamagra Sharma
How to calculate LLM inference cost per successful request
A worked comparison using accepted requests, latency requirements, and the full serving bill.InferenceSamagra Sharma
When does GPU scale-to-zero save money?
Calculate the break-even idle gap and check whether cold-start latency fits your application.Engineering
CloudFormation won’t fix your life but it can fix your infra
A look into how efficiently managing your cloud infrastructure will definitely make your life easier, even though it may not fix itEngineering
Lazy loading isn’t the magic pill to fix AI Inference
In this post, we look into how lazy loading the containers filesystem, while being beneficial, doesn’t necessarily cause the speedups that might be expected from themEngineering
Reducing GPU Cold Start Time when using vLLM
Learn how to reduce the cold start time of a GPU based application when using vLLM.Learning
Lazy loading GPU containers: why startup is not readiness
Understand on-demand image access and measure its effect on readiness and the first inference request.Learning
SLMs are the Future of Agentic AI
Learn how Small Language Models (SLMs) are powerful enough and practically better for building AI agents compared to LLMs. In this post, we’ll explore the practical aspects of the paper and discuss its relevance for your AI applications.Engineering
Handling Unhealthy Nodes in EKS
Learn how to monitor, alert, and automatically heal EKS nodes using CloudWatch, Lambda, and Karpenter’s Node Repair — complete with pros, cons, and code examples.Engineering
Understanding Multi GPU Communication and Nvidia NCCL for finetuning models
In this post, we’ll break down what NCCL does, why it’s critical for multi-GPU training, and how to tackle one of its common challenges – the dreaded “watchdog timeout” error.Engineering
Selecting Ideal EC2 Instances for GPU Workloads on AWS
Choosing the right EC2 pricing model for your AI/ML workloads can make or break your cloud budget. Machine learning tasks, whether training large models or serving real-time predictions, often require significant computing resources.Learning
Boost LLM Throughput: vLLM vs. Sglang and Other Serving Framework
Serving open-source Large Language Models (LLMs) efficiently requires optimizing across hardware, software, and inference techniques.Learning
Better and Cost Effective Alternative to AWS Sagemaker: Tensorfuse
Discover why Tensorfuse is a better alternative to AWS Sagemaker for AI inference tasks.Learning
Why do GPU Containers have long Cold Starts?
Learn how to minimize cold start times in GPU applications by understanding container runtime, image loading, and lazy loading technique. Discover the limitations of using a Kubernetes and Docker-based approach for GPU images compared to CPU imagesLearning
What is serverless GPU computing?
Lately, serverless GPUs have been gaining a lot of traction among machine learning engineers. In this blog, we’ll dive into what serverless computing is all about and trace the journey that brought us here.Tutorial
Increase GPU Quota on AWS: A Comprehensive Guide
Request a higher AWS GPU quota and understand the instance families involved.Tutorial
