Table of Contents
AI is everywhere.
You can use ChatGPT to write code, generate documentation, troubleshoot an error, create an image, or even build an application. But for DevOps and infrastructure engineers, there is another interesting question:
How do you actually run AI systems?
What hardware do you need?
How do you host an LLM?
Should you use a cloud GPU or your own machine?
Where do Kubernetes, Docker, networking, storage, monitoring, and security fit into the picture?
If you are a DevOps engineer who wants to move into AI, you do not necessarily need to become a machine-learning researcher.
You can start from what you already know.
Your existing knowledge of Linux, containers, Kubernetes, networking, cloud platforms, Infrastructure as Code, observability and automation gives you a strong foundation for AI infrastructure and AI engineering.
This guide explains how I would approach learning AI from an infrastructure background, which books are worth reading, and how to build your own AI environment.
AI Is Not Just About Training Models
When people talk about AI, they often immediately think about training huge models.
That is only one part of the ecosystem.
Modern AI systems can involve:
- Large Language Models (LLMs)
- GPU infrastructure
- Model serving
- APIs
- Vector databases
- RAG pipelines
- AI agents
- Model monitoring
- Data pipelines
- Kubernetes
- Containers
- Cloud infrastructure
- Security
- Cost management
This is where DevOps becomes particularly interesting.
A model might be impressive, but somebody still has to answer questions such as:
Where does it run?
How is it deployed?
How do we monitor it?
What happens when the GPU runs out of memory?
How do we update the model without breaking the application?
How do we secure access to it?
How much does it cost?
These are infrastructure questions.
And that means DevOps engineers already have many of the skills needed to work with AI systems.
What Should a DevOps Engineer Learn First?
If you already know Linux, Docker, Kubernetes and cloud infrastructure, I would not recommend starting by spending six months learning advanced mathematics.
Instead, learn AI progressively.
A practical path looks like this:
Python
↓
Machine Learning fundamentals
↓
Deep Learning fundamentals
↓
LLMs and Transformers
↓
AI Applications
↓
Model Serving
↓
AI Infrastructure
↓
MLOps / LLMOps
The goal is not necessarily to become a research scientist.
The goal is to understand enough about AI to build, deploy and operate AI systems.
1. Start With the Fundamentals
Before jumping directly into LLMs, understand the basic vocabulary.
You should know what these terms mean:
- Dataset
- Feature
- Label
- Training
- Validation
- Inference
- Model
- Parameters
- Loss function
- Overfitting
- Neural network
- Embeddings
- Transformers
- Attention
- Fine-tuning
You do not need to become an expert immediately.
You just need enough understanding to answer:
What is the model actually doing?
For a quick introduction, The Hundred-Page Machine Learning Book by Andriy Burkov is a useful starting point. It gives you a relatively compact overview of machine-learning concepts without forcing you immediately into a massive textbook.
If you prefer a more structured statistical foundation, An Introduction to Statistical Learning is another excellent option and is available with a Python-focused edition.
2. The First Serious Book: Hands-On Machine Learning
If you already know Python and want to actually build things, Hands-On Machine Learning with Scikit-Learn and PyTorch by Aurélien Géron is one of the books I would put near the top of the list.
The newer edition moves toward the modern PyTorch and Hugging Face ecosystem and covers practical machine learning and deep learning concepts.
This is important because AI is not learned particularly well by reading definitions.
You need to run code.
You should be comfortable with a workflow such as:
Dataset
↓
Preprocessing
↓
Training
↓
Evaluation
↓
Model
↓
Inference
Once you understand that lifecycle, the infrastructure side becomes much easier to understand.
3. Learn How LLMs Actually Work
Once you understand basic machine learning and neural networks, move into Large Language Models.
This is where things become much more interesting for infrastructure engineers.
You will encounter concepts such as:
- Transformers
- Tokenization
- Context windows
- Attention
- Quantization
- Embeddings
- Fine-tuning
- LoRA
- RAG
- Inference
- KV cache
- Model serving
A particularly interesting hands-on direction is building an LLM yourself rather than only calling an API.
Sebastian Raschka’s Build a Large Language Model (From Scratch) is a good example of this approach because it focuses on understanding the components behind modern language models rather than treating the model as a black box.
The important thing is not that you become capable of training the next frontier model.
It is that you understand what your infrastructure is actually running.
4. Then Learn AI Engineering
Once you understand models, move one level up.
Instead of asking:
How does a transformer work?
Start asking:
How do I build an application around a transformer?
This introduces a completely different set of problems.
For example:
User
↓
Web Application
↓
API
↓
LLM
↓
Vector Database
↓
Documents
Now you need to think about:
- latency
- caching
- authentication
- storage
- observability
- retries
- rate limiting
- model selection
- cost
- security
- scalability
This is where books such as Designing Machine Learning Systems by Chip Huyen become valuable.
The focus is not simply on training models. It is on the systems surrounding machine learning and how those systems behave in production.
For a DevOps engineer, this is probably one of the most interesting parts of the AI ecosystem.
5. You Don’t Need a $10,000 GPU Server to Start
One of the biggest misconceptions about learning AI is that you immediately need expensive hardware.
You don’t.
For learning, you can start with:
- your existing laptop
- CPU inference
- a small GPU
- cloud GPU instances
- rented GPU servers
- a used workstation
- a local mini PC
The hardware depends heavily on what you want to do.
There is a huge difference between:
Running a 3B or 7B quantized model
and
training a large model from scratch.
For most people learning AI infrastructure, the first goal should be inference, not training.
6. Running Your First Local LLM
One of the easiest ways to start experimenting with local AI is Ollama.
The basic architecture can be extremely simple:
Your Computer
|
+-- Ollama
|
+-- LLM
You can then put an application in front of it:
Browser
↓
Open WebUI
↓
Ollama
↓
Local LLM
This is already a real AI infrastructure project.
You have:
- a service
- a model
- persistent storage
- networking
- an API
- authentication
- monitoring opportunities
- resource constraints
And most importantly:
you can break it and fix it yourself.
That is one of the best ways to learn.
7. From Docker to Kubernetes
If you already know Docker and Kubernetes, the next step is obvious.
Instead of running your AI workload directly on a machine, you can start treating the model server like any other workload.
For example:
Kubernetes
│
├── AI namespace
│
├── Model Server
│
├── Open WebUI
│
├── Vector Database
│
├── API
│
└── Monitoring
A more advanced setup could look like:
Internet
│
▼
Ingress
│
▼
AI Application
│
┌─────────┴─────────┐
▼ ▼
Vector DB LLM API
│
▼
Model Server
│
▼
GPU
This is where your existing Kubernetes knowledge becomes useful.
You already understand:
- Deployments
- Services
- Ingress
- Persistent Volumes
- Secrets
- Resource requests
- Resource limits
- Scheduling
- Networking
- Monitoring
The new part is understanding the workload.
8. GPU Infrastructure Changes the Game
CPU inference can be useful for small models, but GPUs become extremely important as models become larger and inference workloads increase.
Now you need to understand concepts that are less common in traditional web applications.
For example:
- GPU VRAM
- CUDA
- NVIDIA drivers
- GPU utilization
- GPU memory fragmentation
- quantization
- batching
- model loading
- inference throughput
- tokens per second
This changes how you think about infrastructure.
With a traditional application you might worry primarily about:
CPU
RAM
Disk
Network
With AI workloads, you add:
GPU
VRAM
CUDA
Model size
Context length
Batch size
Inference throughput
The GPU is no longer simply another resource.
It can become the most important resource in the entire system.
9. What About vLLM?
Once Ollama becomes too simple for what you want to experiment with, look at dedicated model-serving systems such as vLLM.
vLLM is designed specifically for serving language models and can be deployed on Kubernetes, including GPU-based deployments. Its current documentation also provides a production stack with Helm-based deployment and observability integrations.
A simplified architecture could look like:
Kubernetes
│
┌───────┴───────┐
│ │
API/API Monitoring
│ │
▼ ▼
vLLM Grafana
│
▼
GPU
│
▼
LLM
This is where AI infrastructure starts becoming very similar to the production infrastructure you already know.
10. Don’t Forget Storage
AI systems can consume a surprising amount of storage.
You might need space for:
- model weights
- container images
- embeddings
- datasets
- vector databases
- caches
- logs
- generated content
For example, downloading several different models can quickly consume hundreds of gigabytes.
A serious homelab AI environment should therefore have a storage strategy.
Something like:
AI Node
│
├── NVMe
│ ├── Models
│ ├── Containers
│ └── Cache
│
└── NAS
├── Datasets
├── Backups
└── Generated content
This is another area where traditional infrastructure knowledge transfers directly into AI.
11. Observability Is Still Important
Running an AI model without monitoring is a great way to discover problems after users start complaining.
You should monitor things such as:
Infrastructure
- CPU usage
- RAM
- GPU utilization
- GPU VRAM
- disk usage
- network traffic
- temperature
Application
- request latency
- requests per second
- error rate
- queue length
- tokens per second
Model
- inference duration
- input tokens
- output tokens
- context size
- model loading time
The AI stack needs observability just like any other production system.
Your existing knowledge of Prometheus, Grafana, logs and alerting can therefore be reused.
12. Security Becomes Even More Important
Local AI does not automatically mean secure AI.
If you expose an LLM API directly to the internet, you have created another service that needs protection.
Think about:
- authentication
- TLS
- network segmentation
- API rate limiting
- secrets management
- container security
- model supply chain
- prompt injection
- data leakage
- access control
For a homelab, you might have:
Internet
│
▼
Firewall
│
▼
Reverse Proxy
│
▼
Authentication
│
▼
AI Application
│
▼
Private LLM
Do not expose your model server directly just because it works on your LAN.
13. Cloud or Homelab?
Eventually you will probably ask:
Should I run AI locally or in the cloud?
The answer is:
Use both.
A homelab is excellent for learning.
You control everything:
- hardware
- networking
- storage
- Kubernetes
- GPU
- operating system
- model server
You can experiment without worrying about an hourly GPU bill.
The cloud is useful when you need:
- larger GPUs
- more VRAM
- temporary capacity
- production workloads
- distributed inference
- large-scale training
A good learning strategy is:
Local machine
↓
Homelab
↓
Cloud GPU
↓
Production AI infrastructure
Don’t start with production-scale infrastructure.
Start by getting one model running.
14. A Practical AI Infrastructure Learning Path
If I were starting today as a DevOps engineer, I would follow something like this.
Phase 1: AI fundamentals
Learn:
- Python
- NumPy
- basic statistics
- machine learning concepts
- neural networks
Read: The Hundred-Page Machine Learning Book then move to Hands-On Machine Learning
Phase 2: Deep Learning
Learn:
- PyTorch
- tensors
- neural networks
- training loops
- GPUs
- transformers
Build:
Small dataset
↓
PyTorch model
↓
Training
↓
Evaluation
↓
InferencePhase 3: LLMs
Learn:
- tokenization
- embeddings
- transformers
- attention
- context windows
- quantization
- fine-tuning
- RAG
Build:
Documents
↓
Embeddings
↓
Vector DB
↓
Retriever
↓
LLM
↓
Answer
Phase 4: AI Infrastructure
Now bring your DevOps skills back into the picture.
Learn:
- Docker
- GPU containers
- NVIDIA Container Toolkit
- Kubernetes GPU scheduling
- persistent storage
- model serving
- vLLM
- Ollama
- monitoring
Build:
Kubernetes
│
├── LLM
├── API
├── Vector DB
├── Web UI
└── Monitoring
Phase 5: Production AI
Finally learn:
- MLOps
- LLMOps
- model versioning
- CI/CD
- observability
- security
- cost optimization
- autoscaling
- high availability
At this point, you’re no longer just “learning AI.”
You’re learning how to operate AI systems.
15. My Recommended AI Bookshelf
You don’t need to buy 20 books.
Start with a few books that cover different levels.
| Book | Best for |
|---|---|
| The Hundred-Page Machine Learning Book | Understanding the big picture |
| Hands-On Machine Learning | Building practical ML systems |
| Build a Large Language Model From Scratch | Understanding LLM internals |
| Designing Machine Learning Systems | Production ML architecture |
| Designing Data-Intensive Applications | Distributed systems and data infrastructure |
| Deep Learning | Advanced deep-learning theory |
The last two are particularly interesting for infrastructure engineers.
AI systems are still distributed systems.
They still have:
- databases
- queues
- APIs
- storage
- networks
- compute
- failures
- latency
- consistency problems
The models are different.
The engineering problems are often surprisingly familiar.
16. What I Would Build as a Homelab Project
If you want to learn AI and DevOps at the same time, don’t just read books.
Build something.
For example:
Home Network
│
▼
Traefik
│
┌────────────┴────────────┐
│ │
▼ ▼
Open WebUI AI API
│ │
└────────────┬────────────┘
▼
Ollama
│
▼
LLM
│
┌──────┴──────┐
▼ ▼
Vector DB Storage
│
▼
Documents
Then add:
Prometheus
↓
Grafana
↓
GPU / CPU / RAM / latency metrics
Then add:
Git
↓
CI/CD
↓
Docker image
↓
Kubernetes deployment
And finally:
Terraform
↓
Infrastructure
↓
Kubernetes
↓
AI platform
Now you have a real project that combines DevOps + Cloud + Kubernetes + AI.
That is much more valuable than simply saying:
I have studied AI
17. AI Infrastructure Is a Natural Next Step for DevOps
You don’t have to abandon DevOps to move into AI.
In many cases, the opposite is true.
AI needs infrastructure engineers.
Someone needs to build the platforms that allow developers and data scientists to deploy models reliably.
Someone needs to manage GPU clusters.
Someone needs to automate deployments.
Someone needs to monitor inference workloads.
Someone needs to secure the infrastructure.
Someone needs to control costs.
Someone needs to make the whole system boring and predictable.
That sounds a lot like DevOps.
The difference is that instead of deploying a traditional web application, you might be deploying a model that consumes 20 GB of VRAM and serves thousands of inference requests.
The infrastructure principles remain surprisingly familiar.
Where to Go Next
If you are already learning DevOps, Kubernetes and cloud infrastructure, don’t treat AI as a completely separate career.
Treat it as another workload.
Start small:
- Learn the machine-learning fundamentals.
- Build a few models with Python.
- Understand transformers and LLMs.
- Run an LLM locally.
- Put an application in front of it.
- Containerize it.
- Deploy it to Kubernetes.
- Add GPU acceleration.
- Add monitoring.
- Automate everything.
That path will teach you far more than simply collecting AI certificates.
And if you’re already working with Kubernetes, don’t throw away what you’ve learned. Kubernetes remains a valuable infrastructure skill, particularly when you move toward platform engineering and AI workloads. For a deeper look at that topic, check out Yes, Learning Kubernetes & DevOps Is Still Valuable.
For the cloud side, you can also explore the site’s AWS articles, and when you’re ready to start building instead of reading, the DevOps Tutorials section contains practical infrastructure projects.
The future of AI will not only be built by people who train models. It will also be built by engineers who know how to run them.
