You do not need to sound complicated. Move from the problem to the big
picture, then prove one hard part works. This page gives you the
order, the questions, the math, and the exact words to use.
✓Shared file: several machines require filesystem
behavior.
×Do not put large files directly in normal database rows by
default.
Example: EBS for Postgres; S3 for knowledge PDFs;
NFS for legacy tools.
Consistency vs availability
✓Consistent: reject during a network split rather than
create conflicting truth.
✓Available: accept work, then detect and repair
conflicts.
×CAP matters during a network partition—not every normal
request.
Example: refund ledger favors consistency; “user
is typing” favors availability.
Session vs token vs API key
✓Session: browser app and immediate revocation.
✓Token: mobile/multiple APIs. API key: server
integration.
×A valid token proves identity, not access to every
record.
Example: HttpOnly session cookie; short mobile
JWT; hashed partner key.
Text vs vector search
✓Text: exact names, IDs, filters, and keyword
relevance.
✓Vector: similar meaning when wording differs.
Hybrid: both.
×Vectors do not replace permissions, filters, or the official
database.
Example: order ID via text; “item arrived broken”
via vector; RAG uses hybrid.
OpenSearchpgvectorQdrant
Prompt vs RAG vs tuning
✓Prompt: instructions. RAG: changing facts and
citations.
✓Fine-tune: repeated behavior. Tool: live action
or data.
×Do not fine-tune to memorize facts that change
frequently.
Example: RAG for policy; tool for order status;
human for high-risk refund.
01
Clarify the problem.
Minutes 0–5
Goal: Know what you are building before choosing
technology.
Start by saying
“Let me restate the problem. We are building ___ for ___ so they can
___.”
Who uses it?
Regular users
Administrators
Human reviewers
Other systems
Background jobs
What do they do?
Create or upload
Read or search
Update or delete
Receive notifications
Review or approve
What is in version one?
“Which three or four features should I focus on?”
Say what you will not design: payments, login, mobile apps,
advanced reports, or internal tools.
How big?
Daily users
Actions per user
Busy-hour multiplier
Expected growth
Records or files
How fast and reliable?
Which actions must feel immediate?
Which can run later?
May acknowledged data be lost?
Must another region take over?
How fresh?
“Must users see an update immediately, or is a short delay okay?”
This choice changes how much coordination your system needs.
What if the answer is wrong?
Is a missed result worse than a wrong one?
Can we reverse the decision?
Must a person approve it?
Is the data sensitive?
Personal, medical, or financial?
Keep customers separate?
Need an audit history?
Must data stay in one country?
What else must we call?
Payment or email provider
GitHub
Hospital or government system
AI model provider
💬
Running example used throughout the guide
LLM-powered customer-support bot
Functional requirements · what it does
Keep context across a conversation.
Answer using company knowledge and show sources.
Check orders and request refunds through approved APIs.
Send uncertain or sensitive cases to a human with the full
conversation.
Collect a helpful / not-helpful rating.
Non-functional requirements · how well it works
50 million daily users; five messages each day.
95% of normal answers begin within three seconds.
99.9% monthly availability.
Never invent a completed refund or expose another customer’s
data.
Encrypt conversations; support deletion and audit.
Limit model cost and continue with search or human support
when models fail.
Outside version one: voice calls, training our own
foundation model, every language, agent workforce scheduling, and
fully automatic high-value refunds.
End Step 1 by saying
“We are designing a support bot that answers grounded questions,
performs a small set of safe actions, and hands uncertain cases to
people. We expect 50 million daily users and a three-second
response target. A wrong financial action is worse than a slow
answer, so high-risk actions require deterministic checks or human
approval.”
∑
Do only useful math.
Still inside minutes 0–5
Rule: State an assumption, show the formula, round
the answer, then say what it changes.
requests/sec = daily users × actions/day ÷ 86,400
Sizes the API and database.
peak requests/sec = average × 3 to 10
Capacity must handle the busy period.
model calls/sec = peak messages/sec × model-call %
If 70% need a model, the peak inference path is about
20,000 calls/sec.
Storage and network
250M messages/day × 6KB ≈ 1.5TB/day
At peak: about 58MB/sec incoming and
116MB/sec outgoing.
Therefore we need horizontal API scaling, separate model
capacity, retention rules, caching, and strict cost limits.
Important: Do not assume every daily user sends a
message in the same second. Model-server capacity also depends on
model size, token count, batching, and hardware—measure it rather
than quoting one universal requests-per-second number.
After the math, say
“This tells me ___. Therefore, I would ___.”
Finish this step
“I’ll focus on ___. Peak traffic is about ___. Writes must ___. ___
may be slightly delayed. I’ll leave ___ outside the scope.”
02
Draw the big picture.
Minutes 5–12
Goal: Use a few large boxes. Add a box only when it
solves a requirement.
Customer chat
API gateway
Conversation service
AI orchestrator
Knowledge search
Model + safe tools
Answer or human handoff
1. Users and outside systems
Draw who starts work and the providers your system calls.
2. Entry point
Receive the request, check identity, apply limits, and route it.
3. Application
Start with one application. Split a responsibility only if it must
scale, deploy, or fail separately.
4. Main database
Store users, conversations, messages, feedback, tool-call status,
and human-handoff state.
5. Object storage
Store knowledge documents, attachments, exports, and large audit
evidence.
6. Queue + workers
Index knowledge, process feedback, create analytics, and retry
non-interactive work.
7. Cache
Keep hot conversation context and safe repeated answers. Never
cache private answers under a shared key.
8. Knowledge search
Combine keyword and vector search, filter by tenant and
permission, and return source passages.
9. Outside services
Call model providers, order/refund APIs, and the human-agent
platform through narrow, approved interfaces.
Quick architecture choices
Choose the simpler option until a requirement forces the other
Choice
Use it when
Cost
One application
Parts can scale and release together
Less freedom later
Separate services
Parts must scale, release, or fail alone
More network and operational work
Direct call
The work is quick and the user needs the result now
Caller waits; failures spread
Queue
The work is slow or can finish later
Delayed result; duplicates are possible
Redis cache
Many requests read the same data
May be stale; must be rebuildable
Walk through one normal request
“The customer sends a message. The API checks identity and rate
limits, then saves the message. The orchestrator loads recent
conversation context, searches allowed knowledge, and decides
whether to answer, call a safe business API, or hand off to a
person. It saves the answer, citations, model version, and tool
result before streaming the response.”
03
Design the main APIs.
Minutes 12–15
Goal: Show how people and systems use your design.
Support-bot APIs
POST /conversations
POST /conversations/{id}/messages
GET /conversations/{id}/messages?cursor=
POST /conversations/{id}/handoffs
POST /messages/{id}/feedback
POST /knowledge-documents
For each important API
Who may call it?
What does it receive?
What does it return?
Does it finish now or stream?
Can the caller safely try again?
What errors can occur?
Large conversations
Use cursor pagination. Send only recent messages and a verified
summary to the model.
Repeated messages and actions
Use a client message ID and a separate retry key for refund or
cancellation requests.
Competing updates
Use an expected conversation or handoff version so two workers
cannot both publish conflicting state.
Response choice
Should the API finish the work now?
Style
Best when
Trade-off
Return result
Work is fast and predictable
Request stays open
Return job ID
Work is slow or failure-prone
Client must check status
04
Define the data.
Minutes 15–18
Goal: Explain what the system must remember and how
it will read it.
Choose a database from the main access pattern
Default: if none of the special patterns clearly
dominates, start with PostgreSQL. “NoSQL” is a family of different
databases, not one alternative.
ID · tenant ID · user ID · state · version · timestamps ·
model/prompt version · relationships
Common reads
Recent messages by conversation · document chunks by search · open
handoffs by agent · tool result by retry key
Common updates
Append message · save citations · record tool outcome · request
handoff · rate an answer
Choose storage from those reads and updates
“I would keep conversations, messages, actions, and handoffs in a
durable main database. The knowledge search and cache are
rebuildable copies, not the official record.”
Separate the official data from useful copies
Official state
→
Main database
Searchable copy
→
Search storage
Temporary popular copy
→
Cache
Delayed reporting copy
→
Analytics storage
Say which copies may be slightly old. If two copies disagree, name
which one wins.
Quick database choices
Official database · SQL and NoSQL families
Type
Use it when
System example
Main cost
SQL Postgres · MySQL
Joins, transactions, strict rules, or queries may change
Payments, orders, inventory, users, permissions
Very large write scaling needs planning
Key-value DynamoDB
You know the key and need predictable massive scale
Sessions, carts, request state, game state
Ad-hoc queries and joins are difficult
Document MongoDB · Firestore
One flexible object is usually read and written together
Product catalogs, profiles, content blocks
Cross-document rules and joins become harder
Wide-column Cassandra · Bigtable
Huge time-ordered writes with known partition queries
Telemetry, event history, message timelines
Data model follows queries; limited transactions
Graph Neo4j · Neptune
Multi-hop relationships are the main question
Fraud rings, social links, dependency graphs
Another specialized system to operate
Time series Timescale · InfluxDB
Time-window aggregation and retention dominate
Metrics, sensors, prices
Not a general transaction database
Redis
Temporary key-based state needs extremely low latency
Cache, rate limits, locks, short sessions
RAM cost and durability risk; rarely official truth
Search and derived copies
Choice
Use it when
System example
Main cost
Postgres search
Simple text search on modest data
Admin search and small catalogs
Limited ranking and typo handling
OpenSearch
Full text, filters, facets, typo tolerance, logs
E-commerce search and log exploration
Another copy that can lag
PostGIS
Distance, nearby, and area queries
Delivery, maps, local businesses
Specialized for location
pgvector
Meaning search while PostgreSQL is still enough
Small-to-medium RAG knowledge base
Shares database CPU and memory
Vector service
Vector search is huge or operationally independent
Large recommendation or RAG corpus
Synchronization, vendor, and operations cost
Hybrid search
Exact words and meaning both matter
Support: order codes plus natural-language issues
Two scores must be combined and evaluated
Default: PostgreSQL holds the official data. Redis is
usually a rebuildable cache. Add a separate search database only when
PostgreSQL search is not enough.
Quick AI choices
Two definitions
An embedding is a list of numbers that represents meaning. A vector
database finds items with similar embeddings; it does not replace
the official database.
Vector database choices
Choice
Pick it when
Main cost
pgvector
PostgreSQL already exists; start simple
Shares resources with normal database work
Pinecone
Want a managed service with little operations work
Vendor dependency and recurring cost
Qdrant
Want focused open-source vector search
Another database to run and synchronize
Weaviate
Want objects, vectors, and AI integrations together
More concepts and moving parts
Milvus
Vector search is central and extremely large
More infrastructure and operational work
Redis
Already use Redis and need very low latency
Memory cost; not the official document store
OpenSearch
Need keywords, filters, and vectors together
Heavy if vector search is the only need
How to improve an AI answer
Choice
Pick it when
Watch out for
Prompt only
Task is simple and facts are already supplied
Less control and grounding
Retrieval (RAG)
Facts change or answers need citations
Bad retrieval produces bad context
Fine-tuning
Need repeated behavior, style, or format
Needs training data and release testing
Human review
A wrong answer has a high cost
Slower and limited by reviewer capacity
Model selection
Choice
Pick it when
Watch out for
One model
Start simple; one quality level is enough
May overpay for easy work
Model routing
Tasks need different cost, speed, or quality
Every route needs separate evaluation
Fallback model
Availability matters more than identical output
Behavior and quality can change
AI output handling
Choice
Pick it when
Watch out for
Save evidence
Need audit, debugging, or review
Storage and privacy cost
Cache output
Same safe input repeats often
Stale or sensitive answers may be reused
Generate again
Freshness matters and variation is acceptable
Higher cost and different answers
Strong default: Start without vectors. Add pgvector
when meaning-based search clearly helps. Use retrieval for changing
facts, fine-tuning for repeated behavior, and human approval when a
wrong result is expensive.
05
Follow one request.
Minutes 18–21
Goal: Prove the design works from beginning to end.
1. Receive
Customer asks, “Where is order 123?” with a stable client message
ID.
2. Check and save
The API verifies identity, tenant, rate limit, and saves the
message before expensive work.
3. Build context
Load recent messages, a short verified summary, and allowed
knowledge passages.
4. Choose a path
The orchestrator selects a cached answer, knowledge answer,
approved tool call, or human handoff.
5. Save and respond
Save the answer, citations, model version, and tool outcome, then
stream text to the customer.
6. Continue later
Send feedback, analytics, evaluation, and knowledge-index work
through queues.
Show the state changes
Received→Searching→Answering or calling tool→Answered or handed off
06
Go deep on one hard part.
Minutes 21–25
Goal: Show what makes this system different from a
basic web app.
Use the same four-part story every time
State the problem.
Give a real example.
Explain the solution.
Explain what the solution costs or makes harder.
Grounded answers
Search only allowed company documents, give the model short
passages, require citations, and refuse when evidence is weak.
Safe actions
The model proposes check_order or
request_refund. Server code validates arguments,
permission, amount, and retry key before calling the real API.
Human handoff
Use low evidence, repeated failure, user request, or high-risk
intent as handoff signals. Send a conversation summary with source
links.
Conversation context
Keep recent messages exactly. Summarize older messages, but store
the originals so the summary can be checked and rebuilt.
What to say
“The hard part is preventing a fluent answer from becoming an
unsupported fact or action. The model proposes; trusted code
checks and commits.”
Freshness choice
How quickly must copies agree?
Choice
Use it for
Cost
Immediate
Money, ownership, approvals
More waiting and coordination
Short delay
Search, analytics, notifications
Users may briefly see old data
07
Break the design.
Minutes 25–27
Goal: Show the system stays safe when normal
assumptions fail.
Support-bot failure
Safe response
The client sends one message twice
Deduplicate by tenant, conversation, and client message ID;
return the saved result.
The model times out
Retry only within the latency budget, try an evaluated
fallback, or hand off. Do not invent an answer.
Knowledge search returns weak evidence
Ask a clarifying question or hand off instead of answering
confidently.
A refund succeeds but its response times out
Look up the original refund request ID before retrying so the
customer is not refunded twice.
Two workers answer the same message
Use the expected message/conversation version; only one answer
becomes official.
A document tells the model to ignore policy
Treat retrieved text as untrusted data. Tools and permissions
are enforced outside the model.
The model provider is overloaded
Apply per-tenant limits, use cache or smaller models, shorten
context, and preserve human-support capacity.
A user’s access was removed
Check current permission during retrieval and tool execution,
not only when the session began.
Retry choice
Retries help only when the operation is safe to repeat
Choice
Use it when
Risk
Retry automatically
Failure is temporary and duplicates are blocked
Can overload a sick system
Stop and review
Result is unknown or failure repeats
Slower; needs a person
For every failure, answer
“How do we detect it? Is retrying safe? Where is progress saved?
When do we stop retrying? When does a person step in?”
08
Secure it.
Minutes 27–28
Goal: Protect people, customers, and sensitive
information.
Important sentence
“The server finds the customer from the signed-in user. It never
trusts a customer ID from the browser without checking it.”
09
Explain growth and operation.
Minutes 28–29
Goal: Say how the system grows, how you know it is
healthy, and how you release safely.
Grow
Add stateless conversation-service instances
Scale model capacity separately
Route easy work to smaller models
Cache only safe repeated answers
Limit requests by user and tenant
Protect human-handoff capacity
Measure
Time to first word and complete answer
Grounded-answer and citation accuracy
Tool success and duplicate-action rate
Handoff and repeat-contact rate
Model/provider errors
Tokens and cost per resolved conversation
Release
Test saved real questions and attacks.
Run the new path without showing answers.
Compare quality, cost, and handoffs.
Enable employees or a small customer group.
Increase traffic by tenant or intent.
Keep the old prompt/model and human fallback.
Database growth choices
Scale in this order when measurements justify it
Choice
Helps with
Cost
Bigger server
Early read and write growth
Has a limit
Read copies
Heavy reads
May briefly return old data
Split data
Writes or data exceed one database
Harder queries and operations
10
Summarize.
Minute 29–30
Goal: End with a short answer that proves the design
is complete.
Your closing template
“To summarize, we are building ___ for ___. The
API handles ___. The official state is stored in
___. Slow work goes through ___ and is handled by
___. The main safety mechanism is ___. The system
scales by ___. The main trade-off is ___, which is
acceptable because ___.”
Worked closing answer · support bot
“We designed a multi-tenant support bot that keeps conversation
context, answers from permitted company knowledge, performs a narrow
set of safe actions, and hands uncertain cases to people. A durable
database stores messages, citations, actions, and handoffs; search
and cache are rebuildable copies. Stateless services scale the chat
path, while model capacity scales separately. The main safety rule
is that the model proposes answers and actions, but evidence,
server-side permissions, retry keys, and human approval decide what
becomes official. We trade some speed for grounding and safe
escalation because an unsupported answer or duplicate refund costs
more than a slower response.”
Final ten-question check
Use only when the interviewer goes deeper
Extra cheat sheets.
The interview still follows the ten steps above. This bottom section
is a lookup tool. Do not force every idea into every answer.
?
Requirements
Functional: what it does
Non-functional: how well
Set version-one priorities
↗
Scalability
Scale up: bigger machine
Scale out: more machines
Find the real bottleneck first
✓
Reliability
Handle component failure
Retry only safe work
Practice recovery
◷
Availability
Name uptime target
Remove single failure points
Define RTO and RPO
⇄
Performance
Latency: time per request
Throughput: work per second
Measure p95, not only average
▤
Data design
Main records and relationships
Reads, writes, and indexes
Retention and deletion
PGDDBMDB
⚖
Load balancing
Round robin: similar servers
Least connections: uneven work
Hashing: keep a key together
⚡
Caching
Choose what may be stale
Plan misses and hot keys
Keep database fallback safe
RedisCDN
≋
Queues and events
Move slow work off the request
Expect duplicate delivery
Watch oldest job and queue size
KafkaRMQSQS
⌁
API and communication
REST: common resource API
gRPC: typed internal calls
WebSocket: live two-way updates
⏳
Traffic control
Limit by user and tenant
Protect expensive endpoints
Slow intake when queues grow
▣
Storage
Block: database disks
Object: files and backups
Shared files: legacy workloads
S3EBSNFS
⌾
Security
Identity is not permission
Encrypt stored and moving data
Keep tenants separate
◎
Observability
Logs explain events
Metrics show system health
Traces follow one request
OTelGrafana
🧪
Testing and change
Unit, integration, and system
Load and security tests
Small rollout and rollback
$
Cost and maintenance
Compute, storage, and network
Vendor and people costs
Document ownership and migration
1. Make database reads faster
First name the read you need. Then choose the smallest change that
makes that read fast enough.
Database access choices
Choice
Use it when
Main cost
Normal index
You filter or sort by known fields
More storage; writes become a little slower
Read copy
Reads are much heavier than writes
It may briefly show old data
Split the data
One database cannot hold or write everything
Queries across splits become harder
Saved result view
The same expensive result is requested often
The saved result must be refreshed
Say: “I would first add the index needed by this read. I
would split the database only after one database is truly the
limit.”
2. Put slow work in a queue
A queue lets the API accept work quickly and lets workers finish it
later.
Message delivery choices
Choice
What it means
Use it when
At most once
A job may be lost, but it is not repeated
Losing the work is harmless
At least once
A job is not lost, but it may arrive twice
The worker can safely recognize repeats
One business result
Repeats may arrive, but only one result is saved
Use a stable job ID and a database uniqueness rule
Ordered
Jobs with the same key are handled in order
Order matters more than maximum speed
Good default: “I assume the queue may send a job more than
once. Each job has a stable ID, and the database prevents a second
business result.”
3. Use a cache without trusting it
A cache is a fast copy. The system should still be correct if the
cache is empty or unavailable.
Common cache problems
Problem
Simple response
Why
Old value
Use an expiry time and remove changed entries
Limits how long old data survives
Missing value
Read the main database, then fill the cache
The main database still owns the truth
Many requests miss together
Let one request refill while others wait briefly
Protects the database from a sudden rush
One very popular key
Copy it locally or spread the load
One cache machine should not take all traffic
Cache is down
Fall back to the database and limit traffic
The cache improves speed, not correctness
Say: “Redis improves speed. Correctness does not depend on
Redis being available.”
4. Choose how updates reach the client
Client update choices
Choice
Best for
Main cost
Polling
Simple, occasional status checks
Extra requests; updates arrive late
Long polling
Updates should feel quick but are uncommon
Connections stay open longer
Server-sent events
The server continuously sends updates to the browser
Mostly one-way
WebSocket
Frequent two-way messages, such as chat or live editing
More connection and recovery work
Webhook
One server tells another server that something happened
Must verify sender and handle repeats
5. State the recovery target
RTO
Recovery time objective: how long the service
may take to come back.
RPO
Recovery point objective: how much recent data
may be lost.
Example
“RTO is one hour. RPO is five minutes. We test restoring backups
and switching regions.”
Approximate downtime per year
99.9%8h 46m
99.95%4h 23m
99.99%53m
99.999%5m
100%Not a realistic promise
6. Keep multi-step updates safe
When one action changes several things
Situation
Use
Plain meaning
Several records in one database
Database transaction
Save all changes or save none
Save data, then publish a message
Outbox record
Save the business change and “message to send” together
Several independent services
Steps with undo actions
If a later step fails, reverse earlier work where possible
Provider timed out
Look up by stable request ID
Find whether it already finished before trying again
Two people update one item
Expected version check
Reject an update based on an older copy
7. Choose an ID
ID choices
Choice
Pick it when
Main cost
Database number
One main database creates the records
Harder to create independently in many places
UUID
Many machines create records independently
Larger and less friendly to database indexes
Time-sortable ID
You want distributed IDs that roughly sort by time
Time order is not the same as cause-and-effect order
Dedicated ID service
Extreme distributed scale requires compact IDs
Another important service to operate
8. Protect the API
9. Put AI in production safely
Treat AI output as a proposed result. Decide when software may
accept it, when it must show evidence, and when a person must
approve it.
Say: “The model proposes an answer with evidence.
Deterministic checks and, when needed, a person decide whether it
becomes official.”
10. Requirement → likely component
Large files→Object storage
Slow work→Queue + workers
Repeated reads→Cache
Text ranking→OpenSearch
Meaning search→Vector search
Live two-way updates→WebSocket
Static global assets→CDN
Reports and analysis→Warehouse or object storage
Safe related updates→PostgreSQL
Huge known-key traffic→DynamoDB
These are starting points, not rules. Explain the requirement first,
then name the component.
Standalone cheat sheet · AI/ML inference
Pick one layer at a time.
Do not compare tools from different layers.
Triton can serve a TensorRT or ONNX model. Kubernetes can run
Triton. They are teammates, not direct alternatives.
4 · Execution engineTensorRT / ONNX / PyTorchTurns model operations into hardware work.
5 · FleetGPU + KubernetesPlaces, restarts, scales, and rolls out workers.
Fast default: For a first self-hosted LLM, use one vLLM or
SGLang server on one GPU machine. Add Kubernetes when several
replicas, models, teams, or GPU types make fleet operations the real
problem. Use TensorRT-LLM when measured NVIDIA performance is worth
extra build and compatibility work.
Interactive LLM
Start: vLLM or SGLang, continuous batching,
token streaming. Measure first-token delay and time between
tokens.
Vision / embedding / classifier
Start: ONNX Runtime for portability, or Triton
when one server must host many models and batch requests.
Maximum NVIDIA speed
Start: TensorRT for normal neural models;
TensorRT-LLM for generative LLMs. Expect engine builds and
strict version testing.
Offline millions of items
Start: a job queue or batch platform with large
batches. Optimize items per dollar, not response time.
01
Understand prefill and decode
An LLM request has two very different parts. This is why “GPU use”
alone does not explain performance.
Prefill vs decode
Part
Plain meaning
Main pressure
Measure
Improve with
Prefill
Read the whole input prompt and build reusable attention
state.
KV cacheThe saved attention state from tokens already
processed. It avoids rereading the full conversation for every
new token, but consumes GPU memory.
TTFTTime from request arrival until the first output
token. Queueing and prefill dominate it.
ITLTime between later output tokens. Decode scheduling
and memory traffic dominate it.
02
Keep both phases together—or split them?
Prefill/decode layout
Choice
Pick it when
Benefit
You pay with
Together on each worker
Most systems; short or mixed prompts; moderate scale.
Simple routing. No KV cache transfer. Easier debugging.
Long prompts can delay active token generation. Both
phases scale together.
Split prefill and decode pools
Long prompts interfere with chat; phases need different
GPU shapes or scale independently.
Independent capacity and better isolation.
Move a large KV cache between GPUs; fast networking,
routing, matching versions, and more failure modes.
Chunked prefill
Long prompts cause visible pauses for users already
decoding.
Breaks prompt work into smaller pieces so decode can run
between them.
More scheduler work; may reduce pure prefill efficiency.
Prefix-aware routing
Many requests reuse the same system prompt, document, or
conversation prefix.
Routes work to the worker that already has useful KV
state.
Routing state can become stale and may conflict with load
balancing.
Interview trap: Splitting prefill and decode is
not automatically faster. If KV transfer uses a slow network,
moving state can cost more than the compute saved. Start together;
split only after a benchmark shows interference.
03
Choose the execution engine
The engine runs the model. A server may wrap one of these engines
and add APIs, batching, health checks, and metrics.
Execution engine choices
Choice
Best when
Strength
Main cost / trap
PyTorch eager
Prototyping, custom code, fast model changes.
Best developer flexibility and easiest debugging.
Usually leaves serving performance on the table.
PyTorch compile
You want speed without leaving PyTorch.
Less migration than exporting to another format.
Compile time, graph breaks, dynamic-shape surprises,
warm-up testing.
ONNX Runtime
The same exported model must run across CPU, NVIDIA,
Intel, mobile, or several languages.
Portable model format and pluggable hardware backends.
Exported operators may differ or fall back to a slower
backend; validate accuracy and full graph placement.
TensorRT
A vision, encoder, ranking, or other neural model runs on
NVIDIA GPUs and speed matters.
Strong NVIDIA kernel fusion, precision tuning, and
optimized engines.
Engine build time; model shapes, GPU, CUDA, and TensorRT
versions matter.
TensorRT-LLM
A supported LLM needs maximum NVIDIA throughput/latency at
scale.
A distributed Python platform to operate; not itself a GPU
kernel engine.
KServe
You already operate Kubernetes and want a standard
deployment/control layer across teams.
Model deployment objects, traffic rollout, autoscaling
integrations.
Kubernetes/operator complexity; the runtime beneath it
still matters.
Managed model API
Speed to market, elastic demand, or no GPU operations
team.
No model server or GPU fleet to manage.
Per-token cost, provider limits, privacy/residency, less
control, vendor dependency.
TorchServe
Existing stable installation that you must maintain.
PyTorch handlers and request batching.
Do not start a new critical system:
official docs say no planned fixes, features, or security
patches.
TGI
Existing Hugging Face deployment that already works.
LLM streaming and optimized serving.
Officially in maintenance mode; Hugging Face recommends
vLLM or SGLang going forward.
Remember: vLLM and SGLang are specialized LLM runtime +
server packages. Triton is a broader multi-framework server.
TensorRT-LLM is an LLM execution toolkit that Triton can serve.
05
Choose the batching style
Batching lets one GPU operation serve several requests. Larger
batches usually improve total work per second, but waiting to form
them can hurt user latency.
Batching choices
Choice
Use it for
How it works
Main trade-off
Offline batch
Nightly embeddings, scoring, backfills, reports.
Collect a large input set, run jobs, save outputs.
Best cost/throughput; results arrive later and jobs need
checkpoints/retries.
Client/static batch
The caller already owns several same-shaped items.
Client sends one array as one request.
The slowest item holds the whole batch and padding wastes
work.
Server waits a tiny amount and combines compatible
requests.
Higher throughput versus added queue delay. Set a hard
delay and queue limit.
Sequence batch
Stateful streams where calls from one session must reach
the same model state.
Groups work while keeping each sequence on the correct
instance.
State placement, timeouts, sticky routing, and
head-of-line blocking.
Continuous / in-flight batch
LLMs with different output lengths.
At every token step, finished requests leave and new ones
enter.
Much better GPU use; needs KV-cache-aware scheduling and
fairness controls.
Ragged / packed batch
Inputs have very different lengths and the backend
supports packed data.
Avoids padding every item to the longest input.
More complex input metadata and model/backend support.
average requests/sec = daily requests ÷ 86,400Then multiply by the measured peak factor.
token load/sec = requests/sec × (average input + output
tokens)Better than request count when lengths vary.
batch wait + model time ≤ latency targetThe batcher may spend only the latency budget left after
queue and inference.
Size by tokens, not only requests. One
20,000-token prompt can use more prefill compute and KV memory
than dozens of short prompts. Cap input tokens, output tokens,
active tokens, queue time, and customer share.
06
Choose precision and memory optimizations
Number precision
Choice
Pick it when
You gain
You risk
FP32
Reference accuracy, CPU, or a model sensitive to lower
precision.
Safest numeric baseline.
Most memory and usually slowest.
BF16 / FP16
Normal GPU inference baseline.
Roughly half the weight memory of FP32 and faster GPU
math.
FP16 range/overflow issues; compare outputs.
FP8 / INT8
Supported hardware/model and measured quality remains
acceptable.
Lower memory, more throughput.
Calibration or conversion work; accuracy can move.
INT4
Memory/cost is the limit and quality loss is acceptable.
Large weight-memory reduction.
Quality and kernel support vary; dequantization can
limit speed gains.
LLM serving optimizations
Choice
Helps when
Main cost / trap
Paged KV cache
Many sequences of different lengths share GPU memory.
Runtime complexity, but usually a strong LLM default.
Prefix cache
System prompts or document prefixes repeat exactly.
Memory, eviction, tenant/privacy-safe cache keys.
KV offload
GPU cache capacity is the limit.
CPU/network transfer can hurt token latency.
Speculative decode
A small draft model can predict several tokens that the
main model accepts.
Extra model/memory; weak acceptance can make it slower.
Flash/fused attention
Attention kernels dominate.
Hardware, dtype, shape, and model support constraints.
CUDA graphs / compile
Repeated shapes make launch overhead noticeable.
Warm-up, memory, and dynamic-shape limits.
07
Use one GPU or many
Parallelism choices
Choice
Pick it when
Benefit
Main cost
One model per GPU
The model and KV cache fit.
Simplest and no cross-GPU communication.
Cannot serve models larger than one GPU; spare memory may
be unused.
Data parallel replicas
The model fits one GPU but traffic is high.
Easy horizontal throughput and failure isolation.
Each GPU stores full weights; router must balance token
load.
Tensor parallel
One model layer/operation must span GPUs.
Lower single-request latency and fits larger models.
Frequent fast GPU-to-GPU communication; topology matters.
Pipeline parallel
Model layers must be split across GPUs/nodes.
Fits very large models with less all-reduce traffic.
Manual failover, rollout, placement, and capacity work.
Docker Compose / simple scheduler
A few services share one or a small number of machines.
Repeatable packaging without a full cluster.
Limited fleet automation.
Kubernetes
Many replicas/models/teams, mixed GPU pools, rolling
updates, quotas, and high availability.
Placement, restart, service discovery, rollout, policy,
and custom autoscaling.
Cluster, driver, device-plugin, image, networking, and
observability complexity.
Managed inference platform
Operations time costs more than platform fees.
Autoscaling, rollout, and hardware supply are outsourced.
Price, limits, cold starts, less control, vendor
dependency.
Kubernetes schedules GPUsA vendor device plugin
advertises GPUs. Pods request them as integer resources.
Kubernetes does not optimize model kernels or batch requests.
Warm capacity mattersLoading large weights may take
minutes. Keep a minimum warm pool and scale before the queue is
already late.
Topology mattersTensor parallel and KV transfer need fast
links. A scheduler that merely finds “any GPU” may choose a slow
cross-node path.
09
Scale on work, not CPU
Autoscaling signals
Signal
What it tells you
Use / warning
Queued tokens
How much real model work is waiting.
Excellent primary signal when request lengths differ.
Queue wait
Whether users are about to miss the latency target.
Scale early; reject or shed work at a hard limit.
Active sequences / tokens
How full the current scheduler and KV cache are.
Useful per-replica load signal.
KV cache used
Whether memory, not compute, is the limit.
High use can cause eviction, rejection, or pause/resume.
GPU compute use
Whether math units are busy.
Helpful, but low use may still hide memory or queue
bottlenecks.
Request rate
Arrival pressure.
Only useful when token lengths are stable.
CPU
Tokenizer, networking, or Python overhead.
Do not use as the only GPU inference scaling signal.
10
Measure quality, speed, capacity, and cost together
User and capacity metrics
Metric
Question answered
TTFT p50/p95/p99
How long until users see the answer start?
ITL p50/p95/p99
Does the stream feel smooth after it starts?
End-to-end latency
When is the full response done?
Input/output tokens/sec
How much useful work does the fleet finish?
Goodput
How many requests meet the latency target, not merely
finish?
Queue depth/wait
Is demand outrunning warm capacity?
Batch size distribution
Is batching actually combining work?
KV cache use/eviction
Is active context exhausting GPU memory?
Correctness and business metrics
Metric
Question answered
Saved evaluation pass rate
Did an engine, precision, model, or prompt change
behavior?
Acceptance / task success
Did the output solve the real user job?
Grounded/cited answer rate
Are factual answers supported?
Fallback / rejection rate
How often can capacity or safety not serve the request?
Cost per successful task
Did optimization reduce business cost, not just GPU
time?
Tokens per watt / GPU-hour
How efficient is the serving fleet?
Cold-load time
How quickly can a new model replica become useful?
Error by model/version/hardware
Which exact release or GPU path failed?
11
Common failure traps
What breaks and what to do
Failure
Why it happens
Design response
GPU is busy but users wait
Batch or queue favors throughput over latency.
Short queue deadline, priority classes, token caps,
separate interactive from batch traffic.
One long prompt hurts everyone
Prefill blocks decode work.
Input cap, chunked prefill, fair scheduling; split pools
only if measured.
Warm minimum, predictive scaling, local weight cache,
readiness only after warm-up.
New optimized engine changes answers
Quantization, export, unsupported ops, or kernels changed
numerics.
Golden evaluation set, shadow traffic, compare task
quality, gradual rollout.
Request disappears during rollout
Streaming connection or in-flight state was killed.
Drain workers, stop new admissions, allow
completion/cancel, client reconnect rules.
One tenant fills the cache
Long contexts or bursts consume all KV memory.
Per-tenant token/concurrency quotas and fair scheduler.
Split prefill/decode is slower
KV transfer or network topology dominates.
Benchmark together vs split; use fast links/RDMA; keep
pools topology-aware.
A strong 45-second interview answer
“I would separate the choices by layer. The API authenticates,
limits, and streams requests. A token-aware scheduler applies
admission control and continuous batching. For a common
open-source LLM I would start with vLLM or SGLang on one GPU
worker; for a mixed model fleet I would consider Triton, and for
maximum measured NVIDIA performance I would test TensorRT-LLM. I
would keep prefill and decode together first, then split them only
if long prompts hurt decode and fast KV transfer makes the
benchmark better. I would add Kubernetes only when fleet
placement, rollout, and scaling justify it. I would scale on
queued tokens and KV pressure, and watch TTFT, inter-token delay,
goodput, quality, and cost per successful task.”