Skip to content
samit

Interview Preparation

Questions I need to prepare for interviews 2026.

234 questions across 17 sections

234 questions

Math Foundations

20 questions
Linear Algebra
What is an eigenvector/eigenvalue? Why do they matter in ML?→
Explain SVD and its applications in ML.frontier→
Why divide attention scores by √d_k? Derive it.frontier→
What is the Jacobian? The Hessian? When do you use each in ML?frontier→
How are matrix multiplications used inside a single Transformer layer?frontier→
What is the rank of a matrix? Why does low rank matter in LoRA?→
Explain orthogonality. Why do we want weight matrices to be orthogonal?→
Derive why PCA is the eigendecomposition of the covariance matrix.→
Probability & Statistics
MLE vs MAP estimation - when do they coincide? What are they equivalent to in regularization?→
Explain KL divergence. Is it symmetric? Forward vs reverse KL - when to use each?frontier→
Explain bias-variance tradeoff.→
What is entropy? How does it relate to cross-entropy loss?→
State Bayes' theorem. Why does it matter for ML?→
Type I vs Type II errors. Map to precision and recall.→
What is the Central Limit Theorem and why does it matter for ML?→
Explain p-values and their limitations for ML evaluation.→
Calculus & Optimization
Derive the gradient of cross-entropy loss w.r.t. logits.frontier→
Explain gradient descent variants: SGD, Momentum, RMSprop, Adam.→
Why do vanishing gradients happen? What fixes them?→
What is a saddle point? Are they actually a problem?frontier→

Classical ML

14 questions
Algorithms
Decision trees: how do you choose the best split? Entropy vs Gini impurity.→
Random Forest vs Gradient Boosting - when to choose which?→
Explain the kernel trick in SVMs.→
K-Means: does it converge? What is the initialization problem?→
LASSO vs Ridge - which gives sparse solutions and why? Geometric intuition.→
Explain the EM algorithm with an ML example.→
What is the curse of dimensionality? How does it affect ML algorithms?→
Evaluation
Precision, Recall, F1 - when to prioritize which?→
ROC-AUC vs PR-AUC - which to use for imbalanced datasets?→
How do you compare two models statistically?→
k-fold cross-validation - when would you use stratified k-fold?→
Regression metrics: MAE vs MSE vs RMSE - when does each matter?→
How do you evaluate clustering quality without ground truth labels?→
What is Cohen's Kappa? When is it better than accuracy?→

Deep Learning

18 questions
Fundamentals
Explain backpropagation step by step.→
Sigmoid vs Tanh vs ReLU vs Leaky ReLU - when does each fail?→
Explain batch normalization - what does it normalize and why does it help?→
Layer norm vs batch norm - why do transformers use layer norm?→
Explain residual/skip connections. Why do they help with deep networks?→
Explain knowledge distillation.→
What is contrastive learning? Give an example.→
What is dropout? How does it differ at train vs. inference?→
Weight initialization: why does it matter? Xavier vs He.frontier→
What is label smoothing? When does it help?→
Derive the backward pass of BatchNorm.→
Derive the backward pass of LayerNorm. Why is it simpler than BatchNorm in practice?→
CNNs
Explain the convolution operation - output size formula, stride, padding.→
How does a 1×1 convolution work and why is it useful?→
Depthwise separable convolutions - how do they work and why are they more efficient?→
RNNs & Sequential Models
What is the vanishing gradient problem in RNNs? How do LSTMs solve it?→
Why did Transformers replace LSTMs for NLP?→
What is teacher forcing? When does it hurt?→

Transformers & Attention

12 questions
Architecture
Walk through the full Transformer architecture block-by-block.→
Encoder vs decoder Transformers - differences and use cases. Why are SOTA models decoder-only?frontier→
What is Flash Attention? How does it reduce memory from O(n²) to O(n)?frontier→
What is KV cache? Memory scaling, speedup, and how to reduce it.frontier→
Explain positional encoding - sinusoidal, learned, and RoPE.frontier→
What is Grouped Query Attention (GQA)? Why does LLaMA use it?frontier→
What is Mixture of Experts (MoE)? Dense vs sparse models.frontier→
Explain scaling laws (Chinchilla). How do you decide model size vs data budget?frontier→
Implementation
Implement scaled dot-product attention in PyTorch.frontier→
Implement multi-head attention from scratch.frontier→
Implement causal mask for decoder-only Transformer.→
Why is tokenization the root of most LLM weirdness?→

LLMs & Modern AI

39 questions
Training & Alignment
Explain RLHF step by step.frontier→
What is DPO? How does it simplify RLHF?frontier→
What is GRPO? Why does DeepSeek use it?frontier→
Explain Constitutional AI (Anthropic's approach).frontier→
What is the difference between SFT and alignment training?→
Pre-training objectives: how does GPT differ from BERT?frontier→
What is chain-of-thought prompting? Why does it improve reasoning?frontier→
RLAIF vs RLHF - what is the difference?frontier→
Explain the policy gradient theorem and REINFORCE.→
How does PPO work mechanically - clipped objective, value function, GAE?→
Inference & Optimization
Explain continuous batching and why it improves throughput.frontier→
Model parallelism: tensor, pipeline, and data. When to use each?frontier→
Quantization in LLMs: INT8, INT4, FP16, BF16 - trade-offs.→
What is Paged Attention (vLLM)?frontier→
What is speculative decoding?frontier→
Fine-tuning & Adaptation
Explain LoRA: what matrices are trained? What is the rank r?startup→
QLoRA - how does 4-bit quantization + LoRA work together?startup→
When should you choose RAG over fine-tuning?startup→
What is DoRA and how does it improve on LoRA?→
How do you choose the LoRA rank r in practice?→
RAG & Retrieval
Explain the full RAG pipeline.startup→
Chunking strategies - fixed-size vs semantic vs recursive.startup→
What is hybrid search? Dense vs sparse retrieval.startup→
What is the "lost in the middle" problem in RAG?startup→
How do you evaluate a RAG system?startup→
AI Agents
Explain the ReAct (Reason + Act) agent architecture.startup→
What is MCP (Model Context Protocol)?startup→
How do you prevent infinite loops in agents? What are the failure modes?startup→
Single-agent vs multi-agent systems - when do you split?startup→
What is human-in-the-loop (HITL) for agents? When is it required?startup→
What are security risks of agentic AI systems?startup→
How do you evaluate agent quality?startup→
LLM Evaluation & Safety
What is perplexity? What other benchmarks do we use for LLMs?→
What are hallucinations? How do you detect and reduce them?startup→
What is prompt injection? How do you defend against it?startup→
BLEU, ROUGE, BERTScore - differences and when to use each.→
What is LLM-as-a-judge? Strengths and weaknesses.startup→
What is red teaming for LLMs?frontier→
What is the EU AI Act? How does it affect ML engineers?startup→

Computer Vision

14 questions
Detection & Segmentation
Two-stage (Faster R-CNN) vs one-stage (YOLO) detectors.→
What is mAP (mean Average Precision)?→
How does SAM (Segment Anything Model) work?frontier→
Vision-Language & Multimodal
How does CLIP work? Why is it powerful?frontier→
How does Whisper process audio? What happens to long audio?→
How does ViT tokenize images? Compare to CNNs.→
Explain the LLaVA architecture - how do vision encoders connect to LLMs?→
Frontier CV
How do you use SAM at inference? Point, box, and mask prompts.frontier→
SAM 2 for video - how does it track objects across frames?frontier→
Open-vocabulary detection - how do Grounding DINO and YOLO-World work?frontier→
DINOv2 - why use it over supervised ImageNet pretrain?frontier→
Video understanding - how do models capture temporal information?frontierstartup→
When would you fine-tune SAM vs use zero-shot prompts vs train a U-Net?startup→
Contrastive learning for re-identification - what matters for multi-camera tracking?startup→

Generative Models

8 questions
GANs & VAEs
Explain GAN architecture - generator and discriminator objectives.→
Explain VAE - the ELBO objective and the reparameterization trick.frontier→
What is mode collapse in GANs? How do you fix it?→
Diffusion Models
Explain diffusion models - forward process, reverse process, training.frontier→
DDPM vs DDIM - what does DDIM improve?frontier→
What is classifier-free guidance?frontier→
What is flow matching? How does it differ from diffusion?frontier→
What is latent diffusion? How does Stable Diffusion use it?→

System Design

13 questions
Framework
What framework do you use for ML system design problems?→
LLM System Design
Design ChatGPT end-to-end: training to serving.frontier→
Design a RAG system for 1M internal documents.startup→
Design an LLM inference platform (vLLM-as-a-service).frontier→
Classic ML System Design
Design a YouTube video recommendation system.→
Design a fraud detection system.→
Profile-Specific ML System Design (CV + Startups)
Design a real-time traffic CV system : detect, track, and alert in <100ms.startup→
How would you design model rollback and canary release under drift for a CV production pipeline?startup→
Design an annotation + active learning loop for low-data CV startup environments.startup→
Given a fixed monthly budget, how do you allocate spend between training, inference, and data labeling?startup→
Design an end-to-end MLOps stack for a early-career ML engineer at an Indian AI startup.startup→
Distributed Training
What are ZeRO stages 1/2/3? How does FSDP relate?→
What is gradient checkpointing? How does it trade compute for memory?→

MLOps & Production

20 questions
Deployment & Monitoring
What is model drift? How do you monitor for it?→
What is LLMOps? How does it differ from classical MLOps?startup→
How do you implement A/B testing for LLM applications?startup→
What is CI/CD for ML models?→
How do you handle PII in ML pipelines?→
GPTQ vs AWQ - how do post-training quantization methods differ?→
ONNX, TorchScript, Triton Inference Server - when do you use each?→
Inference Engineering & AIOps
TTFT vs tokens/sec - which do you optimize for, and why?frontierstartup→
vLLM vs TensorRT-LLM vs plain PyTorch serve - when to pick each?startup→
Walk through exporting a YOLO detector to TensorRT for production.startup→
How do you calculate cost per 1M tokens for an LLM API?startup→
What signals do you use to autoscale GPU inference workers?frontier→
Design a canary rollout for a new model version in production.startup→
What do you log on every inference request in production?startup→
Sarvam / Krutrim-style: what breaks when you serve Indic LLMs at scale?startupfrontier→
Frontier lab inference round: "7B model, 10K concurrent users, 2s P99 latency - design it."frontier→
AIOps for ML: how is it different from classical DevOps monitoring?startupfrontier→
early-career CV/ML at a startup: what will they actually ask in a technical round?startup→
What is prefix caching? When does it help and when does it not?→
Explain disaggregated prefill/decode serving. Why split them?frontier→

Coding Challenges

6 questions
Implementations from Scratch
Implement softmax with numerical stability.→
Implement cross-entropy loss and its gradient.→
Implement a 2-layer MLP forward and backward pass in pure NumPy.frontier→
Implement K-Means clustering with K-Means++ initialization.→
Implement precision, recall, F1, and BLEU score from scratch.→
Implement a basic RAG pipeline in Python.startup→

Behavioral

16 questions
Common Questions
Tell me about a challenging ML project and what you learned.→
How do you stay current with ML research?frontier→
A model worked great in testing but failed in production. Walk me through the debugging.→
How do you approach a project with very limited labeled data?→
How do you debug a model with a flat loss curve?→
How do you explain AI limitations to a non-technical PM who wants 99% accuracy?startup→
How do you decide whether a problem needs ML at all?→
Tell me about a time you improved model performance significantly.→
How do you handle disagreements about model design with teammates?→
Describe the experience teaching AI. What's the hardest concept to explain?→
What interests you specifically about this company's work? (Prepare per company.)frontier→
Where do you see the AI field heading in the next 3 years?frontier→
Tell me about a time you made a decision with incomplete data.→
How do you measure ROI of an AI feature?startup→
A PM wants 90% accuracy but the model achieves 82%. What do you do?startup→
Tell me about a time you pushed back on a technical direction and were wrong. What did you learn?→

Company Guide

read
Open section →

Reading List

read
Open section →

Resume Deep Dive

16 questions
Beltech AI - Production CV System
You replaced anchor-based YOLO with NMS-free YOLOv10 and cut latency 54%. What is NMS-free detection and why is it faster?frontier→
How does TensorRT optimize inference? Walk through the full pipeline.frontier→
How does Triton Inference Server handle concurrent requests? How does it compare to Flask/FastAPI?frontier→
Walk through the continuous learning pipeline. How do you handle data quality and catastrophic forgetting?→
Explain DeepSORT. What does the "deep" add over SORT? What is the Mahalanobis distance term?→
mAP50 vs mAP50-95 - when does each matter? Which did you optimize at Beltech and why?→
MakeMyTrip - Recommendation & Bandits
Compare epsilon-greedy, UCB, and Thompson Sampling. Which suits high-traffic personalization?→
What is the exploration-exploitation tradeoff? How does it compare to bias-variance?→
Walk through how you trained the RLHF reward model at MakeMyTrip. What data, what loss, what did it output?startup→
How did you build the NLP-based resume screening pipeline at MakeMyTrip? What signals did you use?startup→
MiniTorch & MathLM - Build From Scratch
Explain reverse-mode autodiff. How does topological sort guarantee correct gradient computation?frontier→
Walk through the BPE tokenization algorithm. How does it differ from WordPiece?→
GQA vs MHA vs MQA - tradeoffs and KV cache memory math for LLaMA-3 70B.frontier→
What is SwiGLU? Why do modern LLMs prefer it over ReLU for FFN blocks?frontier→
SFT loss masking on assistant tokens - what is it and what breaks without it?→
Explain Sparse MoE routing: Top-K selection, load balancing loss, expert capacity.frontier→

Karpathy Questions

26 questions
Core Questions
Implement scaled dot-product attention backward pass without autograd. Given dO, compute dQ, dK, dV.frontier→
Build a minimal autograd engine. Value class with +, *, relu, backward using topological sort.frontier→
Implement AdamW from scratch. What is the precise difference between Adam and AdamW?frontier→
What happens when you call loss.backward in PyTorch? Trace the actual call chain.frontier→
A 7B model in fp32 = 28GB. You have a 24GB GPU. All options for inference and fine-tuning.frontier→
Implement top-p nucleus sampling with temperature. Why does temperature work?→
Walk through a nanoGPT training loop line by line: data → forward → loss → backward → step → eval.→
Implement BPE (byte-pair encoding) tokenizer from scratch.→
Trace tensor shapes through one GPT-2 block forward pass (batch B, seq T, model dim C).→
Why tie input token embeddings to the LM head weights?→
Training loss is NaN after ~100 steps. Walk through the debug checklist.→
Implement LayerNorm forward and backward in NumPy (no autograd).→
Estimate Pi with Monte Carlo - implement it and explain the variance.→
Matrix multiply: O(n³) naive, then how do you actually make it fast on a GPU?→
Your network won't overfit a single batch. Walk the debug checklist.→
How do you gradient-check an autograd implementation?→
Explain tensor strides, views, and what .contiguous() does.→
Why does weight initialization matter? What breaks with bad init?→
.data vs .detach() vs .item() vs torch.no_grad() - precise differences.→
What is the residual stream? Why do residual connections make deep nets trainable?→
Hardware & GPUs
What does a CUDA matmul kernel actually do? Why are Tensor Cores faster than regular CUDA cores?frontier→
Why is LLM inference memory-bandwidth bound? What optimization strategies follow from this?frontier→
What is arithmetic intensity? Explain the roofline model.frontier→
Estimate the tokens/sec ceiling for a 7B model on one A100. Show the arithmetic.frontier→
GPU memory hierarchy - the numbers that actually matter.frontier→
What is MFU (Model FLOPs Utilization)? What caps training throughput?frontier→

Projects to Do

12 questions
Frontier Personal Projects
MathLM (the repo) - train and evaluate the own mini LLM stackfrontier→
vLLM Serving Lab - deploy an open model with batching, KV cache, and latency budgetsfrontier→
Multimodal CV Agent - combine detector + VLM + retrieval for practical QAfrontier→
Systems & Kernels
Triton FlashAttention - write fused attention as a GPU kernelfrontier→
Transformers in CUDA - a GPT forward/backward in raw C++/CUDAfrontier→
Contribute to tinygrad - read and modify a real DL frameworkfrontier→
Models From Scratch
Llama 3 from scratch - implement the architecture and load real weightsfrontier→
Stable Diffusion - read the paper and implement a text-to-image sampler→
Multimodal Vision-Language Model - code a VLM from scratch (PaliGemma-style)frontier→
Applied & Agents
Coding harness - build an agentic plan/edit/run/test loop→
RAG system (pi.dev) - retrieval over your own data, done properly→
Ship one AI app end to end that people actually usestartup→

NeetCode 150

read
Open section →