Deep Learning Profiling

Understanding DL Inference with Pytorch Profiler

Feb 15, 2026

Deep Learning workloads demand high compute throughput and memory bandwidth. GPUs are well suited for such workloads due to massive parallelism and specialized tensor compute units. However, achieving optimal performance is not easy.

Even when running on powerful hardware, models may suffer from:

Low GPU utilization
Memory bandwidth bottlenecks
Kernel Launch overhead
Synchronization stall etc.

To achieve optimal performance, Profiling is a necessary step, as it allows us to identify bottlenecks in an application.

In short,

Profiling → an approach to measure application performance.

Simple Profiling → How long does an application take?
Advance Profiling → Why does an application takes such long time?^[1]

Challenges^[1]

Profiling is hard, cumbersome and time-consuming
Profiling tools generate lot of data and is hard to understand
With complex models, we have a large volumes of data.
Need proper strategies to use the right tools and insights to how to analyze the profile data.

Scope of the blog

Task: Deep Learning Inference
Profiler Tool: Pytorch Profiler
GPU used: RTX3060(single GPU)
Model: ViT-Tiny(from timm)(patch_size=16, image_dimension=224)
Goal: Understand and optimize inference performance
Code: Pytorch Profiler Code

Why do we need to warmup before doing profiling?

CUDA Context Initialization
- When we run our first GPU operation, CUDA performs several one-time setup operations:
  - CUDA runtime initializes
  - CUDA context created
  - Allocate device memory pools
  - Kernels are loaded
- Can take hundreds of milliseconds, and should be avoided while profiling and be considered as a startup cost
JIT Compilation and Kernel Optimization
- Many DL Frameworks(like Pytorch) use Just-In-Time(JIT) compilers or dynamic graph mechanisms
- First few iterations often trigger code compilation, kernel selection, and hardware specific optimization(like cuDNN autotuning on GPUs)
- cuDNN autotuning
  - Tests multiple algos to find the fastest one
  - Caches it
- After that → kernels are cached
- Without warmup, we profile compilation time instead of execution time.
GPU Power State & Clocks
- GPUs often sit in a low-power, “idle” state, and it takes time for them to transition to the required mode.
- After load, clocks boost, power stabilizes and thermal equilibrium improves.
- Profiling before boost → wrong throughput numbers

Without warmup

True inference time + initialization + autotune + compilation

Instead of

True steady state inference time

Pytorch Profiler

Trace-based profiling tool
Integrated inside Pytorch
Less setup than other monitoring or profiling tools.
Records both CUDA and CPU activity by default

Why model needs to be in eval mode while profiling inference?

Most deep learning models contains layers whose behavior changes during training and inference.

For e.g.:

Dropout Layer
- In training mode:
  - Dropout randomly zeros activations
  - Random masks are generated
- In Evaluation Mode:
  - Dropout is disabled
  - Activations passes through unchanged
- If left in training mode(which is the default mode):
  - Additional random number generation occurs
  - Execution becomes non-deterministic
BatchNorm
- In Training Mode:
  - Batch statistics(mean & variance) are computed per batch
  - Running statistics are updated
- In Evaluation Mode:
  - Stored running mean & variance are used
  - No statistics are recomputed
Autograd graph
- Default: Pytorch tracks operations for backprop
- In Training mode:
  - Computation graph is built
  - Intermediate tensors are stored for backprop
  - Memory usage increases
- During inference, we don’t need to calculate the backprop. Disable autograd during inference.
Kernel behaviour & Determinism
- Some operations behave differently b/w training and inference:
  - Numerical behaviour
  - may vary.
  - Different code paths may be taken
  - Certain fused kernels may only activate in eval mode

In short, inference profiling should measure:

No gradient tracking
No training-specific behaviour
Deterministic forward pass
Steady-state GPU execution

Warmup-code

model = timm.create_model("vit_tiny_patch16_224", pretrained=True).to(device)
x = torch.randn(BATCH_SIZE, 3, 224, 224).to(device)
model.eval() # Critical for accurate inference profiling results
print("Warming up...")

with torch.inference_mode():
    for _ in range(WARMUP_ITER):
        with torch.autocast(device.type):
            _ = model(x)

if torch.cuda.is_available():
    torch.cuda.synchronize()

print("Warming up done. Starting profiling...")

A natural question arises, where we should synchronize? After each warmup iteration or after completion of warmup iterations?

Version 1(Correct)

with torch.inference_mode():
    for _ in range(WARMUP_ITER):
        with torch.autocast(device_type=device.type):
            _ = model(x)

if torch.cuda.is_available():
    torch.cuda.synchronize()

Version 2(Incorrect/Slower)

with torch.inference_mode():
    for _ in range(WARMUP_ITER):
        with torch.autocast(device_type=device.type):
            _ = model(x)

        if torch.cuda.is_available():
            torch.cuda.synchronize()

Key: CUDA kernel launches are asynchronous

when we call model(x), Pytorch:

launches CUDA kernels
Returns immeadiately
Does NOT wait for kernels to finish

CPU continues execution while GPU works in parallel

Essentially what we want

Launch → Launch → Launch → Wait once

Instead of

Launch → Wait → Launch → Wait → Launch → Wait

In version 2, after every forward pass, CPU waits for GPU to finish before launching the next iteration.

This causes:

Artificial serialization
Reduced GPU utilization
Slower warmup
Different execution behaviour than steady state.

Latency Measurement

if torch.cuda.is_available():
    torch.cuda.reset_peak_memory_stats()

print("Latency measurement")
starter = torch.cuda.Event(enable_timing=True)
ender = torch.cuda.Event(enable_timing=True)
torch.cuda.synchronize()
starter.record()
with torch.inference_mode():
    for _ in range(PROFILER_ITER ):
        _ = model(x)
ender.record()
torch.cuda.synchronize()

total_time_ms = starter.elapsed_time(ender)
avg_latency = total_time_ms / PROFILER_ITER

print(f"\n Average latency per batch: {avg_latency:.3f} msec")
print(f"Throughput: {(1000/avg_latency) * 8:.2f} samples/sec")

Profiler Code

Will profile single forward pass.

with torch.inference_mode():
    with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA], record_shapes=True, profile_memory=True, with_stack=True) as prof:
        with record_function("model_inference"):
            _ = model(x)

if torch.cuda.is_available():
    torch.cuda.synchronize()
    print(f"Peak GPU memory usage: {torch.cuda.max_memory_allocated() / (1024 ** 2):.2f} MB") # Measures memory allocated by tensors by PyTorch, not total GPU memory usage
    torch.cuda.reset_peak_memory_stats()

prof.key_averages() → Pytorch Profiler utility that aggregates profiling events into a summarized table. Will be using it to analyze the results and sort based on metrics.

Key metrics to analyze from profiling:

self_device_time_total: Exclusive GPU time spent in the operator.
- Tells us where the GPU is actually spending time
- Excludes wrapper/module overhead
device_time_total: Inclusive GPU time(includes children)
- Useful for
  - Layer-level bottlenecks
  - Understanding module-level cost
- If self_device_time_total is small but device_time_total is large:
  - Operator is a wrapper
  - Real work is happening in sub-ops

Profiling Observations

For the base results, I start profiling FP32 precision, i.e.

torch.backends.cuda.matmul.allow_tf32 = False
torch.backends.cudnn.allow_tf32 = False
torch.set_float32_matmul_precision("highest")

print(prof.key_averages().table()
- At the bottom of table, we will find the last lines printed under “Self CPU time total” and “Self CUDA time total”. These represents global totals across the entire profiling window.
- Self CPU time total
  - Sum of self CPU time across every operator
  - Includes:
    - operator dispatch
    - kernel launches
    - CPU-only ops
    - Python overhead etc.
  - But excludes time waiting inside child ops(since it’s self)
- Self CUDA time total
  - Total GPU kernel execution time recorded.
  - Sum of all CUDA kernels launched across all operators and iterations.
    They don’t represent elapsed time because GPU kernels overlap
  - E.g.
```
Kernel A = 5 ms
Kernel B = 5 ms
Both overlap → wall time = 5 ms
Profiler CUDA total = 10 ms
```
Stabilizing Profiling Results
- Initial observation was that large fluctuations in GPU execution time across runs. Even with warm-up iterations, repeated profiler runs produced noticeably different timing results.
- Instability pointed to two possible causes:
  - insufficient GPU workload per iteration
  - GPU’s runtime state
    - Increased batchSize from 16 → 64 for a single forward pass instead of multiple passes(owing to similar base).
      - Small batches: individual kernels execute very quickly, making timings sensitive to scheduling overhead, launch latency and clock fluctuations.
      - Larger batches: increase kernel runtime, reducing the relative impact of these overhead
      - Observing outputs becoming more consistent
        Small batch ⇒ kernels too small ⇒ GPU not saturated.
- Observed profiling changed after long system idle periods or after shutting down the laptop
  - When the system is restarted, GPU begins in cold state: CUDA contexts must be recreated, cuDNN algo reselected, memory pools reinitialized and GPU clocks ramped up from low-power mode, despite warmp.
    - Warmup fixes framework-level cold start, but not always hardware-level cold start.
  - First few runs therefore include hidden initialization costs.
  - After several executions, GPU reaches a steady-state.
- Summary:
  - Use a sufficiently large batch size to reduce launch overhead dominance
  - Perform several warm-up iterations before profiling
  - Expect different timings after system restarts or long idle periods
  - Only record results once the GPU is in steady state
Now looking at the table, sort it by self_cuda_time_total:
- Major GPU time contributors
  - Linear layers dominate
    - aten::addmm(~50% CUDA Time)
    - SGEMM kernels(~36% CUDA Time)
      Majority time of the GPU execution time is spent on matrix multiplication.
      When reading profiler tables, note that framework-level operators and low-level CUDA kernels often represent the same work at different abstraction levels. So, they often overlap rather than add-up.
      For ViT:
      - QKV Projections
      - attention output projection
      - MLP Layers
    - In short:
      - Transformer MLPs + projections dominate runtime
      - Model is compute-bound on matrix multiplications
  - AttentionLayers are the second bottleneck
    - aten::_efficient_attention_forward(~27% CUDA Time)
      This confirms:
      - Pytorch using optimized fused attention kernels
      - Despite fusion, it is still expensive.
  - Elementwise ops
    - GELU → ~7% CUDA Time
    - LayerNorm → ~6% CUDA Time
    - Residual adds → ~5% CUDA Time
    - Are small and expected for transformers
- Memory Footprint
  - Linear layers dominate memory too
    - aten::addmm → ~1000 MB CUDA Mem
      - MLP activations dominate memory usage
      - Linear Layers create
        large intermediate activations
        Q/K/V projections
        MLP hidden states
      ViT inference is activation-memory heavy not parameter-heavy.
  - Activations dominate memory
    - Activation functions scale with: batch × sequence length × hidden dim
    - GELU(~443 MB), Residual adds(~230 MB), LayerNorm(~233 MB) show high memory usage
    - GELU stores:
      - input tensor
      - output tensor
      and often intermediate buffers.
    - LayerNorm needs:
      - input tensor
      - normalized output
      - mean + variance buffers
    - Residual connections duplicate tensors briefly.
    - So large batches → big activation footprint.
    - The above operate on full feature maps. They don’t allocate new tensors always, but profiler shows memory touched, not only allocated.
    - So elementwise ops = memory bandwidth heavy
  - Attention also surprisingly small (~110 MB)
    - From trace it shows using efficient attention kernals
      - aten::_efficient_attention_forward
      - scaled_dot_product_efficient_attention
      - fmha_cutlassF_f32_aligned_64x64_rf_sm80
    - These indicate:
      Flash / fused attention kernels are being used
      These kernels:
      - do not materialize the full attention matrix
      - compute attention in blocks
      - stream intermediate results
      - avoid storing the full QK^T matrix

Conclusion

Transformer inference performance is primarily limited by matrix multiplications, with attention and elementwise ops forming secondary contributors.
Inference memory consumption scales primarily with activation size (batch, sequence length, hidden dimension), rather than parameter count.

Trace-Level Observations

Export chrome trace visualization as:

prof.export_chrome_trace("trace.json")

File can be found at: Tiny-vit_chrome_trace_file (Download and run it on chrome chrome://tracing )

From the Chrome trace visualization, several system-level behaviors became evident:

CPU–GPU scheduling behaviour
- GPU kernels don’t start immediately after CPU dispatch
- Small gaps exist between launches
- This shows
  - launch latency
  - runtime overhead
  - scheduling delays
Kernel execution patterns
- From the trace, can see
  - used attention kernels
  - GEMM blocks dominating timeline
  - elementwise ops interleaved between big kernels
- Confirming what profiler tables hinted at
Kernel fusion / optimization evidence
- Trace directly shows:
  - fused attention kernels
  - batched GEMMs
  - optimized cuDNN kernels
- Proves modern DL frameworks apply heavy optimization.
Synchronization overhead
- cudaDeviceSynchronize blocking CPU
- GPU finishing work while CPU waits
- Explains:
  - Why CPU time can look large
  - Why async execution matters
- Synchronization points show how CPU waits for GPU completion, highlighting the importance of minimizing forced syncs

Precision Optimization

Enabling TF32 for Matrix Multiplications

torch.backends.cuda.matmul.allow_tf32 = True 
torch.backends.cudnn.allow_tf32 = True 
torch.set_float32_matmul_precision("high")

Result:

Enabling TF32 did not significantly change the compute distribution or execution time in this experiment. Matrix multiplications continued to dominate GPU execution, followed by attention kernels and elementwise operations.

It suggests that the model is already executing near hardware efficiency and that TensorCore Acceleration alone does not always give measurable gains.

Mixed Precision(Autocast/FP16)

with torch.inference_mode():
    with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA], record_shapes=True, profile_memory=True, with_flops=True) as prof:
        with record_function("model_inference"):
            # for _ in range(PROFILER_ITER):
            with torch.autocast(device.type, dtype=torch.float16):
                _ = model(x)

if torch.cuda.is_available():
    torch.cuda.synchronize()
    print(f"Peak GPU memory usage: {torch.cuda.max_memory_allocated() / (1024 ** 2):.2f} MB") # Measures memory allocated by tensors by PyTorch, not total GPU memory usage
    torch.cuda.reset_peak_memory_stats()