在本教程中，我们在 Nsight Python上实现，它是 NVIDIA 面向其内核分析器的开源 Python 接口。我们不是在 Nsight Compute 命令行下运行脚本之后再翻阅报告，而是用 nsight.annotate 标记我们关心的代码，用 nsight.analyze.kernel 装饰一个基准测试函数，然后得到一个 pandas DataFrame，其中包含每个注释区域、每个配置、每次运行的硬件指标。自 1.0 版本起，该库可以将 Nsight Compute 附加到正在运行的 Python 进程，使其可在 Jupyter 和 Colab 中使用，而即将发布的 1.1 版本增加了实验性的 CUPTI 后端、显式的工具管理以及自定义信息收集器。我们在免费的 Colab T4 上运行所有内容：我们在 Colab 自带的那个旁边安装当前版本的 Nsight Compute，分析第一个内核，确定一次测量需要多少次运行，测量缓存和时钟控制对同一内核的影响，在 fp32 和 fp16 下扫描不同矩阵大小并推导 TFLOP/s 指标，使用硬件计数器区分计算受限与内存受限的工作负载，以移动的字节数测量内核融合效果，记录每次运行的时钟频率和温度，比较 CUPTI 后端与 Nsight Compute，并保留原始报告以供更深入的分析。
复制代码import os
import sys
import glob
import json
import time
import shutil
import textwrap
import traceback
import subprocess
RESULTS = {}
def banner(title):
print("\n" + "=" * 78)
print(title)
print("=" * 78)
def section(name):
def wrap(fn):
def run(*a, **kw):
banner(name)
try:
out = fn(*a, **kw)
RESULTS[name] = out if isinstance(out, str) else "ok"
return out
except Exception as e:
RESULTS[name] = f"SKIPPED / FAILED -> {type(e).__name__}: {str(e).splitlines()[0][:160]}"
print(f"\n[!] {name} did not complete: {type(e).__name__}: {e}")
traceback.print_exc(limit=3)
return None
return run
return wrap
def sh(cmd):
"""Run a shell command and return its stdout, stripped."""
return subprocess.run(cmd, shell=True, capture_output=True, text=True).stdout.strip()
banner("1. Install a current Nsight Compute next to Colab's, then import nsight before CUDA")
print(f" GPU: {sh('nvidia-smi --query-gpu=name,driver_version,memory.total,compute_cap --format=csv,noheader')}")
preinstalled = shutil.which("ncu")
print(f" ncu already on PATH: {preinstalled} -> {sh('ncu --version | tail -1') if preinstalled else 'none'}")
# nsight-python needs Nsight Compute 2026.2.1 or newer. Colab ships an older ncu, but its image already
# carries NVIDIA's CUDA apt repository, so a current release is one apt-get away. It installs into
# /opt/nvidia/nsight-compute/<version>/ and leaves the preinstalled one untouched.
subprocess.run("apt-get update -qq > /tmp/ncu_install.log 2>&1", shell=True)
for NCU_VERSION in ("2026.3.1", "2026.2.1"):
if subprocess.run(f"apt-get install -y -qq nsight-compute-{NCU_VERSION} >> /tmp/ncu_install.log 2>&1",
shell=True).returncode == 0:
break
else:
print(sh("tail -5 /tmp/ncu_install.log"))
raise RuntimeError("could not install Nsight Compute 2026.2.1+ from NVIDIA's apt repository")
# ask dpkg where the package put the ncu launcher rather than assuming /opt/nvidia/nsight-compute/<version>/
ncu_paths = sh(f"dpkg -L nsight-compute-{NCU_VERSION} | grep -E '/ncu$'").split()
if not ncu_paths:
raise RuntimeError(f"dpkg lists no ncu executable in nsight-compute-{NCU_VERSION}")
NCU_DIR = os.path.dirname(min(ncu_paths, key=len)) # the top-level launcher, not a per-platform copy
os.environ["PATH"] = NCU_DIR + os.pathsep + os.environ["PATH"] # nsight-python looks ncu up on PATH
print(f" installed: {shutil.which('ncu')} -> {sh('ncu --version | tail -1')}")
# nsight-python 1.1.0 (CUPTI backend, tool APIs, info collectors) is tagged but not yet on PyPI;
# pin the commit this notebook was written against. The [cupti] extra is used in step 9.
NSIGHT_COMMIT = "bb96db4260b1823d86ecab9acfed0646d7af79ad"
subprocess.run([sys.executable, "-m", "pip", "install", "-q",
f"nsight-python @ git+https://github.com/NVIDIA/nsight-python@{NSIGHT_COMMIT}"], check=True)
# The CUPTI backend needs cupti-python 13.4, which wants a newer libcupti than Colab's torch is pinned to.
# Install it into a separate directory that only step 9's child process puts on its path.
CUPTI_DIR = "/content/cupti_env"
CUPTI_EXTRA = subprocess.run([sys.executable, "-m", "pip", "install", "-q", "--target", CUPTI_DIR, "--no-deps",
"cupti-python>=13.4.0", "nvidia-cuda-cupti~=13.4", "pyelftools"]).returncode == 0
import nsight # activates Nsight Compute's injection library: must come before anything initializes CUDA
from nsight.collection import ncu as ncu_backend
import numpy as np
import pandas as pd
import torch
SILENT = nsight.VerbosityLevel.SILENT
pd.set_option("display.width", 200)
pd.set_option("display.max_columns", 30)
print(f" nsight-python {nsight.__version__} | active tool: {nsight.get_active_tool()}"
f" | injection error at import: {ncu_backend.injection_load_error}")
print(f" torch {torch.__version__} (CUDA {torch.version.cuda}) | {torch.cuda.get_device_name(0)}"
f" | cupti-python installed: {CUPTI_EXTRA}")
print("\n Order matters: `import nsight` loads Nsight Compute into this Python process, and it can only")
print(" attach to CUDA work started after that. Every profiled function below runs in this same")
print(" process; nsight-python starts `ncu --mode attach` on its own PID for each one.")
我们首先检查 GPU 和分析器。Nsight Python 需要 Nsight Compute 命令行工具 ncu，版本为 2026.2.1 或更高，并通过在 PATH 上查找来找到它。Colab 的 GPU 镜像自带较旧版本的 Nsight Compute。不过，它已经配置了 NVIDIA 的 CUDA 软件包仓库，所以我们用 apt-get 安装 Nsight Compute 2026.3.1，用 dpkg 查询 ncu 启动器所在的目录，并把该目录放在 PATH 的最前面。然后我们安装 Nsight Python（固定到编写本笔记本时所用 1.1.0 预发布版本的 commit），并将 cupti-python 包安装到一个单独的目录中，因为它需要比 Colab 的 PyTorch 所固定的版本更新的 CUPTI 库。导入顺序很重要：import nsight 会将 Nsight Compute 的注入库加载到 Python 进程中，它只能附加到之后启动的 CUDA 工作，因此必须在 torch 触碰 GPU 之前导入 nsight。
复制代码ADD = lambda x, y: x + y # sums a metric if cuBLAS splits one matmul into several kernels
@nsight.analyze.kernel(thermal_mode="off", combine_kernel_metrics=ADD)
def first_matmul(n: int) -> None:
a = torch.randn(n, n, device="cuda")
b = torch.randn(n, n, device="cuda")
with nsight.annotate("matmul"):
_ = a @ b
@section("2. Profile one kernel: what a ProfileResults row contains")
def first_profile():
df = first_matmul(2048).to_dataframe()
print(f"\n to_dataframe() columns: {list(df.columns)}\n")
r = df.iloc[0]
t = float(r["AvgValue"])
print(f" annotation {r['Annotation']!r}, n = {r['n']}, kernel {r['Kernel']}")
print(f" {r['Metric']} = {t:,.0f} {r['Unit']} (runs {r['NumRuns']}, stable: {r['StableMeasurement']})")
print(f" GPU {r['GPU']}, ComputeClock {r['ComputeClock']}, MemoryClock {r['MemoryClock']}")
print(f" -> {2 * 2048 ** 3 / (t * 1e-9) / 1e12:.2f} TFLOP/s for a 2048 x 2048 fp32 matmul")
print("\n Only the kernel launched inside nsight.annotate() is profiled; torch.randn above it is not.")
print(" With a single run, StableMeasurement is <NA>: one sample has no spread to judge.")
return f"fp32 2048 matmul {t / 1e6:.3f} ms ({2 * 2048 ** 3 / (t * 1e-9) / 1e12:.2f} TFLOP/s)"
first_profile()
第一次性能分析是单个 fp32 矩阵乘法。基准测试函数先创建输入，然后仅将乘法包裹在 nsight.annotate 中，这样构建输入的随机数内核就不会被测量。调用被装饰的函数会在笔记本自身的进程上以附加模式启动 ncu，运行该函数，并返回一个 ProfileResults 对象，其 to_dataframe 方法为每个注释、配置和指标提供一行数据。我们打印列名并读取该行：内核名称、以纳秒为单位并带有单位的 gpu__time_duration.sum 测量值、运行次数、GPU 名称和时钟属性，然后我们手动将时间换算为 TFLOP/s。combine_kernel_metrics 参数会在 cuBLAS 把一次乘法拆分为多个内核时对时间求和；在单次运行时，StableMeasurement 标志是未定义的，因为单个样本没有离散度。
复制代码def show(df, cols, scale=None, fmt="{:,.2f}"):
out = df[cols].copy()
for c, f in (scale or {}).items():
out[c] = out[c] * f
print(textwrap.indent(out.to_string(index=False, float_format=lambda v: fmt.format(v)), " "))
@nsight.analyze.kernel(configs=[1 << 14, 1 << 24], runs=20, thermal_mode="off", verbosity=SILENT,
combine_kernel_metrics=ADD)
def noisy_or_not(n: int) -> None:
x = torch.randn(n, device="cuda")
y = torch.randn(n, device="cuda")
a = torch.randn(2048, 2048, device="cuda")
with nsight.annotate("vector add"):
_ = x + y
with nsight.annotate("matmul 2048"):
_ = a @ a
@section("3. How many runs? Spread, confidence intervals and the stability flag")
def stability():
df = noisy_or_not().to_dataframe()
print(" 20 runs per configuration; times in microseconds:\n")
show(df, ["Annotation", "n", "AvgValue", "StdDev", "CI95_Lower", "CI95_Upper", "RelativeStdDevPct",
"StableMeasurement"], scale={"AvgValue": 1e-3, "StdDev": 1e-3, "CI95_Lower": 1e-3, "CI95_Upper": 1e-3})
tiny = df[(df.Annotation == "vector add") & (df.n == 1 << 14)].iloc[0]
print(f"\n StableMeasurement is simply RelativeStdDevPct < 2. The smallest kernel ({tiny['AvgValue'] / 1e3:.1f} us)"
f" varies by {tiny['RelativeStdDevPct']:.1f}%:")
print(" at microsecond scale, launch and clock jitter are a large share of the time. Judge small kernels")
print(" with more runs, and compare their confidence intervals rather than single averages.")
return f"16K-element add RSD {tiny['RelativeStdDevPct']:.1f}%; " + \
f"2048 matmul RSD {df[df.Annotation == 'matmul 2048'].RelativeStdDevPct.max():.1f}%"
stability()
单次测量无法说明噪声情况，因此下一步对每个配置运行二十次。此时 DataFrame 会包含均值、标准差、95% 置信区间、以百分比表示的相对标准差，以及一个 StableMeasurement 标志，在源代码中该标志就是相对标准差低于 2%。我们分析两种大小的向量加法——一个以微秒运行的 16,000 元素版本和一个 16,000,000 元素版本——以及一个 2048×2048 矩阵乘法，并比较各自的变化程度。实用的规则是：对短内核运行更多次，并通过置信区间而非单个平均值进行比较。
复制代码def make_scale_bench(cache_control, clock_control):
@nsight.analyze.kernel(configs=[1 << 18, 1 << 24], runs=10, cache_control=cache_control,
clock_control=clock_control, thermal_mode="off", verbosity=SILENT,
metrics=["gpu__time_duration.sum", "gpc__cycles_elapsed.avg.per_second"])
def scale(n: int) -> None:
x = torch.randn(n, device="cuda")
with nsight.annotate("x * 2"):
_ = x * 2.0
return scale
@section("4. Cache and clock control: two defaults that change the answer")
def cache_and_clocks():
rows = []
for cache in ("all", "none"):
for clock in ("none", "base"):
try:
df = make_scale_bench(cache, clock)().to_dataframe()
except Exception as e:
print(f" cache_control={cache!r}, clock_control={clock!r}: failed -> {type(e).__name__}: {str(e)[:120]}")
continue
t = df[df.Metric == "gpu__time_duration.sum"].set_index("n").AvgValue
hz = df[df.Metric == "gpc__cycles_elapsed.avg.per_second"].set_index("n").AvgValue
for n in t.index:
rows.append({"cache_control": cache, "clock_control": clock, "MB": 4 * n / 2 ** 20,
"time_us": t[n] / 1e3, "GB/s": 2 * 4 * n / t[n], "SM_MHz": hz[n] / 1e6})
tab = pd.DataFrame(rows)
print(textwrap.indent(tab.to_string(index=False, float_format=lambda v: f"{v:,.1f}"), " "))
print("\n The T4's L2 cache is 4 MB. A 1 MB input stays in L2 between runs unless caches are flushed,")
print(" so cache_control='none' measures a warm-cache kernel and 'all' (the default) a cold one.")
print(" clock_control='base' locks the SM clock to base frequency for every replay (the plain ncu CLI")
print(" default); nsight-python defaults to 'none', so the GPU boosts. SM_MHz is measured per kernel")
print(" (gpc__cycles_elapsed.avg.per_second). The ComputeClock column is not: it is the device's")
print(" rated clock attribute and stays the same whatever the kernel ran at.")
globals()["CLOCK_TABLE"] = tab
small = tab[tab.MB < 2].set_index(["cache_control", "clock_control"]).time_us
return (f"1 MB scale: cold/boost {small.get(('all', 'none'), float('nan')):.1f} us, "
f"warm/boost {small.get(('none', 'none'), float('nan')):.1f} us, "
f"cold/base {small.get(('all', 'base'), float('nan')):.1f} us")
cache_and_clocks()
有两个分析器默认设置决定了一次测量的含义，本步骤在同一个内核上测量这两者：将向量乘以 2。cache_control 决定在每次内核重放之前是否刷新缓存：Nsight Python 默认会刷新它们，因此原本可能留在 T4 的 4 MB L2 缓存中的小输入每次都要从内存中读取。clock_control 决定是否锁定 GPU 时钟：Nsight Python 默认不锁定它们，而普通的 ncu 命令行会将时钟锁定到基础频率。我们在 1 MB 和 64 MB 下运行全部四种组合，并额外收集 gpc__cycles_elapsed.avg.per_second，它测量内核实际运行时的时钟频率。这很重要，因为正如源代码所示，DataFrame 的 ComputeClock 列是设备的额定时钟属性，而不是实际测量值。
复制代码SIZES = [(512,), (1024,), (2048,), (4096,)]
def tflops(time_ns: float, n: int) -> tuple[float, str]:
return 2 * n ** 3 / (time_ns * 1e-9) / 1e12, "TFLOP/s"
def make_precision_body():
def precision(n: int) -> None: # a fresh function object for each decorator
a = torch.randn(n, n, device="cuda")
b = torch.randn(n, n, device="cuda")
a16, b16 = a.half(), b.half()
with nsight.annotate("fp32"):
_ = a @ b
with nsight.annotate("fp16"):
_ = a16 @ b16
return precision
precision_tflops = nsight.analyze.kernel(configs=SIZES, runs=5, derive_metric=tflops, thermal_mode="off",
verbosity=SILENT, combine_kernel_metrics=ADD)(make_precision_body())
precision_ratio = nsight.analyze.kernel(configs=SIZES, runs=5, normalize_against="fp32", thermal_mode="off",
verbosity=SILENT, combine_kernel_metrics=ADD)(make_precision_body())
@section("5. A size sweep, a derived TFLOP/s metric, and normalisation")
def precision_sweep():
df = precision_tflops().to_dataframe()
tf = df[df.Metric == "tflops"].pivot(index="n", columns="Annotation", values="AvgValue")
print(" derive_metric=tflops receives (gpu__time_duration.sum, n) and returns (value, 'TFLOP/s'):\n")
print(textwrap.indent(tf[["fp32", "fp16"]].to_string(float_format=lambda v: f"{v:,.2f}"), " "))
print("\n T4 datasheet peaks: 8.1 TFLOP/s fp32, 65 TFLOP/s fp16 on tensor cores.")
rd = precision_ratio().to_dataframe()
fp16 = rd[rd.Annotation == "fp16"][["n", "AvgValue", "StdDev", "Unit", "Normalized"]]
print("\n normalize_against='fp32' divides each AvgValue by the fp32 AvgValue of the same configuration:\n")
print(textwrap.indent(fp16.to_string(index=False, float_format=lambda v: f"{v:,.3f}"), " "))
print("\n Read the row carefully: AvgValue is now a ratio (fp16 time / fp32 time), but StdDev and Unit are")
print(" still those of the raw nanosecond measurement, because only AvgValue is divided.")
best = tf.loc[4096]
return f"n=4096: fp32 {best['fp32']:.2f}, fp16 {best['fp16']:.2f} TFLOP/s"
precision_sweep()
接下来，我们把矩阵大小从 512 扫描到 4096，并在一个基准测试函数中同时运行 fp32 和 fp16，每种精度各加一条注解。一个 derive_metric 函数接收测得的时间，随后是基准测试的参数，并返回一个带单位的值，这里是 TFLOP/s，DataFrame 会将它作为时间之外的一个额外指标存储。T4 的数据手册峰值是 fp32 下 8.1 TFLOP/s 以及其 tensor cores 上 fp16 下 65 TFLOP/s，这为测得的数值提供了参照尺度。然后我们再次运行同样的扫描，将 normalize_against 设置为 fp32 注解，这样每个 AvgValue 都会除以相同配置下的 fp32 值。正如转换的源代码所示，只有 AvgValue 被除：StdDev、置信区间和 Unit 列仍然描述的是原始的纳秒测量值，这很容易被误读。
Copy CodeBOUND_METRICS = ["gpu__time_duration.sum",
"sm__throughput.avg.pct_of_peak_sustained_elapsed",
"dram__throughput.avg.pct_of_peak_sustained_elapsed"]
@nsight.analyze.kernel(metrics=BOUND_METRICS, runs=3, thermal_mode="off", verbosity=SILENT)
def what_limits(n: int) -> None:
a = torch.randn(n, n, device="cuda").half()
x = torch.randn(64 * 2 ** 20, device="cuda")
s = torch.randn(8192, 1024, device="cuda")
with nsight.annotate("fp16 matmul 4096"):
_ = a @ a
with nsight.annotate("x * 2 (256 MB)"):
_ = x * 2.0
with nsight.annotate("softmax 8192x1024"):
_ = torch.softmax(s, dim=-1)
@section("6. Compute-bound or memory-bound? Three metrics from one replayed kernel")
def bound():
df = what_limits(4096).to_dataframe()
tab = df.pivot(index="Annotation", columns="Metric", values="AvgValue")
tab = tab.rename(columns={"gpu__time_duration.sum": "ms",
"sm__throughput.avg.pct_of_peak_sustained_elapsed": "SM % of peak",
"dram__throughput.avg.pct_of_peak_sustained_elapsed": "DRAM % of peak"})
tab["ms"] = tab["ms"] / 1e6
tab["limited by"] = np.where(tab["SM % of peak"] > tab["DRAM % of peak"], "compute", "memory")
print(textwrap.indent(tab.to_string(float_format=lambda v: f"{v:,.2f}"), " "))
x2 = tab.loc["x * 2 (256 MB)"]
gbs = 2 * 4 * 64 * 2 ** 20 / (x2["ms"] * 1e6)
print(f"\n Cross-check: x * 2 moves 2 x 256 MB in {x2['ms']:.2f} ms = {gbs:.0f} GB/s, against the T4's 320 GB/s")
print(f" datasheet peak ({100 * gbs / 320:.0f}%); Nsight Compute reports {x2['DRAM % of peak']:.0f}% DRAM throughput.")
print(" Collecting several metrics makes Nsight Compute replay each kernel once per pass it needs;")
print(" nsight-python still returns one row per annotation and metric.")
return f"x*2: {gbs:.0f} GB/s ({x2['DRAM % of peak']:.0f}% DRAM); fp16 matmul SM {tab.loc['fp16 matmul 4096', 'SM % of peak']:.0f}%"
bound()
硬件计数器告诉我们一个 kernel 为什么需要那么长的时间。传入一个指标列表会让 Nsight Compute 按照需要将每个 kernel 重放多次以收集全部指标，而 Nsight Python 仍然为每条注解和每个指标返回一行。我们收集三个不同特点的 kernel 的时长、SM 吞吐量和 DRAM 吞吐量（均以持续峰值的百分比表示）：一个 fp16 矩阵乘法、一个对 256 MB 向量的缩放，以及对 8192 行、每行 1024 个值的 softmax，并按其中较高的吞吐量为每个 kernel 命名。对于向量缩放，我们还根据它必须移动的字节数和测得的时间计算有效带宽，并将其与 T4 的 320 GB/s 数据手册峰值以及 Nsight Compute 报告的 DRAM 百分比进行比较。
Copy CodeBYTES = ["gpu__time_duration.sum", "dram__bytes_read.sum", "dram__bytes_write.sum"]
@nsight.analyze.kernel(metrics=BYTES, runs=5, thermal_mode="off", verbosity=SILENT,
combine_kernel_metrics=lambda x, y: x + y)
def fusion(n: int) -> None:
a, b, c = (torch.randn(n, device="cuda") for _ in range(3))
with nsight.annotate("a + b * c (unfused)"):
_ = a + b * c
with nsight.annotate("torch.addcmul (fused)"):
_ = torch.addcmul(a, b, c)
@nsight.analyze.kernel(runs=1, thermal_mode="off", verbosity=SILENT)
def fusion_without_combine(n: int) -> None:
a, b, c = (torch.randn(n, device="cuda") for _ in range(3))
with nsight.annotate("a + b * c"):
_ = a + b * c
@section("7. Kernel fusion, measured in bytes: combine_kernel_metrics")
def fused():
n = 32 * 2 ** 20
try:
fusion_without_combine(n)
print(" (the unfused expression profiled without combine_kernel_metrics)")
except Exception as e:
print(f" Profiled without combine_kernel_metrics, the unfused expression raises:\n {type(e).__name__}: {str(e).splitlines()[0][:200]}\n")
df = fusion(n).to_dataframe()
tab = df.pivot(index="Annotation", columns="Metric", values="AvgValue")
tab = pd.DataFrame({"ms": tab["gpu__time_duration.sum"] / 1e6,
"MB read": tab["dram__bytes_read.sum"] / 2 ** 20,
"MB written": tab["dram__bytes_write.sum"] / 2 ** 20})
print(f" {n:,} float32 elements per tensor (128 MB each); metrics summed over the kernels in each region:\n")
print(textwrap.indent(tab.to_string(float_format=lambda v: f"{v:,.1f}"), " "))
u, f = tab.loc["a + b * c (unfused)"], tab.loc["torch.addcmul (fused)"]
print(f"\n The unfused version writes b * c to memory and reads it back: {u['MB read'] + u['MB written']:,.0f} MB of")
print(f" traffic against {f['MB read'] + f['MB written']:,.0f} MB, and {u['ms'] / f['ms']:.2f}x the time.")
return f"fusion: {u['ms'] / f['ms']:.2f}x faster, {(u['MB read'] + u['MB written']) / (f['MB read'] + f['MB written']):.2f}x less traffic"
fused()
Kernel 融合通常以时间来描述；这里我们用字节数来衡量它。在 eager PyTorch 中，表达式 a + b * c 会先把 b * c 物化到内存中再与 a 相加，而 torch.addcmul 则在一次操作中计算出相同的结果。我们首先在不使用 combine_kernel_metrics 的情况下分析未融合的表达式，以展示当一个注解区域包含多于一个 kernel 时 Nsight Python 抛出的错误，然后在 combine_kernel_metrics 对每个区域中所有 kernel 的时长以及读写的 DRAM 字节数求和的情况下分析两个版本，所用的张量为 2^25（约 33.5 百万）个 float32 元素。
Copy Codeimport pynvml
pynvml.nvmlInit()
NVML = pynvml.nvmlDeviceGetHandleByIndex(0)
@nsight.experimental.collect(scope=nsight.CollectionScope.ONCE)
def driver_version(*args):
return pynvml.nvmlSystemGetDriverVersion()
@nsight.experimental.collect(scope=nsight.CollectionScope.RUN)
def sm_clock_mhz(*args):
return pynvml.nvmlDeviceGetClockInfo(NVML, pynvml.NVML_CLOCK_SM)
@nsight.experimental.collect(scope=nsight.CollectionScope.RUN)
def gpu_temp_c(*args):
return pynvml.nvmlDeviceGetTemperature(NVML, pynvml.NVML_TEMPERATURE_GPU)
def make_heat_body():
def heat(n: int) -> None:
a = torch.randn(n, n, device="cuda").half()
with nsight.annotate("fp16 matmul"):
_ = a @ a
return heat
@section("8. Custom info collectors and thermal control (experimental API)")
def collectors():
out = {}
for mode in ("off", "auto"):
bench = nsight.analyze.kernel(configs=[4096], runs=30, thermal_mode=mode, thermal_timeout=60,
info_collectors=[driver_version, sm_clock_mhz, gpu_temp_c],
verbosity=SILENT, output_prefix=f"/content/nspy_thermal_{mode}/",
output_csv=True)(make_heat_body())
t0 = time.time()
try:
df = bench().to_dataframe()
except Exception as e:
print(f" thermal_mode={mode!r}: failed -> {type(e).__name__}: {str(e).splitlines()[0][:150]}")
continue
wall = time.time() - t0
csvs = sorted(glob.glob(f"/content/nspy_thermal_{mode}/profiled_data-*.csv"))
raw = pd.read_csv(csvs[-1]) if csvs else pd.DataFrame()
out[mode] = (df.iloc[0], wall, raw)
r = df.iloc[0]
print(f" thermal_mode={mode!r}: {r['AvgValue'] / 1e6:.3f} ms +/- {r['RelativeStdDevPct']:.1f}% "
f"wall {wall:.0f}s driver {r['driver_version']}")
print(f"\n The collectors became DataFrame columns; RUN-scoped ones were called before every run.")
if "off" not in out:
return "info-collector run failed"
raw = out["off"][2]
tcol = "Value" if "Value" in raw.columns else raw.columns[-1]
print(f" raw per-run CSV columns: {list(raw.columns)}")
if {"sm_clock_mhz", "gpu_temp_c"} <= set(raw.columns):
corr = raw[[tcol, "sm_clock_mhz", "gpu_temp_c"]].corr().iloc[0]
print(f" over 30 runs: SM clock {raw.sm_clock_mhz.min()}-{raw.sm_clock_mhz.max()} MHz, "
f"temperature {raw.gpu_temp_c.min()}-{raw.gpu_temp_c.max()} C")
print(f" correlation of kernel time with SM clock {corr['sm_clock_mhz']:+.2f}, with temperature {corr['gpu_temp_c']:+.2f}")
print("\n thermal_mode='auto' (the default) pauses between runs when the GPU's thermal headroom runs low;")
print(" the wall-time difference is the price of that, the spread difference what it buys.")
return "; ".join(f"{m} {out[m][0]['RelativeStdDevPct']:.1f}% RSD in {out[m][1]:.0f}s" for m in out)
collectors()
版本 1.1 增加了自定义信息收集器，这是一个用于将应用特定的值附加到结果上的实验性 API。一个用 nsight.experimental.collect 和收集作用域装饰的函数会在每个会话、每个配置、每次运行或每条注解时被调用一次，其返回值会成为一列。我们记录一次驱动版本，并在一个长时间 fp16 乘法的每次运行之前通过 NVML 读取 SM 时钟和 GPU 温度，同时启用 output_csv，以便每次运行的行写在聚合行旁边。我们将基准测试运行两次，一次关闭热控制，一次使用默认的 auto 模式（在该模式下，当 GPU 的热余量过低时 Nsight Python 会在运行之间暂停），并比较两者的波动幅度和挂钟时间。
Copy CodeCUPTI_SCRIPT = r'''
import os, json, time
os.environ["NSPY_NCU_INIT_AT_IMPORT"] = "0" # do not load Nsight Compute in this process
import nsight
nsight.activate(nsight.Tool.CUPTI)
import torch
@nsight.analyze.kernel(configs=[512, 1024, 2048, 4096], runs=10, thermal_mode="off",
verbosity=nsight.VerbosityLevel.SILENT, combine_kernel_metrics=lambda x, y: x + y)
def matmul(n):
a = torch.randn(n, n, device="cuda")
b = torch.randn(n, n, device="cuda")
with nsight.annotate("matmul"):
_ = a @ b
t0 = time.time()
df = matmul().to_dataframe()
wall = time.time() - t0
nsight.deactivate()
print(json.dumps({"wall": wall, "rows": df[["n", "AvgValue", "RelativeStdDevPct"]].to_dict("records")}))
'''
@nsight.analyze.kernel(configs=[512, 1024, 2048, 4096], runs=10, thermal_mode="off", verbosity=SILENT,
combine_kernel_metrics=ADD)
def matmul_ncu(n: int) -> None:
a = torch.randn(n, n, device="cuda")
b = torch.randn(n, n, device="cuda")
with nsight.annotate("matmul"):
_ = a @ b
@section("9. The experimental CUPTI backend, in its own process, against Nsight Compute")
def cupti_vs_ncu():
t0 = time.time()
ncu_df = matmul_ncu().to_dataframe().set_index("n")
ncu_wall = time.time() - t0
if not CUPTI_EXTRA:
return f"cupti-python could not be installed; Nsight Compute run took {ncu_wall:.0f}s"
with open("/content/cupti_matmul.py", "w") as f:
f.write(CUPTI_SCRIPT)
env = {k: v for k, v in os.environ.items() if k not in ("NVTX_INJECTION64_PATH", "NV_COMPUTE_PROFILER_PERFWORKS_DIR",
"NV_COMPUTE_PROFILER_IN_PROCESS")}
env["PYTHONPATH"] = CUPTI_DIR + os.pathsep + env.get("PYTHONPATH", "")
p = subprocess.run([sys.executable, "/content/cupti_matmul.py"], capture_output=True, text=True, env=env, timeout=900)
if p.returncode != 0:
print(f" CUPTI run failed (exit {p.returncode}); last lines of its stderr:")
print(textwrap.indent("\n".join(p.stderr.strip().splitlines()[-8:]), " "))
return f"CUPTI backend failed (exit {p.returncode}); Nsight Compute run took {ncu_wall:.0f}s"
res = next((json.loads(line) for line in reversed(p.stdout.strip().splitlines()) if line.startswith('{"wall"')), None)
if res is None:
return f"CUPTI run printed no result; Nsight Compute run took {ncu_wall:.0f}s"
cu = pd.DataFrame(res["rows"]).set_index("n")
print(f" {'n':>5s} {'ncu ms':>8s} {'cupti ms':>9s} {'ratio':>6s} {'ncu RSD%':>9s} {'cupti RSD%':>11s}")
for n in ncu_df.index:
a, b = ncu_df.loc[n, "AvgValue"] / 1e6, cu.loc[n, "AvgValue"] / 1e6
print(f" {n:5d} {a:8.3f} {b:9.3f} {b / a:6.2f} {ncu_df.loc[n, 'RelativeStdDevPct']:9.1f} {cu.loc[n, 'RelativeStdDevPct']:11.1f}")
print(f"\n wall time for 4 sizes x 10 runs: Nsight Compute {ncu_wall:.1f}s, CUPTI {res['wall']:.1f}s"
f" ({ncu_wall / res['wall']:.1f}x)")
print(" The two tools are mutually exclusive in one process and Nsight Compute cannot be unloaded, so")
print(" CUPTI runs in a child process with NSPY_NCU_INIT_AT_IMPORT=0. CUPTI records one execution per")
print(" launch, with no replay, no cache flush and no clock control: cheaper, and warmer caches.")
return f"CUPTI {ncu_wall / res['wall']:.1f}x faster to profile; durations within {abs(1 - cu.AvgValue.div(ncu_df.AvgValue)).max():.0%}"
cupti_vs_ncu()
CUPTI 后端是 1.1 的另一项核心特性：通过 CUPTI 的 activity API 测量 kernel 时长，而不使用 Nsight Compute。这两个工具在单个进程中互斥。由于 Nsight Compute 一旦激活就无法卸载，我们编写了一个小脚本，在导入 nsight 之前将 NSPY_NCU_INIT_AT_IMPORT 设置为 0，激活 CUPTI 工具，分析同样的四个矩阵大小，并将其作为子进程运行，且把 cupti-python 目录加入其路径。CUPTI 只支持 duration 指标，并且每次启动只记录一次执行，没有重放、缓存刷新或时钟控制，因此我们将两者的时长以及分析的挂钟开销与 Nsight Compute 下相同的基准测试进行比较。
Copy Code@section("10. Keep the evidence: CSVs and the full Nsight Compute report")
def evidence():
files = sorted(glob.glob("/content/nspy_thermal_off/*"))
for f in files:
print(f" {os.path.basename(f):60s} {os.path.getsize(f) / 1024:9.1f} KB")
import ncu_report
rep_path = next((f for f in files if f.endswith(".ncu-rep")), None)
if rep_path is None:
raise RuntimeError("no .ncu-rep in /content/nspy_thermal_off/ (step 8 did not complete)")
rep = ncu_report.load_report(rep_path)
actions = [rep.range_by_idx(i).action_by_idx(j) for i in range(rep.num_ranges())
for j in range(rep.range_by_idx(i).num_actions())]
a0 = actions[0]
print(f"\n the .ncu-rep holds {len(actions)} profiled kernel launches; the first is")
print(f" {a0.name()}")
print(f" gpu__time_duration.sum = {a0['gpu__time_duration.sum'].value():,.0f} {a0['gpu__time_duration.sum'].unit()}")
print(f" {len(a0.metric_names())} metrics available in the report for this launch")
print("\n output_csv=True writes profiled_data (one row per run) and processed_data (the aggregated")
print(" DataFrame). The .ncu-rep opens in the Nsight Compute GUI for source-level analysis.")
return f"{len(actions)} launches in the report, {len(a0.metric_names())} metrics each"
evidence()
每一次被分析的调用都会留下证据。通过一个输出前缀和 output_csv，目录中会保存 Nsight Compute 报告、其日志、每次运行的 profiled_data CSV 以及聚合的 processed_data CSV。我们列出这些文件并用 ncu_report 打开报告，这是 Nsight Compute 的 Python 报告接口，Nsight Python 本身也在使用它：它列出了被分析的 kernel 启动，并且每一个都携带了所收集的每个指标。同一个 .ncu-rep 文件也可以在 Nsight Compute 界面中打开以进行源码级分析。
Copy Codebanner("SUMMARY")
for name, res in RESULTSitems():
print(f" {name:<76s} {res}")
print("""
What to carry over
- import nsight first; it loads Nsight Compute into the process before CUDA starts.
- Annotate exactly the work you mean to measure; one kernel per region unless you pass
combine_kernel_metrics.
- Decide cache and clock control on purpose: nsight-python flushes caches and leaves clocks
unlocked by default, which is not what a plain ncu run does.
- Use runs and the CI95 / RelativeStdDevPct columns; one run tells you nothing about noise.
- After normalize_against, only AvgValue is a ratio.
Where to go next
- Triton and cuTile kernels: examples/07_triton_minimal.py and examples/12_cutile.py.
- Plots: @nsight.analyze.plot(...) on top of @nsight.analyze.kernel, with row/col panels.
- Docs: docs.nvidia.com/nsight-python Code: github.com/NVIDIA/nsight-python
""")
总结部分打印每个步骤的单行结果以及值得保持的习惯：在任何东西初始化 CUDA 之前导入 nsight，只注解你确实想要测量的工作，审慎地选择缓存和时钟控制，使用足够的运行次数并阅读置信区间，并且记住只有 AvgValue 会被归一化。最后它指向仓库中的 Triton 和 cuTile 示例以及绘图装饰器。
总而言之，Nsight Python 将内核性能分析从单独的命令行会话转变为被测量代码中的几行内容：一个装饰器、一个注解，以及一个承载测量值、其离散程度及其上下文的 DataFrame。在 Colab T4 上实践它的过程也揭示了这种便利性中隐藏着哪些会改变数字的决策。分析器必须比 Colab 自带的版本更新，并且必须在 CUDA 启动之前加载。缓存刷新和时钟锁定的默认设置与直接运行 ncu 时相反；clock 列报告的是显卡的额定值而非内核实际运行的时钟频率；归一化只对平均值进行除法。除非你说明如何合并，否则每个注解区域应恰好包含一个内核。一旦这些选择变得明确，同样这几行代码就能回答实践中真正重要的问题：计时有多稳定、什么限制了内核、融合究竟节省了多少，以及像新的 CUPTI 后端这样更便宜的分析器为更低的额外开销牺牲了什么。
请查看 完整代码。所有功劳归于该项目的研究者。此外，欢迎在 Twitter 上关注我们，别忘了加入我们的 150k+ML SubReddit 并订阅 我们的通讯。等等！你在用 telegram 吗？ 现在你也可以在 telegram 上加入我们了。