# Performance Testing Framework The `perf` sub-module provides a framework for writing and running performance tests. ## Default Command Options The flags below match the .NET `Azure.Test.Perf` runner (`PerfOptions.cs`) so that perf-automation can drive both languages from a single matrix. | Flag | Short | Default | Description | | ---- | ----- | ------- | ----------- | | `--duration` | `-d` | 10 | Measurement-phase duration in seconds. | | `--warmup` | `-w` | 5 | Warmup duration in seconds. Zero skips warmup. | | `--parallel` | `-p` | 1 | Number of goroutines executing the test concurrently. | | `--iterations` | `-i` | 1 | How many times the measurement phase is repeated and aggregated into the final results. | | `--rate` | `-r` | 0 | Target throughput in ops/sec aggregated across workers. Zero = unlimited. | | `--status-interval` | | 1 | Seconds between live status lines. | | `--latency` | `-l` | false | Track per-operation latency and print a percentile summary. | | `--job-statistics` | | false | Print the `#StartJobStatistics` / `#EndJobStatistics` block parsed by perf-automation. | | `--no-cleanup` | | false | Skip `Cleanup` / `GlobalCleanup` at end of run. | | `--sync` | | false | Accepted for CLI parity; no-op in Go. | | `--insecure` | | false | Accepted for CLI parity; the default transport already skips TLS verification for the test proxy. | | `--test-proxies` | `-x` | | Semicolon-separated list of test-proxy URLs. | | `--results-file` | | | When combined with `--latency`, writes per-operation results as JSON. | | `--max-results` | | 1000000 | Maximum sampled operation records retained across all workers. Zero is unbounded. | | `--output-file-prefix` | | | Writes run summary artifacts to `.json/.csv/.txt/.md`. | | `--profile` | | false | Collect a Go CPU profile across setup, warmup, measurement, and cleanup. | | `--profile-path` | | `cpu.pprof` | CPU profile destination when `--profile` is set. | | `--resource-telemetry` | | false | Print a `runtime.MemStats` / goroutine-count delta at end of run. | | `--config` | | | Path to a workload-config JSON file. | | `--workload` | | | Workload name to select from the config file. | | `--debug` | | false | Print extra debug output. | | `--maxprocs` | | `runtime.NumCPU()` | Override `GOMAXPROCS` for the run. | | `--max-io-completion-threads` | | 0 | Accepted for CLI parity with .NET ThreadPool tuning; no-op in Go. | | `--max-worker-threads` | | 0 | Accepted for CLI parity with .NET ThreadPool tuning; no-op in Go. | | `--min-io-completion-threads` | | 0 | Accepted for CLI parity with .NET ThreadPool tuning; no-op in Go. | | `--min-worker-threads` | | 0 | Accepted for CLI parity with .NET ThreadPool tuning; no-op in Go. | The runner always samples process CPU and memory in the background; both are shown live in the status line (`CPU`, `Memory(MiB)`) and as `averageCpuPercent` / `averageMemoryBytes` in the run-summary artifacts. Latency percentile collection retains a bounded sample across all workers to avoid memory growth during long, high-throughput runs. ## Testing the azblob Performance Tests The following examples exercise upload, download, and list operations while collecting a CPU profile. Run them from the azblob perf-test module: ```bash cd sdk/storage/azblob/testdata/perf ``` For OAuth authentication, sign in with the Azure CLI and set the storage account name. The identity must have permission to create containers and blobs, such as the Storage Blob Data Contributor role. ```bash az login export AZURE_STORAGE_ACCOUNT_NAME="" unset AZURE_STORAGE_CONNECTION_STRING ``` For sovereign clouds or custom storage endpoints, set the full account URL instead of the account name: ```bash export AZURE_STORAGE_ACCOUNT_URL="https://" unset AZURE_STORAGE_ACCOUNT_NAME ``` Alternatively, unset `AZURE_STORAGE_ACCOUNT_NAME` and set `AZURE_STORAGE_CONNECTION_STRING` to use shared-key authentication. ### Upload ```bash go run . UploadBlobTest \ --size 10240 \ --iterations 1 \ --warmup 0 \ --profile \ --profile-path ./upload-blob.pprof ``` ### Download ```bash go run . DownloadBlobTest \ --size 10240 \ --iterations 1 \ --warmup 0 \ --profile \ --profile-path ./download-blob.pprof ``` ### List ```bash go run . ListBlobTest \ --num-blobs 100 \ --num-blobs-parallelism 8 \ --iterations 1 \ --warmup 0 \ --profile \ --profile-path ./list-blobs.pprof ``` Verify and inspect any generated profile with `go tool pprof`: ```bash test -s ./list-blobs.pprof go tool pprof -top ./list-blobs.pprof ``` ## Adding Performance Tests to an SDK 1. Create a performance test directory at `testdata/perf` within your module. For example, the storage performance tests live in `sdk/storage/azblob/testdata/perf`. 2. Run `go mod init` to create a new module. Add the `sdk/internal` module and the module you are testing for to your go.mod file and run `go mod tidy` 3. Create a `struct` that maintains all global values for the performance test (ie. account name, blob name, etc.). 4. Implement `GlobalPerfTest` interface by adding the `NewPerfTest(context.Context, *PerfTestOptions) (PerfTest, error)` and the `GlobalCleanup(context.Context)` functions on the global struct. The `NewPerfTest` method creates a new struct that is responsible for running a performance test in its own `goroutine`. `GlobalCleanup` takes care of any session level cleanup. 5. Implement the `PerfTest` interface on the struct returned from the `NewPerfTest` method. Add a `Run(context.Context)` and a `Cleanup(context.Context)` function. The `Run` method is the method you want to measure the performance of. The `Cleanup` method is for cleaning up within a single `goroutine`, most of the time this will be empty. 6. Create a main package that uses the `perf.Run` function to provide all the global constructor methods. `perf.Run` takes a map[string]PerfMethods 6. (Optional): Create a method that registers local flags. ```golang package main import ( "github.com/Azure/azure-sdk-for-go/sdk/internal/perf" ) // uploadTestRegister registers flags for the "UploadBlobTest" // This is optional and does not to be included for every test. func uploadTestRegister() { flag.IntVar(&uploadTestOpts.size, "size", 10240, "Size in bytes of data to be transferred in upload or download tests. Default is 10240.") } func main() { perf.Run(map[string]perf.PerfMethods{ "UploadBlobTest": {Register: uploadTestRegister, New: NewUploadTest}, "ListBlobTest": {Register: listTestRegister, New: NewListTest}, "DownloadBlobTest": {Register: downloadTestRegister, New: NewDownloadTest}, }) } ``` ### Writing a test The following walks through the `azblob` Blob Upload performance test: #### `GlobalPerfTest` Interface This struct handles global set up for your account and spawning structs for each `goroutine`. Make sure to embed the `perf.PerfTestOptions` struct in your global struct. The `NewUploadTest` will be run once per process and is used to spawn each parallel test. ```go type uploadTestGlobal struct { perf.PerfTestOptions containerName string blobName string globalContainerClient azblob.ContainerClient } // NewUploadTest is called once per process func NewUploadTest(ctx context.Context, options perf.PerfTestOptions) (perf.GlobalPerfTest, error) { u := &uploadTestGlobal{ PerfTestOptions: options, containerName: "uploadcontainer", blobName: "uploadblob", } connStr, ok := os.LookupEnv("AZURE_STORAGE_CONNECTION_STRING") if !ok { return nil, fmt.Errorf("the environment variable 'AZURE_STORAGE_CONNECTION_STRING' could not be found") } containerClient, err := azblob.NewContainerClientFromConnectionString(connStr, u.containerName, nil) if err != nil { return nil, err } u.globalContainerClient = containerClient _, err = u.globalContainerClient.Create(context.Background(), nil) if err != nil { return nil, err } return u, nil } ``` The `GlobalCleanup` method cleans up the account, resetting it into the default. ```go func (m *uploadTestGlobal) GlobalCleanup(ctx context.Context) error { connStr, ok := os.LookupEnv("AZURE_STORAGE_CONNECTION_STRING") if !ok { return fmt.Errorf("the environment variable 'AZURE_STORAGE_CONNECTION_STRING' could not be found") } containerClient, err := azblob.NewContainerClientFromConnectionString(connStr, m.containerName, nil) if err != nil { return err } _, err = containerClient.Delete(context.Background(), nil) return err } ``` `NewPerfTest` is called once per `goroutine`, and creates a new `PerfTest` interface which will be used by each goroutine. This method should also include the setup for the eventual returned struct. ```go // uploadPerfTest implements perf.PerfTest and is used to test performance of your event type uploadPerfTest struct { *uploadTestGlobal perf.PerfTestOptions data io.ReadSeekCloser blobClient azblob.BlockBlobClient } // NewPerfTest is called once per goroutine func (g *uploadTestGlobal) NewPerfTest(ctx context.Context, options *perf.PerfTestOptions) (perf.PerfTest, error) { u := &uploadPerfTest{ uploadTestGlobal: g, PerfTestOptions: *options, } connStr, ok := os.LookupEnv("AZURE_STORAGE_CONNECTION_STRING") if !ok { return nil, fmt.Errorf("the environment variable 'AZURE_STORAGE_CONNECTION_STRING' could not be found") } containerClient, err := azblob.NewContainerClientFromConnectionString( connStr, u.uploadTestGlobal.containerName, &azblob.ClientOptions{ Transporter: u.PerfTestOptions.Transporter, }, ) if err != nil { return nil, err } bc := containerClient.NewBlockBlobClient(u.blobName) u.blobClient = bc data, err := perf.NewRandomStream(uploadTestOpts.size) if err != nil { return nil, err } u.data = data return u, nil } ``` #### `PerfTest` Interface The `PerfTest` interface is an interface that is responsible for running a single performance test. Each performance test in run within a single `goroutine`, these `goroutine`s are created by the `perf` framework. The `Run` method is the method that is being measured and the `Cleanup` method is responsible for cleanup work after each goroutine completes or errors out. ```go func (m *uploadPerfTest) Run(ctx context.Context) error { _, err := m.data.Seek(0, io.SeekStart) // rewind to the beginning if err != nil { return err } _, err = m.blobClient.Upload(ctx, m.data, nil) return err } func (m *uploadPerfTest) Cleanup(ctx context.Context) error { return nil } ``` #### Constructor Each method needs to have a constructor that returns an instantialized `perf.GlobalPerfTest`. ```go // NewUploadTest is called once per process func NewUploadTest(ctx context.Context, options perf.PerfTestOptions) (perf.GlobalPerfTest, error) { u := &uploadTestGlobal{ PerfTestOptions: options, containerName: "uploadcontainer", blobName: "uploadblob", } connStr, ok := os.LookupEnv("AZURE_STORAGE_CONNECTION_STRING") if !ok { return nil, fmt.Errorf("the environment variable 'AZURE_STORAGE_CONNECTION_STRING' could not be found") } containerClient, err := azblob.NewContainerClientFromConnectionString(connStr, u.containerName, nil) if err != nil { return nil, err } _, err = containerClient.Create(context.Background(), nil) if err != nil { return nil, err } return u, nil } ``` #### `main.go` file The `main` function must have the `perf.Run` method with a map of each performance test constructor. Flag parsing, test selection, reporting, proxy interfaces, and all other portions are taken care of for the user in the `perf.Run` method. ```go package main import ( "github.com/Azure/azure-sdk-for-go/sdk/internal/perf" ) func main() { perf.Run(map[string]perf.PerfMethods{ "UploadBlobTest": {Register: uploadTestRegister, New: NewUploadTest}, "ListBlobTest": {Register: listTestRegister, New: NewListTest}, "DownloadBlobTest": {Register: downloadTestRegister, New: NewDownloadTest}, }) } ``` ### Running with the test proxy To run with the test proxy, configure your client to route requests to the test proxy using the `Transporter` parameters for the `Transport` in your client options. If the `--test-proxy` flag is not specified, `Transporter` is the same default HTTP client from `azcore`. ```go func (k *keysPerfTest) GlobalSetup() error { ... options = &azkeys.ClientOptions{ ClientOptions: azcore.ClientOptions{ Transport: k.Transporter, }, } client, err := azkeys.NewClient("", cred, options) if err != nil {return err} ... } ``` ### Registering Local Arguments We use the `flag` library for argument parsing. Each perf test suite can have optional arguments and they are registered using the `perf.RegisterArguments` method, which takes a simple function that registers arguments using `flag.Int`, `flag.String`, etc. If you have no local arguments, you can skip this step. ```go type uploadTestOptions struct { size int } var uploadTestOpts uploadTestOptions = uploadTestOptions{size: 10240} // uploadTestRegister is called once per process func uploadTestRegister() { flag.IntVar(&uploadTestOpts.size, "size", 10240, "Size in bytes of data to be transferred in upload or download tests.") } ``` The `main.go` file changes to: ```go package main import "github.com/Azure/azure-sdk-for-go/sdk/internal/perf" func main() { perf.Run(map[string]perf.PerfMethods{ "UploadBlobTest": {Register: uploadTestRegister, New: NewUploadTest}, "ListBlobTest": {Register: listTestRegister, New: NewListTest}, "DownloadBlobTest": {Register: downloadTestRegister, New: NewDownloadTest}, }) } ``` ## Running Performance Tests First, make sure you have go version 1.17 or later installed. You can install go from the [go.dev](https://go.dev/doc/install) site. Navigate to your SDK performance test e.g., `sdk/security/keyvault/azkeys/testdata/perf` folder and run `go mod download` to download all the requirements for the performance framework. To run a single performance test specify the test as the first argument: ```pwsh go run . CreateEntityTest ``` To specify flags for a performance test, add them after the first argument: ```pwsh go run . CreateEntityTest --duration 7 --test-proxies https://localhost:5001 --num-blobs 100 ``` To run against multiple proxies and with multiple parallel tests running at the same time use the `-p/--parallel` flag ```pwsh go run . CreateEntityTest --duration 7 --parallel 4 --test-proxies "https://localhost:5001;http://localhost:5000" --num-blobs 100 ``` This runs four goroutines of the same test and splits the traffic to two different proxy addresses. The first and third routine will target `https://localhost:5001`, the second and fourth will use `http://localhost:5000`. For help run `go run . --help`. A list of registered performance tests, global command flags, and local command flags will print out.