24f44838a8
* Fix issue where 2d index arrays with length 1 get flattened by powershell * Pass NonSparseParameters to imported matrix generation Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
534 lines
17 KiB
Markdown
534 lines
17 KiB
Markdown
# Azure Pipelines Matrix Generator
|
|
|
|
* [Usage in a pipeline](#usage-in-a-pipeline)
|
|
* [Matrix config file syntax](#matrix-config-file-syntax)
|
|
* [Fields](#fields)
|
|
* [matrix](#matrix)
|
|
* [include](#include)
|
|
* [exclude](#exclude)
|
|
* [displayNames](#displaynames)
|
|
* [$IMPORT](#import)
|
|
* [Matrix Generation behavior](#matrix-generation-behavior)
|
|
* [all](#all)
|
|
* [sparse](#sparse)
|
|
* [include/exclude](#includeexclude)
|
|
* [displayNames](#displaynames-1)
|
|
* [Filters](#filters)
|
|
* [Replace](#replace-values)
|
|
* [NonSparseParameters](#nonsparseparameters)
|
|
* [Under the hood](#under-the-hood)
|
|
* [Testing](#testing)
|
|
|
|
|
|
This directory contains scripts supporting dynamic, cross-product matrix generation for azure pipeline jobs.
|
|
It aims to replicate the [cross-product matrix functionality in github actions](https://docs.github.com/free-pro-team@latest/actions/reference/workflow-syntax-for-github-actions#example-running-with-more-than-one-version-of-nodejs),
|
|
but also adds some additional features like sparse matrix generation, cross-product includes and excludes, and programmable matrix filters.
|
|
|
|
This functionality is made possible by the ability for the azure pipelines yaml to take a [dynamic variable as an input
|
|
for a job matrix definition](https://docs.microsoft.com/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml#multi-job-configuration) (see the code sample at the bottom of the linked section).
|
|
|
|
## Usage in a pipeline
|
|
|
|
In order to use these scripts in a pipeline, you must provide a config file and call the matrix creation script within a powershell job.
|
|
|
|
For a single matrix, you can include the `/eng/common/pipelines/templates/jobs/archetype-sdk-tests-generate.yml` template in a pipeline (see /eng/common/scripts/job-matrix/samples/matrix-test.yml for a full working example):
|
|
|
|
```
|
|
jobs:
|
|
- template: /eng/common/pipelines/templates/jobs/archetype-sdk-tests-generate.yml
|
|
parameters:
|
|
MatrixConfigs:
|
|
- Name: base_product_matrix
|
|
Path: eng/scripts/job-matrix/samples/matrix.json
|
|
Selection: all
|
|
NonSparseParameters:
|
|
- framework
|
|
GenerateVMJobs: true
|
|
- Name: sparse_product_matrix
|
|
Path: eng/scripts/job-matrix/samples/matrix.json
|
|
Selection: sparse
|
|
GenerateVMJobs: true
|
|
JobTemplatePath: /eng/common/scripts/job-matrix/samples/matrix-job-sample.yml
|
|
AdditionalParameters: []
|
|
CloudConfig:
|
|
SubscriptionConfiguration: $(sub-config-azure-cloud-test-resources)
|
|
Location: eastus2
|
|
Cloud: Public
|
|
MatrixFilters: []
|
|
MatrixReplace: []
|
|
```
|
|
|
|
## Matrix config file syntax
|
|
|
|
Matrix parameters can either be a list of strings, or a set of grouped strings (represented as a hash). The latter parameter
|
|
type is useful for when 2 or more parameters need to be grouped together, but without generating more than one matrix permutation.
|
|
|
|
```
|
|
"matrix": {
|
|
"<parameter1 name>": [ <values...> ],
|
|
"<parameter2 name>": [ <values...> ],
|
|
"<parameter set>": {
|
|
"<parameter set 1 name>": {
|
|
"<parameter set 1 value 1": "value",
|
|
"<parameter set 1 value 2": "<value>",
|
|
},
|
|
"<parameter set 2 name>": {
|
|
"<parameter set 2 value 1": "value",
|
|
"<parameter set 2 value 2": "<value>",
|
|
}
|
|
}
|
|
}
|
|
"include": [ <matrix>, <matrix>, ... ],
|
|
"exclude": [ <matrix>, <matrix>, ... ],
|
|
"displayNames": { <parameter value>: <human readable override> }
|
|
```
|
|
|
|
See `samples/matrix.json` for a full sample.
|
|
|
|
### Fields
|
|
|
|
#### matrix
|
|
|
|
The `matrix` field defines the base cross-product matrix. The generated matrix can be full or sparse.
|
|
|
|
Example:
|
|
```
|
|
"matrix": {
|
|
"operatingSystem": [
|
|
"windows-2019",
|
|
"ubuntu-18.04",
|
|
"macOS-10.15"
|
|
],
|
|
"framework": [
|
|
"net461",
|
|
"netcoreapp2.1",
|
|
"net50"
|
|
],
|
|
"additionalTestArguments": [
|
|
"",
|
|
"/p:UseProjectReferenceToAzureClients=true",
|
|
]
|
|
}
|
|
```
|
|
|
|
#### include
|
|
|
|
The `include` field defines any number of matrices to be appended to the base matrix after processing exclusions.
|
|
|
|
#### exclude
|
|
|
|
The `include` field defines any number of matrices to be removed from the base matrix. Exclude parameters can be a partial
|
|
set, meaning as long as all exclude parameters match against a matrix entry (even if the matrix entry has additional parameters),
|
|
then it will be excluded from the matrix. For example, the below entry will match the exclusion and be removed:
|
|
|
|
```
|
|
matrix entry:
|
|
{
|
|
"a": 1,
|
|
"b": 2,
|
|
"c": 3,
|
|
}
|
|
|
|
"exclude": [
|
|
{
|
|
"a": 1,
|
|
"b": 2
|
|
}
|
|
]
|
|
```
|
|
|
|
#### displayNames
|
|
|
|
Specify any overrides for the azure pipelines definition and UI that determines the matrix job name. If some parameter
|
|
values are too long or unreadable for this purpose (e.g. a command line argument), then you can replace them with a more
|
|
readable value here. For example:
|
|
|
|
```
|
|
"displayNames": {
|
|
"/p:UseProjectReferenceToAzureClients=true": "UseProjectRef"
|
|
},
|
|
"matrix": {
|
|
"additionalTestArguments": [
|
|
"/p:UseProjectReferenceToAzureClients=true"
|
|
]
|
|
}
|
|
```
|
|
|
|
#### $IMPORT
|
|
|
|
Matrix configs can also import another matrix config. The effect of this is the imported matrix will be generated,
|
|
and then the importing config will be combined with that matrix (as if each entry of the imported matrix was a parameter).
|
|
To import a matrix, add a parameter with the key `$IMPORT`:
|
|
|
|
```
|
|
"matrix": {
|
|
"$IMPORT": "path/to/matrix.json",
|
|
"JavaVersion": [ "1.8", "1.11" ]
|
|
}
|
|
```
|
|
|
|
Importing can be useful, for example, in cases where there is a shared base matrix, but there is a need to run it
|
|
once for each instance of a language version. Importing does not support overriding duplicate parameters. To achieve
|
|
this, use the [Replace](#replace-values) argument instead.
|
|
|
|
The `Selection` and `NonSparseParameters` parameters are respected when generating an imported matrix.
|
|
|
|
The processing order is as follows:
|
|
|
|
Given a matrix and import matrix like below:
|
|
```
|
|
{
|
|
"matrix": {
|
|
"$IMPORT": "example-matrix.json",
|
|
"endpointType": [ "storage", "cosmos" ],
|
|
"JavaVersion": [ "1.8", "1.11" ]
|
|
},
|
|
"include": [
|
|
{
|
|
"operatingSystem": "windows",
|
|
"mode": "TestFromSource",
|
|
"JavaVersion": "1.8"
|
|
}
|
|
]
|
|
}
|
|
|
|
### example-matrix.json to import
|
|
{
|
|
"matrix": {
|
|
"operatingSystem": [ "windows", "linux" ],
|
|
"client": [ "netty", "okhttp" ]
|
|
},
|
|
"include": [
|
|
{
|
|
"operatingSystem": "mac",
|
|
"client": "netty"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
1. The base matrix is generated (sparse in this example):
|
|
```
|
|
{
|
|
"storage_18": {
|
|
"endpointType": "storage",
|
|
"JavaVersion": "1.8"
|
|
},
|
|
"cosmos_111": {
|
|
"endpointType": "cosmos",
|
|
"JavaVersion": "1.11"
|
|
}
|
|
}
|
|
```
|
|
1. The imported base matrix is generated (sparse in this example):
|
|
```
|
|
{
|
|
"windows_netty": {
|
|
"operatingSystem": "windows",
|
|
"client": "netty"
|
|
},
|
|
"linux_okhttp": {
|
|
"operatingSystem": "linux",
|
|
"client": "okhttp"
|
|
}
|
|
}
|
|
```
|
|
1. Includes/excludes from the imported matrix get applied to the imported matrix
|
|
```
|
|
{
|
|
"windows_netty": {
|
|
"operatingSystem": "windows",
|
|
"client": "netty"
|
|
},
|
|
"linux_okhttp": {
|
|
"operatingSystem": "linux",
|
|
"client": "okhttp"
|
|
},
|
|
"mac_netty": {
|
|
"operatingSystem": "mac",
|
|
"client": "netty"
|
|
}
|
|
}
|
|
```
|
|
1. The base matrix is multipled by the imported matrix (in this case, the base matrix has 2 elements, and the imported
|
|
matrix has 3 elements, so the product is a matrix with 6 elements:
|
|
```
|
|
"storage_18_windows_netty": {
|
|
"endpointType": "storage",
|
|
"JavaVersion": "1.8",
|
|
"operatingSystem": "windows",
|
|
"client": "netty"
|
|
},
|
|
"storage_18_linux_okhttp": {
|
|
"endpointType": "storage",
|
|
"JavaVersion": "1.8",
|
|
"operatingSystem": "linux",
|
|
"client": "okhttp"
|
|
},
|
|
"storage_18_mac_netty": {
|
|
"endpointType": "storage",
|
|
"JavaVersion": "1.8",
|
|
"operatingSystem": "mac",
|
|
"client": "netty"
|
|
},
|
|
"cosmos_111_windows_netty": {
|
|
"endpointType": "cosmos",
|
|
"JavaVersion": "1.11",
|
|
"operatingSystem": "windows",
|
|
"client": "netty"
|
|
},
|
|
"cosmos_111_linux_okhttp": {
|
|
"endpointType": "cosmos",
|
|
"JavaVersion": "1.11",
|
|
"operatingSystem": "linux",
|
|
"client": "okhttp"
|
|
},
|
|
"cosmos_111_mac_netty": {
|
|
"endpointType": "cosmos",
|
|
"JavaVersion": "1.11",
|
|
"operatingSystem": "mac",
|
|
"client": "netty"
|
|
}
|
|
}
|
|
```
|
|
1. Includes/excludes from the top-level matrix get applied to the multiplied matrix, so the below element will be added
|
|
to the above matrix, for an output matrix with 7 elements:
|
|
```
|
|
"windows_TestFromSource_18": {
|
|
"operatingSystem": "windows",
|
|
"mode": "TestFromSource",
|
|
"JavaVersion": "1.8"
|
|
}
|
|
```
|
|
|
|
## Matrix Generation behavior
|
|
|
|
#### all
|
|
|
|
`all` will output the full matrix, i.e. every possible permutation of all parameters given (p1.Length * p2.Length * ...).
|
|
|
|
#### sparse
|
|
|
|
`sparse` outputs the minimum number of parameter combinations while ensuring that all parameter values are present in at least one matrix job.
|
|
Effectively this means the total length of a sparse matrix will be equal to the largest matrix dimension, i.e. `max(p1.Length, p2.Length, ...)`.
|
|
|
|
To build a sparse matrix, a full matrix is generated, and then walked diagonally N times where N is the largest matrix dimension.
|
|
This pattern works for any N-dimensional matrix, via an incrementing index (n, n, n, ...), (n+1, n+1, n+1, ...), etc.
|
|
Index lookups against matrix dimensions are calculated modulus the dimension size, so a two-dimensional matrix of 4x2 might be walked like this:
|
|
|
|
```
|
|
index: 0, 0:
|
|
o . . .
|
|
. . . .
|
|
|
|
index: 1, 1:
|
|
. . . .
|
|
. o . .
|
|
|
|
index: 2, 2 (modded to 2, 0):
|
|
. . o .
|
|
. . . .
|
|
|
|
index: 3, 3 (modded to 3, 1):
|
|
. . . .
|
|
. . . o
|
|
```
|
|
|
|
#### include/exclude
|
|
|
|
Include and exclude support additions and subtractions off the base matrix. Both include and exclude take an array of matrix values.
|
|
Typically these values will be a single entry, but they also support the cross-product matrix definition syntax of the base matrix.
|
|
|
|
Include and exclude are parsed fully. So if a sparse matrix is called for, a sparse version of the base matrix will be generated, but
|
|
the full matrix of both include and exclude will be processed.
|
|
|
|
Excludes are processed first, so includes can be used to add back any specific jobs to the matrix.
|
|
|
|
#### displayNames
|
|
|
|
In the matrix job output that azure pipelines consumes, the format is a dictionary of dictionaries. For example:
|
|
|
|
```
|
|
{
|
|
"net461_macOS1015": {
|
|
"framework": "net461",
|
|
"operatingSystem": "macOS-10.15"
|
|
},
|
|
"net50_ubuntu1804": {
|
|
"framework": "net50",
|
|
"operatingSystem": "ubuntu-18.04"
|
|
},
|
|
"netcoreapp21_windows2019": {
|
|
"framework": "netcoreapp2.1",
|
|
"operatingSystem": "windows-2019"
|
|
},
|
|
"UseProjectRef_net461_windows2019": {
|
|
"additionalTestArguments": "/p:UseProjectReferenceToAzureClients=true",
|
|
"framework": "net461",
|
|
"operatingSystem": "windows-2019"
|
|
}
|
|
}
|
|
```
|
|
|
|
The top level keys are used as job names, meaning they get displayed in the azure pipelines UI when running the pipeline.
|
|
|
|
The logic for generating display names works like this:
|
|
|
|
- Join parameter values by "_"
|
|
a. If the parameter value exists as a key in `displayNames` in the matrix config, replace it with that value.
|
|
b. For each name value, strip all non-alphanumeric characters (excluding "_").
|
|
c. If the name is greater than 100 characters, truncate it.
|
|
|
|
#### Filters
|
|
|
|
Filters can be passed to the matrix as an array of strings, each matching the format of `<key>=<regex>`. When a matrix entry
|
|
does not contain the specified key, it will default to a value of empty string for regex parsing. This can be used to specify
|
|
filters for keys that don't exist or keys that optionally exist and match a regex, as seen in the below example.
|
|
|
|
Display name filters can also be passed as a single regex string that runs against the [generated display name](#displaynames) of the matrix job.
|
|
The intent of display name filters is to be defined primarily as a top level variable at template queue time in the azure pipelines UI.
|
|
|
|
For example, the below command will filter for matrix entries with "windows" in the job display name, no matrix variable
|
|
named "ExcludedKey", a framework variable containing either "461" or "5.0", and an optional key "SupportedClouds" that, if exists, must contain "Public":
|
|
|
|
```
|
|
./Create-JobMatrix.ps1 `
|
|
-ConfigPath samples/matrix.json `
|
|
-Selection all `
|
|
-DisplayNameFilter ".*windows.*" `
|
|
-Filters @("ExcludedKey=^$", "framework=(461|5\.0)", "SupportedClouds=^$|.*Public.*")
|
|
```
|
|
|
|
#### Replace values
|
|
|
|
Replacements for values can be passed to the matrix as an array of strings, each matching the format of `<keyRegex>=<valueRegex>/<replacementValue>`.
|
|
The replace argument will find any permutations where the key fully matches the key regex and the value fully matches the value regex, and replace the value with
|
|
the replacement specified.
|
|
|
|
NOTE:
|
|
- The replacement value supports regex capture groups, enabling substring transformations, e.g. `Foo=(.*)-replaceMe/$1-replaced`. See the below examples for usage.
|
|
- For each key/value, the first replacement provided that matches will be the only one applied.
|
|
- If `=` or `/` characters need to be part of the regex or replacement, escape them with `\`.
|
|
|
|
For example, given a matrix config like below:
|
|
|
|
```
|
|
{
|
|
"matrix": {
|
|
"Agent": {
|
|
"ubuntu-1804": { "OSVmImage": "MMSUbuntu18.04", "Pool": "azsdk-pool-mms-ubuntu-1804-general" }
|
|
},
|
|
"JavaTestVersion": [ "1.8", "1.11" ]
|
|
}
|
|
}
|
|
|
|
```
|
|
|
|
The normal matrix output (without replacements), looks like:
|
|
|
|
```
|
|
$ ./Create-JobMatrix.ps1 -ConfigPath <test> -Selection all
|
|
{
|
|
"ubuntu1804_18": {
|
|
"OSVmImage": "MMSUbuntu18.04",
|
|
"Pool": "azsdk-pool-mms-ubuntu-1804-general",
|
|
"JavaTestVersion": "1.8"
|
|
},
|
|
"ubuntu1804_111": {
|
|
"OSVmImage": "MMSUbuntu18.04",
|
|
"Pool": "azsdk-pool-mms-ubuntu-1804-general",
|
|
"JavaTestVersion": "1.11"
|
|
}
|
|
}
|
|
```
|
|
|
|
Passing in multiple replacements, the output will look like below. Note that replacing key/values that appear nested within a grouping
|
|
will not affect that segment of the job name, since the job takes the grouping name (in this case "ubuntu1804").
|
|
|
|
The below example includes samples of regex grouping references, and wildcard key/value regexes:
|
|
|
|
```
|
|
$ $replacements = @('.*Version=1.11/2.0', 'Pool=(.*ubuntu.*)-general/$1-custom')
|
|
$ ../Create-JobMatrix.ps1 -ConfigPath ./test.Json -Selection all -Replace $replacements
|
|
{
|
|
"ubuntu1804_18": {
|
|
"OSVmImage": "MMSUbuntu18.04",
|
|
"Pool": "azsdk-pool-mms-ubuntu-1804-custom",
|
|
"JavaTestVersion": "1.8"
|
|
},
|
|
"ubuntu1804_20": {
|
|
"OSVmImage": "MMSUbuntu18.04",
|
|
"Pool": "azsdk-pool-mms-ubuntu-1804-custom",
|
|
"JavaTestVersion": "2.0"
|
|
}
|
|
}
|
|
```
|
|
|
|
#### NonSparseParameters
|
|
|
|
Sometimes it may be necessary to generate a sparse matrix, but keep the full combination of a few parameters. The
|
|
NonSparseParameters argument allows for more fine-grained control of matrix generation. For example:
|
|
|
|
```
|
|
./Create-JobMatrix.ps1 `
|
|
-ConfigPath /path/to/matrix.json `
|
|
-Selection sparse `
|
|
-NonSparseParameters @("JavaTestVersion")
|
|
```
|
|
|
|
Given a matrix like below with `JavaTestVersion` marked as a non-sparse parameter:
|
|
|
|
```
|
|
{
|
|
"matrix": {
|
|
"Agent": {
|
|
"windows-2019": { "OSVmImage": "MMS2019", "Pool": "azsdk-pool-mms-win-2019-general" },
|
|
"ubuntu-1804": { "OSVmImage": "MMSUbuntu18.04", "Pool": "azsdk-pool-mms-ubuntu-1804-general" },
|
|
"macOS-10.15": { "OSVmImage": "macOS-10.15", "Pool": "Azure Pipelines" }
|
|
},
|
|
"JavaTestVersion": [ "1.8", "1.11" ],
|
|
"AZURE_TEST_HTTP_CLIENTS": "netty",
|
|
"ArmTemplateParameters": [ "@{endpointType='storage'}", "@{endpointType='cosmos'}" ]
|
|
}
|
|
}
|
|
```
|
|
|
|
A matrix with 6 entries will be generated: A sparse matrix of Agent, AZURE_TEST_HTTP_CLIENTS and ArmTemplateParameters
|
|
(3 total entries) will be multipled by the two `JavaTestVersion` parameters `1.8` and `1.11`.
|
|
|
|
NOTE: NonSparseParameters are also applied when generating an imported matrix.
|
|
|
|
#### Under the hood
|
|
|
|
The script generates an N-dimensional matrix with dimensions equal to the parameter array lengths. For example,
|
|
the below config would generate a 2x2x1x1x1 matrix (five-dimensional):
|
|
|
|
```
|
|
"matrix": {
|
|
"framework": [ "net461", "netcoreapp2.1" ],
|
|
"additionalTestArguments": [ "", "/p:SuperTest=true" ]
|
|
"pool": [ "ubuntu-18.04" ],
|
|
"container": [ "ubuntu-18.04" ],
|
|
"testMode": [ "Record" ]
|
|
}
|
|
```
|
|
|
|
The matrix is stored as a one-dimensional array, with a row-major indexing scheme (e.g. `(2, 1, 0, 1, 0)`).
|
|
|
|
## Testing
|
|
|
|
The matrix functions can be tested using [pester](https://pester.dev/). The test command must be run from within the tests directory.
|
|
|
|
```
|
|
$ cd tests
|
|
$ Invoke-Pester
|
|
|
|
Starting discovery in 3 files.
|
|
Discovery finished in 75ms.
|
|
[+] /home/ben/sdk/azure-sdk-tools/eng/common/scripts/job-matrix/tests/job-matrix-functions.filter.tests.ps1 750ms (309ms|428ms)
|
|
[+] /home/ben/sdk/azure-sdk-tools/eng/common/scripts/job-matrix/tests/job-matrix-functions.modification.tests.ps1 867ms (250ms|608ms)
|
|
[+] /home/ben/sdk/azure-sdk-tools/eng/common/scripts/job-matrix/tests/job-matrix-functions.tests.ps1 2.71s (725ms|1.93s)
|
|
Tests completed in 4.33s
|
|
Tests Passed: 141, Failed: 0, Skipped: 4 NotRun: 0
|
|
```
|