* Get-PRPkgProperties now honors artifact-list specific absolute/relative paths to other triggering folders. This will supplant the common "if eng/ changes include template and core" in a follow-up PR.
---------
Co-authored-by: Scott Beddall <scbedd@microsoft.com>
* adjustments to iterating across matrix configs for packages that belong to multiple matrices
---------
Co-authored-by: Scott Beddall <scbedd@microsoft.com>
* replace PkgProps CIMatrixConfigs property in favor of a more generalized CIParameters property. Original CIMatrixConfigs will exist as an item of same key within CIParameters.
* update Create-PRJobMatrix to honor new CIMatrixConfigs property location
---------
Co-authored-by: Scott Beddall <scbedd@microsoft.com>
* update Get-ObjectKey to coalesce boolean-like values to their true boolean value
* update Get-ObjectKey to call Get-ObjectKey on each item of an array or pscustom object - versus utilizing their values directly
---------
Co-authored-by: Scott Beddall <scbedd@microsoft.com>
* Pass arm template parameter for TME context
* Use supportsSafeSecretStandard parameter instead
---------
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* allow the indirect matrix to generate non-sparse based on new parameter to generate-job-matrix
---------
Co-authored-by: Scott Beddall <scbedd@microsoft.com>
* Added label handle sdk-gen pipeline template
Added common script to delete label from a PR
* Update eng/common/scripts/Invoke-GitHubAPI.ps1
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
---------
Co-authored-by: ray chen <raychen@microsoft.com>
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* Support incrementing semver prereleases with 'zero' versions
* Make tests more explicit
---------
Co-authored-by: Patrick Hallisey <pahallis@microsoft.com>
* updates to dynamic matrix generation. moves indirectly included packages to sparse usage. adds additional parameters to GeneratePRMatrix that enable further filtering of the matrix generated for the indirect packages
---------
Co-authored-by: Scott Beddall <scbedd@microsoft.com>
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* Use -Mail for test resources local user lookup to support TME
* Auto-select TME subscription based on user context
---------
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* ensure that packages that come back from AdditionalValidationPackages are NOT counted as 'includedForValidation: true' if they also exist in the original 'these packages changed' set
Co-authored-by: Ben Broderick Phillips <ben@benbp.net>
---------
Co-authored-by: Scott Beddall <scbedd@microsoft.com>
Co-authored-by: Scott Beddall <45376673+scbedd@users.noreply.github.com>
Co-authored-by: Ben Broderick Phillips <ben@benbp.net>
* add and use new function to PSModule-Helpers.ps1 -- `InstallAndImport-ModuleIfNotInstalled` which smartly short circuits the install and module import to be as efficient as possible. This largely has impact of improving Save-Package-Properties performance on windows machines.
---------
Co-authored-by: Scott Beddall <scbedd@microsoft.com>
* Use new generation tool that supports service connections
* Fix condition for generating weekly test pipelines
---------
Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com>
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* change our verifyChangelogs over to utilizing the artifact details. if we don't have an artifact details at all, don't verify changelog. if we do, but skipverifychangelog is present, don't verify changelog. if we have artifact details, but no disable, verify the changelog
---------
Co-authored-by: Scott Beddall <scbedd@microsoft.com>
* ensure that ChangedServices value within the generated pr diff is always properly an array of strings
* upload the changed services as an attachment named AdditionalTags in save-package-properties.yml, to allow PipelineWitness to tag public builds during log processing
---------
Co-authored-by: Scott Beddall <scbedd@microsoft.com>
Co-authored-by: Scott Beddall <45376673+scbedd@users.noreply.github.com>
Co-authored-by: Patrick Hallisey <hallipr@gmail.com>
* Add step for emitting rate limits for a token
* add step to eng/common sync template
* Fix resource label output
* Use propery name instead of nested 'resource' property
* Add percent metric too
* Add divide by zero safety
---------
Co-authored-by: Patrick Hallisey <pahallis@microsoft.com>
* Common cleanup with the docs docker images no longer being used
* Remove docker-pull-image.yml
---------
Co-authored-by: James Suplizio <jasupliz@microsoft.com>
* Retry container deletion
* Do not try to purge keyvaults with purge protection
* Delete all blobs when container has immutability
* Skip missing blob container properties
* Fix null container
---------
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* identify and resolve missing function GenerateMatrixForConfig
* identify and resolve miss-used null-coalesce operator
---------
Co-authored-by: Scott Beddall <scbedd@microsoft.com>
* package-properties are now populated with matrix configurations from their ci.yml if present
* create new code path for generate-job-matrix.yml which combines Create-JobMatrix and the "distribute-packages-to-matrix" action to generate dynamic matrices for PRs
---------
Co-authored-by: Scott Beddall <scbedd@microsoft.com>
* Run perf tests under federated auth
Run the tests under the same federated auth used to deploy the tests and setup the variables needed to configure AzurePipelineCredential.
* Fix indention for working directory
---------
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* Refactor PackageProps to make the Yaml loading and processing methods common
* Updates for feedback
---------
Co-authored-by: James Suplizio <jasupliz@microsoft.com>
* Remove default sub config sub-config-azure-cloud-test-resources
We no longer want to default the configuration and instead
default from the service connection which had the info we need.
* Fix quoting
---------
Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com>
* save-package-properties.yml only activates PR diff mechanism if the service directory is "auto"
---------
Co-authored-by: Scott Beddall <scbedd@microsoft.com>
* Escape parentheses in includes to fix issues in publishing
* Only escape close parens
* Comments
---------
Co-authored-by: Daniel Jurek <djurek@microsoft.com>
* Changes required for sdk-repo based emitter pipelines
* Rename template
* Convert emitter archetype to pipeline template
* Move the archetype template out of the stages folder
---------
Co-authored-by: Patrick Hallisey <pahallis@microsoft.com>
* Add CompatibleConvertFrom-Yaml to common Package-Helpers
* Add CI artifact parsing to Save-Package-Properties, now each individual packageinfo.json file contains the relevant ci artifact lines for the package. This short circuits needing to re-parse this information in common checks.
---------
Co-authored-by: Scott Beddall <scbedd@microsoft.com>
* Export the subscription data from the service connection
* Update deploy-test-resources.yml
---------
Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com>
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* don't verify changelog for packages that aren't actively published anymore
---------
Co-authored-by: Scott Beddall <scbedd@microsoft.com>
Co-authored-by: Scott Beddall <45376673+scbedd@users.noreply.github.com>
Co-authored-by: Ben Broderick Phillips <ben@benbp.net>
* updating package properties with direct/indirect (if named differently) as well as pulling BuildDocs from ci.yml artifact list if it exists
* eliminate the addition of buildDocs property. it requires powershell-yaml to be present on our base function. not good
* remove call to InitializeBuildDocs
---------
Co-authored-by: Scott Beddall <scbedd@microsoft.com>
* Add descriptive error when variable groups are not added to pipeline
* Add emoji to deploy step title to make it easier to find
* Move yaml json checking into powershell script
* Fix base sub config empty/string checking
---------
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* Refactor prepare-pipelines
* Convert pipeling generation single to use template
* Convert to pwsh for setting the variables
* Fix switch expression
* Add path separator in front of devopspath
* Continue on error for individual step failures
* Remove unnecessary parameters
---------
Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com>
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* Apply a few changes to Get-PRPackageProperties to make it more customizable for our users. Remove wonky gap + extra spacing that should pass the patch application process now that we've manually aligned common.ps1
* account for feedback -- rename CanaryPackages to AdditionalValidationPackages
* dependentPackagesforInclusion -> additionalValidationPackages
---------
Co-authored-by: Scott Beddall <scbedd@microsoft.com>
* bump proxy version
* bump proxy version
* restore most of the old performance, only fall back to json matching when something goes wrong
* use newest version with protection from invalid json
---------
Co-authored-by: Scott Beddall <scbedd@microsoft.com>
Given we cannot create subfolders under .github\workflow folder
and we have repos that want additional workflows then we have
in the tools repo we need to setup a file pattern matching for
the sync processing to only sync the ones we want to be common.
For now just use the existing *event* pattern but we can move them
to another pattern similar to *common* or *sync* in the future as
we add more common workflows.
Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com>
* Add RestAPISpecsDocsRepos to branch cleanup.
* Remove commented out yml, add pre-message to limit output and output core limit at the start of processing
* updates for feedback
---------
Co-authored-by: James Suplizio <jasupliz@microsoft.com>
* we want the path that we write to to be defined by literalpath, otherwise files with [ or ] will break powershell when writing the file to disk
* slight wording adjustment so the command can be invoked directly
---------
Co-authored-by: Scott Beddall <scbedd@microsoft.com>
* more compatibility with expanding/contracting packages. add ability for packages to have DependentPackages that must be included in the set of packages that should be built given a changeset
---------
Co-authored-by: Scott Beddall <scbedd@microsoft.com>
Co-authored-by: Scott Beddall <45376673+scbedd@users.noreply.github.com>
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* Do not mark values as secret that are from git-hosted sub config files
* Simplify sub config build yaml
---------
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* missed a condition passthrough
---------
Co-authored-by: Scott Beddall <scbedd@microsoft.com>
Co-authored-by: Scott Beddall <45376673+scbedd@users.noreply.github.com>
* add additional argument to Save-Package-Properties to allow for usage in pull request context
---------
Co-authored-by: Scott Beddall (from Dev Box) <scbedd@microsoft.com>
Co-authored-by: Scott Beddall <45376673+scbedd@users.noreply.github.com>
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* Set storage account test resources to disable blob public access
* Skip adding network rules to storage accounts that don't need them during cleanup
* Add succeeded check to set pipeline subnet info step
* Disable network firewall by default in resource creation/removal
---------
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* Support storage network access and worm removal in remove test resources script
* Move storage network access script to common resource helpers file
* Improve storage container deletion resilience
* Plumb through pool variable to live test cleanup template
* Add sleep for network rule application
---------
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* commit the file changes so that we can see them running
* use standalone tool
---------
Co-authored-by: Scott Beddall (from Dev Box) <scbedd@microsoft.com>
* Ensure subConfigFiles is not an empty string
* Skip instances where $file is an empty string
---------
Co-authored-by: Daniel Jurek <djurek@microsoft.com>
* Fix default value for env vars in build-test-resource-config template
* Add empty pool condition
---------
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* Restrict live test storage account access to client IP
* Add storage pools to test resources vnet allowlist
* Use pool subnet map to reduce number of subnets added to live test resources
* snap
* Add test resource parameter to add ip ranges to storage firewall
* Validate and update ip firewall rules
* Get pool subnet using separate workload identity
---------
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
Add logging and fix display name given Agent.JobStatus doesn't evaluate currently in the display name property.
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* Merge subscription configuration file values with the supplied subscription config
* Bring forward working parts
* Collapse into a single task
* SubConfig-Helpers.ps1 changes
* Update remove-test-resources.yml
---------
Co-authored-by: Daniel Jurek <djurek@microsoft.com>
* Removal of the devops release PAT
* Don't pass in the access token, just use the AzureCLI task and do everything in the scripts
* Updates for feedback
---------
Co-authored-by: James Suplizio <jasupliz@microsoft.com>
* Plumb env vars through deploy test resources
* Update eng/common/TestResources/deploy-test-resources.yml
Co-authored-by: Daniel Jurek <djurek@microsoft.com>
* Update eng/common/TestResources/deploy-test-resources.yml
Co-authored-by: Daniel Jurek <djurek@microsoft.com>
---------
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
Co-authored-by: Daniel Jurek <djurek@microsoft.com>
* PAT Removal from pipeline generation yml
* update version of PipelineGenerator to install
---------
Co-authored-by: James Suplizio <jasupliz@microsoft.com>
* Reuse same branch name for previews of a PR
* Default AddBuildSummary parameter using pipeline variable existence
---------
Co-authored-by: Patrick Hallisey <pahallis@microsoft.com>
* Changes to scripts to deal with PATs and AccessTokens
* swap access and auth for add-retention-lease
* AuthToken to BearerToken and remove unused Base64EncodedAuthToken from the script parameters
* remove unneccsary if not null check for the mandatory parameter
---------
Co-authored-by: James Suplizio <jasupliz@microsoft.com>
* Migrate stress cluster to use workload identity
* Only sleep to sync fed creds when namespace is new
---------
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* consume newest stable version of the test-proxy
* remove override of proxy pins to resume honoring standard proxy version in CI
---------
Co-authored-by: Scott Beddall <scbedd@microsoft.com>
* ensure we write the log to the same root directory as what is being used for the proxy invocation
* fix spacing
* we should write these proxy logs to the same root folder that we start the proxy in
---------
Co-authored-by: Scott Beddall <scbedd@microsoft.com>
* Add support for Federated Auth to test resources scripts
* Default -- UseFederatedAuth: false
* Clear secrets if FederatedAuth is set
* Template conditions use AzurePowerShell only when a service connection is needed
* Review feedback and pair with Ben
* Update docs
* http:// -> https://
---------
Co-authored-by: Daniel Jurek <djurek@microsoft.com>
* allow an override FILE to be honored by test-proxy-tool
* update local test-framework to honor proxy override version by default
---------
Co-authored-by: Scott Beddall (from Dev Box) <scbedd@microsoft.com>
* Only make one call to GetPackageInfoJson
* fix copy/paste, it should be packageInfoLocation not packageInfoJsonLocation
---------
Co-authored-by: James Suplizio <jasupliz@microsoft.com>
Moving away from SAS tokens for connecting to storage so switching to using a Workload Identity Federation connection to the container to download the needed files.
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* Pass flag to suppress errors correctly when verifying release change log
* Changes as per review comments
---------
Co-authored-by: Praveen Kuttappan <prmarott@microsoft.com>
* Pipeline template to validate package and update package work item
* Changes to restructure validations
* Additional fixes as per comments
* Remove explicit exit code
* Set erroractionpreference for change log check
---------
Co-authored-by: Praveen Kuttappan <prmarott@microsoft.com>
In the case there is exactly one previous release PS will return the single object and thus the Count property will not exist. Instead this change ensures that we always have a list.
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* Run publish on failed or succeeded
* Expand agent os string detection
* Check agent job status env var for artifact name detection
* Add sbomEnabled flag to publish template
* Fix image and artifact name conditional
---------
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* Add 1es changes for job/matrix generation and publish
* Use more flexible pool filter for prev/next pool matches
* consolidate displayname definition
* use linux pool variables for generate matrix job
* Fix publish artifact
* Use single publish task for publish artifact
---------
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* Support resolving environment variable references in matrix config
* Improve type and null handling
* Fix reference bug
* Change behavior on missing env vars to throw
---------
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* Fix sparse-checkout git command line behavior
* Use command arg parsing behavior in sparse-checkout.yml
---------
Co-authored-by: Daniel Jurek <djurek@microsoft.com>
* Prepare-Release.ps1: Make dateTime.ToString("MM/dd/yyyy") to work on exotic set-ups
On my machine, I experimented with the registry, and the worst part is that I don't remember/know how to reset it back.
The work items that script produces, do have datetimes for the upcoming releases in the `MM-dd-yyyy` format, and then I have to correct them by hand.
`dateTime.ToString("MM/dd/yyyy")` does produce the date in the format of `MM-dd-yyyy` on my machine. This also happens if I write a corresponding .NET app.
The fix that I am proposing makes it work on my specific setup and hopefully breaks no one else. I understand if you are hesitant to take it. Let me know, I'll see how I can restore my setting.
But on the other hand, I don't think it makes anything worse, it only makes things more robust, so maybe take it?
* Use [CultureInfo]::InvarialtCulture
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* Update eng/common/scripts/Prepare-Release.ps1
* Update eng/common/scripts/Prepare-Release.ps1
---------
Co-authored-by: Anton Kolesnyk <41349689+antkmsft@users.noreply.github.com>
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* Remove package retrieval when verify pkg version
* Modified description of PackageName parameter
---------
Co-authored-by: Ray Chen <raychen@microsoft.com>
* Correct the name of JS package folder
* Uncomment the package verification
* Logging more info for troubleshooting
* Get sdkType and directory from the package info
---------
Co-authored-by: Ray Chen <raychen@microsoft.com>
* Support creating resources with user auth
* Log warning if TestApplicationId is set
* missing space
* regenerate md file
* Rename
* Update link
---------
Co-authored-by: jolov <jolov@microsoft.com>
* Add ConflictedFile to git-helpers.ps1, add git-helpers.tests.ps1 to exercise basic functionality.
* Add `resolve-asset-conflict.ps1` a script that can autoresolve an assets.json file.
---------
Co-authored-by: Scott Beddall (from Dev Box) <scbedd@microsoft.com>
Co-authored-by: Scott Beddall <45376673+scbedd@users.noreply.github.com>
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* Added script and pipeline for spec location validation
SDK release pipeline would run this validation to ensure the spec comes from
the main branch of Azure/azure-rest-api-specs repo
* Update parameter
* Use github rest api to validate commit
* Added token parameter
* Support more yaml cases and other languages
* Removed the default setting in yaml template
* Only validate in case of GA package
* Follow APIView to retrieve package version for verification
* Get github token from env variable
* Removed obsolete parameter
---------
Co-authored-by: raychen <raychen@microsoft.com>
* Add package filter parameter to select recursively found packages in release
* Handle empty value for packageFilter
---------
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* check for the presence of a compatible powershell. ensure that we always return a list of tags
* allow the script to require pshell6+
* remove the en-us from the link
---------
Co-authored-by: Scott Beddall (from Dev Box) <scbedd@microsoft.com>
* Update organization of the eng/common/testproxy folder
* Add merge-proxy-tags.ps1 and readme to eng/common/testproxy/scripts/tag-merge/
* Extract common assets script functionality to common-asset-functions.ps1
---------
Co-authored-by: Scott Beddall (from Dev Box) <scbedd@microsoft.com>
Http Status code 401 just be unauthorized and not necessarily that the link isn't valid. Removing this from the list of codes to error from.
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* Create new pipeline to create package work item
* Update powershell script to pass tag
* Fixes as per review comments
* Update as per review comments
* Login as az devops
* Create new entry point script to create package work item
* Load common ps module
* Add devops login step
* Pipe login token to devops login
* Pipe login token to devops login
* Add looging
* more logging
* Remove duplicate login from new script and add more logging
* More logging to debug create work item
* Remove script added for debugging
* Skip az interactive login when PAT is passed
* Remove debug logging
---------
Co-authored-by: praveenkuttappan <praveenkuttappan@gmail.com>
* Force capitalize all deployment/env outputs in live test
* Update eng/common/TestResources/New-TestResources.ps1
Co-authored-by: Heath Stewart <heaths@outlook.com>
---------
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
Co-authored-by: Heath Stewart <heaths@outlook.com>
* Add CODEOWNERS linter yml file to eng/common/pipelines
* Add filter baseline option to run command line
* Update version of linter to 1.0.0-dev.20231107.2 which will install correctly
---------
Co-authored-by: James Suplizio <jasupliz@microsoft.com>
* update proxy version to 20231030.1
* update test-proxy-tool.yml to record both stderr and stdout to the proxy log
---------
Co-authored-by: Scott Beddall (from Dev Box) <scbedd@microsoft.com>
