* Apply a few changes to Get-PRPackageProperties to make it more customizable for our users. Remove wonky gap + extra spacing that should pass the patch application process now that we've manually aligned common.ps1
* account for feedback -- rename CanaryPackages to AdditionalValidationPackages
* dependentPackagesforInclusion -> additionalValidationPackages
---------
Co-authored-by: Scott Beddall <scbedd@microsoft.com>
* bump proxy version
* bump proxy version
* restore most of the old performance, only fall back to json matching when something goes wrong
* use newest version with protection from invalid json
---------
Co-authored-by: Scott Beddall <scbedd@microsoft.com>
Given we cannot create subfolders under .github\workflow folder
and we have repos that want additional workflows then we have
in the tools repo we need to setup a file pattern matching for
the sync processing to only sync the ones we want to be common.
For now just use the existing *event* pattern but we can move them
to another pattern similar to *common* or *sync* in the future as
we add more common workflows.
Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com>
* Add RestAPISpecsDocsRepos to branch cleanup.
* Remove commented out yml, add pre-message to limit output and output core limit at the start of processing
* updates for feedback
---------
Co-authored-by: James Suplizio <jasupliz@microsoft.com>
* we want the path that we write to to be defined by literalpath, otherwise files with [ or ] will break powershell when writing the file to disk
* slight wording adjustment so the command can be invoked directly
---------
Co-authored-by: Scott Beddall <scbedd@microsoft.com>
* more compatibility with expanding/contracting packages. add ability for packages to have DependentPackages that must be included in the set of packages that should be built given a changeset
---------
Co-authored-by: Scott Beddall <scbedd@microsoft.com>
Co-authored-by: Scott Beddall <45376673+scbedd@users.noreply.github.com>
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* Do not mark values as secret that are from git-hosted sub config files
* Simplify sub config build yaml
---------
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* missed a condition passthrough
---------
Co-authored-by: Scott Beddall <scbedd@microsoft.com>
Co-authored-by: Scott Beddall <45376673+scbedd@users.noreply.github.com>
* add additional argument to Save-Package-Properties to allow for usage in pull request context
---------
Co-authored-by: Scott Beddall (from Dev Box) <scbedd@microsoft.com>
Co-authored-by: Scott Beddall <45376673+scbedd@users.noreply.github.com>
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* Set storage account test resources to disable blob public access
* Skip adding network rules to storage accounts that don't need them during cleanup
* Add succeeded check to set pipeline subnet info step
* Disable network firewall by default in resource creation/removal
---------
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* Support storage network access and worm removal in remove test resources script
* Move storage network access script to common resource helpers file
* Improve storage container deletion resilience
* Plumb through pool variable to live test cleanup template
* Add sleep for network rule application
---------
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* commit the file changes so that we can see them running
* use standalone tool
---------
Co-authored-by: Scott Beddall (from Dev Box) <scbedd@microsoft.com>
* Ensure subConfigFiles is not an empty string
* Skip instances where $file is an empty string
---------
Co-authored-by: Daniel Jurek <djurek@microsoft.com>
* Fix default value for env vars in build-test-resource-config template
* Add empty pool condition
---------
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* Restrict live test storage account access to client IP
* Add storage pools to test resources vnet allowlist
* Use pool subnet map to reduce number of subnets added to live test resources
* snap
* Add test resource parameter to add ip ranges to storage firewall
* Validate and update ip firewall rules
* Get pool subnet using separate workload identity
---------
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
Add logging and fix display name given Agent.JobStatus doesn't evaluate currently in the display name property.
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* Merge subscription configuration file values with the supplied subscription config
* Bring forward working parts
* Collapse into a single task
* SubConfig-Helpers.ps1 changes
* Update remove-test-resources.yml
---------
Co-authored-by: Daniel Jurek <djurek@microsoft.com>
* Removal of the devops release PAT
* Don't pass in the access token, just use the AzureCLI task and do everything in the scripts
* Updates for feedback
---------
Co-authored-by: James Suplizio <jasupliz@microsoft.com>
* Plumb env vars through deploy test resources
* Update eng/common/TestResources/deploy-test-resources.yml
Co-authored-by: Daniel Jurek <djurek@microsoft.com>
* Update eng/common/TestResources/deploy-test-resources.yml
Co-authored-by: Daniel Jurek <djurek@microsoft.com>
---------
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
Co-authored-by: Daniel Jurek <djurek@microsoft.com>
* PAT Removal from pipeline generation yml
* update version of PipelineGenerator to install
---------
Co-authored-by: James Suplizio <jasupliz@microsoft.com>
* Reuse same branch name for previews of a PR
* Default AddBuildSummary parameter using pipeline variable existence
---------
Co-authored-by: Patrick Hallisey <pahallis@microsoft.com>
* Changes to scripts to deal with PATs and AccessTokens
* swap access and auth for add-retention-lease
* AuthToken to BearerToken and remove unused Base64EncodedAuthToken from the script parameters
* remove unneccsary if not null check for the mandatory parameter
---------
Co-authored-by: James Suplizio <jasupliz@microsoft.com>
* Migrate stress cluster to use workload identity
* Only sleep to sync fed creds when namespace is new
---------
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* consume newest stable version of the test-proxy
* remove override of proxy pins to resume honoring standard proxy version in CI
---------
Co-authored-by: Scott Beddall <scbedd@microsoft.com>
* ensure we write the log to the same root directory as what is being used for the proxy invocation
* fix spacing
* we should write these proxy logs to the same root folder that we start the proxy in
---------
Co-authored-by: Scott Beddall <scbedd@microsoft.com>
* Add support for Federated Auth to test resources scripts
* Default -- UseFederatedAuth: false
* Clear secrets if FederatedAuth is set
* Template conditions use AzurePowerShell only when a service connection is needed
* Review feedback and pair with Ben
* Update docs
* http:// -> https://
---------
Co-authored-by: Daniel Jurek <djurek@microsoft.com>
* allow an override FILE to be honored by test-proxy-tool
* update local test-framework to honor proxy override version by default
---------
Co-authored-by: Scott Beddall (from Dev Box) <scbedd@microsoft.com>
* Only make one call to GetPackageInfoJson
* fix copy/paste, it should be packageInfoLocation not packageInfoJsonLocation
---------
Co-authored-by: James Suplizio <jasupliz@microsoft.com>
Moving away from SAS tokens for connecting to storage so switching to using a Workload Identity Federation connection to the container to download the needed files.
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* Pass flag to suppress errors correctly when verifying release change log
* Changes as per review comments
---------
Co-authored-by: Praveen Kuttappan <prmarott@microsoft.com>
* Pipeline template to validate package and update package work item
* Changes to restructure validations
* Additional fixes as per comments
* Remove explicit exit code
* Set erroractionpreference for change log check
---------
Co-authored-by: Praveen Kuttappan <prmarott@microsoft.com>
In the case there is exactly one previous release PS will return the single object and thus the Count property will not exist. Instead this change ensures that we always have a list.
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* Run publish on failed or succeeded
* Expand agent os string detection
* Check agent job status env var for artifact name detection
* Add sbomEnabled flag to publish template
* Fix image and artifact name conditional
---------
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* Add 1es changes for job/matrix generation and publish
* Use more flexible pool filter for prev/next pool matches
* consolidate displayname definition
* use linux pool variables for generate matrix job
* Fix publish artifact
* Use single publish task for publish artifact
---------
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* Support resolving environment variable references in matrix config
* Improve type and null handling
* Fix reference bug
* Change behavior on missing env vars to throw
---------
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* Fix sparse-checkout git command line behavior
* Use command arg parsing behavior in sparse-checkout.yml
---------
Co-authored-by: Daniel Jurek <djurek@microsoft.com>
* Prepare-Release.ps1: Make dateTime.ToString("MM/dd/yyyy") to work on exotic set-ups
On my machine, I experimented with the registry, and the worst part is that I don't remember/know how to reset it back.
The work items that script produces, do have datetimes for the upcoming releases in the `MM-dd-yyyy` format, and then I have to correct them by hand.
`dateTime.ToString("MM/dd/yyyy")` does produce the date in the format of `MM-dd-yyyy` on my machine. This also happens if I write a corresponding .NET app.
The fix that I am proposing makes it work on my specific setup and hopefully breaks no one else. I understand if you are hesitant to take it. Let me know, I'll see how I can restore my setting.
But on the other hand, I don't think it makes anything worse, it only makes things more robust, so maybe take it?
* Use [CultureInfo]::InvarialtCulture
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* Update eng/common/scripts/Prepare-Release.ps1
* Update eng/common/scripts/Prepare-Release.ps1
---------
Co-authored-by: Anton Kolesnyk <41349689+antkmsft@users.noreply.github.com>
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* Remove package retrieval when verify pkg version
* Modified description of PackageName parameter
---------
Co-authored-by: Ray Chen <raychen@microsoft.com>
* Correct the name of JS package folder
* Uncomment the package verification
* Logging more info for troubleshooting
* Get sdkType and directory from the package info
---------
Co-authored-by: Ray Chen <raychen@microsoft.com>
* Support creating resources with user auth
* Log warning if TestApplicationId is set
* missing space
* regenerate md file
* Rename
* Update link
---------
Co-authored-by: jolov <jolov@microsoft.com>
* Add ConflictedFile to git-helpers.ps1, add git-helpers.tests.ps1 to exercise basic functionality.
* Add `resolve-asset-conflict.ps1` a script that can autoresolve an assets.json file.
---------
Co-authored-by: Scott Beddall (from Dev Box) <scbedd@microsoft.com>
Co-authored-by: Scott Beddall <45376673+scbedd@users.noreply.github.com>
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* Added script and pipeline for spec location validation
SDK release pipeline would run this validation to ensure the spec comes from
the main branch of Azure/azure-rest-api-specs repo
* Update parameter
* Use github rest api to validate commit
* Added token parameter
* Support more yaml cases and other languages
* Removed the default setting in yaml template
* Only validate in case of GA package
* Follow APIView to retrieve package version for verification
* Get github token from env variable
* Removed obsolete parameter
---------
Co-authored-by: raychen <raychen@microsoft.com>
* Add package filter parameter to select recursively found packages in release
* Handle empty value for packageFilter
---------
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* check for the presence of a compatible powershell. ensure that we always return a list of tags
* allow the script to require pshell6+
* remove the en-us from the link
---------
Co-authored-by: Scott Beddall (from Dev Box) <scbedd@microsoft.com>
* Update organization of the eng/common/testproxy folder
* Add merge-proxy-tags.ps1 and readme to eng/common/testproxy/scripts/tag-merge/
* Extract common assets script functionality to common-asset-functions.ps1
---------
Co-authored-by: Scott Beddall (from Dev Box) <scbedd@microsoft.com>
Http Status code 401 just be unauthorized and not necessarily that the link isn't valid. Removing this from the list of codes to error from.
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* Create new pipeline to create package work item
* Update powershell script to pass tag
* Fixes as per review comments
* Update as per review comments
* Login as az devops
* Create new entry point script to create package work item
* Load common ps module
* Add devops login step
* Pipe login token to devops login
* Pipe login token to devops login
* Add looging
* more logging
* Remove duplicate login from new script and add more logging
* More logging to debug create work item
* Remove script added for debugging
* Skip az interactive login when PAT is passed
* Remove debug logging
---------
Co-authored-by: praveenkuttappan <praveenkuttappan@gmail.com>
* Force capitalize all deployment/env outputs in live test
* Update eng/common/TestResources/New-TestResources.ps1
Co-authored-by: Heath Stewart <heaths@outlook.com>
---------
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
Co-authored-by: Heath Stewart <heaths@outlook.com>
* Add CODEOWNERS linter yml file to eng/common/pipelines
* Add filter baseline option to run command line
* Update version of linter to 1.0.0-dev.20231107.2 which will install correctly
---------
Co-authored-by: James Suplizio <jasupliz@microsoft.com>
* update proxy version to 20231030.1
* update test-proxy-tool.yml to record both stderr and stdout to the proxy log
---------
Co-authored-by: Scott Beddall (from Dev Box) <scbedd@microsoft.com>
* Update branch deletion code
- Update to use the gh cli for github operations
- Switch to using graphql for pulling branches as the rest api
starts to fail after a certain number of branches and we can
also pull back all the associated data like pull request info
and commit date all in one query.
- switc to running on ubuntu 1ES pool
Improve query error handling
Fix condition
Fix slash escaping on the command line
clean-up pipeline
* Update eng/common/scripts/Delete-RemoteBranches.ps1
Co-authored-by: Ben Broderick Phillips <ben@benbp.net>
* Update eng/common/scripts/Delete-RemoteBranches.ps1
Co-authored-by: Ben Broderick Phillips <ben@benbp.net>
---------
Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com>
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
Co-authored-by: Ben Broderick Phillips <ben@benbp.net>
* Remove ARM deployment after deploy
* Update eng/common/TestResources/New-TestResources.ps1
Co-authored-by: Heath Stewart <heaths@outlook.com>
* Apply suggestions from code review
---------
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
Co-authored-by: Heath Stewart <heaths@outlook.com>
Fixes https://github.com/Azure/azure-sdk-tools/issues/5882
We need to set the difffilter to empty instead of the default of exclude deleted files when we are trying to verify there are no changes under eng/common.
See test PR https://github.com/Azure/azure-sdk-for-python/pull/32348 which demonstrates us not detecting a deleted file under eng/common. I'll use that same test PR to verify this now catches that issue.
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* Remove codeowners resolution logic from docs scripts
* Remove parameters that aren't needed
* Remove auth information from update-docsms-metadata.yml
---------
Co-authored-by: Daniel Jurek <djurek@microsoft.com>
* Add autorest-preview pipeline
* Add emitternpminstall
* Always install
* Use shorter leg name
* Add short circuiting to emitter install
* Use language matrix function
* Don't look for subfolders in the matrix package folders
* Revert unnecessary eng/common changes
* Rewrite GetPullRequestUrl to Get-BuildSourceDescription
* Remove alias from Invoke-LoggedCommand
* Use invoke-expression instead of shelling out
* Add better job splitting
* Replace Folder with Directory
---------
Co-authored-by: Patrick Hallisey <pahallis@microsoft.com>
* Add LockDeletionForDays parameter to set PodDisruptionBudget and cleanup job
* Use matrix for parallel tests. PDB improvements+docs.
* Fix kubectl namespace context preservation on login
---------
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
