* Added yaml support for job matrix creation
* autogen scenario matrix for stress test
* Temporary Working State
* update to default sparse
* pr comments and some error handling
* custom matrixfilename and ordering of generatedValues.yaml
* common module import
* JobMatrix write host
Co-authored-by: Albert Cheng <albertcheng@microsoft.com>
We have hit a few 50x errors while pulling a page to get
links and that has caused the rest of the link checking to
terminate early. To fix that we switching to LogError which will
output an error in devops instead of Write-Error which terminates
immediately. We also add some retry count to the page retrieval
and cache file retrieval to help with these transitent 50x issues.
Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com>
This should fix https://github.com/Azure/azure-sdk-tools/issues/3538
as it will now upgrade the devops extension to the latest version
which has the auth fix needed.
Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com>
* Detect API changes using new snadboxinx approach
* Added strictmode
* Remove unset variable
* Changes as per strict mode 3
* Rervert strict mode to allow language level fixes to merge first
Co-authored-by: praveenkuttappan <prmarott@microsoft.com>
* Fixed the common contract.
* Fix for current regex
* Add new line
* Update eng/common/scripts/Service-Level-Readme-Automation.ps1
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* move compare and merge into gerenerate function
* Fix on new lines
Co-authored-by: sima-zhu <sizhu@microsoft.com>
Co-authored-by: Sima Zhu <48036328+sima-zhu@users.noreply.github.com>
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* stress test script and docs migration from test to pg cluster
* comments for stress-test-deployment-lib script
Co-authored-by: Albert Cheng <albertcheng@microsoft.com>
* Update verify-readme to take a single or multiple paths
* chance ScanPaths to a comma delimited list and coalesce ScanPath/ScanPaths into a single variable for the script
* VS Code was nice enough to add an extra single quote when adding a quote to the end of the line'
* Capture ScanPaths.Split into an array so we don't have to call Split again
Co-authored-by: James Suplizio <jasupliz@microsoft.com>
* Support local addons path override in stress test deployment
* Support username based deployId in local stress deployment
* Support WhatIf in stress infrastructure provision script
* Simplify stress user detection
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* Run helm plugin add with helper
* Add WhatIf support to ps module install helper function
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* Use seperate scripts
* address comments.
* do compare and update
* save on the service level readme
* have the helper for reuse function
* remove mgmt table
* changes
* fix
* no return on error
* return if no contents
* Address comments
* change the table
* address wes comments.
* address wes comments.
* address more comments.
Co-authored-by: sima-zhu <sizhu@microsoft.com>
* Use the batch version of ValidateDocsMsPackagesFn
* Use proper pipelining for GetPackageInfoJson function
Co-authored-by: Patrick Hallisey <pahallis@microsoft.com>
* Verify changelog like a release is about to take place if a vlaid date is present
* Update eng/common/scripts/ChangeLog-Operations.ps1
Co-authored-by: Ben Broderick Phillips <ben@benbp.net>
* Update eng/common/scripts/ChangeLog-Operations.ps1
Co-authored-by: Ben Broderick Phillips <ben@benbp.net>
* Update eng/common/scripts/ChangeLog-Operations.ps1
Co-authored-by: Ben Broderick Phillips <ben@benbp.net>
* Update eng/common/scripts/ChangeLog-Operations.ps1
Co-authored-by: Ben Broderick Phillips <ben@benbp.net>
* Update eng/common/scripts/ChangeLog-Operations.ps1
Co-authored-by: Ben Broderick Phillips <ben@benbp.net>
* Remove strict date verification
* Add function for sorting changelog entries
* Minor cleanups
Co-authored-by: Chidozie Ononiwu <chononiw@microsoft.com>
Co-authored-by: Chidozie Ononiwu (His Righteousness) <31145988+chidozieononiwu@users.noreply.github.com>
Co-authored-by: Ben Broderick Phillips <ben@benbp.net>
* Fix bug where job matrices with leading numbers generated duplicate job names
* Fail matrix generation when config path or import paths are not found
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* Use common script for git diff changes
* remove extra files
* suppress spell check
* suppress false positive cspelling
* Address comments
* make changes to cspell
* Suppress to right values
* More fix
Co-authored-by: sima-zhu <sizhu@microsoft.com>
* archetype-sdk-tests-generate.yml now provides an opportunity for additional steps after the target has been cloned but before matrix generation is invoked
Co-authored-by: scbedd <45376673+scbedd@users.noreply.github.com>
* Verify changelog like a release is about to take place if a vlaid date is present
* Update eng/common/scripts/ChangeLog-Operations.ps1
Co-authored-by: Ben Broderick Phillips <ben@benbp.net>
* Update eng/common/scripts/ChangeLog-Operations.ps1
Co-authored-by: Ben Broderick Phillips <ben@benbp.net>
* Update eng/common/scripts/ChangeLog-Operations.ps1
Co-authored-by: Ben Broderick Phillips <ben@benbp.net>
* Update eng/common/scripts/ChangeLog-Operations.ps1
Co-authored-by: Ben Broderick Phillips <ben@benbp.net>
* Update eng/common/scripts/ChangeLog-Operations.ps1
Co-authored-by: Ben Broderick Phillips <ben@benbp.net>
Co-authored-by: Chidozie Ononiwu <chononiw@microsoft.com>
Co-authored-by: Chidozie Ononiwu (His Righteousness) <31145988+chidozieononiwu@users.noreply.github.com>
Co-authored-by: Ben Broderick Phillips <ben@benbp.net>
* Common scripts for git diff changes
* Remove the localizer.
* Check not PR triggered case
* Pretty print
Co-authored-by: sima-zhu <sizhu@microsoft.com>
* Improve Cosmos Emulator logic
* Run cosmos db emulator in two steps
* Using povershell < core for emulator
* add retry logic to cosmosdb-emulator launch
* Add Retry step fir cosmos emulator launch
Co-authored-by: Chidozie Ononiwu <chononiw@microsoft.com>
* Non stop on readme checking
* Print the right variable name
Co-authored-by: sima-zhu <sizhu@microsoft.com>
Co-authored-by: Sima Zhu <48036328+sima-zhu@users.noreply.github.com>
* Get ms alias from github identity.
* Update Update-DocsMsMetadata.ps1
Co-authored-by: sima-zhu <sizhu@microsoft.com>
Co-authored-by: Sima Zhu <48036328+sima-zhu@users.noreply.github.com>
- Fix bug in Add-ReleaseLease helper script
- Enable better local logging for handling token
- Enable strict mode to help catch issues like this in the future
* Use stress test environment defaults for group and subscription
* Fix parameter passing from deploy-stress-tests.ps1 script
* Redact stress deployment logs and simplify image handling
* Use DevopsLogging parameter to prevent secret logging in non-devops CI environments
* Use switch type for DevopsLogging parameter
* Remove boolean parameter usage in favor of [switch]
* Add default parameter set usage comment
* Throw when clusterGroup and/or subscription is not specified for custom environments
* Add helper function for logging azure pipelines vso commands
* Invert SuppressVsoCommands binary default value
* Vso command fixes
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* Add disk metrics script and template
* Support linux disk matrics
* Add continueOnError to write-filesystemmetrics.yml
Co-authored-by: Patrick Hallisey <pahallis@microsoft.com>
* Add PowerShell Script for Validating READMEs
* Remember my VSCode doesn't use autosave
Co-authored-by: Alan Zimmer <48699787+alzimmermsft@users.noreply.github.com>
* Add spell check pin changes
* Apply suggestions from code review
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* Add package-lock.json
* Set node version and upgrade npm in check-spelling.yml
* Do not mutate local NPM environment
* Remove unnecessary comment, we no longer use npm exec
Co-authored-by: Daniel Jurek <djurek@microsoft.com>
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* Consume Codeowners parser library, ceperate users from teams in codeownerse
* Update get-pr-owners and related logic
Co-authored-by: Chidozie Ononiwu <chononiw@microsoft.com>
* Mitigate relative path calculation error on multiple iterations
* Revert "Mitigate relative path calculation error on multiple iterations"
This reverts commit 45baedd990c6a3085742a38a4891d8706a93be77.
* GetRelativePath should check if path is already relative before calling [IO.Path]::GetRelativePath
Co-authored-by: Daniel Jurek <djurek@microsoft.com>
* Handle assembly already loaded errors in deploy stress test script
* Add dependency checks and installation links to deploy-stress-tests.ps1
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* Remove empty sections in changelog entry as part of prepare release
* Update SECTIONS_HEADER_REGEX to accomodate various header levels
Co-authored-by: Chidozie Ononiwu <chononiw@microsoft.com>
* Common pipeline template and script to detect API changes
* Cahnges as per review comments
* Review comment changes
* CCahgnes as per review comments
* Show warning for list of failed packages to detect API changes
* Apply suggestions from code review
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
Co-authored-by: praveenkuttappan <prmarott@microsoft.com>
Co-authored-by: praveenkuttappan <55455725+praveenkuttappan@users.noreply.github.com>
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
Since we are treating v0 versions as prerelease we need to make sure
we don't accidently start to add the bogus prerelease label in cases where
we call ToString() on the version.
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* Add ability to exit gracefully when all files in the diff are excluded
* Address case where cspell exits with an error when all files from the 'files' config list are excluded by the expressions in 'ignorePaths'
* Add tests
* Review feedback: impl goes at the bottom and should be treated as a script, logic for testing should happen above that and exit appropriately if running tests
* Import common instead of logging
* Enable strict mode
Co-authored-by: Daniel Jurek <djurek@microsoft.com>
* Stress test deployment script resiliency fixes
* Use PSModule-Helpers to install powershell-yaml
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* Attempt to purge all vaults, managed HSMs
Reverts #1910. Vaults and managed HSMs are automatically purged on their purge date. The point was to purge them daily to preserve capacity. The default purge date is +90 days.
* Add timeout and more logging
* Pass required -Resource
* Fix log message
* Ensure the $Resource is correctly captured
Added comment to new code explaining why, since ScriptBlock.GetNewClosure() is not working as expected.
* Add -ErrorAction to Receive-Job
Worked without terminating when run locally, but failed on the first error in the AzDO agent.
* Use $using:r instead of creating ScriptBlock
More idiomatic for passing ScriptBlocks to jobs.
* Resolve PR feedback
* Change default DeleteAfterHours to 120
Resolves#1917
* Use the Az cmdlets built-in -AsJob
Co-authored-by: Heath Stewart <heaths@microsoft.com>
- Fix rg.Name to rg.ResourceGroupName
- Add more verbose logging for better debugging
- Handle deleted resource groups when gathering puragable resource
- Remove coerce now that we are collecting in functions
Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com>
* Deploy for stress tests at resource group scope
* Automatically compile bicep files in stress test deploy script
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* Update change log headers based on guideline update
Updates based on https://github.com/Azure/azure-sdk/pull/3103
- Renamed "Key Bugs Fixed" to "Bugs Fixed"
- Renamed "Fixed" to "Other Changes"
Added a warning in validation if at lease one of the recommended
headers aren't used.
* Update eng/common/scripts/ChangeLog-Operations.ps1
Co-authored-by: JoshLove-msft <54595583+JoshLove-msft@users.noreply.github.com>
Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com>
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
Co-authored-by: JoshLove-msft <54595583+JoshLove-msft@users.noreply.github.com>
* Bring changes from JS docs metadata
* Move business logic inside Update-DocsMsMetadata.ps1
* Update with the latest changes in JS PR
* Update from latest PR feedback
* Add check for empty path to prevent crashes when creating relative paths
Co-authored-by: Daniel Jurek <djurek@microsoft.com>
- Switch to using rest instead of cli for querying work items
as cli is limited to 1000 items only.
- Fix issues with Generated fields not being set.
- Correctly sort older workitems by version isntead of string.
Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com>
* Check for API review status only if release date is set in changelog
* Change property name to ReleaseStatus
Co-authored-by: Praveen Kuttappan <praveen.kuttappan@gmail.com>
* Add API status check
* Revert temp change
* Update as per review comments
* Removed blank line
* Review comments to use az cli
* Changes to move az cli commands to caller script and other review comments
* Skip languages that's not supported in APIView
* Fix bug in language mapping
Co-authored-by: praveenkuttappan <prmarott@microsoft.com>
- Remove copied AzPowershell utilities
- Add latest AZ module path already on hosted agents to PSModulePath
- Rename setup-az-modules template setup-environments to reflect what is is doing
- Add support for Caching the current user PS Module folder
- Add support for install-module if not already present in module folder
- Organize the live test clean-up script to be in the standard location
Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com>
* Update common engineering system for docs onboarding
* Add updates from review feedback
* Space nit
Co-authored-by: Daniel Jurek <djurek@microsoft.com>
DevOps helper scripts were updated in azure-sdk repo
so syncing those changes into the tools repo so they
can be used prepare-release script.
Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com>
This will allow us to point our verify-link script at a raw
MD file in a github repo and have it parsed correctly for
links.
Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com>
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
- Update link checking pipeline to enable caching
- Add loading cache file from a http endpoint
- Add retry logic when hitting 429
Fix verify link renaming scenarios
- Correctly error if link format is incorrect
- Handle case where the replace regex is not passed
- Improve the error logging so it is easier to
identify the broken links
- This removes the ci.yml filtering which means all projects can be discovered
- This eliminates the need for the yml parsing module for most common scenarios
- This does introduce some potential issues with duplicate package names for
languages like java. The only known case currently is azure-storage-blobs and
if there is ever a need to explicitly pick the correct one someone can pass
in the ServiceDirectory filter similar to "storage/azure*" for new and "storage/microsoft*"
for the older sdk. By default azure usually comes first so the new one gets discovered
which is likely all we need for now. If this turns out not to be true we might
need another way to disambiguate.
Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com>
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* Handle empty files in Test-SampleMetadata
Relates to Azure/azure-sdk-for-java#20684
* Remove condition from verify-samples
* Use forward slash in path
* Fix directory typo
Co-authored-by: Heath Stewart <heaths@microsoft.com>
* Verify samples during CIs
Replaces Azure/azure-sdk-for-python#17999 for use in all language repos.
* Move exit to proper scope
Co-authored-by: Heath Stewart <heaths@microsoft.com>
* Fix issue where 2d index arrays with length 1 get flattened by powershell
* Pass NonSparseParameters to imported matrix generation
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* Add chagelogpath to Update-ChangeLog.ps1
* Make PackageName and ServiceDirectory not mandatory, test ChangelogPath
Co-authored-by: Chidozie Ononiwu <chononiw@microsoft.com>
* Handle null values when generating display names
* Fix issue processing matrices where the base matrix is only an import
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* Add 'replace' support. Improve handling of dynamic parameter types.
* Support display name import and overrides
* Support regex capture groups for replace. Force fully matching regex.
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
Pull only artifacts from `ci.yml` to resolve issue with conflicting package names.
Co-authored-by: Chidozie Ononiwu <31145988+chidozieononiwu@users.noreply.github.com>
* Update Package Properties Logic, make prepare release require only 1 argument, make rerun of Get-PkgProperties in Update-Changelog not required
* Remove ServiceDirectory parameter from PrepareRelease.ps1
* changes to argument names
* pass service directory to Get-AllPackageInfoFromRepo
* Remove changes in Update-ChangeLog.ps1
* updates to Package-Properties.ps1
Co-authored-by: Chidozie Ononiwu <chononiw@microsoft.com>
Co-authored-by: Chidozie Ononiwu <31145988+chidozieononiwu@users.noreply.github.com>
* Added more logging to API view create
* Updated logs with arch board rescue email address
* Fix as per review comments
* Update logging to show API URL
Co-authored-by: praveenkuttappan <prmarott@microsoft.com>
- Remove BuildType parameter as we can default it from package properties
- Stop passing BuildType and GroupId and instead defaul them from package properties
- Enable StrictMode to help identify potential errors
- Start passing sdktype and isnewsdk properties to devops script
- Sync latest changes with devops work item to fix a couple bugs
Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com>
* Add common tools for spell check
* Updates from further iteration
* Review feedback: Add check for git as well
* Review feedback * Use common approach to resolving base branch name * Eliminate default base branch "master" as this will be changed later, providing no default and using a mandatory parameter means a dev must provide the value * Check for existence of $CspellConfigPath * No need to Set-Location if run from the root of the repo (most common scenario) * -join
* Review feedback: Rename TargetRef -> TargetBranch, add SourceBranch, Add reference to spelling docs, exit 0, Rename to check-spelling-in-changed-files.ps1,
* Review feedback: Remove ValidateNotNullOrEmpty (we do more definitive validation farther down that will also catch these errors), Update comments and script name
Co-authored-by: Daniel Jurek <djurek@microsoft.com>
* Create package property file using artifact name
* Add option to pass config file dir
* Set default config file directory to avoid script execution failure
Co-authored-by: praveenkuttappan <prmarott@microsoft.com>
