* Common pipeline template and script to detect API changes
* Cahnges as per review comments
* Review comment changes
* CCahgnes as per review comments
* Show warning for list of failed packages to detect API changes
* Apply suggestions from code review
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
Co-authored-by: praveenkuttappan <prmarott@microsoft.com>
Co-authored-by: praveenkuttappan <55455725+praveenkuttappan@users.noreply.github.com>
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* Make ServiceName optional for verify-changelog
In cases like go where we don't pass a service name having this default to "not-specified" breaks things so we should allow for ServiceName and/or ServiceDirectory to be empty.
* Put quotes around the strings to allow for empty
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
Since we are treating v0 versions as prerelease we need to make sure
we don't accidently start to add the bogus prerelease label in cases where
we call ToString() on the version.
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* Add ability to exit gracefully when all files in the diff are excluded
* Address case where cspell exits with an error when all files from the 'files' config list are excluded by the expressions in 'ignorePaths'
* Add tests
* Review feedback: impl goes at the bottom and should be treated as a script, logic for testing should happen above that and exit appropriately if running tests
* Import common instead of logging
* Enable strict mode
Co-authored-by: Daniel Jurek <djurek@microsoft.com>
* Doc Updates and Revisions for External Use
The focus of these changes is to revise the script to better support use
by external contributors and others outside of the Azure SDK ecosystem and
without access to the Microsoft AAD Tenant.
Changes include:
- Creation of a new Test Application service principal is now possible
from a non-Microsoft AAD tenant.
- When a new Test Application principal is created, the principle of least
privilege is now applied; the new Test Application is granted ownership
of the resource group associated with the test resources and no longer
has access to any other resources in the subscription.
- If an existing Test Application principal is specified, it will be
assigned ownership of the resource group created. This supports using
a Test Application principal without privileges at the subscription-level.
- When no provisioner is specified, the script is now executed in the
context of the caller rather than the Test Application principal.
This supports using a Test Application principal that has restricted
privileges and better aligns to the purpose of the Test Application
principal.
- The `$TestApplicationOid` is now explicitly bound at the time a new Test
Application principal is created rather than having to query for it later.
- Common error scenarios resulting from lack of permissions now provide
messaging with more context of why the failure occurred and suggest
remediation.
- Added new examples to illustrate the common call patterns needed by
external contributors running the script, outside of the Microsoft tenant
and Azure SDK ecosystem.
- Documentation has been enhanced with additional context to detail the
permissions and roles assigned by the script.
- Added documentation details for Bicep template use.
* Add the provisioner OID to the deployment params
Key Vault needs this to deploy Managed HSMs. There's a corresponding change necessary in test-resources.json I'll roll out across languages.
* Fixing typos and spelling mistakes
Co-authored-by: Jesse Squire <jesse.squire@gmail.com>
Co-authored-by: Heath Stewart <heaths@microsoft.com>
* Stress test deployment script resiliency fixes
* Use PSModule-Helpers to install powershell-yaml
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* Attempt to purge all vaults, managed HSMs
Reverts #1910. Vaults and managed HSMs are automatically purged on their purge date. The point was to purge them daily to preserve capacity. The default purge date is +90 days.
* Add timeout and more logging
* Pass required -Resource
* Fix log message
* Ensure the $Resource is correctly captured
Added comment to new code explaining why, since ScriptBlock.GetNewClosure() is not working as expected.
* Add -ErrorAction to Receive-Job
Worked without terminating when run locally, but failed on the first error in the AzDO agent.
* Use $using:r instead of creating ScriptBlock
More idiomatic for passing ScriptBlocks to jobs.
* Resolve PR feedback
* Change default DeleteAfterHours to 120
Resolves#1917
* Use the Az cmdlets built-in -AsJob
Co-authored-by: Heath Stewart <heaths@microsoft.com>
- Fix rg.Name to rg.ResourceGroupName
- Add more verbose logging for better debugging
- Handle deleted resource groups when gathering puragable resource
- Remove coerce now that we are collecting in functions
Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com>
* Deploy for stress tests at resource group scope
* Automatically compile bicep files in stress test deploy script
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
Explicitly add '/*' '!/*/' to the list of paths so we include
the files in the root of the repo.
Added some extra command logging so we can easily repro locally without
needing to find the code that ran.
Update to '/eng' so that we only get paths that start with eng instead
of every path that has eng as a folder.
Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com>
* Support building and deploying bicep templates
* Add bicep powershell install aka link to deployment error message
* Write bicep compiled arm templates to temp directory
* Simplify bicep building code/function usage
* Use bicep location for compiled arm templates, and remove them on success
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* Update change log headers based on guideline update
Updates based on https://github.com/Azure/azure-sdk/pull/3103
- Renamed "Key Bugs Fixed" to "Bugs Fixed"
- Renamed "Fixed" to "Other Changes"
Added a warning in validation if at lease one of the recommended
headers aren't used.
* Update eng/common/scripts/ChangeLog-Operations.ps1
Co-authored-by: JoshLove-msft <54595583+JoshLove-msft@users.noreply.github.com>
Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com>
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
Co-authored-by: JoshLove-msft <54595583+JoshLove-msft@users.noreply.github.com>
* Bring changes from JS docs metadata
* Move business logic inside Update-DocsMsMetadata.ps1
* Update with the latest changes in JS PR
* Update from latest PR feedback
* Add check for empty path to prevent crashes when creating relative paths
Co-authored-by: Daniel Jurek <djurek@microsoft.com>
- Switch to using rest instead of cli for querying work items
as cli is limited to 1000 items only.
- Fix issues with Generated fields not being set.
- Correctly sort older workitems by version isntead of string.
Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com>
* Check for API review status only if release date is set in changelog
* Change property name to ReleaseStatus
Co-authored-by: Praveen Kuttappan <praveen.kuttappan@gmail.com>
* Update pipeline generation tool version
Consume latest changes from pipeline generation tool in https://github.com/Azure/azure-sdk-tools/pull/1708
* Update tool version to include fix for public ci
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* Add API status check
* Revert temp change
* Update as per review comments
* Removed blank line
* Review comments to use az cli
* Changes to move az cli commands to caller script and other review comments
* Skip languages that's not supported in APIView
* Fix bug in language mapping
Co-authored-by: praveenkuttappan <prmarott@microsoft.com>
- Remove copied AzPowershell utilities
- Add latest AZ module path already on hosted agents to PSModulePath
- Rename setup-az-modules template setup-environments to reflect what is is doing
- Add support for Caching the current user PS Module folder
- Add support for install-module if not already present in module folder
- Organize the live test clean-up script to be in the standard location
Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com>
* Update common engineering system for docs onboarding
* Add updates from review feedback
* Space nit
Co-authored-by: Daniel Jurek <djurek@microsoft.com>
DevOps helper scripts were updated in azure-sdk repo
so syncing those changes into the tools repo so they
can be used prepare-release script.
Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com>
This will allow us to point our verify-link script at a raw
MD file in a github repo and have it parsed correctly for
links.
Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com>
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
- Update link checking pipeline to enable caching
- Add loading cache file from a http endpoint
- Add retry logic when hitting 429
Fix verify link renaming scenarios
- Correctly error if link format is incorrect
- Handle case where the replace regex is not passed
- Improve the error logging so it is easier to
identify the broken links
- This removes the ci.yml filtering which means all projects can be discovered
- This eliminates the need for the yml parsing module for most common scenarios
- This does introduce some potential issues with duplicate package names for
languages like java. The only known case currently is azure-storage-blobs and
if there is ever a need to explicitly pick the correct one someone can pass
in the ServiceDirectory filter similar to "storage/azure*" for new and "storage/microsoft*"
for the older sdk. By default azure usually comes first so the new one gets discovered
which is likely all we need for now. If this turns out not to be true we might
need another way to disambiguate.
Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com>
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* Handle empty files in Test-SampleMetadata
Relates to Azure/azure-sdk-for-java#20684
* Remove condition from verify-samples
* Use forward slash in path
* Fix directory typo
Co-authored-by: Heath Stewart <heaths@microsoft.com>
* Verify samples during CIs
Replaces Azure/azure-sdk-for-python#17999 for use in all language repos.
* Move exit to proper scope
Co-authored-by: Heath Stewart <heaths@microsoft.com>
* Remove passing -Mode Complete to deployment
When passing Complete it will remove any resources
already in the resource group that weren't part
of the current deployment. That removal breaks a
lot of assumptions, like multiple deployments when
testing things like smoke-tests or if you are reusing
an existing resource group. We don't want that to happen.
* Remove ServiceDirectory as required parameter for remove/update
When this was made mandatory it broke some usages which didn't
pass the value. Those usages don't need to pass it because
the pass required information in other ways so removing the
requirement for the parameter to be passed.
Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com>
Initializing the BaseName with a ServiceDirectory that contains
a "/", because it is a multiple level path, causes the BaseName
initialization code to fail because it doesn't support the
validation pattern.
In all the known cases we already pass the ResourceGroupName
explicitly so don't need to set the BaseName so we can
skip the initialization in those cases.
Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com>
* Use SubscriptionId throughout TestResources
Fixes#1454
* Resolve PR feedback
* Default DeleteAfterHours to 48 for SDK team
Also makes a few other adjustments for subscriptions, like restoring the previous one if available and another was specified.
* Resolve PR feedback
* Change deployment mode to Complete
Also fixes an issue where if the user opted not to deploy to the same resource group, the script would continue execution anyway.
* Use consistent aka links to satisfy link checker
Only need it for the new Update-TestResources.ps1 script, but I wanted them to look consistent.
Co-authored-by: Heath Stewart <heaths@microsoft.com>
* Fix issue where 2d index arrays with length 1 get flattened by powershell
* Pass NonSparseParameters to imported matrix generation
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* Add chagelogpath to Update-ChangeLog.ps1
* Make PackageName and ServiceDirectory not mandatory, test ChangelogPath
Co-authored-by: Chidozie Ononiwu <chononiw@microsoft.com>
* Handle null values when generating display names
* Fix issue processing matrices where the base matrix is only an import
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* Add 'replace' support. Improve handling of dynamic parameter types.
* Support display name import and overrides
* Support regex capture groups for replace. Force fully matching regex.
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
Pull only artifacts from `ci.yml` to resolve issue with conflicting package names.
Co-authored-by: Chidozie Ononiwu <31145988+chidozieononiwu@users.noreply.github.com>
* Update Package Properties Logic, make prepare release require only 1 argument, make rerun of Get-PkgProperties in Update-Changelog not required
* Remove ServiceDirectory parameter from PrepareRelease.ps1
* changes to argument names
* pass service directory to Get-AllPackageInfoFromRepo
* Remove changes in Update-ChangeLog.ps1
* updates to Package-Properties.ps1
Co-authored-by: Chidozie Ononiwu <chononiw@microsoft.com>
Co-authored-by: Chidozie Ononiwu <31145988+chidozieononiwu@users.noreply.github.com>
* Create a publish-artifact.yml template step within eng/common. This template utilizes PublishPipelineArtifactV1 to upload your artifact only in the case where steps up till the invocation are succeeded(). Otherwise, the artifacts will be uploaded to an alternate artifact name.
* Added more logging to API view create
* Updated logs with arch board rescue email address
* Fix as per review comments
* Update logging to show API URL
Co-authored-by: praveenkuttappan <prmarott@microsoft.com>
- Remove BuildType parameter as we can default it from package properties
- Stop passing BuildType and GroupId and instead defaul them from package properties
- Enable StrictMode to help identify potential errors
- Start passing sdktype and isnewsdk properties to devops script
- Sync latest changes with devops work item to fix a couple bugs
Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com>
* Add common tools for spell check
* Updates from further iteration
* Review feedback: Add check for git as well
* Review feedback * Use common approach to resolving base branch name * Eliminate default base branch "master" as this will be changed later, providing no default and using a mandatory parameter means a dev must provide the value * Check for existence of $CspellConfigPath * No need to Set-Location if run from the root of the repo (most common scenario) * -join
* Review feedback: Rename TargetRef -> TargetBranch, add SourceBranch, Add reference to spelling docs, exit 0, Rename to check-spelling-in-changed-files.ps1,
* Review feedback: Remove ValidateNotNullOrEmpty (we do more definitive validation farther down that will also catch these errors), Update comments and script name
Co-authored-by: Daniel Jurek <djurek@microsoft.com>
* Change other version to all versions
* Exclude the packages which already show up in latest GA and preview
* Update main.js
Co-authored-by: Sima Zhu <sizhu@microsoft.com>
Co-authored-by: Sima Zhu <48036328+sima-zhu@users.noreply.github.com>
* Create package property file using artifact name
* Add option to pass config file dir
* Set default config file directory to avoid script execution failure
Co-authored-by: praveenkuttappan <prmarott@microsoft.com>
* Improve TestResources docs and logging
Resolves#1388Resolves#1407
Also ignores cached service principal if it no longer exists. I ran into this while testing since I cleaned up old SPs.
* Add ADP test sub to look-up
Co-authored-by: Heath Stewart <heaths@microsoft.com>
* Hide the package which set to true instead of putting to Other page
* Hide the package not showing in csv
* Added logging message
* fix the variable
* leave an extra space
* Update eng/common/docgeneration/Generate-DocIndex.ps1
Co-authored-by: Sima Zhu <sizhu@microsoft.com>
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
- Sync devops helper script changes
- Fix issue in SemVer script when using strict mode with Language not set
Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com>
* Support sub directory for artifact location
* Fix as per review comment to avoid artifact sub path
* Package property file is not created for track1 and management pacakges.
* Fix property name case
* Fix as per review comments
* Fix as per review comments
* Use Config File param value as is
* Change to make path joineasily readable
Co-authored-by: praveenkuttappan <prmarott@microsoft.com>
* Use a helper function to normalize the package name for readme
* Get the readme name from artifact-metadata-parsing
* Change to the readme
* Update eng/common/scripts/artifact-metadata-parsing.ps1
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
Co-authored-by: Sima Zhu <sizhu@microsoft.com>
Co-authored-by: Sima Zhu <48036328+sima-zhu@users.noreply.github.com>
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* Enforce API approval status for GA and include SDK type in package properties
* Fix per review comment
* Fixes as per review comments to avoid changing constructor signature
* Fixes as per review comments to handle track 1 packages
* Fix yaml format error
Co-authored-by: praveenkuttappan <prmarott@microsoft.com>
* Run the link verification step even previous step failed.
* Make the tab right
* Address comments
* Run default condition if no RunOnFailure set.
* Changed back to simple condition
Co-authored-by: Sima Zhu <sizhu@microsoft.com>
* Make the user agent configurable.
* Address comments
* Default to current Chrome version
* Revert the yaml file changes
Co-authored-by: Sima Zhu <sizhu@microsoft.com>
* Update latest folder with one index.html includes the redirect links
* Update the redirect links and remove everything in latest
* Address feedback
* Have the entire copy for latest GA.
Co-authored-by: Sima Zhu <sizhu@microsoft.com>
- Add set of helpers to work with devops work items
- Add script to create devops release package items
- Update SemVer to support version type
Co-authored-by: Wes Haggard <Wes.Haggard@microsoft.com>
* Support alpha and azure pipeline build version formats for SemVer parsing and sorting
* Minor build number fixes
* Support zero-padding of build versions more generically in SemVer script
* Fix pre-release label conditional for HasValidPrereleaseLabel in semver script
* Set default convention for build number separator in semver script
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* reformatted parameter arguments to both update-docs scripts. added parameter requirements and renamed CIRepository to DocRepoLocation which is a lot clearer. Added additional CIConfig parameter to update-docs-metadata to support specific write locations per moniker.
Co-authored-by: scbedd <45376673+scbedd@users.noreply.github.com>
* Fixed the docindex.yml and rename the files
* Reset the mutated changes
* Force checkout as the changes have already copied out
* Minor changes for parameters
* Update Generate-DocIndex.ps1
* Update Generate-DocIndex.ps1
* Update Generate-DocIndex.ps1
* Added function to common
* Update Generate-DocIndex.ps1
Co-authored-by: Sima Zhu <sizhu@microsoft.com>
Co-authored-by: Sima Zhu <48036328+sima-zhu@users.noreply.github.com>
* Move entire docgeneration into common tools
* Move docindex to common
* Added the package replacement logic
* Fixed on parameters
* Fixed param
* Change function to dash
* Added regex on function
* Added display name.
* Update eng/common/docgeneration/Generate-DocIndex.ps1
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* Deal with js
* Add no new line args
* revert some test changes
* Need to default to the double quotes for JS regex
* Update Generate-DocIndex.ps1
* Added the appTitle
* type
Co-authored-by: Sima Zhu <sizhu@microsoft.com>
Co-authored-by: Sima Zhu <48036328+sima-zhu@users.noreply.github.com>
Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com>
* Cache created service principal for iteration
Useful when testing changes over and over again without passing your own -TestApplicationId and -TestApplicationSecret.
* Restore initial AzContext for New-TestResources
* Make sure PSBoundParameters is correct
Fixes#1177
Co-authored-by: Heath Stewart <heaths@microsoft.com>
* Add debugging link on resource deployment failures to log output
* Update aka link for live test help docs. Use here string and empty throw.
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* Add debug flag to arm deployment command
* Only set debug preference when $CI is true
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* Update subscription configuration schema to include new parameters
* Support platform specific arm template parameters and legacy hashtable format
* Update arm template parameter comment to include top level key
* Restore AdditionalParameters. Merge ArmTemplateParameters from stringified hash literal
* Handle duplicate keys more explicitly for arm and env vars
* Regenerate New-TestResources.ps1 markdown
* revert variable name to environmentVariables to fix post-scripts
* Handle empty arm template parameters better
* Remove arm template parameter merge logic from deploy template
* Add merge hashes function to New-TestResources.ps1
* Add merge hashes function to New-TestResources.ps1
* Add env variable overwrite warning. Use ContainsKey checks
* Temporarily manually fix invalid generated markdown links
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
This also adds previously-required parameters back into
@PSBoundParameters to pass down to pre- and post-scripts.
Co-authored-by: Heath Stewart <heaths@microsoft.com>
* Remove resource group asynchronously and do not wait for completion
* Verify resource group has reached Deleting state before exiting script
* Use $_ instead of $Error[0] for resource group removal handling
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
* Simplify Net-TestResources usage
* docs and windows check
* Update eng/common/TestResources/New-TestResources.ps1
Co-authored-by: Heath Stewart <heaths@outlook.com>
* update markdown
* make service directory the default parameter
* Fix links
* Doc change
Co-authored-by: Pavel Krymets <pavel@krymets.com>
Co-authored-by: Heath Stewart <heaths@outlook.com>
* Update cache-control for newly created blobs
* remove extra
* remove extra space
* Added the cache control for existing blobs as well
* Update copy-docs-to-blobstorage.ps1
Co-authored-by: Sima Zhu <sizhu@microsoft.com>
Co-authored-by: Sima Zhu <48036328+sima-zhu@users.noreply.github.com>
* Update location of patch files
* Add some changes in eng/common to test things
* Switch to pushing to upstream
* Change workflow to use upstream branches
Co-authored-by: Chidozie Ononiwu <chononiw@microsoft.com>
* update version to dev.1 for test release
* Update release date in CHANGELOG.md
* Add checkout step
* Faster matrix
* Add Skip.Test to improve inner loop
* Show more info for tagging
* Revert "Faster matrix"
This reverts commit 713afd3d64743cbc1841e1ee965e936c049f7cf7.
* Add correct folder
* Add fixes for tagging repo
* Use Invoke-WebRequest for GetExistingTags
* Use -Uri
* return empty list
* Add vcpkg caching
* Set contents to copy
* Add doc uploads for C++
* Revert "Update release date in CHANGELOG.md"
This reverts commit 6fb302701381758cbb3d3828c2ce8d29d6b77950.
* Revert "update version to dev.1 for test release"
This reverts commit 4e188e4da99cb1246f195a5362e47d6ff80c3219.
* Revert "Add vcpkg caching"
This reverts commit 1f5f4d95e21d07d5ade1df470f39ceb478849f06.
