Commit Graph

592 Commits

Author SHA1 Message Date
Victor Vazquez
f90ab8a9b5
Add Storage Performance tests for DownloadTo API (#2987)
Adding 3 new tests for Storage Performance
The next tests are expected to find out the overhead between each layer of the Azure SDK client while performing a download operation. The current performance test is measuring the client + azure core pipeline + http transport.

The next tests removes authentication policy and then each layer up to the HTTP transport adapter directly.

DownloadBlobSas
Generates a SaS token to download a blob. This way the authentication policy "sharedKey" won't run in the pipeline.

DownloadBlobWithPipelineOnly
Use the SaS token as well but send the request without the BlobClient. Instead it uses the Azure Core Pipeline directly

DownloadBlobWithTransportOnly
Same as before, the SaS token is used to download the blob but consuming only the libcurl-transport adapter. This option skips the HttpPipeline. The only elements used from Azure Core are the HttpRequest and the Azure::Context

Note
porting tests from the .NET equivalent:

https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/storage/Azure.Storage.Blobs/perf/Azure.Storage.Blobs.Perf/Scenarios/DownloadSasUriBlobClient.cs

https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/storage/Azure.Storage.Blobs/perf/Azure.Storage.Blobs.Perf/Scenarios/DownloadSasUriHttpClient.cs

https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/storage/Azure.Storage.Blobs/perf/Azure.Storage.Blobs.Perf/Scenarios/DownloadSasUriHttpPipeline.cs
2021-10-25 11:48:55 -07:00
Daniel Jurek
c534a1e160
Cognitive Matrix Consolidation (#2947)
* Refactor template to use multiple builds on the same machine resetting the machine and runtime varaibles along the way

Move template expression out of default matrix parameters

Remove extra reference to ci/jobs.yml

Remove unnecessary ci.yml, change param declaration in devops-variables-set.yml

Heredoc

Fix set/reset scripts after testing with devops outputs

-ErrorAction Ignore

Smaller matrix

OsVMImage comes from machine, not parameters

Add Ubuntu 20 to matrix

Remove checkout: self, it is implied

Move machine-setup steps outside of loop

CODE_COVERAGE handling

tab

Remove creation of CODE_COVERAGE

Use AptDependencies and add apt update to dependency install step

Remove extra preparation steps

Enable code coverage check scenario

Add MacOS 11

Disable vcpkg cache

Coverage configuration

Pass parameter properly

bool -> boolean

Install coverage tools if it's possible that coverage may run

More logging on code coverage

Show contents of Makefile

One vcpkg to build them all

Verbose

SourcesDirectory

Nesting VcpkgInstall

workingDirectory for vcpkg install

AptDependencies

complete the conditional

Enable all matrix entries

sudo

Remove extra chatter

More scrubbing post-build to help with Windows scenarios

sudo echo

Get-ChildItem

ErrorAction

sudo sh -c

More ErrorAction

output sourceListFile

Disable additional sources.list

Install coverage tools when used for coverage

COndition

Do not build testing for Windows arm64

Correct path for coverage tools

ne -> eq

Add more scenarios to the matrix

Remove libxml2

Add MacOS 10.15 to matrix

Revert "Remove libxml2"

This reverts commit 178af8b89abb21ee84dbcd25b4d54bd90ec1d44c.

Add libxml2 to MacOS 10.15 matrix entry

Display name decorator, refine the matrix

Remove OsVmImage variable

Add commentary to tests about matrix

Rebalance matrix, nest EnvVars, DisplayNameDecorator

Correct parameter access

AptDependencies

Ensure building for WindowsStore in UWP builds, Only build tests for debug builds in altered matrix entires, Linux build type specified as environment variable

Rebalance matrix for better timing

Remove all instances of libxml2 in vcpkg dependencies

Move common devops tasks to eng/common

Set eng/common paths in pipelines

Revert "Remove all instances of libxml2 in vcpkg dependencies"

This reverts commit 35ffbc7aac87f400fdf3351076346750c1c750ed.

https://github.com/Azure/azure-sdk-for-cpp/issues/2946

Permit words like iname in more pipeline .yml files

Remove libxml2 from vcpkg dependencies for Windows

Perf test matrix maxParallel: 10

Validate

apt install first

apt install first

maxParallel: 12

-j 8

-j 10

Use matrix generator

Add expected parameters

Move parameters to appropriate file, fix cloud spec

Specify correct platform matrix location

DependsOn

Platform matrix

Remove DisplayNameDecorator, try more clang information

Rapid iteration matrix

clang --version

install clang-11

Use correct pool

Use full matrix, revert additional logic in jobs/ci.yml

* Review feedback: newline

* Review feedback: Test in release, too

* Remove DisplayNameDecorator (it was useful when we were putting multiple builds into a single job, now it is redundant)

* Remove displayNames

* Matrix review feedback

* Conditions

* Rename ci.yml -> ci.tests.yml
2021-10-22 13:49:10 -07:00
Anton Kolesnyk
b477373a9c
Remove extra semicolons (#2979)
Co-authored-by: Anton Kolesnyk <antkmsft@users.noreply.github.com>
2021-10-19 16:08:42 -07:00
JinmingHu
9dac89c675
make DefaultEncryptionScope and PreventEncryptionScopeOverride optional (#2968) 2021-10-16 10:06:18 +08:00
Victor Vazquez
f4e6390dff
Enable no-rtti (#2963)
* Enable no-rtti
2021-10-14 18:12:03 -07:00
JinmingHu
88500317b7
Add ut for queue message containing special characters (#2858) 2021-09-09 12:04:58 +08:00
Azure SDK Bot
ef1dd860d5
Increment version for storage releases (#2846)
* Increment package version after release of azure-storage-blobs

* Increment package version after release of azure-storage-files-datalake

* Increment package version after release of azure-storage-files-shares

* Increment package version after release of azure-storage-queues
2021-09-09 10:42:50 +08:00
Azure SDK Bot
18ec808a25
Increment package version after release of azure-storage-common (#2847) 2021-09-09 10:41:41 +08:00
JinmingHu
a1f69cd1fb
Update vcpkg.json (#2845) 2021-09-08 12:01:47 +08:00
JinmingHu
e6a488193b
Storage Sept Release (#2836) 2021-09-08 09:39:24 +08:00
JinmingHu
841675b622
Fix a bug where blobs with some characters cannot be listed (#2837)
* fix bug

* CL

* fix bug
2021-09-07 15:14:37 +08:00
JinmingHu
6611b8f03f
Improvement: Create less threads when there isn't enough data to transfer (#2838)
* Improvement: Create only necessary threads when there isn't enough data to transfer

* Update sdk/storage/azure-storage-common/inc/azure/storage/common/internal/concurrent_transfer.hpp

* CL
2021-09-07 14:42:17 +08:00
JinmingHu
47d04c3ff9
New XML parser on Windows (#2830) 2021-09-04 23:04:12 +08:00
JinmingHu
b0ebf21a9f
small fix in checking nullable variable (#2832) 2021-09-04 23:03:24 +08:00
JinmingHu
98b188330b
StartsOn and ExpiresOn were changed to nullable (#2828) 2021-09-03 12:57:50 +08:00
Kan Tang
b2b1bdffd0
Make AccessPolicy::StartsOn and ExpiresOn nullable values. (#2829)
# Pull Request Checklist

Please leverage this checklist as a reminder to address commonly occurring feedback when submitting a pull request to make sure your PR can be reviewed quickly:

See the detailed list in the [contributing guide](https://github.com/Azure/azure-sdk-for-cpp/blob/main/CONTRIBUTING.md#pull-requests).

- [x] [C++ Guidelines](https://azure.github.io/azure-sdk/cpp_introduction.html)
- [x] Doxygen docs
- [x] Unit tests
- [x] No unwanted commits/changes
- [x] Descriptive title/description
  - [x] PR is single purpose
  - [x] Related issue listed
- [x] Comments in source
- [x] No typos
- [x] Update changelog
- [x] Not work-in-progress
- [x] External references or docs updated
- [x] Self review of PR done
- [] Any breaking changes?
2021-09-03 04:57:36 +00:00
JinmingHu
29659f9350
Storage Queue Review Feedback (#2823)
* Remove CreateIfNotExists

* GetServicePropertiesResult/GetServiceStatisticsResult

* GetQueuePropertiesResult/GetQueueAccessPolicyResult

* CL
2021-09-03 09:49:24 +08:00
JinmingHu
bee0dc2654
ignore specific words rather than skip the whole file for spell-checking augo-generated files (#2805) 2021-08-30 15:05:01 +08:00
JinmingHu
b8013185d0
Storage Queue API review feedback (#2801)
* fix

* sample

* CL

* service version

* fix

* ACL
2021-08-27 04:46:51 +00:00
JinmingHu
8cae42999e
cspell for storage (#2800)
* cspell for storage

* aaa
2021-08-27 11:55:07 +08:00
Anton Kolesnyk
7332432efd
Re-add vcpkg version constraints (#2781)
Co-authored-by: Anton Kolesnyk <antkmsft@users.noreply.github.com>
2021-08-24 16:10:07 -07:00
Anton Kolesnyk
f1ae8a9cdf
Rename exceptions message parameter to whatArg (#2782)
Co-authored-by: Anton Kolesnyk <antkmsft@users.noreply.github.com>
2021-08-23 18:41:16 -07:00
Victor Vazquez
8fe306e15f
missing include (#2778) 2021-08-20 15:41:48 -07:00
JinmingHu
f72c86cc11
Use std::chrono::seconds for time duration in Queue (#2766)
* Use std::chrono::seconds for time duration in Queue

* add include header

* add const val
2021-08-18 11:50:02 +08:00
JinmingHu
b27cabf20f
queue sample (#2758) 2021-08-18 09:57:47 +08:00
JinmingHu
68100a543c
Disable storage queue vcpkg (#2749) 2021-08-17 09:45:40 +08:00
JinmingHu
4a224e9e8d
Storage Queue Service (#2671)
* Queue clinet interface

* clang-format

* impl

* queue impl

* ut

* fix Create()

* remove unused options

* fix ut

* fix

* more test cases

* fix build errors

* fix build error

* update status code check in generated protocol layer code

* CreateQueueResult.Created should be false on 204

* queue message encoding option

* add some doc

* clang-format

* more doc

* queue sas

* disable some test case

* Revert "queue message encoding option"

This reverts commit a3e6fb1e40e04f4fa3090598066dbb16a4938b85.

* Some API change

* update some docs

* clang-format

* CL

* update vcpkg portfile to be consistent

* queues.hpp

* clang-format

* add file doxygen

* account sas queue ut
2021-08-16 12:25:39 +08:00
JinmingHu
a9507ae88d
Fix build error on some platforms (#2734)
* Move uuid impl to .cpp file

* CL
2021-08-13 07:50:25 +08:00
Azure SDK Bot
e5c60f913c
Increment version for storage releases (#2733)
* Increment package version after release of azure-storage-common

* Increment package version after release of azure-storage-blobs

* Increment package version after release of azure-storage-files-datalake

* Increment package version after release of azure-storage-files-shares
2021-08-11 01:26:35 +00:00
JinmingHu
8fcc1df085
Update Changelog for storage 12.1.0 release (#2731) 2021-08-10 16:01:00 -07:00
JinmingHu
d65724b6dc
storage 12.1.0 release (#2714)
* storage 12.1.0 release

* a

* Apply suggestions from code review

Co-authored-by: Anton Kolesnyk <41349689+antkmsft@users.noreply.github.com>

Co-authored-by: Anton Kolesnyk <41349689+antkmsft@users.noreply.github.com>
2021-08-10 13:05:10 +00:00
Anton Kolesnyk
6bf52bdbde
Update vcpkg version dependencies (#2718) 2021-08-09 13:15:27 -07:00
JinmingHu
9b3b63df47
improve status code check logic in protocol layer (#2703) 2021-08-04 19:53:05 -07:00
JinmingHu
f3f798058e
small fix to blob doc (#2709) 2021-08-04 18:33:50 +08:00
JinmingHu
7a82f92465
remove superfluous (void) (#2693) 2021-08-02 08:46:02 -07:00
JinmingHu
79143d4a1a
Remove BlobItemDetails::TagCount (#2684) 2021-07-29 12:41:11 +08:00
Victor Vazquez
5c14975957
remove prefer ninja (#2675) 2021-07-27 19:26:11 -07:00
JinmingHu
2f9ab65526
Fixed a bug where GetBlockList throws exception if the blob is not committed (#2664)
* Fix a bug where GetBlockList threw exception if the blob is not committed.

* CL

* f
2021-07-27 14:12:43 +08:00
John Heffner
1839061e0d
Avoid time domain casting exception during request cancellation (#2645)
Context::Cancel() sets the deadline to DateTime::min(), which throws a range
exception when cast to system_clock::time_point.  Instead, cast
system_clock::now() to DateTime.
2021-07-22 13:10:37 +08:00
JinmingHu
9806e4692e
move constant definition to bottom to get around a bug in APIView tool (#2648) 2021-07-21 17:51:52 +08:00
Victor Vazquez
41c79fb2c1
adding listBlobs perf test (#2639) 2021-07-20 13:34:15 -07:00
JinmingHu
c4b88f933e
Fixed a bug where sequence number access conditions didn't work for page blob operations (#2643)
* PageBlobClient::Resize doesn't support sequence number access conditions

* PageBlobClient::UpdateSequenceNumber doesn't support seq access conditions

* Fixed a bug where sequence number access conditions didn't work for page blob operations
2021-07-16 07:43:51 +00:00
JinmingHu
b4037ee591
Storage unit test improvements (#2642)
* add test for downloading empty file/blob

* Unit test improvements

* f
2021-07-16 04:34:39 +00:00
JinmingHu
59c46e2ab1
Fixed a bug where unspecified SMB properties got overwritten rather than preserved by SetProperties() (#2632)
* fix bug: Set{File/Directory}Properties overwrites unspecified values

* CL
2021-07-16 10:08:50 +08:00
JinmingHu
5b87554736
PageBlobClient::SetBlobSize doesn't support encryption headers (#2603)
* PageBlobClient::SetBlobSize doesn't support encryption headers

* update ut
2021-07-14 11:01:00 +08:00
JinmingHu
46e333d32d
Remove sha256 hash from storage internal code (#2585) 2021-07-12 10:07:26 +08:00
JinmingHu
ae112cd8a4
UpdateSequenceNumber (#2588) 2021-07-10 02:11:04 +00:00
JinmingHu
da755c9192
Add queue service protocol layer (#2531)
* Add queue service protocol layer

* Update sdk/storage/azure-storage-queues/NOTICE.txt

* fix typo, add comments

* clang-format

* Update sdk/storage/azure-storage-queues/CHANGELOG.md

Co-authored-by: Anton Kolesnyk <41349689+antkmsft@users.noreply.github.com>

* Update sdk/storage/azure-storage-queues/CMakeLists.txt

Co-authored-by: Anton Kolesnyk <41349689+antkmsft@users.noreply.github.com>

* Update sdk/storage/azure-storage-queues/vcpkg/vcpkg.json

Co-authored-by: Anton Kolesnyk <41349689+antkmsft@users.noreply.github.com>

Co-authored-by: Anton Kolesnyk <41349689+antkmsft@users.noreply.github.com>
2021-07-09 06:07:00 +00:00
JinmingHu
186ccc7164
Fix misplaced entry in datalake changelog (#2584) 2021-07-09 04:15:42 +00:00
JinmingHu
bc16d4d812
Some API changes to blob service (#2557) 2021-07-09 11:26:36 +08:00