Commit Graph

2206 Commits

Author SHA1 Message Date
Victor Vazquez
400ca5ae29
Adding retry policy to unit tests (#707) 2020-10-07 14:10:11 -07:00
Victor Vazquez
190a0e80b1
fixing typos from #686 (#704)
Follow up from #686 to address PR comments about typos
2020-10-06 23:10:42 +00:00
Kan Tang
a898fd819a
Added SetExpiry DataLake convenience layer. (#696) 2020-09-30 16:13:37 +08:00
JinmingHu
e238e28b0d
[Storage Common] Storage Retry Policy (#588)
* storage retry policy

* revert changes to azure core

* refactor storage retry policy

* fix typo

* change the logic a little bit

* adapt storage retry policy in blob batch client

* fix build error

* Add StorageRetryWithSecondaryOptions
2020-09-30 13:14:28 +08:00
JinmingHu
8c4c4e75a9
[Storage Blobs Service] STG 74 features (#652)
* SetExpiry protocol layer

* add test

* last access time

* upgrade blob sas to 2020-02-10

* datalake sas

* fix build error

* tag count

* enable tag test

* fix build error

* enable container soft delete test
2020-09-30 11:38:25 +08:00
Victor Vazquez
7c32578798
Perf fix windows (#686)
* sample for repro

* adding repro app

* reset win socket for perf windows
2020-09-29 11:37:31 -07:00
Casey Carter
21d4e4c5b4
Response<T> stores Nullable<T> (#654)
* Response<T> stores Nullable<T>

Drive-by: Replace `Nullable(const T&)` with `Nullable(T)` to avoid extra copy when initialized with an rvalue.

* Update sdk/core/azure-core/inc/azure/core/response.hpp

Co-authored-by: Rick Winter <rick.winter@microsoft.com>

Co-authored-by: Rick Winter <rick.winter@microsoft.com>
2020-09-29 10:01:00 -07:00
JinmingHu
fbe95f36c5
fix variable casing (#689) 2020-09-28 11:10:32 +08:00
Rick Winter
8b22d3f478
Add Notice and cgmanifest for Core (#687)
* Add Notice and cgmanifest for Core
2020-09-25 10:57:06 -07:00
Rick Winter
d867d7e8ad
Standardize header includes (#685)
* Standardize header includes

Update sources to match guidelines

- `#include ""` when file is relative to the project
- `#include <>` when file is external to the project

Updated tests to mirror how a customer will include the header

- `#include <>`
2020-09-25 10:56:06 -07:00
Rick Winter
f73f61119b
Add cgmanifest and NOTICE file to template project (#682) 2020-09-24 18:31:55 +00:00
Victor Vazquez
fdcb150621
Replace select for epoll (#676)
* adding poll instead of select

* Windows support for polling

* fix for define check
2020-09-22 16:24:26 -07:00
Sima Zhu
8fed524b96
Replaced relative link with absolute links and remove locale (#668) 2020-09-17 22:28:36 -07:00
Sima Zhu
caf9fbc7d6
Enable master link replacement feature for cpp (#632) 2020-09-17 17:51:27 -07:00
Daniel Jurek
39e8dfd6e7
Vcpkg publish framework (#645)
* Add vcpkg artifact creation and port template copying
* Skip staging if vcpkg port directory doesn't exist
2020-09-17 15:45:43 -07:00
Kan Tang
34944552ad
Resolved an issue where path is not encoded. (#657) 2020-09-16 11:13:12 +08:00
JinmingHu
7584c208ab
Fix bug: encode blob name (#655) 2020-09-16 11:08:59 +08:00
JinmingHu
95f4034490
fix bug in blob tags (#653) 2020-09-15 10:52:23 +08:00
Victor Vazquez
9c9b2c242a
throw transport exception when getting new connection fails (#640)
* throw transport exception when getting new connection fails

* use function instead of macro

* Update function to PascalCase
2020-09-14 13:44:04 -07:00
JinmingHu
73ba0029bc
Add date in Changelog (#637)
* Add date in Changelog

* Update changelog

* Update changelog
2020-09-10 13:51:31 +08:00
Daniel Jurek
01cb68f763
Fix release tagging (#636)
* add ReleaseTag
* pkgId -> pkgName
* pkgId -> pkgName
* beta.2
2020-09-09 22:27:49 -07:00
Azure SDK Bot
9fa18c4174
Increment package version after release of azure-core (#630) 2020-09-09 21:41:10 -07:00
Rick Winter
6b1043fc3f
Update the repo readmes and changelog (#627)
* Update the repo readmes and changelog

* Lowercase beta

* Fix a couple instances of `preview`
2020-09-09 16:55:57 -07:00
Victor Vazquez
db4f318ffa
throw transport error when write error (#623) 2020-09-09 09:26:34 -07:00
JinmingHu
29c58ca4bf
[Storage Blobs Service] update changelog (#615)
* update changelog

* Add changelog

* Update changelog
2020-09-09 10:32:35 +08:00
JinmingHu
a5d09b5f47
disable blob tags ut (#613)
* disable blob tags ut

* fix ut

* disable tags in sas
2020-09-09 09:04:44 +08:00
Anton Kolesnyk
949d143284
Add doxygen comments (#611) 2020-09-08 11:00:06 -07:00
Ahson Khan
40121dd78a
Fix typo in comment. (#610) 2020-09-04 17:55:15 -07:00
Victor Vazquez
864582b10e
fix the url init in credentials (#599)
* fix the url init in credentials

* do not encode tenantId
2020-09-04 16:13:17 -07:00
Victor Vazquez
0dce4de5a1
throw runtime error if response is shorter than expected or gets closed (#577)
* throw runtime error if response is shorter than expected or gets closed

* trhow Azure::Core::Http::TransportException
2020-09-04 14:51:12 -07:00
Victor Vazquez
9eb4d41bfc
fix format and remove inline (#601) 2020-09-04 14:21:44 -07:00
Victor Vazquez
68ee875fca
Legal/adding notice for libcurl sample (#372)
* Adding libcurl sample usage to Notice
2020-09-04 14:15:35 -07:00
JinmingHu
c949400dba
fix bug in file sas (#597) 2020-09-04 17:30:07 +08:00
JinmingHu
bcdeeb1434
[Storage Blobs Service] Blob Tags (#416)
* blob tags

* add TagConditions for CreateSnapshot and GetBlockList

* update tags ut

* remove unsupported conditional headers from PutBlockFromUriOptions

* sas permission ut for tags

* add comment
2020-09-04 14:47:34 +08:00
JinmingHu
911a879a00
[Storage Blobs Service] Blob Batch (#582)
* blob batch

* fix build error

* Update sdk/storage/azure-storage-blobs/test/blob_batch_client_test.cpp
2020-09-04 14:47:11 +08:00
JinmingHu
07208d1720
[Storage Files Service] File sas (#568)
* File sas

* rebase on lastest master
2020-09-04 14:46:24 +08:00
Victor Vazquez
caf705c863
reuse code from tolower char (#583)
* reuse code from tolower char

* c_LocaleInvariant table to do tolower instead of toUpper
2020-09-03 22:46:28 -07:00
Kan Tang
d12db7f413
Added ChangeLog/Readme for file service. (#573) 2020-09-04 04:31:07 +08:00
Kan Tang
d9b58fd935
Resolved breaking change introduced by URL. (#584) 2020-09-04 04:18:48 +08:00
Victor Vazquez
38d90d62f3
URL encode and URL Unification (#561)
* Update URL from CORE with uri builder from storage and make Request to take URL to be created
2020-09-02 23:57:03 -07:00
Daniel Jurek
919610f429
Use version.hpp for package versions (#572)
* Add version.txt
* Add doc generation for template
* Add version.hpp parsing and update capabilities to cmake and engsys
* Get-SdkVersion -> Get-PkgVersion
* Move Update-PkgVersion.ps1 under eng/scripts
* Get-PackageVersion -> Get-PkgVersion
* Update paths, params, verbosity
* Couple fixes to output and make use of new SemVer version
* Add fallback support for verison.txt in cases where we still use it to unblock release artifact generation
* Use version information in release pipeline
* Add workaround to generate storage pipeline artifacts
* eng/scripts/
* Write warning
* Haven't released storage-file-shares yet according to releases on GitHub
* Set release date on changelog.md
* Update CHANGELOG.md to indicate this is a test release
* Remove fallback exception for storage
* Re-add Rick's suggestions
* Revert "Remove fallback exception for storage"
2020-09-02 18:53:46 -07:00
Daniel Jurek
67b21fed12
Template ci.yml (#569)
* Add ci.yml to template project
* triggers on sdk/template
2020-09-01 21:44:02 -07:00
Victor Vazquez
b46a544a7b
fix for unit tests for cleanner (#558)
* fix for unit tests for cleanner
2020-08-28 16:17:16 -07:00
Kan Tang
97bfdf91d2
Rewritten the user agent. (#555) 2020-08-28 21:23:31 +08:00
Kan Tang
f9fee4f56a
Added file/datalake changelog (#554) 2020-08-28 18:26:29 +08:00
JinmingHu
9a1f84fbfb
Update changelog (#551) 2020-08-28 17:09:05 +08:00
Victor Vazquez
6e33262be5
Fix for connection re-use on response error (#548)
* Fix for connection re-use on response error
2020-08-27 22:45:33 -07:00
Victor Vazquez
e782efbb47
perf stress basic struct folder and no op herarchy structure (#390)
* perf stress basic struct folder and no op herarchy structure
2020-08-27 15:27:31 -07:00
JinmingHu
66c331e52b
Resolve review comments (#545)
* Rename UndeleteContainer->Undelete

* Remove concurrent upload page blob

* move offset to the first parameter

* Rename: TransactionalMd5 TransactionalCrc64

* add batch protocol layer
fix x-ms-server-encrypted
transactional-md5

* transcational md5

* Remove Transactional headers from ClearPagesResult

* parse x-ms-blob-content-md5

* Remove redundant space
2020-08-27 22:44:29 +08:00
Kan Tang
62dd39c1be
Resolved review comments. (#544)
* Fixed review comments.

* Resolved some review comments
2020-08-27 16:54:35 +08:00
JinmingHu
3a4e1f3b49
Restructure storage directory (#536)
* Restructure azure storage directory

* Add tests and sample

* reformat with clang-format

* Reorder files in CMake

* load versions from version.txt

* fix crash

* Update README.md

* Use relative links

* fix CI

* Fix CI

* Try to fix CI

* Move option to the top of the file
2020-08-27 16:16:08 +08:00
Kan Tang
53bd00034b
Added samples and more test cases to file service and resolved some issues. (#539) 2020-08-26 19:56:15 -07:00
Victor Vazquez
1f0da6fcc4
Clean routine (#530)
* adding cleaner routine
2020-08-26 16:51:24 -07:00
Kan Tang
2c84e3f094
Resolve some casing issue (#534) 2020-08-26 12:09:06 +08:00
Rick Winter
c5438d623b
Restructure the azure inc folder (#514)
* Restructure the azure inc folder
2020-08-25 20:08:34 -07:00
Kan Tang
ae6ada5cfb
Added lease support for DataLake's Path (#521) 2020-08-26 07:43:29 +08:00
Kan Tang
b87bd0816c
Added GetSubDirectory. (#524) 2020-08-26 07:43:08 +08:00
Kan Tang
9c9e631c0d
Fixed datalake/file service protocol layer remaining issues. (#523)
* Fixed datalake build warnings on linux

* Pass context by reference instead.

* Added metadata key case check

* Fixed acronym casing.

* Fixes more acronym casing.
2020-08-26 07:42:46 +08:00
Victor Vazquez
760c645af0
remove double checking mutex (#528) 2020-08-25 15:52:02 -07:00
Victor Vazquez
80f2c2f407
Reuse connection when calling ReadToCount() (#518)
* change EOF for IsEOF function to avoid setting EOF after checking read everytime

* fix setting size to read

* move IsEOF to be private

* Update comment about not moving a connection back to the pool
2020-08-25 10:06:47 -07:00
JinmingHu
cf08d39d14
Remove context from StorageError::CreateFromResponse (#511)
* Remove context from StorageError::CreateFromResponse

* Resolved issues in datalake/file protocol layer

Co-authored-by: Tank Tang <kat@microsoft.com>
2020-08-25 14:38:18 +08:00
Kan Tang
38241bb152
Added support for directory/file client for file service. (#510)
* Added support for directory/file client for file service.

* Resolve multiple comments

* Resolved some further comments

* Removed ShareSnapshot

* Fixed protocol layer and integrated refined concurrent download test from blob service.

* Fixed CI issue on Linux/Mac

* Adapted to change of Storage per-retry policy.

* Removed inconsistency comparing towards other SDKs.
2020-08-25 12:17:21 +08:00
JinmingHu
f0ece5fd7a
[Storage Blobs] Optimize blob ut (#513) 2020-08-25 10:14:20 +08:00
JinmingHu
b2dfbabd04
StoragePerRetryPolicy (#512) 2020-08-25 09:52:11 +08:00
Victor Vazquez
70eeec5984
Connection pool for keep alive (#500)
* keep-alive. reuse same connection based on host
2020-08-21 15:41:45 -07:00
JinmingHu
755085f25d
GetBlobServiceStatistics (#496) 2020-08-21 09:29:26 +08:00
Victor Vazquez
67abbb3728
fix clang warnings on e2e tests (#495) 2020-08-20 00:07:44 -07:00
JinmingHu
b9299478ee
[Storage Blob Service] Concurrent upload for Page Blob (#486) 2020-08-20 10:54:28 +08:00
JinmingHu
070772e7c5
Remove Content-MD5 and ContentCRC64 from Create{PageBlob|AppendBlob}Result (#487) 2020-08-20 10:52:21 +08:00
Mitch Denny
420a72a9d5
Adjust CI YAML headers (#448) 2020-08-19 13:49:45 +10:00
JinmingHu
4c5286803b
[Storage Blob Service] ORS (#467) 2020-08-19 09:13:13 +08:00
Victor Vazquez
5cb3a644a4
Thow runtime error if reading fails at FileStream (#481)
* Handle non eof errors on Windows

* trhow on pread error for linux

* adding errno.h to fix macOS
2020-08-18 15:16:47 -07:00
Kan Tang
e0bb037cdd
Fixed test failure due to network latency too low. (#466) 2020-08-18 11:13:39 +08:00
JinmingHu
42ffbc853e
[Storage Blobs Service] Seal Append Blob (#465)
* seal append blob

* copy seal append blob
2020-08-18 09:55:12 +08:00
Kan Tang
3042f69d5b
Rename return types for Datalake and file services. (#464) 2020-08-18 09:28:31 +08:00
JinmingHu
1dce74c642
[Storage Blobs Service] Undelete Container (#462)
* Undelete Container
2020-08-18 09:11:59 +08:00
JinmingHu
24959f30da
[Storage Blobs Service] Rename Results and Options (#460)
* rename UploadFrom

* Rename CommitBlockListResult/Options GetBlockListResult/Options

* Rename DownloadBlobTo{Options/Result} UploadBlockBlobFrom{Options/Result}
2020-08-17 10:06:58 +08:00
JinmingHu
86a22901e9
[Storage Blobs Service] Transactional MD5 and CRC64 (#456)
* MD5

* Crc64

* crc64 concatenate
2020-08-16 00:30:49 +08:00
Rick Winter
0b2b688e72
Context improvements (#457)
* Expose the ContextValueType so callers can use it to get at key values.

* Add a const char * overload for construction

* Remove explicit conversion which adds an extra move construction.
const char* implicitly converts to std::string
2020-08-14 13:02:16 -07:00
Victor Vazquez
e031696bce
handle size 0 chunked response (#451) 2020-08-14 09:57:42 -07:00
Victor Vazquez
311902fe25
Fix waitforsocket usage when receiving from socket (#453) 2020-08-14 09:03:54 -07:00
JinmingHu
069e359562
Rename return types and option types (#449)
* rename return types and option types

* define ConcurrentUploadBlockBlobFromBufferOptions and ConcurrentUploadBlockBlobFromFileOptions

* use std::move

* Rename DownloadTo

* update doc

* Resolved datalake build issues.

* Resolved some cmake issue when linking

Co-authored-by: Tank Tang <kat@microsoft.com>
2020-08-14 13:02:10 +08:00
Kan Tang
d77ecf7917
Added support for share service (#450) 2020-08-14 13:01:51 +08:00
Anton Kolesnyk
cc78776bcb
EnvironmentCredential with ClientSecretCredential support (#442) 2020-08-13 14:28:08 -07:00
Rick Winter
e3394ec3f5
Null out the union for the Undefined type (#444) 2020-08-13 13:11:17 -07:00
JinmingHu
698c4f5848
Replace old RFC1123 impl which cannot be built until GCC 5 (#440) 2020-08-13 08:17:27 +08:00
JinmingHu
a848c65933
x-ms-client-request-id header (#439) 2020-08-13 08:16:52 +08:00
JinmingHu
4c549b1a29
Fix build error (#438) 2020-08-13 08:16:21 +08:00
Rick Winter
1a32179fec
Create template project (#443)
* Create template project
Support retrieving version from code
2020-08-12 15:08:46 -07:00
JinmingHu
2027d5102f
add test cases for base64 encode/decode, hash wrapper (#431) 2020-08-12 10:13:16 +08:00
JinmingHu
65bd5c6bf4
use #include "" instead of #include <> (#418) 2020-08-12 10:12:48 +08:00
Anton Kolesnyk
83174eeec3
Context tweaks (#378) 2020-08-11 17:11:27 -07:00
JinmingHu
ee0b8d07b8
pass Context by const ref (#432) 2020-08-12 07:02:28 +08:00
Rick Winter
8821aadb80
Rename UUID -> Uuid (#436)
* Rename UUID -> Uuid
Update header references
2020-08-11 13:23:14 -07:00
Anton Kolesnyk
302bbf251e
Add authority URL support for ClientSecretCredential (#424) 2020-08-11 11:22:09 -07:00
JinmingHu
68643e3ee2
[Storage Blobs Service] Add test cases for LastModifiedTime and ETag and lease id (#425)
* add test cases for LastModifiedTime and ETag and lease id
2020-08-11 10:40:59 +08:00
JinmingHu
8372d39cc1
[Storage Blobs Service] Customer-Provided Key (CPK-V) and Encryption Scope (CPK-N) (#427)
* cpk-v and cpk-n
2020-08-11 09:45:35 +08:00
JinmingHu
a47789c311
Fix bug: CopyBlob always sends empty access-tier header (#426) 2020-08-11 07:25:09 +08:00
JinmingHu
329a30f0e3
[Storage Blob Service] integrate reliable stream (#421)
* integrate reliable stream

* define c_reliableStreamRetryCount
2020-08-10 16:07:38 +08:00
JinmingHu
cc0ff27eec
[Storage Blobs Service] Only make one HTTP request if the blob size is small for Concurrent Upload API (#419)
* Only make one HTTP request if the blob size is small for Concurrent Upload API
2020-08-08 08:44:03 +08:00
KarishmaGhiya
3060f7341c
clean up links and add link verification (#410) 2020-08-07 11:55:11 -07:00
JinmingHu
c700b7811c
[Storage Blobs Service] Blob Versioning (#409)
* blob versioning
2020-08-07 12:14:26 +08:00
Rick Winter
75590c6d96
Add requestId policy implementation (#239)
* Add requestId policy implementation
Includes uuid class
2020-08-06 15:38:53 -07:00
Victor Vazquez
a5c3f10b7d
204 http client support (#408)
* 204 http client support

* adding test for 204 response
2020-08-06 09:00:09 -07:00
JinmingHu
355184ef4f
[Storage Blob Service] Container Lease and Blob Lease (#402)
* container lease

* blob lease
2020-08-06 13:50:15 +08:00
Kan Tang
46c0af9a70
Added service prefix to file names. (#399) 2020-08-06 13:09:02 +08:00
JinmingHu
1081ed6979
Fix compiler warning (#404) 2020-08-06 07:47:36 +08:00
JinmingHu
5e596b723e
Fix compiler warning (#403) 2020-08-06 07:47:20 +08:00
JinmingHu
52dbc6b2f2
Pascal case for acronyms (#391) 2020-08-05 10:32:03 +08:00
Kan Tang
626a8856f0
Added support for get/set file service properties. (#384) 2020-08-05 10:31:48 +08:00
Victor Vazquez
eeffd6248f
update context to const ref (#389) 2020-08-04 09:45:13 -07:00
Kan Tang
3de57667d4
Added support for create/delete file shares. (#382)
* Added support for create/delete file shares.

* Removed nullable for metadata.
2020-08-04 09:54:04 +08:00
JinmingHu
8867c7abcb
[Storage Blob Service] Get/SetBlobContainerAccessPolicy (#381)
* Get/SetBlobContainerAccessPolicy

* Remove ResetPermissions()
2020-08-04 09:53:34 +08:00
JinmingHu
18b02094ed
Parse error message only from bodyBuffer (#380) 2020-08-04 09:53:22 +08:00
Victor Vazquez
5426cf7a07
Retriable stream (#369)
* Adding retriable_stream
2020-08-03 10:09:12 -07:00
JinmingHu
22f4ba12fc
GetAccountInfo (#371) 2020-08-03 10:32:43 +08:00
Kan Tang
1b4ea7ac70
Added file protocol layer. (#367)
* Added file protocol layer.

* Resolved some comments.
2020-08-01 20:23:50 -07:00
Victor Vazquez
fb729fcf10
Http/download on error (#376)
* Download body on error response codes from request

* Adding test case for downloading body
2020-07-31 22:11:31 -07:00
Anton Kolesnyk
4bf1f62ba8
Core logging infrastructure + logging policy + retry logging (#330) 2020-07-31 16:43:47 -07:00
JinmingHu
b9212922e7
GetBlobServiceProperties and SetBlobServiceProperties (#366) 2020-07-30 16:39:06 +08:00
JinmingHu
cbec5c6af6
Add unit test cases that require SAS token (#359) 2020-07-29 17:42:14 +08:00
JinmingHu
b0b5a18d18
User Delegation SAS (#358) 2020-07-29 16:11:21 +08:00
Victor Vazquez
6e3c3d9acf
avoid making a copy from vectorBody (#364) 2020-07-29 00:50:55 -07:00
Victor Vazquez
9563d9dd3c
Http/making upload chunk size overridle with Request option (#342)
* adding http request option to customize the upload chunk size for bodyStream
2020-07-29 00:16:36 -07:00
JinmingHu
b0facd3dd5
adapt to breaking change in Azure Core (#356)
* adapt to breaking change in Azure Core

* Adapt breaking change for Datalake service.

* Adapt to the breaking change

* Adapt breaking change for Datalake service.

* fix ut

* Fixed UT.

Co-authored-by: Tank Tang <kat@microsoft.com>
2020-07-29 14:55:53 +08:00
Victor Vazquez
d9e39a0e6e
Http/moving DownloadViaStream option to request (#362)
* move DownloadViaStream option to Request

* Fix credentials request

* Fix unit tests

* use constructor instead of set method

* fix tests
2020-07-28 14:10:26 -07:00
JinmingHu
9d1566c5ea
Add Account SAS and Blob SAS (#353)
* Add Account SAS and Blob SAS

* add doc for sas

* remove Nones

* remove Nones
2020-07-28 13:43:04 +08:00
JinmingHu
b380dfb76a
Fix inconsistent RFC1123 date format in different system locales (#352) 2020-07-27 18:49:42 -07:00
Victor Vazquez
ef0c530ee0
Pipeline/download content on pipeline to let Retry policy to handle read/get issues (#323)
make upload chunk size custumizable
2020-07-27 11:40:40 -07:00
Kan Tang
573ffdf1f7
Added Readme/Changelog and 3rd party notice. (#340) 2020-07-24 11:14:10 +08:00
JinmingHu
ef03d3b3ec
Rename XxxRespone->XxxInfo (#337)
* Rename XxxRespone->XxxInfo

* fix doc, remove unused include headers, remove fab4

* Rename SetAccessTierInfo->SetBlobAccessTierInfo
2020-07-24 09:40:15 +08:00
Kan Tang
8568327a56
Refined DataLake sample (#336)
* Refined DataLake sample

* Added DirectoryClient::GetFileClient, resolved some comment and test failures.

* Resolved an issue introduced by lowercasing response headers resulting in error code/message not being passed, and refined sample.

* Refined the structure according to comments.

* Changed buffer initialziation to stack allocated.

* Resolved some further comments.
2020-07-23 15:40:16 +08:00
JinmingHu
29f5e30ad3
integrate TelemetryPolicy into Blob Service (#328)
* integrate TelemetryPolicy into Blob Service

* Also add telemetry support to ADLS Gen 2.

Co-authored-by: Tank Tang <kat@microsoft.com>
2020-07-23 15:39:51 +08:00
Kan Tang
b0b348e9cb
Resolved API review comments and added FileClient and DirectoryClient… (#308)
* Added API to extract raw response.

* Resolved API review comments and added FileClient and DirectoryClient for DataLake.

* Added some blob/adls Gen2 interop functionality.

* Resolved some issues/comments

* Adds more interop, support retry and concurrent upload/download.

* Added support for Azure Core's token credential to support bearer token credential.

* Added integration on RawResponse/Response<T> and resolved some test issues.

* Added validation for Client secret authentication.

* Resolved some review comments and resolved CI issue.

* Resolved a UT failure in Azure Core.

* Resolved some further comments.
2020-07-22 16:03:44 +08:00
JinmingHu
cd023c49f0
Remove MD5 and CRC64 function interface (#324) (#329) 2020-07-22 11:27:27 +08:00
JinmingHu
4d19f3b078
Integrate Response<T> into Storage Blobs Service (#327)
* Revert changes to storage code

* adapt to breaking changes in Azure::Core, Response->RawResponse, pipeline clone returns unique_ptr

* just to make build pass, remove this commit after https://github.com/Azure/azure-sdk-for-cpp/pull/326 is merged

* adapt to Response<T>

* concurrent upload/download also return Response<T>

* add BlobInfo and BlobContainerInfo
2020-07-21 10:31:00 +08:00
Daniel Jurek
1a9918c4db
add CMakeLists.txt and eng to CI and PR triggers (#314) 2020-07-20 10:09:50 -07:00
JinmingHu
2c21d4b018
Remove MD5 and CRC64 function interface (#324) 2020-07-20 10:35:09 +08:00
Anton Kolesnyk
b259db5a3b
Telemetry policy (#264) 2020-07-17 17:03:47 -07:00
Victor Vazquez
6122529bac
Core/response t (#316)
* rename http response to RawResponse

* Adding Response T
2020-07-17 14:43:17 -07:00
JinmingHu
0f37994e44
add perf benchmark code (#319)
* Add multithread perf testcase

* add large scale upload testcase

* Fix compiler error on GCC 9

* Fix failed testcase
2020-07-17 11:16:18 +08:00
Rick Winter
bbbaa06b5c
Exception objects should be thrown and not the pointer to the exception object. (#322)
Exception objects should be thrown and not the pointer to the exception object. 
   https://azure.github.io/azure-sdk/cpp_design.html#c-exceptions
The pattern for the catch expects a reference to the exception and not the pointer.

Example:
try{
   throw Myxception();
}
catch (MyException const& e) {  //process  }
2020-07-16 15:44:34 -07:00
JinmingHu
204c025d09
suggested changes in API review (#312)
* suggested changes in API review

* Rename UrlBuilder->UriBuilder

* add perf test case

* Integrate retrypolicy into blob clients

* remove `Content-Length` header for GET/HEAD/DELETE http requests

* Remove temp files in ut
2020-07-16 00:49:55 -07:00
Victor Vazquez
12b389a579
Http/throw on read error (#274)
* throw exception on Read from wire Error instead of returning -1

* throw exception if read from socket is not CURLE_OK

* missing header for Win

* fix header on windows

* Add more description to transport errors
2020-07-15 10:27:38 -07:00
JinmingHu
3db59439c9
[Storage Blobs Service] ListBlobsByHierarchy (#265)
* add API ListBlobsByHierarchy

* fix compiler error on GCC 8

* Update Blob API version to 2019-12-12

* use lowercase header key
2020-07-15 10:45:22 +08:00
Victor Vazquez
0eacb230cb
Http/fix reading more than content (#311)
* Fix reading more data than content length

* fix for head request
2020-07-14 16:38:20 -07:00
Victor Vazquez
c0faea5f90
Http transport adapter - read all chunks from server until chunkSize is zero (#273)
* fix for getting all chunks from a chunked response until chuksize is zero

* fix for pulling from wire when inner buffer is empty
2020-07-13 23:00:23 -07:00
Victor Vazquez
05c91d6002
Http/missing change for to lower (#279)
* throw exception on Read from wire Error instead of returning -1

* missing change for tolower
2020-07-13 22:02:32 -07:00
Victor Vazquez
57bdc0f6d6
Adding toLower for header names and unit testd (#270) 2020-07-13 17:21:08 -07:00
Victor Vazquez
59d7ebabe1
Http/fix gcc 8 build (#275)
* throw exception on Read from wire Error instead of returning -1
2020-07-13 17:20:44 -07:00
Kan Tang
a1ec5119a6
Datalake full feature support PR. (#257)
* Added test cases.

* Adapt stream breaking change and adding usage for Nullable.

* Resolve CI issue.

* Resolved comments from review.

* Integrate access condition support and resolve more comments.

* Resolved some further comments.

* Resolved some further comments and removed GetMetadata.
2020-07-13 05:31:02 -07:00
Victor Vazquez
c699888daa
Make sure Read will always start on body start. (#256)
* make stream use int64_t instead of uint64_t
2020-07-10 23:02:18 -07:00
Anton Kolesnyk
43dcc6c495
Retry policy (#241) 2020-07-10 00:23:40 -07:00
JinmingHu
83406f23dd
Storage Blob Service Access Conditions, Bitwise enum class (#252)
* access conditions

* bitwise enum class

* Update x-ms-date on each retry

* add a header for all constant strings

* fix some ut

* rename ModifiedTimeAccessConditons->LastModifiedTimeAccessConditions
2020-07-09 21:03:32 -07:00
Billy O'Neal
f6defdf25a
Add tests and locale invariant string equal function in support of the 'extensible enum' guideline. (#197)
* Add locale invariant string equal function in support of the 'extensible enum' guideline.

* Resolve gcc narrowing warning.

* Fix case flipping table to go the correct direction 😳

* organize tests and add one test for invariant locale compare as exaple

* Add more test cases.

* Move to Details namespace.
2020-07-08 22:34:29 -07:00
JinmingHu
90d4ee3203
Add constructor for HttpPipeline which accepts rvalue (#165) 2020-07-08 09:18:33 -07:00
JinmingHu
95aa6361ef
Concurrent upload block blob from file, concurrent download blob to file (#247)
* concurrent download to file

* concurrent upload block blob from file

* update sample

* Change default download chunk size from 8MB to 4MB

* fix CI
2020-07-08 13:26:22 +08:00
JinmingHu
66c7518dce
Suggested changes in API review (#244)
* Rename Properties->HttpHeaders

* Rename BlobXxxItems->Items

* Remove MaxResults in response of list operations

* specify time format in comments for GetUSerDelegationKey API

* per operation pipelines and per retry pipelines

* Define some response types and rename ListBlobsFlat

* Add different return types for some APIs

* Rename BlobDownloadInfo->BlobDownloadResponse

* Assign default value to BlobContentLength to supporess compiler warnings

* add concurrent download to buffer

* concurrent upload block blob from buffer
2020-07-07 10:54:05 +08:00
Victor Vazquez
9fb4119ccb
Memory Stream - File Stream (win/lin) - HTTP Request / Response - Updates (#236)
* make stream use int64_t instead of uint64_t
2020-07-03 22:22:16 -07:00
Chidozie Ononiwu
c22e4f8386
Clean up azure-sdk-tools and reduce dependency on external resources (#228) 2020-07-02 14:20:12 -07:00
JinmingHu
e61c3ff51a
Documentation for BlockBlobClient and Blob options (#235)
* Documentation for blob options

* Rename struct UploadBlobOptions -> UploadBlockBlobOptions

* documentation for BlockBlobClient
2020-07-02 10:01:27 +08:00
JinmingHu
74ce80659b
Documentation for Storage Blob Service, Nullable, signed int, and others (#213)
* use Nullable

* use C++ style cast

* Add documentation

* use signed int instead of unsigned

* disable storage ut

* add timeout in protocol layer

* enable stroage ut

* undelete test case

* Support copy status in properties

* Remove default value for SequenceNumber in CreatePageBlobOptions
2020-06-30 03:03:42 -07:00
Victor Vazquez
2877c959e1
Enable storage unit tests (#227)
* enable core unit tests to be compiled
* Avoid running core unit tests in CI
2020-06-29 14:19:46 -07:00
Victor Vazquez
d58cbad14d
Unique ptr for body stream (#214)
* change bodyStream to unique ptr
2020-06-29 10:33:31 -07:00
Anton Kolesnyk
ad2e63afe1
Remake ClientSecretCredential (#212) 2020-06-27 15:27:11 -07:00
Daniel Jurek
0cfcdca725
Unified Pipelines for storage (#184)
* Add ci.yml for storage

* OSVmImage

* Use correct name for storage

* Update ci.ymls

* Documentation generation

* Fix tabbing

* More tabbing

* Use correct service directory for storage ci.yml

* Add docs assets

* Use cpp for dropdown generation

* Use cmake to generate documentation

* Use CMake to generate doxygen docs

* BuildArgs -> GenerateArgs

* Correct template path

* More CURL options

* Install curl to satisfy minimum build requirements

* use dependency variable name

* Add VCPKG_DEFAULT_TRIPLET

* Always generate documentaion when -DBUILD_DOCUMENTATION=YES

* Build docs at top level

* Change variable names, simplify cmake-build.yml

* -DBUILD_DOCUMENTATION

* Try using MathJax for formula rendering instead of Latex

* Add version.txt for storage

* artifact.Name -> artifact.Path

* Build docs by target

* Remove Doxyfile

* Remove Doxyfile.template

* Remove generate_docs.py, we are using cmake

* Pass CtestRegex to archetype-sdk-client job template)
2020-06-26 10:42:48 -07:00
Rick Winter
078cbc2205
Fix for non-msvc std lib (#222) 2020-06-26 09:45:29 -07:00
Rick Winter
692cab5651
Nullable type (#204)
* Nullable type
Add Nullable type and tests
2020-06-25 13:58:27 -07:00
Kan Tang
1564e5106e
Resolved comments and rewrote StorageError. (#199)
* Resolved comments and rewrote StorageError.

* Changed Upn name, removed ';', refined error output, moved local function to unamed namespace.
2020-06-24 08:54:06 -07:00
JinmingHu
f74011a2e0
Add Unittest for Blob Service, other fixes and enhancements found during validation (#181)
* Give AccessTier a default value to eliminate warning

* Add blob clients conversion functions

* Fix bug in shared key authentication

* Add GetUri().

* make lease-status and lease-state optional

* Check duplicates in metadata

* add tests for blob service

* AppendBlocb doesn't support AccessTier

* fix bug in "RangeFromXml"

* redefine PageRange in convenience layer

* suppress warnings from gtest

* add API AppendBlockFromUri

* connection string parsing

* Add memorystream

* Rename connection string constants

* Adapt to new bodystream, this will be rewritten when bosystream is changed to unique_ptr

* fix compiler errors

* Change MemoryStream interface

* samples framework

* Add newline at the end of file

* fix compiler error
2020-06-24 01:17:09 -07:00
Victor Vazquez
e1e88b3984
Support for requests with no body (#202)
* avoid fail if no body in request

* request with no body

* fix for win86

* Update sdk/samples/http_client/curl/src/azure_core_with_curl_bodyBuffer.cpp

Co-authored-by: Rick Winter <rick.winter@microsoft.com>

Co-authored-by: Rick Winter <rick.winter@microsoft.com>
2020-06-22 18:38:35 -07:00
Victor Vazquez
90345e090c
fix the number of bytes that are read by stream (#203) 2020-06-22 18:24:09 -07:00
Victor Vazquez
e5e063f88c
support no path request (#201) 2020-06-22 18:23:42 -07:00
Victor Vazquez
614fa143ba
adding host header automatically if missing for libcurl transport (#200) 2020-06-22 18:23:30 -07:00
Kan Tang
a8a200ef26
Added convenience layer for datalake service (#178) 2020-06-20 19:55:41 -07:00
Victor Vazquez
90e57362bf
Victor/http/add streams (#183)
* Adding body Stream for HTTP
* Adding libcurl session with custom send and receive
2020-06-18 21:43:00 -07:00
JinmingHu
5758bedc5c
Export symbols when building dll (#177) 2020-06-16 13:28:37 +08:00
JinmingHu
4366159c50
Add more APIs in storage sdk (#169)
* Add xml serializer/deserializer

* Add blob service client, add undelete, set access tier, start copy, abort copy, create snapshot, stage block from uri

* add support for conditional headers

* support range download

* Change the helper functions order to alphabetical

so that we always get the same result in different environemnts.

* use c++ casting

* Add support for append blob and page blob

* remove redundant ; added by auto-generator

* Remove BlobType in BlockBlob::Upload

* Rename blob_client_options.hpp->blob_options.hpp

* Remove API version in request options
2020-06-15 17:31:36 +08:00
Kan Tang
18408f848a
Adding ADLS gen2 protocol layer code. (#153)
* Adding test framework for azure storage.

* Adding ADLS gen2 protocol layer code.
2020-06-09 06:31:12 -07:00
JinmingHu
401298577e
Shared Key Authentication Policy (#160)
* Remove Date in request options, assign x-ms-version a default value

* reformat files with clang-format

* Add shared key policy and basic request policy

* Add openssl dependency for linux and macOS

* add some comments, some slight fixes

* Rename CommonRequestPolicy -> CommonHeaderRequestPolicy
2020-06-09 03:38:35 -07:00
Kan Tang
6e01498038
Adding test framework for azure storage. (#152) 2020-06-09 02:45:11 -07:00
JinmingHu
11f842bfbe
Add support for HTTP methods to unblock storage team (#158)
* Support all HTTP methods

* Add const qualifier

* fix compiler error on some platforms
2020-06-04 09:44:40 -07:00
JinmingHu
5fc6fea9f4
Auto-generated protocol layer of Container and BlockBlob (#144)
* add blob service auto generated protocol layer

* Add storage uri builder

* integrate with pipeline

* Add libxml2 dependency

* Fix variables name

* Rename variable and function name

* Move xxx_options.hpp out of internal directory

* Rename FromConnectionString -> CreateFromConnectionString
2020-06-04 13:21:08 +08:00
JinmingHu
502d3f15a9
Use move to initialize the body buffer (#150) 2020-06-03 10:13:26 -07:00
Rick Winter
602f75744a
HttpPipeline and HttpPolicies (#63)
* HttpPipeline
2020-05-29 10:57:10 -07:00
Kan Tang
0b1eb288db
Use move to initialize the body buffer (#148) 2020-05-28 09:18:54 -07:00
JinmingHu
39f7c37e90
Discard the space after colon in HTTP respone headers (#143)
* Discard the space after colon in HTTP respone headers

* also discard tab
2020-05-28 09:11:46 -07:00
Victor Vazquez
c19f9e2b05
Adding writing response from curl to buffer (#64)
* write headers and response code to Response
2020-05-20 14:48:26 -07:00
Rick Winter
5350d1dbf8
Fix namespace case and file formating (#98) 2020-05-12 11:04:59 -07:00
Rick Winter
6b7b428e71
Fix casing on enum (#94)
* Fix casing on enum
* Update tests
2020-05-08 10:49:30 -07:00
Anton Kolesnyk
55deee50ec
Move scopes to ClientSecret credential, fix multi threading, 'CoW' token that is shared between copies (#62) 2020-04-06 11:08:53 -07:00
Victor Vazquez
61b8cdc3d4
Adding curl and nohttp transport options (#54)
* http client
2020-04-02 11:05:26 -07:00
Anton Kolesnyk
9631e5978a
Add credential classes (#42) 2020-04-01 15:51:24 -07:00
Danny Amirault
a6bb11ed01
Remove dependancy on azure-sdk-build-tools (#58) 2020-04-01 09:01:40 -07:00
Danny Amirault
00b8ba52b3
Auto update package versions upon release generation (#52)
* Auto update package versions upon release generation
- Adapted from .Net package incrementing
 - https://github.com/Azure/azure-sdk-for-net/blob/master/eng/Update-PkgVersion.ps1
- Seed Changelog file to allow updates to happen

* Spelling fix
2020-03-31 09:58:06 -07:00
Victor Vazquez
9f1493af27
Adding http request (#41)
* adding gtest framework

* http request
2020-03-26 11:38:07 -07:00
Danny Amirault
2708ca8f11
Specify artifact path in YML to locate artifact files correctly (#45)
* Specify artifact path in YML to locate artifact files correctly

* Remove doc generation for now
2020-03-25 12:41:15 -07:00
Victor Vazquez
7971d525df
Victor/core/basic structure for cmake to work (#37)
* moving files and updating cmake to make it compilable
2020-03-18 15:39:36 -07:00
Danny Amirault
5dff0dad92
Initial CI pipeline work (#36)
* Replicate C pipeline files to CPP

* Adjust to build CPP

* Remove pipeline pieces copied from C that are not needed for C++ build.

* Remove extraneous debugging step.
2020-03-18 13:32:30 -07:00
Victor Vazquez
010038a342
moving files and updating cmake to make it compilable (#35)
* moving files and updating cmake to make it compilable

* Adding http namespace
2020-03-18 10:58:28 -07:00
Rick Winter
5ee2b47171
Add simple contract implementation 2020-03-17 11:44:40 -07:00
Rick Winter
5fb8226875
Core namespace casing (#12) 2020-02-25 09:01:21 -08:00
Anton Kolesnyk
67a110a055 End namespace comment 2020-02-24 17:53:51 -08:00
Anton Kolesnyk
50d0998a79 Apply style 2020-02-24 17:46:42 -08:00
Anton Kolesnyk
cf249bcc8d Use the same clangformat we ended up using in the Embedded C99 SDK 2020-02-24 17:42:48 -08:00
Rick Winter
eab3352932
Add Core scaffolding 2020-02-20 12:10:01 -08:00