Skip to content
Merged
Show file tree
Hide file tree
Changes from 143 commits
Commits
Show all changes
148 commits
Select commit Hold shift + click to select a range
1890d91
changing recording to use test-proxy
seankane-msft Jul 19, 2021
c55ef02
updating client and test proxy
seankane-msft Jul 19, 2021
9ef5856
updated Transport Do to send actual request
seankane-msft Jul 19, 2021
4603915
whitespace
seankane-msft Jul 20, 2021
2099dbc
working implementation
seankane-msft Jul 20, 2021
99e3ecf
updating internal recording
seankane-msft Jul 20, 2021
89dea93
now I have a blank recording...
seankane-msft Jul 20, 2021
366abc2
cleaning up
seankane-msft Jul 20, 2021
f4dc6cf
more clean up
seankane-msft Jul 20, 2021
af288f0
transport Do is repeating, proxy is not parsing URL correctly
seankane-msft Jul 20, 2021
22ae67f
small changes
seankane-msft Jul 21, 2021
79b6a2b
merge conflicts
seankane-msft Jul 26, 2021
9e00c51
adding a policy to do the same thing
seankane-msft Jul 26, 2021
8c32067
Merge branch 'track2-tables' of https://github.com/Azure/azure-sdk-fo…
seankane-msft Jul 28, 2021
2f2e990
adding a second test for proxy and transport
seankane-msft Jul 28, 2021
25e262f
further progress with Jeff, switching to only using a policy right now
seankane-msft Jul 28, 2021
756ce72
cleaning up file
seankane-msft Jul 29, 2021
f8965dc
cleaning up
seankane-msft Jul 29, 2021
c06704a
added more debugging, added two required headers that I was missing
seankane-msft Jul 29, 2021
91399e7
proxy working on a single test, need to convert all tests to test proxy
seankane-msft Jul 29, 2021
e236fa1
cleaning up
seankane-msft Jul 29, 2021
b048044
moved over successfully :)
seankane-msft Jul 29, 2021
2abbbcd
more helper functions
seankane-msft Jul 29, 2021
d7d31d9
working for a subtest functionality
seankane-msft Jul 29, 2021
7c3be8d
fixed up a service client test
seankane-msft Jul 30, 2021
18997bc
converting more service tests
seankane-msft Jul 30, 2021
1868088
converted all service client tests
seankane-msft Jul 30, 2021
f5de97a
converted table client tests
seankane-msft Jul 30, 2021
71b10ac
fixed last two client tests
seankane-msft Jul 30, 2021
214e477
entity test
seankane-msft Jul 30, 2021
e61297f
entity tests
seankane-msft Jul 30, 2021
26eb607
converting access policy tests
seankane-msft Aug 2, 2021
992f303
all passing in record mode
seankane-msft Aug 2, 2021
8bb758c
batch tests
seankane-msft Aug 2, 2021
1b9c3bd
liveonly for batch tests
seankane-msft Aug 2, 2021
4e7a0e7
adding a sanitizer and more methods for recording
seankane-msft Aug 2, 2021
d23cb92
adding main method to start and stop the proxy automatically
seankane-msft Aug 2, 2021
5d50021
adding test proxy step to pipeline
seankane-msft Aug 2, 2021
bc5c63a
double dash on version arg
seankane-msft Aug 2, 2021
6d022ed
need a better way to get userful error reporting
seankane-msft Aug 2, 2021
ca7f9d6
explicitly running test-proxy in the background
seankane-msft Aug 2, 2021
43d8c43
adjust the proxy start and install to be compatible with specific pro…
scbedd Aug 2, 2021
5dd22bc
Merge branch 'test-proxy' of https://github.com/seankane-msft/azure-s…
seankane-msft Aug 2, 2021
8580096
dont run test proxy in background
seankane-msft Aug 2, 2021
6178b38
didnt remove an import
seankane-msft Aug 2, 2021
9ed3455
header to proxy_test.go
seankane-msft Aug 2, 2021
38a1a6d
fixes for pipeline
seankane-msft Aug 2, 2021
ecab2a5
added loggin on accident
seankane-msft Aug 2, 2021
fd20e49
use nohup on linux machines
scbedd Aug 2, 2021
f355f82
further simplify nohup usage
scbedd Aug 2, 2021
2e2110c
there are no exes on windows
scbedd Aug 2, 2021
53d3b1c
fixing errors, passing in default env variable
seankane-msft Aug 2, 2021
76053b8
Merge branch 'test-proxy' of https://github.com/seankane-msft/azure-s…
seankane-msft Aug 2, 2021
3390179
force background for linux task. may need another iteration to place …
scbedd Aug 2, 2021
a6190d6
Merge branch 'test-proxy' of https://github.com/seankane-msft/azure-s…
scbedd Aug 2, 2021
82a3230
fake aad credential
seankane-msft Aug 2, 2021
9feac51
Merge branch 'test-proxy' of https://github.com/seankane-msft/azure-s…
seankane-msft Aug 2, 2021
416e257
clean up after test-proxy
scbedd Aug 2, 2021
8f0b219
Merge branch 'test-proxy' of https://github.com/seankane-msft/azure-s…
scbedd Aug 2, 2021
fd8c713
add print for record mode
seankane-msft Aug 2, 2021
df1d4d7
Merge branch 'test-proxy' of https://github.com/seankane-msft/azure-s…
seankane-msft Aug 2, 2021
97fac55
another different attempt at starting the proxy
scbedd Aug 2, 2021
bfc545c
Merge branch 'test-proxy' of https://github.com/seankane-msft/azure-s…
scbedd Aug 2, 2021
06febdd
jobs apparently is an unrecognized command on our linux boxes
scbedd Aug 2, 2021
a30493d
correct dumb typo in run tests
scbedd Aug 2, 2021
f6574b5
replace nuget install with docker
scbedd Aug 2, 2021
3b57585
working fake credentials for client delete entity test
seankane-msft Aug 3, 2021
31de5d4
fixing fake credentials portion, passing locally
seankane-msft Aug 3, 2021
c937f2e
trying an insecureskipverify transport
seankane-msft Aug 3, 2021
2213bd5
debugging statements to find the cwd
seankane-msft Aug 3, 2021
d1a7ab6
removing blank recording
seankane-msft Aug 3, 2021
1a70cb1
working implementation of sas
seankane-msft Aug 3, 2021
a0b9850
working sas credential for account signature
seankane-msft Aug 3, 2021
06f7399
table sas is failing
seankane-msft Aug 4, 2021
ad4ad61
leverage proxy tool, not docker
scbedd Aug 4, 2021
2b02ea7
remove apostrophes
scbedd Aug 4, 2021
f53f912
Merge branch 'test-proxy' of https://github.com/seankane-msft/azure-s…
seankane-msft Aug 4, 2021
321800f
merge conflicts
seankane-msft Aug 4, 2021
939289c
moving recordings one directory up
seankane-msft Aug 4, 2021
130209d
transitioning to using the sas written by storage team
seankane-msft Aug 4, 2021
056adee
using storage methods
seankane-msft Aug 4, 2021
cf8f99a
copied storage code, sas needs to be appended in a different way
seankane-msft Aug 4, 2021
f2f3ef8
changing directory for test-proxy to start from
seankane-msft Aug 4, 2021
5019877
removing parenthesis
seankane-msft Aug 4, 2021
9dbe47d
adding steps for validating ssl
seankane-msft Aug 4, 2021
a0a9ba7
switching to docker
seankane-msft Aug 4, 2021
ecb9e2b
adding start server script
seankane-msft Aug 5, 2021
f861f5c
working implementation if we can fix the prepending of 'recording' to…
seankane-msft Aug 5, 2021
ab07220
big refactor, moving stuff into recording file, cleaning up proxy_tes…
seankane-msft Aug 5, 2021
a192402
last fixings
seankane-msft Aug 5, 2021
9c97403
convert back to docker. windows and linux images now present
scbedd Aug 6, 2021
49c446b
double condition
scbedd Aug 6, 2021
3bb4b37
moving configuration to a separate file. preparing for eng/common move
scbedd Aug 6, 2021
8e76a04
tier 0 of trust. I don't expect this to work, but it's still worth a …
scbedd Aug 6, 2021
567588a
update condition to use variable syntax
scbedd Aug 6, 2021
6d95e96
guess I'm not using variables. removing
scbedd Aug 6, 2021
e2c78ee
call start-server.ps1
scbedd Aug 6, 2021
fafb7c5
adjust build-test and configure-proxy to run the docker container in …
scbedd Aug 6, 2021
7c4c3d7
wrap in quotes for the container create. it's apparently a bad
scbedd Aug 6, 2021
0219745
adjust the initialize call
scbedd Aug 6, 2021
105e29e
disable vet temporarily
scbedd Aug 6, 2021
808885b
looking for cert file in env variable PROXY_CERT
seankane-msft Aug 6, 2021
c881539
set proxy_cert environment variable to find that certificate
scbedd Aug 6, 2021
797a804
changing to crt file
seankane-msft Aug 6, 2021
b0f82b3
use crt cert
scbedd Aug 6, 2021
69f0e4a
resolve conflicts. use crt
scbedd Aug 6, 2021
de2c846
want to see errors
seankane-msft Aug 6, 2021
bcf19b4
Merge branch 'test-proxy' of https://github.com/seankane-msft/azure-s…
seankane-msft Aug 6, 2021
5018d31
powershell errors
seankane-msft Aug 6, 2021
fbc39c4
correcting how the volume binds to the windows container
scbedd Aug 6, 2021
8865646
Merge branch 'test-proxy' of https://github.com/seankane-msft/azure-s…
scbedd Aug 6, 2021
69258a4
small modification for windows container
scbedd Aug 6, 2021
400a97b
finally got the magic sauce
scbedd Aug 6, 2021
6f63a6c
cleaning up to remove azcore from internal
seankane-msft Aug 9, 2021
8789596
removing recordings that use vcr
seankane-msft Aug 9, 2021
a1f1115
Merge branch 'test-proxy' of https://github.com/seankane-msft/azure-s…
seankane-msft Aug 9, 2021
e6c5c21
issue with the url creation
seankane-msft Aug 9, 2021
9c04890
persist query params correctly in azcore.JoinPaths
seankane-msft Aug 10, 2021
44e70a4
removing print statements
seankane-msft Aug 10, 2021
680a4c4
return root
seankane-msft Aug 10, 2021
b7942f4
changing location of script
seankane-msft Aug 10, 2021
7f28fae
forgot the stop command
seankane-msft Aug 10, 2021
d915223
working table level sas implementation
seankane-msft Aug 10, 2021
a86cb1c
adding start/end rk/pk functionality
seankane-msft Aug 10, 2021
b5bab2c
removing print statements
seankane-msft Aug 10, 2021
0fa39dc
removing more non-tables code
seankane-msft Aug 10, 2021
5437cd0
Merge branch 'test-proxy' of https://github.com/seankane-msft/azure-s…
seankane-msft Aug 10, 2021
1327130
docstrings and removing storage only code
seankane-msft Aug 10, 2021
9161e6d
making sas tests live only
seankane-msft Aug 10, 2021
56ae489
Fixing sas table name to be lowercase always, adding test to verify r…
seankane-msft Aug 11, 2021
85a89a2
Fixing sas table name to be lowercase always, adding test to verify r…
seankane-msft Aug 11, 2021
d09bbb9
adding cosmos sas test
seankane-msft Aug 11, 2021
2fc15c1
fix for params
seankane-msft Aug 12, 2021
f3d341d
apiview fixes
seankane-msft Aug 12, 2021
19b22cd
fixing test
seankane-msft Aug 12, 2021
346d401
merge conflicts
seankane-msft Aug 12, 2021
db64747
undoing change
seankane-msft Aug 12, 2021
ff5fb77
updates
seankane-msft Aug 12, 2021
672008c
error naming
seankane-msft Aug 12, 2021
6f0d419
fixed service client, was adding percallpolicies twice
seankane-msft Aug 12, 2021
1ad58a8
simplification
seankane-msft Aug 12, 2021
69b236f
aligning with main
seankane-msft Aug 12, 2021
2b5dcd0
converting from query pager to list pager for naming consistency
seankane-msft Aug 12, 2021
2eb8dbf
updating objects for consistency
seankane-msft Aug 12, 2021
72bbddc
jeffs feedback
seankane-msft Aug 13, 2021
a4e44bc
small fixes
seankane-msft Aug 13, 2021
a7e8b6b
autorest.md changes
seankane-msft Aug 13, 2021
479a88c
fixing tablename
seankane-msft Aug 16, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions eng/pipelines/templates/steps/build-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ steps:
foreach ($td in $testDirs) {
pushd $td

$(Build.SourcesDirectory)/eng/scripts/start-server.ps1 start
$(Build.SourcesDirectory)/eng/scripts/proxy-server.ps1 start

Write-Host "##[command]Executing go test -run "^Test" -v -coverprofile coverage.txt $td | go-junit-report -set-exit-code > report.xml"
go test -run "^Test" -v -coverprofile coverage.txt . > temp.txt
Expand All @@ -49,7 +49,7 @@ steps:
rm coverage.txt
}

$(Build.SourcesDirectory)/eng/scripts/start-server.ps1 stop
$(Build.SourcesDirectory)/eng/scripts/proxy-server.ps1 stop
}
displayName: 'Run Tests'
workingDirectory: '${{parameters.GoWorkspace}}'
Expand Down
File renamed without changes.
40 changes: 25 additions & 15 deletions sdk/azcore/request.go
Original file line number Diff line number Diff line change
Expand Up @@ -66,24 +66,34 @@ func (ov opValues) get(value interface{}) bool {
return ok
}

func stripQueryParams(path string) (string, string) {
if strings.Contains(path, "?") {
splitPath := strings.Split(path, "?")
return splitPath[0], splitPath[1]
}
return path, ""
}

// JoinPaths concatenates multiple URL path segments into one path,
// inserting path separation characters as required.
func JoinPaths(paths ...string) string {
// inserting path separation characters as required. JoinPaths will preserve
// query parameters in the root path
func JoinPaths(root string, paths ...string) string {
if len(paths) == 0 {
return ""
}
path := paths[0]
for i := 1; i < len(paths); i++ {
if path[len(path)-1] == '/' && paths[i][0] == '/' {
// strip off trailing '/' to avoid doubling up
path = path[:len(path)-1]
} else if path[len(path)-1] != '/' && paths[i][0] != '/' {
// add a trailing '/'
path = path + "/"
}
path += paths[i]
return root
}

root, qps := stripQueryParams(root)

for i := 0; i < len(paths); i++ {
root = strings.TrimRight(root, "/")
paths[i] = strings.TrimLeft(paths[i], "/")
root += "/" + paths[i]
}

if qps != "" {
return strings.Join([]string{root, qps}, "?")
}
return path
return root
}

// NewRequest creates a new Request with the specified input.
Expand Down
2 changes: 1 addition & 1 deletion sdk/azcore/request_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -507,7 +507,7 @@ func TestNewRequestFail(t *testing.T) {
}

func TestJoinPaths(t *testing.T) {
if path := JoinPaths(); path != "" {
if path := JoinPaths(""); path != "" {
t.Fatalf("unexpected path %s", path)
}
const expected = "http://test.contoso.com/path/one/path/two/path/three/path/four/"
Expand Down
64 changes: 58 additions & 6 deletions sdk/tables/aztable/byte_array_response.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ import (
"time"
)

// ByteArrayResponse converts the MapOfInterfaceResponse.Value from a map[string]interface{} to a []byte
// ByteArrayResponse is the return type for a GetEntity operation. The entities properties are stored in the Value property
type ByteArrayResponse struct {
// ClientRequestID contains the information returned from the x-ms-client-request-id header response.
ClientRequestID *string
Expand Down Expand Up @@ -45,6 +45,7 @@ type ByteArrayResponse struct {
XMSContinuationNextRowKey *string
}

// newByteArrayResponse converts a MapofInterfaceResponse from a map[string]interface{} to a []byte.
func newByteArrayResponse(m MapOfInterfaceResponse) (ByteArrayResponse, error) {
marshalledValue, err := json.Marshal(m.Value)
if err != nil {
Expand All @@ -65,8 +66,8 @@ func newByteArrayResponse(m MapOfInterfaceResponse) (ByteArrayResponse, error) {
}, nil
}

// TableEntityQueryByteResponseResponse is the response envelope for operations that return a TableEntityQueryResponse type.
type TableEntityQueryByteResponseResponse struct {
// TableEntityListByteResponseResponse is the response envelope for operations that return a TableEntityQueryResponse type.
type TableEntityListByteResponseResponse struct {
// ClientRequestID contains the information returned from the x-ms-client-request-id header response.
ClientRequestID *string

Expand Down Expand Up @@ -101,12 +102,12 @@ type TableEntityQueryByteResponse struct {
Value [][]byte
}

func castToByteResponse(resp *TableEntityQueryResponseResponse) (TableEntityQueryByteResponseResponse, error) {
func castToByteResponse(resp *TableEntityQueryResponseResponse) (TableEntityListByteResponseResponse, error) {
marshalledValue := make([][]byte, 0)
for _, e := range resp.TableEntityQueryResponse.Value {
m, err := json.Marshal(e)
if err != nil {
return TableEntityQueryByteResponseResponse{}, err
return TableEntityListByteResponseResponse{}, err
}
marshalledValue = append(marshalledValue, m)
}
Expand All @@ -116,7 +117,7 @@ func castToByteResponse(resp *TableEntityQueryResponseResponse) (TableEntityQuer
Value: marshalledValue,
}

return TableEntityQueryByteResponseResponse{
return TableEntityListByteResponseResponse{
ClientRequestID: resp.ClientRequestID,
Date: resp.Date,
RawResponse: resp.RawResponse,
Expand All @@ -127,3 +128,54 @@ func castToByteResponse(resp *TableEntityQueryResponseResponse) (TableEntityQuer
XMSContinuationNextRowKey: resp.XMSContinuationNextRowKey,
}, nil
}

type TableListResponse struct {
// The metadata response of the table.
OdataMetadata *string `json:"odata.metadata,omitempty"`

// List of tables.
Value []*TableResponseProperties `json:"value,omitempty"`
}

func tableListResponseFromQueryResponse(q *TableQueryResponse) *TableListResponse {
return &TableListResponse{
OdataMetadata: q.OdataMetadata,
Value: q.Value,
}
}

// TableListResponseResponse stores the results of a ListTables operation
type TableListResponseResponse struct {
// ClientRequestID contains the information returned from the x-ms-client-request-id header response.
ClientRequestID *string

// Date contains the information returned from the Date header response.
Date *time.Time

// RawResponse contains the underlying HTTP response.
RawResponse *http.Response

// RequestID contains the information returned from the x-ms-request-id header response.
RequestID *string

// The properties for the table query response.
TableListResponse *TableListResponse

// Version contains the information returned from the x-ms-version header response.
Version *string

// XMSContinuationNextTableName contains the information returned from the x-ms-continuation-NextTableName header response.
XMSContinuationNextTableName *string
}

func listResponseFromQueryResponse(q TableQueryResponseResponse) *TableListResponseResponse {
return &TableListResponseResponse{
ClientRequestID: q.ClientRequestID,
Date: q.Date,
RawResponse: q.RawResponse,
RequestID: q.RequestID,
TableListResponse: tableListResponseFromQueryResponse(q.TableQueryResponse),
Version: q.Version,
XMSContinuationNextTableName: q.XMSContinuationNextTableName,
}
}
6 changes: 3 additions & 3 deletions sdk/tables/aztable/cosmos_patch_transform_policy.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,10 @@ import (
"github.com/Azure/azure-sdk-for-go/sdk/azcore"
)

// CosmosPatchTransformPolicy transforms PATCH requests into POST requests with the "X-HTTP-Method":"MERGE" header set.
type CosmosPatchTransformPolicy struct{}
// cosmosPatchTransformPolicy transforms PATCH requests into POST requests with the "X-HTTP-Method":"MERGE" header set.
type cosmosPatchTransformPolicy struct{}

func (p CosmosPatchTransformPolicy) Do(req *azcore.Request) (*azcore.Response, error) {
func (p cosmosPatchTransformPolicy) Do(req *azcore.Request) (*azcore.Response, error) {
transformPatchToCosmosPost(req)
return req.Next()
}
Expand Down
4 changes: 4 additions & 0 deletions sdk/tables/aztable/errors.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@

package aztable

import "errors"

var errInvalidUpdateMode = errors.New("invalid EntityUpdateMode")

func checkEntityForPkRk(entity *map[string]interface{}, err error) error {
if _, ok := (*entity)[partitionKey]; !ok {
return partitionKeyRowKeyError
Expand Down
1 change: 1 addition & 0 deletions sdk/tables/aztable/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ go 1.13

replace github.com/Azure/azure-sdk-for-go/sdk/internal => ../../internal
replace github.com/Azure/azure-sdk-for-go/sdk/azidentity => ../../azidentity
replace github.com/Azure/azure-sdk-for-go/sdk/azcore => ../../azcore

require (
github.com/Azure/azure-sdk-for-go/sdk/azcore v0.16.2
Expand Down
10 changes: 5 additions & 5 deletions sdk/tables/aztable/proxy_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -91,10 +91,10 @@ func createTableServiceClientForRecording(t *testing.T, serviceURL string, cred
func initClientTest(t *testing.T, service string, createTable bool) (*TableClient, func()) {
var client *TableClient
var err error
if service == string(StorageEndpoint) {
if service == string(storageEndpoint) {
client, err = createStorageTableClient(t)
require.NoError(t, err)
} else if service == string(CosmosEndpoint) {
} else if service == string(cosmosEndpoint) {
client, err = createCosmosTableClient(t)
require.NoError(t, err)
}
Expand All @@ -118,10 +118,10 @@ func initClientTest(t *testing.T, service string, createTable bool) (*TableClien
func initServiceTest(t *testing.T, service string) (*TableServiceClient, func()) {
var client *TableServiceClient
var err error
if service == string(StorageEndpoint) {
if service == string(storageEndpoint) {
client, err = createStorageServiceClient(t)
require.NoError(t, err)
} else if service == string(CosmosEndpoint) {
} else if service == string(cosmosEndpoint) {
client, err = createCosmosServiceClient(t)
require.NoError(t, err)
}
Expand Down Expand Up @@ -226,7 +226,7 @@ func clearAllTables(service *TableServiceClient) error {
pager := service.ListTables(nil)
for pager.NextPage(ctx) {
resp := pager.PageResponse()
for _, v := range resp.TableQueryResponse.Value {
for _, v := range resp.TableListResponse.Value {
_, err := service.DeleteTable(ctx, *v.TableName, nil)
if err != nil {
return err
Expand Down
Loading