Add debug logging messages on retry by ericphanson · Pull Request #549 · JuliaCloud/AWS.jl

ericphanson · 2022-05-11T18:16:57Z

Closes #514

Based on #548

The logging messages are structured so the string is a human-readable summary of the message, and then the fields are more structured for later analysis, e.g. retry::Bool, reason::String, etc. Additionally, the id of the log message will be useful for aggregation.

ericphanson · 2022-05-11T18:17:07Z

Let's see if CI likes it:

bors try

bors · 2022-05-11T18:23:39Z

try

Build succeeded:

christopher-dG

LGTM!

ericphanson · 2022-05-13T15:08:51Z

ok great! I think we should probably first review/merge #548, then merge this on top.

mattBrzezinski

bors r+

bors · 2023-03-20T20:33:13Z

Merge conflict.

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

mattBrzezinski

bors r+

549: Add debug logging messages on retry r=mattBrzezinski a=ericphanson Closes #514 Based on #548 The logging messages are structured so the string is a human-readable summary of the message, and then the fields are more structured for later analysis, e.g. `retry::Bool`, `reason::String`, etc. Additionally, the id of the log message will be useful for aggregation. Co-authored-by: Eric Hanson <5846501+ericphanson@users.noreply.github.com> Co-authored-by: Matt Brzezinski <matt.brzezinski@beacon.bio>

bors · 2023-03-20T20:49:45Z

Build failed:

Julia 1 - ubuntu-latest - x64

mattBrzezinski · 2023-03-20T20:57:29Z

bors r+

549: Add debug logging messages on retry r=mattBrzezinski a=ericphanson Closes #514 Based on #548 The logging messages are structured so the string is a human-readable summary of the message, and then the fields are more structured for later analysis, e.g. `retry::Bool`, `reason::String`, etc. Additionally, the id of the log message will be useful for aggregation. Co-authored-by: Eric Hanson <5846501+ericphanson@users.noreply.github.com> Co-authored-by: Matt Brzezinski <matt.brzezinski@beacon.bio>

github-actions · 2023-03-20T20:58:53Z

        return nothing
    end

+    check = (s, e) -> begin


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

check = (s, e) -> begin

check =

(s, e) -> begin

github-actions · 2023-03-20T20:58:53Z


+    check = (s, e) -> begin
+            if isa(e, HTTP.StatusError) && AWS._http_status(e) >= 500
+                @debug "AWS.jl Downloads inner retry for status >= 500" retry=true reason="status >= 500" status=_http_status(e) exception=e


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

@debug "AWS.jl Downloads inner retry for status >= 500" retry=true reason="status >= 500" status=_http_status(e) exception=e

@debug "AWS.jl Downloads inner retry for status >= 500" retry = true reason = "status >= 500" status = _http_status(

e

) exception = e

github-actions · 2023-03-20T20:58:53Z

+                @debug "AWS.jl Downloads inner retry for status >= 500" retry=true reason="status >= 500" status=_http_status(e) exception=e
+                return true
+            elseif isa(e, Downloads.RequestError)
+                @debug "AWS.jl Downloads inner retry for Downloads.RequestError" retry=true reason="Downloads.RequestError" exception=e


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

@debug "AWS.jl Downloads inner retry for Downloads.RequestError" retry=true reason="Downloads.RequestError" exception=e

@debug "AWS.jl Downloads inner retry for Downloads.RequestError" retry =

true reason = "Downloads.RequestError" exception = e

github-actions · 2023-03-20T20:58:53Z

+                @debug "AWS.jl Downloads inner retry for Downloads.RequestError" retry=true reason="Downloads.RequestError" exception=e
+                return true
+            else
+                @debug "AWS.jl Downloads inner retry declined" retry=false reason="Exception passed onwards" exception=e


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

@debug "AWS.jl Downloads inner retry declined" retry=false reason="Exception passed onwards" exception=e

@debug "AWS.jl Downloads inner retry declined" retry = false reason = "Exception passed onwards" exception =

e

github-actions · 2023-03-20T20:58:54Z

+    check = (s, e) -> begin
        # Pass on non-AWS exceptions.
        if !(e isa AWSException)
+            @debug "AWS.jl declined to retry non-AWSException" retry=false reason="Non-AWSException" exception=e
            return false
        end


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

check = (s, e) -> begin

# Pass on non-AWS exceptions.

if !(e isa AWSException)

@debug "AWS.jl declined to retry non-AWSException" retry=false reason="Non-AWSException" exception=e

return false

end

check =

(s, e) -> begin

# Pass on non-AWS exceptions.

if !(e isa AWSException)

@debug "AWS.jl declined to retry non-AWSException" retry = false reason = "Non-AWSException" exception =

e

return false

end

github-actions · 2023-03-20T20:58:56Z

+            if isa(e, error)
+                @debug "AWS.jl HTTP inner retry for $error" retry = true reason = "$error" exception =
+                    e
+                return true


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

return true

) exception = e

return true

github-actions · 2023-03-20T20:58:56Z

+        end
+        if (isa(e, HTTP.StatusError) && _http_status(e) >= 500)
+            @debug "AWS.jl HTTP inner retry for status >= 500" retry = true reason = "status >= 500" status = AWS._http_status(


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

end

if (isa(e, HTTP.StatusError) && _http_status(e) >= 500)

@debug "AWS.jl HTTP inner retry for status >= 500" retry = true reason = "status >= 500" status = AWS._http_status(

@debug "AWS.jl HTTP inner retry declined" retry = false reason = "Exception passed onwards" exception =

github-actions · 2023-03-20T20:58:56Z

+            ) exception = e
+            return true


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

) exception = e

return true

return false

github-actions · 2023-03-20T20:58:56Z

+        @debug "AWS.jl HTTP inner retry declined" retry = false reason = "Exception passed onwards" exception =
+            e
+        return false
+    end


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

@debug "AWS.jl HTTP inner retry declined" retry = false reason = "Exception passed onwards" exception =

e

return false

end

github-actions · 2023-03-20T20:58:56Z

+        record.level == Logging.Debug &&
+        get(record.kwargs, :retry, nothing) == successful


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

record.level == Logging.Debug &&

get(record.kwargs, :retry, nothing) == successful

record.level == Logging.Debug && get(record.kwargs, :retry, nothing) == successful

bors · 2023-03-20T21:02:37Z

Build failed:

Julia 1 - ubuntu-latest - x64

mattBrzezinski · 2023-03-20T21:17:53Z

bors try

bors · 2023-03-20T21:22:29Z

try

Build failed:

Julia 1 - ubuntu-latest - x64

github-actions · 2023-03-20T21:26:29Z

                e = AWSException(e, stream)
                rethrow(e)
+            elseif !(e isa AWSException)
+                @debug "AWS.jl declined to retry non-AWSException" retry=false reason="Non-AWSException" exception=e


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

@debug "AWS.jl declined to retry non-AWSException" retry=false reason="Non-AWSException" exception=e

@debug "AWS.jl declined to retry non-AWSException" retry = false reason = "Non-AWSException" exception =

e

mattBrzezinski · 2023-03-20T21:27:22Z

bors try

bors · 2023-03-20T21:31:57Z

try

Build failed:

Julia 1.6 - ubuntu-latest - x64

mattBrzezinski · 2023-03-20T21:40:07Z

bors try

github-actions · 2023-03-20T21:41:40Z


+    check = (s, e) -> begin
+            if isa(e, HTTP.StatusError) && AWS._http_status(e) >= 500
+                @debug "AWS.jl Downloads inner retry for status >= 500" retry=true reason="status >= 500" status=AWS._http_status(e) exception=e


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

@debug "AWS.jl Downloads inner retry for status >= 500" retry=true reason="status >= 500" status=AWS._http_status(e) exception=e

@debug "AWS.jl Downloads inner retry for status >= 500" retry = true reason = "status >= 500" status = AWS._http_status(

e

) exception = e

github-actions · 2023-03-20T21:41:40Z

+    check = (s, e) -> begin
+        errors = (Sockets.DNSError, HTTP.ParseError, Base.IOError)
+        for error in errors
+            if isa(e, error)
+                @debug "AWS.jl HTTP inner retry for $error" retry=true reason="$error" exception=e


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

check = (s, e) -> begin

errors = (Sockets.DNSError, HTTP.ParseError, Base.IOError)

for error in errors

if isa(e, error)

@debug "AWS.jl HTTP inner retry for $error" retry=true reason="$error" exception=e

check =

(s, e) -> begin

errors = (Sockets.DNSError, HTTP.ParseError, Base.IOError)

for error in errors

if isa(e, error)

@debug "AWS.jl HTTP inner retry for $error" retry = true reason = "$error" exception =

e

return true

end

end

if (isa(e, HTTP.StatusError) && _http_status(e) >= 500)

@debug "AWS.jl HTTP inner retry for status >= 500" retry = true reason = "status >= 500" status = _http_status(

e

) exception = e

github-actions · 2023-03-20T21:41:40Z

+                @debug "AWS.jl HTTP inner retry for $error" retry=true reason="$error" exception=e
+                return true
+            end
+        end


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

end

@debug "AWS.jl HTTP inner retry declined" retry = false reason = "Exception passed onwards" exception =

e

return false

end

github-actions · 2023-03-20T21:41:40Z

+        if (isa(e, HTTP.StatusError) && _http_status(e) >= 500)
+            @debug "AWS.jl HTTP inner retry for status >= 500" retry=true reason="status >= 500" status=_http_status(e) exception=e
+            return true
+        end
+        @debug "AWS.jl HTTP inner retry declined" retry=false reason="Exception passed onwards" exception=e
+        return false
+    end


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

if (isa(e, HTTP.StatusError) && _http_status(e) >= 500)

@debug "AWS.jl HTTP inner retry for status >= 500" retry=true reason="status >= 500" status=_http_status(e) exception=e

return true

end

@debug "AWS.jl HTTP inner retry declined" retry=false reason="Exception passed onwards" exception=e

return false

end

mattBrzezinski · 2023-03-20T21:43:15Z

bors try

bors · 2023-03-20T21:43:16Z

try

Already running a review

bors · 2023-03-20T21:45:17Z

try

Build failed:

Julia 1.6 - ubuntu-latest - x64

mattBrzezinski · 2023-03-21T14:29:55Z

bors try

bors · 2023-03-21T14:34:38Z

try

Build failed:

Julia 1 - ubuntu-latest - x64

mattBrzezinski · 2023-03-21T16:31:41Z

Discussed offline, but we'll be revising how to properly introduce debugging here, closing this PR for now.

bors Bot added a commit that referenced this pull request May 11, 2022

Try #549:

921ada6

github-actions Bot reviewed May 11, 2022

View reviewed changes

Comment thread src/utilities/request.jl Outdated

christopher-dG approved these changes May 12, 2022

View reviewed changes

ericphanson mentioned this pull request May 19, 2022

Increase some retries from 4 to 10 #550

Closed

omus reviewed May 19, 2022

View reviewed changes

Comment thread test/issues.jl Outdated

Comment thread test/issues.jl

Base automatically changed from eph/base-retry to master May 25, 2022 15:08

mattBrzezinski approved these changes Mar 20, 2023

View reviewed changes

ericphanson and others added 6 commits March 20, 2023 15:45

wip

5a94611

add more logging

5d69160

wip tests

ba7a65a

add tests

c4741ba

Update src/utilities/request.jl

f960817

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Longer line lengths

f5c8cc7

mattBrzezinski force-pushed the eph/log-retries branch from 952895b to f5c8cc7 Compare March 20, 2023 20:46

Rebase fix

95ccc75

mattBrzezinski approved these changes Mar 20, 2023

View reviewed changes

Rebase fixes x2

d8a00dd

github-actions Bot reviewed Mar 20, 2023

View reviewed changes

bors Bot added a commit that referenced this pull request Mar 20, 2023

Try #549:

fc02ce8

Fix?

5b62567

github-actions Bot reviewed Mar 20, 2023

View reviewed changes

bors Bot added a commit that referenced this pull request Mar 20, 2023

Try #549:

b47f367

one more pass

5b6c716

bors Bot added a commit that referenced this pull request Mar 20, 2023

Try #549:

e307bcb

github-actions Bot reviewed Mar 20, 2023

View reviewed changes

debug statements

fcc394b

bors Bot added a commit that referenced this pull request Mar 21, 2023

Try #549:

48e59f1

mattBrzezinski closed this Mar 21, 2023

	@debug "AWS.jl Downloads inner retry for Downloads.RequestError" retry=true reason="Downloads.RequestError" exception=e
	@debug "AWS.jl Downloads inner retry for Downloads.RequestError" retry =
	true reason = "Downloads.RequestError" exception = e

	@debug "AWS.jl Downloads inner retry declined" retry=false reason="Exception passed onwards" exception=e
	@debug "AWS.jl Downloads inner retry declined" retry = false reason = "Exception passed onwards" exception =
	e

		record.level == Logging.Debug &&
		get(record.kwargs, :retry, nothing) == successful

	record.level == Logging.Debug &&
	get(record.kwargs, :retry, nothing) == successful
	record.level == Logging.Debug && get(record.kwargs, :retry, nothing) == successful

	@debug "AWS.jl declined to retry non-AWSException" retry=false reason="Non-AWSException" exception=e
	@debug "AWS.jl declined to retry non-AWSException" retry = false reason = "Non-AWSException" exception =
	e

Uh oh!

Conversation

ericphanson commented May 11, 2022

Uh oh!

ericphanson commented May 11, 2022

Uh oh!

Uh oh!

bors Bot commented May 11, 2022

try

Uh oh!

christopher-dG left a comment

Choose a reason for hiding this comment

Uh oh!

ericphanson commented May 13, 2022

Uh oh!

Uh oh!

Uh oh!

mattBrzezinski left a comment

Choose a reason for hiding this comment

Uh oh!

bors Bot commented Mar 20, 2023

Uh oh!

mattBrzezinski left a comment

Choose a reason for hiding this comment

Uh oh!

bors Bot commented Mar 20, 2023

Uh oh!

mattBrzezinski commented Mar 20, 2023

Uh oh!

github-actions Bot Mar 20, 2023

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Mar 20, 2023

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Mar 20, 2023

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Mar 20, 2023

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Mar 20, 2023

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Mar 20, 2023

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Mar 20, 2023

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Mar 20, 2023

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Mar 20, 2023

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Mar 20, 2023

Choose a reason for hiding this comment

Uh oh!

bors Bot commented Mar 20, 2023

Uh oh!

mattBrzezinski commented Mar 20, 2023

Uh oh!

bors Bot commented Mar 20, 2023

try

Uh oh!

github-actions Bot Mar 20, 2023

Choose a reason for hiding this comment

Uh oh!

mattBrzezinski commented Mar 20, 2023

Uh oh!

bors Bot commented Mar 20, 2023

try

Uh oh!

mattBrzezinski commented Mar 20, 2023

Uh oh!

github-actions Bot Mar 20, 2023

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Mar 20, 2023