Check and delete corrupted blobs in cache #533

munishchouhan · 2024-06-14T17:51:23Z

This PR will implement the functionality to check the size of uploaded blob to cache, if its not same delete it

munishchouhan · 2024-06-17T14:52:50Z

Tested locally using dev cache, objects deleted from cache
by temporarily changing the condition to delete when sizes are equal:
Logs:

16:44:26.769 [io-executor-thread-1] DEBUG i.s.w.c.RegistryProxyController - Forwarding blobs cache request 'cr.seqera.io/public/nf-jdk@sha256:xxxx'
16:44:26.769 [io-executor-thread-1] DEBUG i.s.w.c.RegistryProxyController - == Blob cache upstream RegistryProxyService$DelegateResponse(statusCode:200, headers:[accept-ranges:[bytes], cache-control:[max-age=31536000], connection:[keep-alive], content-length:[52320215], content-security-policy:[frame-ancestors 'none'], content-type:[application/octet-stream], date:[Mon, 17 Jun 2024 14:44:26 GMT], docker-content-digest:[sha256:xxxx], docker-distribution-api-version:[registry/2.0], etag:["sha256:xxxx"], server:[nginx], set-cookie:[sid=be3c93a3f8aaa90f94de0e9d479783fe; Path=/; HttpOnly], x-frame-options:[DENY], x-request-id:[d46ea2b6-9ea7-4bc0-ab01-93b360196718]], body:null, location:null, redirect:false)
16:44:26.888 [io-executor-thread-3] DEBUG i.s.w.c.RegistryProxyController - Forwarding blobs cache request 'cr.seqera.io/public/nf-jdk@sha256:xxxx'
16:44:26.889 [io-executor-thread-3] DEBUG i.s.w.c.RegistryProxyController - == Blob cache upstream RegistryProxyService$DelegateResponse(statusCode:200, headers:[accept-ranges:[bytes], cache-control:[max-age=31536000], connection:[keep-alive], content-length:[157044297], content-security-policy:[frame-ancestors 'none'], content-type:[application/octet-stream], date:[Mon, 17 Jun 2024 14:44:26 GMT], docker-content-digest:[sha256:xxxx], docker-distribution-api-version:[registry/2.0], etag:["sha256:xxxx"], server:[nginx], set-cookie:[sid=fedf7d774304013df6ea91fcd9c54d4f; Path=/; HttpOnly], x-frame-options:[DENY], x-request-id:[c80ef3ba-9671-4f61-8c5d-a456343c7678]], body:null, location:null, redirect:false)
16:44:26.912 [io-executor-thread-6] DEBUG i.s.w.c.RegistryProxyController - Forwarding blobs cache request 'cr.seqera.io/public/nf-jdk@sha256:xxxx'
16:44:26.912 [io-executor-thread-6] DEBUG i.s.w.c.RegistryProxyController - == Blob cache upstream RegistryProxyService$DelegateResponse(statusCode:200, headers:[accept-ranges:[bytes], cache-control:[max-age=31536000], connection:[keep-alive], content-length:[25435122], content-security-policy:[frame-ancestors 'none'], content-type:[application/octet-stream], date:[Mon, 17 Jun 2024 14:44:26 GMT], docker-content-digest:[sha256:xxxx], docker-distribution-api-version:[registry/2.0], etag:["sha256:xxxx"], server:[nginx], set-cookie:[sid=75d3327d658a3de1c6db3a38ae55982e; Path=/; HttpOnly], x-frame-options:[DENY], x-request-id:[c47581cc-2dda-47cb-a313-d7515d73ea77]], body:null, location:null, redirect:false)
16:44:26.912 [io-executor-thread-5] DEBUG i.s.w.c.RegistryProxyController - Returning blobs from repository: 'cr.seqera.io/public/nf-jdk@sha256:3ab66f79f71f82e926e6e768fb63b6a0e60f98537e3cd185087c4d12debebebd'
16:44:27.085 [io-executor-thread-1] DEBUG i.s.w.s.b.impl.BlobCacheServiceImpl - == Blob cache begin for object 'https://wave-cache-dev-cloudflare.seqera.io/cr.seqera.io/v2/public/nf-jdk/blobs/sha256:xxxx?verify=xxxx'
16:44:27.086 [io-executor-thread-1] DEBUG i.s.wave.core.RegistryProxyService - Credentials for route path=cr.seqera.io/public/nf-jdk@sha256:xxxx; identity=PlatformId(user=null, workspaceId=null, accessToken=null, towerEndpoint=null) => RegistryCredentials[username=munish.chouhan; password=nxa****]
16:44:27.102 [io-executor-thread-1] INFO  i.s.wave.service.k8s.K8sConfigClient - Creating K8s config with path: /Users/munish.chouhan/.kube/config -- context: 'colima'
16:44:27.121 [io-executor-thread-3] DEBUG i.s.w.s.b.impl.BlobCacheServiceImpl - == Blob cache begin for object 'https://wave-cache-dev-cloudflare.seqera.io/cr.seqera.io/v2/public/nf-jdk/blobs/sha256:xxxx?verify=xxxx'
16:44:27.122 [io-executor-thread-3] DEBUG i.s.wave.core.RegistryProxyService - Credentials for route path=cr.seqera.io/public/nf-jdk@sha256:xxxx; identity=PlatformId(user=null, workspaceId=null, accessToken=null, towerEndpoint=null) => RegistryCredentials[username=munish.chouhan; password=nxa****]
16:44:27.125 [io-executor-thread-6] DEBUG i.s.w.s.b.impl.BlobCacheServiceImpl - == Blob cache begin for object 'https://wave-cache-dev-cloudflare.seqera.io/cr.seqera.io/v2/public/nf-jdk/blobs/sha256:xxxx?verify=xxxx'
16:44:27.125 [io-executor-thread-6] DEBUG i.s.wave.core.RegistryProxyService - Credentials for route path=cr.seqera.io/public/nf-jdk@sha256:xxxx; identity=PlatformId(user=null, workspaceId=null, accessToken=null, towerEndpoint=null) => RegistryCredentials[username=munish.chouhan; password=nxa****]
16:44:47.486 [io-executor-thread-6] DEBUG i.s.w.s.b.impl.BlobCacheServiceImpl - == Blob cache completed for object 'cr.seqera.io/v2/public/nf-jdk/blobs/sha256:xxxx'; status=0
16:44:47.872 [io-executor-thread-6] WARN  i.s.w.s.b.impl.BlobCacheServiceImpl - Blob size mismatch for object 'https://wave-cache-dev-cloudflare.seqera.io/cr.seqera.io/v2/public/nf-jdk/blobs/sha256:xxxx?verify=xxxx'
16:44:47.874 [io-executor-thread-6] DEBUG i.s.w.c.RegistryProxyController - == Blob cache response [succeeded=true] BlobCacheInfo(locationUri:https://wave-cache-dev-cloudflare.seqera.io/cr.seqera.io/v2/public/nf-jdk/blobs/sha256:xxxx?verify=xxxx, headers:[Host:986c-85-136-67-71.ngrok-free.app, User-Agent:docker/24.0.9 go/go1.20.13 git-commit/fca702d kernel/6.5.0-15-generic os/linux arch/arm64 UpstreamClient(Docker-Client/26.1.4 \(darwin\)), Accept-Encoding:gzip, X-Forwarded-For:85.136.67.71, X-Forwarded-Host:986c-85-136-67-71.ngrok-free.app, X-Forwarded-Proto:https], contentLength:25435122, contentType:application/octet-stream, cacheControl:max-age=31536000, creationTime:2024-06-17T14:44:26.912883Z, completionTime:2024-06-17T14:44:47.486378Z, exitStatus:0, logs:{"operation":"pipe","success":true,"destination":"s3://wave-cache-dev/cr.seqera.io/v2/public/nf-jdk/blobs/sha256:xxxx","object":{"type":"file"}}
)
16:44:48.179 [io-executor-thread-2] DEBUG i.s.w.s.b.impl.BlobCacheServiceImpl - Deleted object cr.seqera.io/v2/public/nf-jdk/blobs/sha256:xxxx from bucket s3://wave-cache-dev
16:44:48.411 [io-executor-thread-5] INFO  i.s.w.c.RegistryProxyController - > Request [GET] /v2/wt/773b4f78592d/public/nf-jdk/blobs/sha256:xxxx
16:44:48.413 [io-executor-thread-5] DEBUG i.s.wave.core.RegistryProxyService - Credentials for route path=cr.seqera.io/public/nf-jdk@sha256:xxxx; identity=PlatformId(user=null, workspaceId=null, accessToken=null, towerEndpoint=null) => RegistryCredentials[username=munish.chouhan; password=nxa****]
16:44:48.580 [io-executor-thread-5] DEBUG i.s.w.c.RegistryProxyController - Forwarding blobs cache request 'cr.seqera.io/public/nf-jdk@sha256:xxxx'
16:44:48.581 [io-executor-thread-5] DEBUG i.s.w.c.RegistryProxyController - == Blob cache upstream RegistryProxyService$DelegateResponse(statusCode:200, headers:[accept-ranges:[bytes], cache-control:[max-age=31536000], connection:[keep-alive], content-length:[25435122], content-security-policy:[frame-ancestors 'none'], content-type:[application/octet-stream], date:[Mon, 17 Jun 2024 14:44:48 GMT], docker-content-digest:[sha256:xxxx], docker-distribution-api-version:[registry/2.0], etag:["sha256:xxxx"], server:[nginx], set-cookie:[sid=5800f70187d1e009a3461a962815a3d0; Path=/; HttpOnly], x-frame-options:[DENY], x-request-id:[5f91f519-d3c7-4831-a9cf-3a7e5c395966]], body:null, location:null, redirect:false)
16:44:48.597 [io-executor-thread-5] DEBUG i.s.w.c.RegistryProxyController - == Blob cache response [succeeded=true] BlobCacheInfo(locationUri:https://wave-cache-dev-cloudflare.seqera.io/cr.seqera.io/v2/public/nf-jdk/blobs/sha256:xxxx?verify=xxxx, headers:[Host:986c-85-136-67-71.ngrok-free.app, User-Agent:docker/24.0.9 go/go1.20.13 git-commit/fca702d kernel/6.5.0-15-generic os/linux arch/arm64 UpstreamClient(Docker-Client/26.1.4 \(darwin\)), Accept-Encoding:gzip, X-Forwarded-For:85.136.67.71, X-Forwarded-Host:986c-85-136-67-71.ngrok-free.app, X-Forwarded-Proto:https], contentLength:25435122, contentType:application/octet-stream, cacheControl:max-age=31536000, creationTime:2024-06-17T14:44:26.912883Z, completionTime:2024-06-17T14:44:47.486378Z, exitStatus:0, logs:{"operation":"pipe","success":true,"destination":"s3://wave-cache-dev/cr.seqera.io/v2/public/nf-jdk/blobs/sha256:xxxx","object":{"type":"file"}}
)
16:45:07.503 [io-executor-thread-3] DEBUG i.s.w.s.b.impl.BlobCacheServiceImpl - == Blob cache completed for object 'cr.seqera.io/v2/public/nf-jdk/blobs/sha256:xxxx'; status=0
16:45:07.503 [io-executor-thread-1] DEBUG i.s.w.s.b.impl.BlobCacheServiceImpl - == Blob cache completed for object 'cr.seqera.io/v2/public/nf-jdk/blobs/sha256:xxxx'; status=0
16:45:07.647 [io-executor-thread-1] WARN  i.s.w.s.b.impl.BlobCacheServiceImpl - Blob size mismatch for object 'https://wave-cache-dev-cloudflare.seqera.io/cr.seqera.io/v2/public/nf-jdk/blobs/sha256:xxxx?verify=xxxx'
16:45:07.649 [io-executor-thread-1] DEBUG i.s.w.c.RegistryProxyController - == Blob cache response [succeeded=true] BlobCacheInfo(locationUri:https://wave-cache-dev-cloudflare.seqera.io/cr.seqera.io/v2/public/nf-jdk/blobs/sha256:xxxx?verify=xxxx, headers:[Host:986c-85-136-67-71.ngrok-free.app, User-Agent:docker/24.0.9 go/go1.20.13 git-commit/fca702d kernel/6.5.0-15-generic os/linux arch/arm64 UpstreamClient(Docker-Client/26.1.4 \(darwin\)), Accept-Encoding:gzip, X-Forwarded-For:85.136.67.71, X-Forwarded-Host:986c-85-136-67-71.ngrok-free.app, X-Forwarded-Proto:https], contentLength:52320215, contentType:application/octet-stream, cacheControl:max-age=31536000, creationTime:2024-06-17T14:44:26.783101Z, completionTime:2024-06-17T14:45:07.503752Z, exitStatus:0, logs:{"operation":"pipe","success":true,"destination":"s3://wave-cache-dev/cr.seqera.io/v2/public/nf-jdk/blobs/sha256:xxxx","object":{"type":"file"}}
)
16:45:07.733 [io-executor-thread-3] WARN  i.s.w.s.b.impl.BlobCacheServiceImpl - Blob size mismatch for object 'https://wave-cache-dev-cloudflare.seqera.io/cr.seqera.io/v2/public/nf-jdk/blobs/sha256:xxxx?verify=xxxx'
16:45:07.733 [io-executor-thread-3] DEBUG i.s.w.c.RegistryProxyController - == Blob cache response [succeeded=true] BlobCacheInfo(locationUri:https://wave-cache-dev-cloudflare.seqera.io/cr.seqera.io/v2/public/nf-jdk/blobs/sha256:xxxx?verify=xxxx, headers:[Host:986c-85-136-67-71.ngrok-free.app, User-Agent:docker/24.0.9 go/go1.20.13 git-commit/fca702d kernel/6.5.0-15-generic os/linux arch/arm64 UpstreamClient(Docker-Client/26.1.4 \(darwin\)), Accept-Encoding:gzip, X-Forwarded-For:85.136.67.71, X-Forwarded-Host:986c-85-136-67-71.ngrok-free.app, X-Forwarded-Proto:https], contentLength:157044297, contentType:application/octet-stream, cacheControl:max-age=31536000, creationTime:2024-06-17T14:44:26.889502Z, completionTime:2024-06-17T14:45:07.503791Z, exitStatus:0, logs:{"operation":"pipe","success":true,"destination":"s3://wave-cache-dev/cr.seqera.io/v2/public/nf-jdk/blobs/sha256:xxxx","object":{"type":"file"}}
)
16:45:08.047 [io-executor-thread-5] DEBUG i.s.w.s.b.impl.BlobCacheServiceImpl - Deleted object cr.seqera.io/v2/public/nf-jdk/blobs/sha256:xxxx from bucket s3://wave-cache-dev
16:45:08.047 [io-executor-thread-2] DEBUG i.s.w.s.b.impl.BlobCacheServiceImpl - Deleted object cr.seqera.io/v2/public/nf-jdk/blobs/sha256:xxxx from bucket s3://wave-cache-dev
16:45:08.242 [io-executor-thread-5] INFO  i.s.w.c.RegistryProxyController - > Request [GET] /v2/wt/773b4f78592d/public/nf-jdk/blobs/sha256:xxxx
16:45:08.244 [io-executor-thread-5] DEBUG i.s.wave.core.RegistryProxyService - Credentials for route path=cr.seqera.io/public/nf-jdk@sha256:xxxx; identity=PlatformId(user=null, workspaceId=null, accessToken=null, towerEndpoint=null) => RegistryCredentials[username=munish.chouhan; password=nxa****]
16:45:08.288 [io-executor-thread-1] INFO  i.s.w.c.RegistryProxyController - > Request [GET] /v2/wt/773b4f78592d/public/nf-jdk/blobs/sha256:xxxx
16:45:08.289 [io-executor-thread-1] DEBUG i.s.wave.core.RegistryProxyService - Credentials for route path=cr.seqera.io/public/nf-jdk@sha256:xxxx; identity=PlatformId(user=null, workspaceId=null, accessToken=null, towerEndpoint=null) => RegistryCredentials[username=munish.chouhan; password=nxa****]
16:45:08.521 [io-executor-thread-5] DEBUG i.s.w.c.RegistryProxyController - Forwarding blobs cache request 'cr.seqera.io/public/nf-jdk@sha256:xxxx'
16:45:08.522 [io-executor-thread-5] DEBUG i.s.w.c.RegistryProxyController - == Blob cache upstream RegistryProxyService$DelegateResponse(statusCode:200, headers:[accept-ranges:[bytes], cache-control:[max-age=31536000], connection:[keep-alive], content-length:[52320215], content-security-policy:[frame-ancestors 'none'], content-type:[application/octet-stream], date:[Mon, 17 Jun 2024 14:45:08 GMT], docker-content-digest:[sha256:xxxx], docker-distribution-api-version:[registry/2.0], etag:["sha256:xxxx"], server:[nginx], set-cookie:[sid=b6e6e3bd5bef374c30b0138aad1e3f28; Path=/; HttpOnly], x-frame-options:[DENY], x-request-id:[243b05eb-5997-4de1-8b1e-656139355389]], body:null, location:null, redirect:false)
16:45:08.525 [io-executor-thread-5] DEBUG i.s.w.c.RegistryProxyController - == Blob cache response [succeeded=true] BlobCacheInfo(locationUri:https://wave-cache-dev-cloudflare.seqera.io/cr.seqera.io/v2/public/nf-jdk/blobs/sha256:xxxx?verify=xxxx, headers:[Host:986c-85-136-67-71.ngrok-free.app, User-Agent:docker/24.0.9 go/go1.20.13 git-commit/fca702d kernel/6.5.0-15-generic os/linux arch/arm64 UpstreamClient(Docker-Client/26.1.4 \(darwin\)), Accept-Encoding:gzip, X-Forwarded-For:85.136.67.71, X-Forwarded-Host:986c-85-136-67-71.ngrok-free.app, X-Forwarded-Proto:https], contentLength:52320215, contentType:application/octet-stream, cacheControl:max-age=31536000, creationTime:2024-06-17T14:44:26.783101Z, completionTime:2024-06-17T14:45:07.503752Z, exitStatus:0, logs:{"operation":"pipe","success":true,"destination":"s3://wave-cache-dev/cr.seqera.io/v2/public/nf-jdk/blobs/sha256:xxxx","object":{"type":"file"}}
)
16:45:08.565 [io-executor-thread-1] DEBUG i.s.w.c.RegistryProxyController - Forwarding blobs cache request 'cr.seqera.io/public/nf-jdk@sha256:xxxx'
16:45:08.565 [io-executor-thread-1] DEBUG i.s.w.c.RegistryProxyController - == Blob cache upstream RegistryProxyService$DelegateResponse(statusCode:200, headers:[accept-ranges:[bytes], cache-control:[max-age=31536000], connection:[keep-alive], content-length:[157044297], content-security-policy:[frame-ancestors 'none'], content-type:[application/octet-stream], date:[Mon, 17 Jun 2024 14:45:08 GMT], docker-content-digest:[sha256:xxxx], docker-distribution-api-version:[registry/2.0], etag:["sha256:xxxx"], server:[nginx], set-cookie:[sid=e18338f143ce84336933ff707bfc4ba3; Path=/; HttpOnly], x-frame-options:[DENY], x-request-id:[e6699ce3-c4d4-463f-8be2-3b721db0d1f0]], body:null, location:null, redirect:false)
16:45:08.568 [io-executor-thread-1] DEBUG i.s.w.c.RegistryProxyController - == Blob cache response [succeeded=true] BlobCacheInfo(locationUri:https://wave-cache-dev-cloudflare.seqera.io/cr.seqera.io/v2/public/nf-jdk/blobs/sha256:xxxx?verify=xxxx, headers:[Host:986c-85-136-67-71.ngrok-free.app, User-Agent:docker/24.0.9 go/go1.20.13 git-commit/fca702d kernel/6.5.0-15-generic os/linux arch/arm64 UpstreamClient(Docker-Client/26.1.4 \(darwin\)), Accept-Encoding:gzip, X-Forwarded-For:85.136.67.71, X-Forwarded-Host:986c-85-136-67-71.ngrok-free.app, X-Forwarded-Proto:https], contentLength:157044297, contentType:application/octet-stream, cacheControl:max-age=31536000, creationTime:2024-06-17T14:44:26.889502Z, completionTime:2024-06-17T14:45:07.503791Z, exitStatus:0, logs:{"operation":"pipe","success":true,"destination":"s3://wave-cache-dev/cr.seqera.io/v2/public/nf-jdk/blobs/sha256:xxxx","object":{"type":"file"}}

Client:

munish.chouhan@Munishs-MacBook-Pro ~ % wave -i cr.seqera.io/public/nf-jdk:XXX --wave-endpoint http://localhost:9090
986c-85-136-67-71.ngrok-free.app/wt/xxxx/public/nf-jdk:corretto-17-al2023-jemalloc
munish.chouhan@Munishs-MacBook-Pro ~ % docker pull 986c-85-136-67-71.ngrok-free.app/wt/XXXX/public/nf-jdk:XXX§
corretto-17-al2023-jemalloc: Pulling from wt/xxxx/public/nf-jdk
f6175f9c503b: Pulling fs layer
22dd896ed7d2: Pulling fs layer
0ebe931f9599: Downloading
8b2e91afa5e5: Waiting
8b144ddc09ee: Waiting
unknown blob

munishchouhan · 2024-06-17T14:53:43Z

If I pull again , the caching starts again

src/main/groovy/io/seqera/wave/service/blob/BlobCacheService.groovy

src/main/groovy/io/seqera/wave/service/blob/impl/BlobCacheServiceImpl.groovy

Signed-off-by: munishchouhan <[email protected]>

Signed-off-by: Paolo Di Tommaso <[email protected]>

src/main/groovy/io/seqera/wave/service/blob/impl/BlobCacheServiceImpl.groovy

…iceImpl.groovy Co-authored-by: Paolo Di Tommaso <[email protected]>

munishchouhan · 2024-07-03T12:49:27Z

Tested it locally, one issue found, when docker client retries wave find the entry in cache and try to fetch the blob in cloudflare which does not exists

pditommaso · 2024-07-03T12:59:58Z

How could that happen, it should continue to find the BlobCacheInfo in failed status here

wave/src/main/groovy/io/seqera/wave/controller/RegistryProxyController.groovy

Lines 321 to 326 in 00e88cd

    
           final blobCache = blobCacheService .retrieveBlobCache(route, headers, resp.headers) 
        
           log.debug "== Blob cache response [succeeded=${blobCache.succeeded()}] $blobCache" 
        
           if( !blobCache.succeeded() ) { 
        
               final String msg = blobCache.logs ?: "Unable to cache blob ${blobCache.locationUri}" 
        
               return badRequest(msg) 
        
           }

munishchouhan · 2024-07-03T13:17:46Z

How could that happen, it should continue to find the BlobCacheInfo in failed status here

correct, But shouldn't it restart blob transfer in the next retry?

munishchouhan · 2024-07-03T13:19:30Z

blob deletion is working correctly; tested using the dev Cloudflare cache in the local

pditommaso · 2024-07-03T13:27:03Z

ok, but why you say

and try to fetch the blob in cloudflare which does not exists

If find the entry marked as failed should continue to return an error

munishchouhan · 2024-07-03T13:38:16Z

ok, but why you say

and try to fetch the blob in cloudflare which does not exists

If find the entry marked as failed should continue to return an error

I miss understood the logs, because it shows the locationURI of deleted blob, when docker retries

pditommaso · 2024-07-03T14:31:00Z

ok, so the behaviour is correct. What it could be done is to use a shorted TTL when a job is failing. For example here

wave/src/main/groovy/io/seqera/wave/service/blob/impl/BlobCacheServiceImpl.groovy

Line 205 in 00e88cd

: blobConfig.statusDelay.multipliedBy(10)

use instead

       final ttl = result.succeeded()
                    ? blobConfig.statusDuration
                    : blobConfig.statusDelay

Could you try this ☝️ at attached the log/screenshot of the retry (tho I think 5secs it's still too long)

munishchouhan · 2024-07-03T15:00:36Z

Could you try this ☝️ at attached the log/screenshot of the retry (tho I think 5secs it's still too long)

attached retry log, with changed token and URL
bloblcache_retry.log

I will try with shorter ttl

munishchouhan · 2024-07-03T15:03:26Z

correction the above log is without the ttl, i will try with the change and attach the log again

munishchouhan · 2024-07-03T15:11:45Z

after shorting the status delay, blob cache upload begins again with docker retry, here is the log
bloblcache_retry.log

pditommaso · 2024-07-03T17:28:22Z

What ttl have you used and what the Docker retry interval?

munishchouhan · 2024-07-03T17:36:36Z

What ttl have you used and what the Docker retry interval?

ttl - 5s
docker retry interval - first retry 5s, then interval gets incremented by 5 secs in every retry like 10s, 15s, 20s

pditommaso · 2024-07-03T17:46:51Z

it maybe better to introduce setting for the failed upload ttl, having 4 secs as default

Signed-off-by: munishchouhan <[email protected]>

munishchouhan · 2024-07-03T20:46:48Z

it maybe better to introduce setting for the failed upload ttl, having 4 secs as default

Added failureTTL setting, now caching begins with every retry when status is failed

Signed-off-by: munishchouhan <[email protected]>

Signed-off-by: Paolo Di Tommaso <[email protected]>

pditommaso · 2024-07-04T08:22:06Z

Ok, changed to failureDuration for consistency with statusDuration

pditommaso · 2024-07-04T08:31:41Z

src/main/groovy/io/seqera/wave/service/blob/impl/BlobCacheServiceImpl.groovy

+                            .bucket(blobConfig.storageBucket.replaceFirst("s3://",""))
+                            .key(key)


This is is to tight to s3, we may want to use other storage in the future. Let's use BucketTokenizer

Same for delete method below

src/main/groovy/io/seqera/wave/service/blob/impl/BlobCacheServiceImpl.groovy

Signed-off-by: Paolo Di Tommaso <[email protected]>

pditommaso · 2024-07-04T14:05:54Z

Rationalised a bit in order to:

Always use the method blobStorePath to determine the S3 object location
delegate to BucketTokenizer the splitting in bucket and key components

Signed-off-by: Paolo Di Tommaso <[email protected]>

Initial commit

ddf6df5

munishchouhan linked an issue Jun 14, 2024 that may be closed by this pull request

Detection and deletion of corrupted uploads #476

Closed

munishchouhan self-assigned this Jun 14, 2024

munishchouhan marked this pull request as draft June 14, 2024 17:51

munishchouhan and others added 9 commits June 14, 2024 19:52

minor change [ci skip]

38e058d

minor change [ci skip]

18453b5

minor change [ci skip]

739aa0e

refactored

6c943d2

condition corrected

636ae4e

Added unit tests

003acf3

fixed tests

f63ab0c

Merge branch 'master' into 476-delete-potential-corrupted-uploads

6e1b972

refactored

11ac455

munishchouhan marked this pull request as ready for review June 17, 2024 14:55

munishchouhan requested a review from pditommaso June 17, 2024 14:55

munishchouhan and others added 5 commits June 17, 2024 16:59

minor change[ci skip]

a6e7743

Merge branch 'master' into 476-delete-potential-corrupted-uploads

f35a55d

Merge branch 'master' into 476-delete-potential-corrupted-uploads

15cf5ff

Merge branch 'master' into 476-delete-potential-corrupted-uploads

9270319

Merge branch 'master' into 476-delete-potential-corrupted-uploads

74d6ffe

pditommaso reviewed Jul 2, 2024

View reviewed changes

src/main/groovy/io/seqera/wave/service/blob/BlobCacheService.groovy Outdated Show resolved Hide resolved

pditommaso reviewed Jul 2, 2024

View reviewed changes

src/main/groovy/io/seqera/wave/service/blob/impl/BlobCacheServiceImpl.groovy Outdated Show resolved Hide resolved

munishchouhan and others added 6 commits July 2, 2024 17:32

refactored

6b9275c

Signed-off-by: munishchouhan <[email protected]>

changes as per review

d62b2ce

Signed-off-by: munishchouhan <[email protected]>

unit test added

387a642

Signed-off-by: munishchouhan <[email protected]>

formatted [ci skip]

fbdf291

Signed-off-by: munishchouhan <[email protected]>

Minor change

e51a97e

Signed-off-by: Paolo Di Tommaso <[email protected]>

Minor refactor

9d10653

Signed-off-by: Paolo Di Tommaso <[email protected]>

munishchouhan added 2 commits July 2, 2024 22:26

Merge branch 'master' into 476-delete-potential-corrupted-uploads

dcab529

Merge branch 'master' into 476-delete-potential-corrupted-uploads

a68d1f5

pditommaso reviewed Jul 3, 2024

View reviewed changes

src/main/groovy/io/seqera/wave/service/blob/impl/BlobCacheServiceImpl.groovy Outdated Show resolved Hide resolved

Update src/main/groovy/io/seqera/wave/service/blob/impl/BlobCacheServ…

f3a2994

…iceImpl.groovy Co-authored-by: Paolo Di Tommaso <[email protected]>

added ttl with default 4s

f49c442

Signed-off-by: munishchouhan <[email protected]>

munishchouhan and others added 2 commits July 3, 2024 23:02

revert unwanted changes

2d2cc7d

Signed-off-by: munishchouhan <[email protected]>

Refactor

67cbd9d

Signed-off-by: Paolo Di Tommaso <[email protected]>

pditommaso reviewed Jul 4, 2024

View reviewed changes

src/main/groovy/io/seqera/wave/service/blob/impl/BlobCacheServiceImpl.groovy Outdated Show resolved Hide resolved

Refactored

05f7018

Signed-off-by: Paolo Di Tommaso <[email protected]>

pditommaso force-pushed the 476-delete-potential-corrupted-uploads branch from 03388ad to 05f7018 Compare July 4, 2024 14:02

Just cases [ci skip]

c62952f

Signed-off-by: Paolo Di Tommaso <[email protected]>

pditommaso merged commit b0c775a into master Jul 4, 2024
1 check passed

pditommaso deleted the 476-delete-potential-corrupted-uploads branch July 4, 2024 14:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Check and delete corrupted blobs in cache #533

Check and delete corrupted blobs in cache #533

munishchouhan commented Jun 14, 2024

munishchouhan commented Jun 17, 2024

munishchouhan commented Jun 17, 2024

munishchouhan commented Jul 3, 2024

pditommaso commented Jul 3, 2024

munishchouhan commented Jul 3, 2024 •

edited

Loading

munishchouhan commented Jul 3, 2024

pditommaso commented Jul 3, 2024

munishchouhan commented Jul 3, 2024

pditommaso commented Jul 3, 2024

munishchouhan commented Jul 3, 2024

munishchouhan commented Jul 3, 2024

munishchouhan commented Jul 3, 2024

pditommaso commented Jul 3, 2024

munishchouhan commented Jul 3, 2024

pditommaso commented Jul 3, 2024

munishchouhan commented Jul 3, 2024

pditommaso commented Jul 4, 2024

pditommaso Jul 4, 2024

munishchouhan Jul 4, 2024

pditommaso commented Jul 4, 2024

		.bucket(blobConfig.storageBucket.replaceFirst("s3://",""))
		.key(key)

Check and delete corrupted blobs in cache #533

Check and delete corrupted blobs in cache #533

Conversation

munishchouhan commented Jun 14, 2024

munishchouhan commented Jun 17, 2024

munishchouhan commented Jun 17, 2024

munishchouhan commented Jul 3, 2024

pditommaso commented Jul 3, 2024

munishchouhan commented Jul 3, 2024 • edited Loading

munishchouhan commented Jul 3, 2024

pditommaso commented Jul 3, 2024

munishchouhan commented Jul 3, 2024

pditommaso commented Jul 3, 2024

munishchouhan commented Jul 3, 2024

munishchouhan commented Jul 3, 2024

munishchouhan commented Jul 3, 2024

pditommaso commented Jul 3, 2024

munishchouhan commented Jul 3, 2024

pditommaso commented Jul 3, 2024

munishchouhan commented Jul 3, 2024

pditommaso commented Jul 4, 2024

pditommaso Jul 4, 2024

Choose a reason for hiding this comment

munishchouhan Jul 4, 2024

Choose a reason for hiding this comment

pditommaso commented Jul 4, 2024

munishchouhan commented Jul 3, 2024 •

edited

Loading