-
Notifications
You must be signed in to change notification settings - Fork 16
/
s3.aws.txt
5066 lines (4129 loc) · 280 KB
/
s3.aws.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
S3
VERSION ==> #2023-11-28
SVERSION #v20180820
S3FS-FUSE ==> #See its doc
SUMMARY ==> #API: virtual hosting, s3|s3-control, high-level CLI
#Buckets: region, accelerate, tags
#Download: range, torrent, request payment
#Upload: simple, streams, browser, multipart, 100-continue
#Copy: contents, headers, batch, sync
#Prefix: list, delimiter, delete
#Attributes: length, checksum, encoding, MIME, content-disposition, [un]conditional caching
#Metadata: custom, tags, language
#Versioning: versionId, delete marker, MFA delete
#Storage class: STANDARD_IA, ONEZONE_IA, GLACIER_IR, intelligent tiering, Express
#Archive restore: GLACIER|DEEP_ARCHIVE, tier, duration, select
#Lifecycle: filter, [non]current, duration, storage class, expiration, delete marker, abort multipart
#Lock: duration|date, bypass, legal hold
#Ownership: bucket|object, BucketOwnerEnforced|ObjectWriter
#Permissions: bucket policy, immutable policy, ACL OBJ|canned, access grants, public access block, CORS
#Pre-signed URL: duration, ignore body
#Static website: error document, index, redirects
#Encryption: SSE-S3, [D]SSE-KMS, SSE-C
#Select: range, CSV|ndjson|Parquet, compression, SQL
#Replication: source|destination, bidirectional, filter, existing OBJECTs, delete markers, RTC, batch
#Inventory: filter, daily|weekly, CSV|ORC|Parquet
#Requests logging
#Metrics: size, I/O, requests
#Storage lens: Cloudwatch|dashboard|file, daily, filter, account|bucket|prefix grouping
#Storage class analysis: daily, CSV
#Notifications: SNS|SQS|Lambda|EventBridge, filter, test event
#Access point: own policy|public_block|vpc, fewer actions
#Multi-region access point: regions, fewer actions, failovers
#Lambda: object|supporting access point, GetObject|HeadObject|ListObjectsV2(), payload, range, batch, builtins (decompress, PII)
#Batch: operations, manifest|inventory, priority, status, progress, report
/=+===============================+=\
/ : : \
)==: API :==(
\ :_______________________________: /
\=+===============================+=/
NAME ==> #Simple Storage Service
s3.amazonaws.com #SERVICE_DOMAIN for BUCKET-specific ACTIONs
BUCKET.s3.amazonaws.com #"Virtual hosting". Same but using BUCKET as subdomain instead of /BUCKET as path.
#Preferred: /BUCKET path is deprecated
#Allows BUCKET-specific:
# - CNAME
# - top-level files: robots.txt, favicon.ico, etc.
# - same origin, CORS
#Requires:
# - specifying REGION in subdomain
# - 301 redirection + x-amz-bucket-region: REGION2 [S] if not in right REGION
# - with JavaScript client, requires COPTS.followRegionRedirects true
# - CNAME == 'BUCKET' exactly, e.g. BUCKET called 'www.example.com'
COPTS.forcePathStyle #BOOL (def: false). Use /BUCKET instead of virtual hosting
S3_CONF.addressing_style #'virtual' (def) or 'path'
ACCOUNT_ID.s3-control #SERVICE_DOMAIN for ACTIONs that not BUCKET-specific
.amazonaws.com/SVERSION #Some ACTIONs have two variants s3[-control]: BUCKET-specific, and not
x-amz-account-id: ACCOUNT_ID [C] #Alternative to specifying ACCOUNT_ID in subdomain
s3express-control
.REGION.amazonaws.com/BUCKET #"Regional endpoint". SERVICE_DOMAIN for Express, with *Bucket*()
BUCKET.s3express-AZ_MID
.REGION.amazonaws.com #"Zonal endpoint". SERVICE_DOMAIN for Express, with *Object|Part*(), CreateSession() and HeadBucket()
FORMAT ==> #XML REST
#Uses some PARAMS|RESP headers
ERRORS ==> #OBJ_ARR: Code, Message, Resource, RequestId
REQ_ID #x-amz-request-id [S] + x-amz-id-2 [S]
PAGINATION ==> #SDK lacks paginateList*() for: ListBucket*(), ListMultipartUploads(), ListObjectVersions()
#CLI lacks --page-size and --starting-token (but not automatic pagination) for ListBucket*() and s3-control
PARAMS.nextToken
RESP.NextToken #'TOKEN'
PARAMS.continuation-token
RESP.[Next]ContinuationToken #'TOKEN', alternative syntax for some ACTIONs
PARAMS.PROP-marker #'TOKEN', alternative syntax for some ACTIONs
RESP.[Next]PROPMarker #Sometimes has multiple PROPs
RESP.IsTruncated #BOOL. Not with RESP.NextToken
PARAMS.maxResults #NUM (def: 100 with INVENTORY|ANALYTICS|MAIN_METRICS, 1000 otherwise)
PARAMS.max-PROPs
RESP.MaxPROPs #NUM (def: 1000), alternative syntax for some ACTIONs
RESP.PROPCount #NUM. Only set with some ACTIONs, together with RESP.MaxPROPs
AmazonS3[Outposts]FullAccess #AWS managed POLICY. Grants all 's3[-object-lambda][-outposts]:*' PACTIONs
AmazonS3[Outposts]ReadOnlyAccess #Same but readonly
/=+===============================+=\
/ : : \
)==: CLIENT :==(
\ :_______________________________: /
\=+===============================+=/
@aws-sdk/client-s3 #SDK CLIENT for s3 SERVICE
@aws-sdk/client-s3-control #SDK CLIENT for s3-control SERVICE
@aws-sdk/
middleware-sdk-s3[-control] #MFUNC used by S3[-control] CLIENT
@aws-sdk/
middleware-bucket-endpoint #Used internally by @aws-sdk/middleware-sdk-s3-control
aws s3api ... #CLI for s3 SERVICE
aws s3control ... #CLI for s3-control SERVICE
CONF.s3 #S3_CONF. Only used by CLI.
/=+===============================+=\
/ : : \
)==: HIGH-LEVEL CLI :==(
\ :_______________________________: /
\=+===============================+=/
aws s3 ... #High-level S3 CLI on top of aws s3api|s3control
--OPT #Same as other CLI commands
--debug #Show HTTP request|response
DIR|FILE|PATH #Local file path, using OS-specific syntax
S3BUCKET #s3://BUCKET or s3://ACCESSPOINT_ARN
S3DIR #S3BUCKET[/PREFIX/]
S3FILE #S3DIR/OBJECT
S3PATH #S3DIR|S3FILE
HS3_CONF #S3_CONF, but only for `aws s3`
HS3_CONF.max_bandwidth #NUM[UNIT] (def: no limit) (def UNIT: 'B/s'). Max bandwidth for GetObject|PutObject()
HS3_CONF.max_concurrent_requests #Max NUM (def: 10) of concurrent requests
HS3_CONF.max_queue_size #Max NUM (def: 1e3) of ACTIONs waiting to be run (due to max_concurrent_requests)
s3p ##Alternative to `aws s3` that is up to ~20x faster by running paginated ACTIONs
##with parallel calls instead of one page at a time
##CLI/programmatic
##Not documented yet
CRT ==> #SDKs for Java|Python|C++ is much faster thanks to logic that parallelizes:
# - reads: using Range [C]
# - writes: using MUPLOADs
/=+===============================+=\
/ : : \
)==: PRICING :==(
\ :_______________________________: /
\=+===============================+=/
STORAGE SPACE ==> #1$/43GB
#Free (first year): 5GB
#4% cheaper after 50TB, 9% cheaper after 500TB
#Express One Zone: 7x more expensive
#Cheaper STORAGE_CLASS:
# - STANDARD_IA: 2x
# - ONEZONE_IA: 2.3x
# - GLACIER_IR: 5.7x
# - GLACIER: 6.4x
# - DEEP_ARCHIVE: 23x
METADATA ==> #8KB for each OBJECT
# - always priced as STORAGE_CLASS STANDARD
#Some additional ones for STANDARD_IA|ONEZONE_IA|GLACIER_IR if file <128KB
#32KB more if GLACIER[_IR]|DEEP_ARCHIVE, with that STORAGE_CLASS
# - i.e. better for multiple files tar'd, or single big files
REQUEST COUNT ==> #1$/2e6 requests
#List|write: 10x more expensive
#Delete: free
#Free (first year): 2e4 requests
#Express One Zone: 2x cheaper
# - igress 1$/125GB, egress 1$/650GB
# - free: first 512KB igress|egress per request
#More expensive STORAGE_CLASS:
# - STANDARD_IA|ONEZONE_IA: 2x for list|write, 2.5x for read
# - GLACIER_IR: 4x for list|write, 25x for read
# - GLACIER: 6x for list|write, 1x for read
# - DEEP_ARCHIVE: 10x for list|write, 1x for read
# - i.e. better for read-only files
READ|WRITE|RESTORE ==> #Express One Zone:
# - write 1$/125GB
# - read 1$/650GB
# - free: first 512KB read|write per request
#STORAGE_CLASS, on read|restore contents:
# - STANDARD: free
# - STANDARD_IA|ONEZONE_IA: 1$/100GB
# - GLACIER_IR: 1$/33GB
# - GLACIER:
# - Bulk: free
# - Standard: 1$/2e4 requests, 1$/100GB
# - Expedited: 1$/1e2 requests, 1$/33GB
# - DEEP_ARCHIVE:
# - Bulk: 1$/4e4 requests, 1$/400GB
# - Standard: 1$/1e4 requests, 1$/50GB
#I.e. better for rarely accessed files, e.g. backups
SELECT ==> #Input: 1$/500GB
# - GLACIER + Expedited: 10x more expensive
# - GLACIER + Standard: 4x more expensive
# - GLACIER + Bulk: 2x cheaper
# - DEEP_ARCHIVE: free
#Output: 1$/1400GB
# - GLACIER_IR, GLACIER + Expedited: 43x more expensive
# - STANDARD_IA|ONEZONE_IA, GLACIER + Standard: 14x more expensive
# - GLACIER + Bulk: 4x more expensive
# - DEEP_ARCHIVE: free
IN|OUT TRAFFIC ==> #Internet -> S3: free
#S3 -> Internet: 1$/11GB
# - free: 100GB
# - after 10TB: 5% cheaper
# - after 50TB: 20% cheaper
# - after 150TB: 50% cheaper
#S3 -> same region: free
#S3 -> CloudFront: free
#S3 -> another region: 4x cheaper
#Max 500 TB/month (flexible)
MULTI-REGION ACCESS POINT ==> #Always: 1$/300GB
#S3 <-> AWS: in|out traffic charge
#S3 <-> internet:
# - same continent:
# - internet -> S3: 1$/400GB
# - S3 -> internet: 1$/200GB
# - ~4x more expensive in Asia, ~10x in South America
# - between continents: 1$/20GB
ACCELERATE ==> #1$/25GB
#For both in|out
#Regardless of whether from|to Internet|AWS
#2x more expensive for Internet -> S3 if client's REGION is not US|Europe|Japan
METRICS ==> #MAIN_METRICS: free
#ADVANCED_METRICS: 1$/5e6 OBJECTs per SLENS
# - 20% cheaper if >2.5e10 OBJECTs
# - 40% cheaper if >1e11 OBJECTs
REPLICATION ==> #Free, except normal traffic|requests|storage cost of copying
#Each write creates up to 5 reads requests
#RTC: 1$/67GB
BATCH ==> #1$/4 JOBs
#1$ per 1e6 tasks (JOB_OP + OBJECT)
#1$ per 7e7 OBJECTs, when using JOB.ManifestGenerator
OTHERS ==> #OBJECT TAGs: 1$/1e6 TAGs
#CAPACITY: 100$/month
#INTELLIGENT_TIERING: 1$/4e5 OBJECTs (with >128KB)
#DSSE-KMS: 1$/333GB
#INVENTORY: 1$/4e8 OBJECTs
#LOGGING: free
#ANALYTICS: 1$/1e7 OBJECTs
#LACCESSPOINT: 1$/200GB
/=+===============================+=\
/ : : \
)==: BUCKET MAIN :==(
\ :_______________________________: /
\=+===============================+=/
PUT /BUCKET #Req: BUCKET
CreateBucket() # - no Name, CreationDate
# - x-amz-acl [C], x-amz-grant-PERMISSION [C], x-amz-object-ownership [C]
# x-amz-bucket-object-lock-enabled [C]
#Res: Location [S]
HEAD /BUCKET #Req: empty
HeadBucket() #Res: x-amz-bucket-region [S], x-amz-access-point-alias [S]
#No own PACTION: use 's3:ListBucket' instead
GET / #Req: empty
ListBuckets() #Res: BUCKETS_LIST
# - no BUCKET.LocationConstraint|Bucket|Location
#No pagination
#PACTION named 's3:ListAllMyBuckets' instead
DELETE /BUCKET #Req: empty
DeleteBucket() #Res: empty
#BUCKET must be empty first
## - with Pulumi, can use RPROPS.forceDestroy true
BUCKETS_LIST.Buckets #BUCKET_ARR
BUCKET #Like a root folder
#Max 100 per REGION (soft)
#WAIT (HeadBucket()) Bucket[Not]Exists: when it exists
BUCKET_ARN #arn:aws:s3:::BUCKET
Location: /BUCKET [S]
BUCKET.Name #'BUCKET'
#3-63 chars, [[:alnum:]-.]
#Must be unique across AWS
BUCKET.CreationDate #'DATE'
aws s3 mb S3BUCKET #CreateBucket()
--region #'REGION'
aws s3 ls #ListBuckets()
aws s3 rb S3BUCKET #DeleteBucket()
--force #If OBJECTs exist, delete them first
#Fail if OBJECTs are versioned
AwsCommunity::S3:: ##With CloudFormation, deletes BUCKET if empty
DeleteBucketContents ##RESPROP: BucketName 'BUCKET'
XW.defaultBucket ##With Pulumi, like new CW.s3.Bucket(...) except:
('BUCKET', OBJ, NRPROPS, NROPTS)## - can set OBJ.existing.arn 'BUCKET_ARN' or name 'BUCKET'
## to retrieve already existing BUCKET instead.
# - OBJ.skip true to be a noop, with no outputs
DEFAULT_BUCKET.bucket ##BUCKET. Undefined if OBJ.existing.arn
DEFAULT_BUCKET.bucketId.arn ##}'BUCKET_ARN'{
DEFAULT_BUCKET.bucketId.name ##}'BUCKET'{
XW.requiredBucket(...) ##Same but cannot use `skip`
AWS::S3[Express]::Bucket #RESPROPs: BucketName
#RESATTRs: Arn
AWS::S3::Bucket #RESATTRs:
# - DomainName 'BUCKET.s3.amazonaws.com'
# - RegionalDomainName 'BUCKET.s3.REGION.amazonaws.com'
# - DualStackDomainName 'BUCKET.s3.dualstack.REGION.amazonaws.com'
#Default RESOURCE.DeletionPolicy is 'Delete' (like other RESOURCEs)
# - but DeleteBucket() fails if there any OBJECTs
/=+===============================+=\
/ : : \
)==: BUCKET TAGS :==(
\ :_______________________________: /
\=+===============================+=/
PUT /BUCKET?tagging #Req: TagSet TAG_PAIRS, Content-MD5 [C], x-amz-checksum-ALGO [C]
PutBucketTagging() #Res: empty
GET /BUCKET?tagging #Req: empty
GetBucketTagging() #Res: TagSet TAG_PAIRS
DELETE /BUCKET?tagging #Req: empty
DeleteBucketTagging() #Res: empty
#No own PACTION: use 's3:PutBucketTagging' instead
AWS::S3::Bucket #Includes RESPROPs: Tags TAG_PAIRS
/=+===============================+=\
/ : : \
)==: OBJECT READ :==(
\ :_______________________________: /
\=+===============================+=/
GET /BUCKET/OBJECT #Req:
GetObject() # - READ_OBJECT
# - READ_OBJECT.response-*: must be authenticated
# - Range [C], If-[None-]Match [C], If-[Un]modified-Since [C],
# x-amz-server-side-encryption-customer-* [C], x-amz-checksum-mode [C]
#Res:
# - RAW_OBJECT
# - Content-Range [S], Accept-Ranges [S], Content-Type [S], Content-Encoding [S],
# Content-Language [S], ETag [S], Last-Modified [S], Expires [S], Cache-Control [S],
# Content-Disposition [S], Content-Length [S], x-amz-meta-* [S], x-amz-version-id [S], x-amz-delete-marker [S],
# x-amz-restore [S], x-amz-restore-request-date [S], x-amz-archive-status [S],
# x-amz-expiration [S], x-amz-missing-meta [S], x-amz-website-redirect-location [S],
# x-amz-tagging-count [S], x-amz-server-side-encryption-* [S], x-amz-checksum-ALGO [S],
# x-amz-storage-class [S], x-amz-replication-status [S], x-amz-mp-parts-count [S]
# x-amz-object-lock* [S]
HEAD /BUCKET/OBJECT #Req: like GetObject()
HeadObject() # - no READ_OBJECT.response-*
#Res: like GetObject()
#No own PACTION: use 's3:GetObject' instead
GET /BUCKET/OBJECT?attributes #Req:
GetObjectAttributes() # - like GetObject()
# - only versionId, x-amz-server-side-encryption-customer-* [C]
# - x-amz-object-attributes [C]
# - same pagination as ListParts(), but as x-amz-max-parts [C], x-amz-part-number-marker [C]
#Res: OBJECT
# - only ETag, StorageClass, Size -> ObjectSize, ChecksumALGO -> Checksum.ChecksumALGO
# - only Last-Modified [S], x-amz-version-id [S], x-amz-delete-marker [S]
# - ObjectParts MUPLOAD: only Parts and pagination
#Requires PACTIONs 's3:GetObjectAttributes' + 's3:GetObject'
MISSING OBJECT ==> #For GetObject|HeadObject|GetObjectAttributes: 404 becomes 403 if no PACTION 's3:ListBucket'
OBJECT #Like a file
#WAIT (HeadObject()) Object[Not]Exists: when it exists
# - not needed after a write, since write waits for it already
# - i.e. OBJECTs use "read-after-write consistency", not eventual consistency
#Unlimited amount
#All OBJECT-related ACTIONs are CloudTrail DATA_ACTIONs. If current ACCOUNT is:
# - OBJECT_OWNER, BUCKET can be in a different ACCOUNT
# - not OBJECT_OWNER, can still log if BUCKET is in current ACCOUNT
OBJECT_ARN #arn:aws:s3:::BUCKET/OBJECT
OBJECT.Bucket #'BUCKET'
OBJECT.Key #'OBJECT', its name
#Is a file path, allows virtual "subfolders" ("prefixes")
# - delimiter can be anything, but is usually /
# - max 5500 reads/sec, 3500 writes/sec per prefix
# - when increases, underlying infrastructure scales up, which might result in some 503s
#Due to internal storage details:
# - more performant when start with a random value
# - as opposed to serial integer, timestamp, etc.
#Max 1KB, [[:alnum:]-_.*'()!] (should URL encode others)
OBJECT.Location #'.../BUCKET/OBJECT' URL
Range: bytes=NUM-NUM2 [C]
Content-Range:
bytes NUM-NUM2/NUM3 [S]
Accept-Ranges: bytes [S] #See HTTP doc
READ_OBJECT.partNumber #NUM, 1-based index, max 1e4, size 5MiB
x-amz-mp-parts-count: NUM3 [S] #Total NUM3 of parts
x-amz-object-attributes:
VAR,... [C] #Only return OBJECT.VAR
RAW_OBJECT #OBJECT, as is
#Max 5GB
#In SDK, returned as RESP.Body RESP_STREAM
##With Pulumi:
## - can be set either with:
## - RPROPS.content[Base64] STR
## - RPROPS.source ASSET
## - RPROPS.sourceHash 'TRIGGER'
PARAMS.Body #As request, RAW_OBJECT can be STR or UINT8ARR|BUFFER|BLOB
#It can also be RSTREAM|ISTREAM:
# - uses Transfer-Encoding: chunked [C]
# and x-amz-content-sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER [C]
# - requires setting PARAMS.checksumAlgorithm
--body PATH #Same for CLI
/=+===============================+=\
/ : : \
)==: TORRENT :==(
\ :_______________________________: /
\=+===============================+=/
GET /BUCKET/OBJECT?torrent #Req: empty
GetObjectTorrent() #Res: TORRENT
TORRENT #OBJECT as Content-Type: application/x-bittorrent [S]
#Goal: readers upload to each other:
# - less server egress
# - faster download speed
#Max OBJECT content 5GB instead of 5TB
#In SDK, returned as RESP.Body RESP_STREAM
#In CLI, must specify output file
Content-Disposition: attachment;
filename=OBJECT.torrent; [S] #
/=+===============================+=\
/ : : \
)==: OBJECT LIST :==(
\ :_______________________________: /
\=+===============================+=/
GET /BUCKET #Req: OBJECTS
ListObjectsV2() # - CamelCase -> dash-case
# - only EncodingType, StartAfter, Prefix, Delimiter, FetchOwner
# - x-amz-optional-object-attributes [C]
#Res: OBJECTS
# - no FetchOwner
#Paginates OBJECTS.Contents with ContinuationToken, MaxKeys, KeyCount
#PACTION named 's3:ListBucket' instead
OBJECTS.Name #'BUCKET'
OBJECTS.EncodingType #'url'. Percent encode the OBJECTS.* with value 'OBJECT'
#I.e. for Key|StartAfter|Prefix|Delimiter
#Needed if contains characters not allowed by XML 1.0
OBJECTS.StartAfter #STR. Only include if 'OBJECT' is STR, or lexicographically after
OBJECTS.Prefix #STR. Only include if 'OBJECT' start with STR
OBJECTS.Delimiter #STR separating 'OBJECT' "subfolders". This is usually '/'
#When specified, distinguishes between:
# - pseudo-regular files:
# - 'OBJECT' starts with Prefix, but does not include Delimiter
# - returned in OBJECTS.Contents
# - pseudo-directories:
# - 'OBJECT' starts with Prefix, but includes Delimiter
# - returned in OBJECTS.CommonPrefixes
OBJECTS.CommonPrefixes #Unique COMMON_PREFIX_ARR
#Max 1e3
COMMON_PREFIX.Prefix #Prefix + 'OBJECT' until next Delimiter (included)
COND_KEY s3:prefix #OBJECTS.Prefix. Only for PACTIONs 's3:ListBucket[Versions]'
COND_KEY s3:delimiter #OBJECTS.Delimiter. Only for PACTIONs 's3:ListBucket[Versions]'
COND_KEY s3:max-keys #OBJECTS.MaxKeys. Only for PACTIONs 's3:ListBucket[Versions]'
OBJECTS.Contents #OBJECT_ARR
# - no Bucket, Location, ChecksumALGO, VersionId, IsLatest
#Sorted by 'OBJECT'
x-amz-optional-object-attributes:
VAR,... [C] #Return OBJECT.VAR (otherwise not) among: 'RestoreStatus'
aws s3 ls S3DIR #ListObjectsV2()
--recursive #Include OBJECTs in subfolders
--human-readable #Print size as KiB|MiB|etc.
--summarize #Print NUM of OBJECTs and total size
--request-payer #x-amz-request-payer: requester [C]
/=+===============================+=\
/ : : \
)==: SIMPLE UPLOAD :==(
\ :_______________________________: /
\=+===============================+=/
PUT /BUCKET/OBJECT #Req:
PutObject() # - RAW_OBJECT (including PARAMS.Body STREAM)
# - Expect [C], Content-Type [C], Content-Encoding [C], Content-Language [C], Expires [C],
# Cache-Control [C], Content-Disposition [C], x-amz-meta-* [C], x-amz-storage-class [C],
# x-amz-acl [C], x-amz-grant-PERMISSION [C], x-amz-website-redirect-location [C],
# x-amz-tagging [C], x-amz-server-side-encryption-* [C], Content-MD5 [C], x-amz-checksum-ALGO [C],
# x-amz-object-lock* [C]
# - max 8KB request headers
#Res:
# - ETag [S], x-amz-version-id [S], x-amz-expiration [S], x-amz-server-side-encryption-* [S],
# x-amz-checksum-ALGO [S]
Expect: 100-continue [C] #See HTTP doc
@aws-sdk/
middleware-expect-continue #Always use Expect: 100-continue [C], used by S3 client
NEVENT s3:ObjectCreated:Put #On PutObject() success
/=+===============================+=\
/ : : \
)==: BROWSER UPLOAD MAIN :==(
\ :_______________________________: /
\=+===============================+=/
POST /BUCKET #Req: BUPLOAD
PostObject() # - BUPLOAD.HEADER: any PutObject() request header
# - strip x-amz-* prefix for: x-amz-acl [C], x-amz-tagging [C]
#Res:
# - OBJECT
# - only Bucket, Key, Location
# - same headers as PutObject()
# - success_action_redirect [S], Redirect [S]
#Not available in SDK|CLI, but available with AWS Amplify
#No own PACTION: use 's3:PutObject' instead
BUPLOAD #Like PutObject() but meant to be performed by a <form> submit (i.e. client-side)
#Must use Content-Type: multipart/form-data [C]
#Max 20KB
BUPLOAD.key #'OBJECT'
BUPLOAD.file #RAW_OBJECT (excluding PARAMS.body STREAM)
#Must be last field
${filename} #Anywhere in BUPLOAD, replaced by FILENAME from BUPLOAD.file, i.e.:
# - Content-Disposition: form-data; name="file"; filename="FILENAME"
# - filename used in <form>
BUPLOAD.x-amz-signature|algorithm #Authenticate request (see AWS signature version 4 doc)
|credential|security-token|date #If not set, anonymous request
BUPLOAD.success_action_redirect
success_action_redirect: URL [S]
Redirect: URL [S] #URL to redirect to when response code is success_action_status NUM
BUPLOAD.success_action_status #200, 201 or 204 (def)
NEVENT s3:ObjectCreated:Post #On PostObject() success
/=+===============================+=\
/ : : \
)==: BROWSER UPLOAD POLICY :==(
\ :_______________________________: /
\=+===============================+=/
BUPLOAD.policy #BUPLOAD_POLICY, as Base64 JSON
#Required with authenticated requests
#Enforces values|constraints for BUPLOAD.*
# - so that client-side can upload, but not change intended request
BUPLOAD_POLICY.expiration #'DATE'. After this, BUPLOAD_POLICY denies all.
BUPLOAD_POLICY.conditions #ARR of either:
# - ['OP', 'VAL1', 'VAL2']
# - { PROP: 'VAL2' }: same as ['eq', '$PROP', 'VAL2']
#Must include at least one PROP|$PROP for each BUPLOAD.PROP
# - except BUPLOAD: file, x-amz-signature, policy
# - also except BUPLOAD.x-ignore-PROP: behaves like BUPLOAD.PROP otherwise
PROP #Value of BUPLOAD.PROP. Can also be:
# - bucket 'BUCKET': as specified in request URI
$PROP #Can be included anywhere, to use its value
OP 'eq' #VAL1 == VAL2
OP 'starts-with' #VAL1 starts with VAL2
#Can use 'VAL1,...' with Content-Type [C]
#VAL2 can be '' to mean "always allow"
#Only with PROP: acl, Cache-Control, Content-*, Expires, key, success_action_redirect, x-amz-meta-*
OP 'content-length-range' #VAL1 <= Content-Length [S] <= VAL2
/=+===============================+=\
/ : : \
)==: BROWSER UPLOAD SIGNATURE :==(
\ :_______________________________: /
\=+===============================+=/
x-amz-signature: STR [C] #Unlike other ACTIONs, request body used in signature is BUPLOAD_POLICY
#I.e. when authenticating with a pre-signed URL, client-side cannot modify BUPLOAD_POLICY
#Using a pre-signed URL is recommended since this is done client-side
@aws-sdk/s3-presigned-post #Part of JavaScript SDK
createPresignedPost
(CLIENT, OPTS)->>OBJ #Create pre-signed URL for PostObject()
OPTS.Bucket #'BUCKET'
OPTS.Key #BUPLOAD.key
OPTS.Expires #BUPLOAD_POLICY.expiration, as NUM (in secs, def: 1h)
OPTS.Conditions #BUPLOAD_POLICY.conditions
OPTS.Fields #BUPLOAD.PROPs, as OBJ
OBJ.fields #OBJ. Same but adds:
# - policy 'BASE64_JSON'
# - key 'OBJECT'
# - X-Amz-* of signature
OBJ.url #'URL'
/=+===============================+=\
/ : : \
)==: MULTIPART UPLOAD :==(
\ :_______________________________: /
\=+===============================+=/
POST /BUCKET/OBJECT?uploads #Req:
CreateMultipartUpload() # - same headers as PutObject()
# - x-amz-checksum-algo-ALGO [C], x-amz-checksum-algorithm [C]
#Res:
# - MUPLOAD
# - only Bucket, Key, UploadId
# - same headers as PutObject()
# - no x-amz-version-id [S]
# - x-amz-checksum-ALGO [S] -> x-amz-checksum-algorithm [S]
# - x-amz-abort* [S]
#No own PACTION: use 's3:PutObject' instead
PUT /BUCKET/OBJECT #Req:
UploadPart() # - RAW_MPART
# - Expect [C], x-amz-server-side-encryption-customer-* [C], Content-MD5 [C], x-amz-checksum-ALGO [C]
# - query variables: MUPLOAD.uploadId, MPART.partNumber
#Res: ETag [S], x-amz-server-side-encryption-* [S], x-amz-checksum-ALGO [S]
#No own PACTION: use 's3:PutObject' instead
GET /BUCKET/OBJECT #Req: MUPLOAD
ListParts() # - only uploadId, x-amz-server-side-encryption-customer-* [C]
#Res:
# - MUPLOAD
# - no Initiated
# - x-amz-abort* [S]
#Paginates MUPLOAD.Parts with PartNumberMarker, MaxParts
#PACTION named 's3:ListMultipartUploadParts' instead (automatically granted to MUPLOAD.Initiator)
GET /BUCKET #Req: MUPLOADS
ListMultipartUploads() # - CamelCase -> dash-case
# - no Bucket, CommonPrefixes, Uploads
#Res: MUPLOADS
# - MUPLOAD.Bucket -> MUPLOADS.Bucket
# - no MUPLOAD.Parts
#Paginates MUPLOADS.Uploads with UploadIdMarker + KeyMarker, MaxUploads
#PACTION named 's3:ListBucketMultipartUploads' instead
POST /BUCKET/OBJECT #Req: MUPLOAD
CompleteMultipartUpload() # - only uploadId (as query variable)
# - only Parts
# - no LastModified, Size
# - x-amz-server-side-encryption-customer-* [C], x-amz-checksum-ALGO [C]
#Res:
# - OBJECT
# - only Bucket, Location, ETag, LastModified, ChecksumALGO
# - x-amz-version-id [S], x-amz-expiration [S], x-amz-server-side-encryption-* [S]
#No own PACTION: use 's3:PutObject' instead
DELETE /BUCKET/OBJECT #Req: MUPLOAD
AbortMultipartUpload() # - only uploadId
#Res: empty
#PACTION 's3:AbortMultipartUpload'
# - automatically granted to MUPLOAD.Initiator, except with Express where it must be explicit
MUPLOADS.Uploads #MUPLOAD_ARR
MUPLOADS.EncodingType|Prefix
|Delimiter|CommonPrefixes #Like OBJECTS, except Contents -> Uploads
MUPLOAD #Upload OBJECT in multiple requests
#Max OBJECT contents 5TB (160GB in UI console) instead of 5GB. Recommended over 100MB
MUPLOAD.UploadId #MUPLOAD_MID
MUPLOAD.Bucket #'BUCKET'
MUPLOAD.Key #'OBJECT'
MUPLOAD.Initiated #'DATE'
MUPLOAD.Initiator #S3_OWNER
RAW_MPART #OBJECT part, as is (including PARAMS.Body STREAM)
#Min 5MB, max 5GB
MUPLOAD.Parts #MPART_ARR
MPART.partNumber #NUM. 1-based index.
#Max 1e4
MPART.Size #NUM
NEVENT s3:ObjectCreated:
CompleteMultipartUpload #On CompleteMultipartUpload() success
HS3_CONF.multipart_threshold #NUM[UNIT] (def: 8MB) (def UNIT: 'B')
#When uploading files larger than this, use a MUPLOAD
HS3_CONF.multipart_chunksize #MPART.Size (def: 8MB)
/=+===============================+=\
/ : : \
)==: MULTIPART HELPER :==(
\ :_______________________________: /
\=+===============================+=/
@aws-sdk/lib-storage #
new Upload(UOPTS) #Upload an OBJECT using a MUPLOAD
#If OBJECT < MPART.Size, use PutObject() instead
UOPTS.client #S3_CLIENT
UOPTS.params #PutObject()'s PARAMS, including PARAMS.Body (including STREAM)
UOPTS.partSize #MPART.Size (in bytes, min|def: 5MB)
UOPTS.queueSize #NUM (def: 4) of MPARTs to upload in parallel
UOPTS.tags #TAG_PAIRS, added with PutObjectTagging()
UPLOAD.done()->>RESP #RESP is same as PutObject()
#RESP also has: Bucket 'BUCKET', Key 'OBJECT', Location 'URI'
UPLOAD.abort()->> #
UOPTS.abortController #ABORT_CONTROLLER calling UPLOAD.abort()
UOPTS.leavePartsOnError #BOOL. If false (def), abort MUPLOAD if any MPART fails
UPLOAD.on
('httpUploadProgress', FUNC(OBJ))#
OBJ.loaded #NUM of bytes sent
OBJ.total #NUM of bytes to be sent
OBJ.part #MPART.partNumber
OBJ.Bucket #'BUCKET'
OBJ.Key #'OBJECT'
/=+===============================+=\
/ : : \
)==: OBJECT COPY MAIN :==(
\ :_______________________________: /
\=+===============================+=/
PUT /BUCKET/OBJECT #Req:
CopyObject() # - like PutObject()
# - x-amz-copy-source* [C], x-amz-metadata-directive [C], x-amz-tagging-directive [C],
# x-amz-checksum-algorithm [C]
# - query variables: OBJECT.versionId (source)
#Res:
# - OBJECT
# - only ETag, LastModified, ChecksumALGO
# - x-amz-version-id [S], x-amz-expiration [S], x-amz-server-side-encryption-* [S],
# x-amz-copy-source* [S]
#No own PACTION: use 's3:GetObject' on source + 's3:PutObject' on destination instead
PUT /BUCKET/OBJECT #Req:
UploadPartCopy() # - like uploadPart()
# - x-amz-copy-source* [C]
# - query variables: OBJECT.versionId (source)
#Res: like copyObject()
# - no x-amz-expiration [S], x-version-id [S]
#Same PACTIONs as CopyObject()
x-amz-copy-source: #Copy OBJECT
/BUCKET/OBJECT [C] #New OBJECT's ACL is 'private'
COND_KEY s3:x-amz-copy-source #x-amz-copy-source [C]. Only for PACTIONs 's3:PutObject|BypassGovernanceRetention'
x-amz-metadata-directive: #If 'COPY' (def):
COPY|REPLACE [C] # - only allows setting request headers related to SSE, storage class and x-amz-website-redirect-location [C]
# - those headers are erased if not specified
COND_KEY
s3:x-amz-metadata-directive #x-amz-metadata-directive [C]. Only for PACTIONs 's3:PutObject|BypassGovernanceRetention'
x-amz-tagging-directive: STR [C] #Same as x-amz-metadata-directive [C] for TAGs
x-amz-copy-source-HEADER: STR [C] #Same as [x-amz-]HEADER: STR [C] but targetting the copied OBJECT instead, from x-amz-copy-source [C]
#For HEADERs:
# - If-[Un]modified-Since [C], If-[None-]Match [C], Range [C],
# x-amz-server-side-encryption-customer-* [C]
# - called x-amz-source-HEADER instead: x-amz-expected-bucket-owner [C]
x-amz-copy-source-HEADER: STR [S] #Reflects same request header [C]
#Only for version-id, to distinguish from x-amz-version-id [S] (of new OBJECT)
NEVENT s3:ObjectCreated:Copy #On CopyObject() success
CW.s3.ObjectCopy ##Pulumi REZ to keep an OBJECT as a copy of another
/=+===============================+=\
/ : : \
)==: OBJECT COPY BATCH :==(
\ :_______________________________: /
\=+===============================+=/
JOB_OP.S3PutObjectCopy #JOB_COPY. Call CopyObject() on multiple OBJECTs
#JOB.RoleArn must be allowed to PACTIONs:
# - s3:GetObject[Version][Acl|Tagging], s3:ListBucket on source BUCKET
# - s3:PutObject[Version][Acl|Tagging] on destination BUCKET
JOB_COPY.TargetResource #Destination BUCKET_ARN
JOB_COPY.TargetKeyPrefix #Destination 'OBJECT' prefix
JOB_COPY.MetadataDirective #STR. x-amz-metadata-directive [C]
JOB_COPY
.[Un]ModifiedSinceConstraint #'DATE'. x-amz-copy-source-if-[un]modified-since [C]
JOB_COPY.NewObjectTagging #TAG_PAIRS. x-amz-tagging [C]
JOB_COPY.StorageClass #STR. x-amz-storage-class [C]
JOB_COPY.AccessControlGrants #Like JOB_ACLIST.Grants. x-amz-grant-PERMISSION [C]
JOB_COPY.CannedAccessControlList #'ACL'. x-amz-acl [C]
JOB_COPY.SSEAwsKmsKeyId #STR. x-amz-server-side-encryption-aws-kms-key-id [C]
JOB_COPY.BucketKeyEnabled #BOOL. x-amz-server-side-encryption-bucket-key-enabled [C]
JOB_COPY.ChecksumAlgorithm #STR. x-amz-checksum-algorithm [C]
JOB_COPY.ObjectLockLegalHoldStatus#STR. x-amz-object-lock-legal-hold [C]
JOB_COPY.ObjectLockMode #STR. x-amz-object-lock-mode [C]
JOB_COPY.ObjectLockRetainUntilDate#'DATE'. x-amz-object-lock-retain-until-date [C]
JOB_COPY.RedirectLocation #STR. x-amz-website-redirect-location [C]
JOB_COPY.RequesterPays #BOOL. x-amz-request-payer [C]
JOB_COPY.NewObjectMetadata #JOB_COPY_PARAMS. Def: copy source's. If set, do not copy any from source's.
JOB_COPY_PARAMS.CacheControl #STR. Cache-Control [C]
JOB_COPY_PARAMS.ContentDisposition#STR. Content-Disposition [C]
JOB_COPY_PARAMS.ContentEncoding #STR. Content-Encoding [C]
JOB_COPY_PARAMS.ContentLanguage #STR. Content-Language [C]
JOB_COPY_PARAMS.ContentLength #STR. Content-Length [C]
JOB_COPY_PARAMS.ContentMD5 #STR. Content-MD5 [C]
JOB_COPY_PARAMS.ContentType #STR. Content-Type [C]
JOB_COPY_PARAMS.HttpExpiresDate #'DATE'. Expires [C]
JOB_COPY_PARAMS.SSEAlgorithm #'AES256|KMS'. x-amz-server-side-encryption [C]
JOB_COPY_PARAMS.RequesterCharged #BOOL. x-amz-request-payer [C]
JOB_COPY_PARAMS.UserMetadata #OBJ. x-amz-meta-* [C]
/=+===============================+=\
/ : : \
)==: OBJECT HELPERS :==(
\ :_______________________________: /
\=+===============================+=/
aws s3 cp S3PATH PATH2 #GetObject()
#PATH2 can be - for stdout
aws s3 cp PATH S3PATH2 #PutObject()
#PATH can be - for stdin
aws s3 cp S3PATH S3PATH2 #CopyObject()
aws s3 mv [S3]PATH [S3]PATH2 #Like aws cp + aws rm
aws s3 sync [S3]PATH [S3]PATH2 #Same as aws s3 cp, but only copy files if either file size differs, or local mtime newer
--delete #If copied file exist in [S3]PATH2, delete it
--size-only #Do not check mtime
--exact-timestamps #Also copy files if local mtime is older (but not equal)
s3-sync-client ##Like `aws s3 sync` but programmatic
##Not documented yet
FLAGS ==> #Following flags are for aws cp|mv|sync
--recursive #Allows [S3]PATH to be a [S3]DIR, targetting also all its OBJECTs
--exclude|include #'GLOB'. With --recursive
#Priority to rightmost
# - --include '*' is always prepended
# - i.e. --include requires using --exclude before it first, e.g. --exclude '*'
#cwd is [S3]PATH
--follow-symlinks #
--source-region #'REGION' of [S3]PATH. Def: same as --region
--region #'REGION' of [S3]PATH2
--dryrun #
--no-progress #Do not print progress
--only-show-errors #Print only warnings|errors
--quiet #Print nothing
--ignore-glacier-warnings #Do not warn when STORAGE_CLASS is GLACIER|DEEP_ARCHIVE
--force-glacier-transfer #RestoreObject() if STORAGE_CLASS is GLACIER|DEEP_ARCHIVE
--no-guess-mime-type #Do not guess Content-Type [C]
--expected-size #NUM (in bytes) of OBJECT.Size. Required if uploading >50GB
--copy-props #Similar to --metadata-directive, but with values:
# - none: x-amz-metadata-directive: REPLACE [C]
# - default: x-amz-metadata-directive: COPY [C]
# - metadata-directive: like x-amz-metadata-directive: COPY [C] but using instead:
# x-amz-* [C], content-* [C], cache-control [C], expires [C]
--acl
--cache-control
--content-disposition
--content-encoding
--content-language
--content-type
--expires
--grants
--metadata
--metadata-directive
--request-payer
--sse
--sse-c
--sse-c-copy-source
--sse-c-copy-source-key
--sse-c-key
--sse-kms-key-id
--storage-class
--website-redirect #Same as request headers
/=+===============================+=\
/ : : \
)==: OBJECT DELETE :==(
\ :_______________________________: /
\=+===============================+=/
DELETE /BUCKET/OBJECT #Req:
DeleteObject() # - OBJECT
# - only versionId
# - x-amz-mfa [C], x-amz-bypass-governance-retention [C]
#Res: x-amz-version-id [S], x-amz-delete-marker [S]
POST /BUCKET?delete #Req:
DeleteObjects() # - DELETE_OBJECTS
# - no Errors, DeleteMarker*
# - x-amz-mfa [C], x-amz-bypass-governance-retention [C], x-amz-checksum-ALGO [C]
#Res: DELETE_OBJECTS
# - no Quiet
# - Objects -> Deleted
#No own PACTION: use 's3:DeleteObject' instead
DELETE_OBJECTS.Objects #DELETE_OBJECT_ARR
#Max 1e3
DELETE_OBJECT.Key #'OBJECT'
DELETE_OBJECTS.Quiet #BOOL. If true (def: false), only return OBJECTs which errored
DELETE_OBJECTS.Errors #DELETE_ERROR_ARR
DELETE_ERROR.Key #'OBJECT'
DELETE_ERROR.Code #STR
DELETE_ERROR.Message #STR
NEVENT s3:ObjectRemoved:Delete #On OBJECT permanent deletion (not from LIFECYCLE)
NEVENT s3:ObjectRemoved:
DeleteMarkerCreated #On delete marker creation (not from LIFECYCLE)
aws s3 rm S3PATH #DeleteObject[s]()
--recursive
--exclude|include
--request-payer
--dryrun
--only-show-errors
--quiet #Like aws s3 cp
/=+===============================+=\
/ : : \
)==: CONTENT LENGTH :==(
\ :_______________________________: /
\=+===============================+=/
OBJECT.Size #NUM, in bytes
Content-Length: NUM [C] #Like other SERVICEs: when there is a request payload
READ_OBJECT
.response-content-encoding
Content-Encoding: STR [C|S] #See HTTP doc
/=+===============================+=\
/ : : \
)==: CONTENT TYPE :==(
\ :_______________________________: /
\=+===============================+=/
READ_OBJECT.response-content-type
Content-Type: STR [C|S] #See HTTP doc
READ_OBJECT
.response-content-disposition
Content-Disposition: STR [C|S] #See HTTP doc
/=+===============================+=\