From 1ff257286a232abb28f290067cc699a504721fba Mon Sep 17 00:00:00 2001 From: Marco Pracucci Date: Thu, 24 Sep 2020 13:08:25 +0200 Subject: [PATCH 01/16] Documented shuffle sharding Signed-off-by: Marco Pracucci --- docs/blocks-storage/store-gateway.template | 2 + docs/guides/shuffle-sharding.md | 94 ++++++++++++++++++ .../guides/shuffle-sharding-probability.png | Bin 0 -> 32049 bytes 3 files changed, 96 insertions(+) create mode 100644 docs/guides/shuffle-sharding.md create mode 100644 website/static/images/guides/shuffle-sharding-probability.png diff --git a/docs/blocks-storage/store-gateway.template b/docs/blocks-storage/store-gateway.template index 3f0454d209c..c2a123d71b6 100644 --- a/docs/blocks-storage/store-gateway.template +++ b/docs/blocks-storage/store-gateway.template @@ -48,6 +48,8 @@ The **`shuffle-sharding`** strategy spreads the blocks of a tenant across a subs The shuffle sharding strategy can be enabled via `-store-gateway.sharding-strategy=shuffle-sharding` and requires the `-store-gateway.tenant-shard-size` flag (or their respective YAML config options) to be set to the default shard size, which is the default number of store-gateway instances each tenant should be sharded to. The shard size can then be overridden on a per-tenant basis setting the `store_gateway_tenant_shard_size` in the limits overrides. +_Please check out the [shuffle sharding documentation](../guides/shuffle-sharding.md) for more information about how it works._ + ### Auto-forget When a store-gateway instance cleanly shutdowns, it automatically unregisters itself from the ring. However, in the event of a crash or node failure, the instance will not be unregistered from the ring, potentially leaving a spurious entry in the ring forever. diff --git a/docs/guides/shuffle-sharding.md b/docs/guides/shuffle-sharding.md new file mode 100644 index 00000000000..9cea8770f63 --- /dev/null +++ b/docs/guides/shuffle-sharding.md @@ -0,0 +1,94 @@ +--- +title: "Shuffle Sharding" +linkTitle: "Shuffle Sharding" +weight: 5 +slug: shuffle-sharding +--- + +Cortex leverages on sharding techniques to horizontally scale both single and multi-tenant clusters beyond the capacity of a single node. + +## Background + +The **default sharding strategy** employed by Cortex distributes the workload across the entire pool of instances running a given service (eg. ingesters). For example, on the write path each tenant series are sharded across all ingesters, regardless how many active series the tenant has or how many different tenants are in the cluster. + +The default strategy allows to have a fair balance on the resources consumed by each instance (ie. CPU and memory) and to maximise these resources across the cluster. + +However, in a **multi-tenant** cluster this approach also introduces some **downsides**: + +1. An outage affects all tenants +1. A mishaving tenant (eg. causing out of memory) could affect all other tenants + +The goal of **shuffle sharding** is to provide an alternative sharding strategy to reduce the blast radius of an outage and better isolate tenants. + +## What is shuffle sharding + +Shuffle sharding is a technique used to isolate different tenants workload and to give each tenant a single-tenant experience even if they're running in a shared cluster. This technique has been publicly shared and clearly explained by AWS in their [builders' library](https://aws.amazon.com/builders-library/workload-isolation-using-shuffle-sharding/) and a reference implementation has been shown in the [Route53 Infima library](https://github.com/awslabs/route53-infima/blob/master/src/main/java/com/amazonaws/services/route53/infima/SimpleSignatureShuffleSharder.java). + +The idea is to assign each tenant a shard composed by a subset of the Cortex service instances, aiming to minimize the overlapping instances between two different tenants. Shuffle sharding brings the following **benefits** over the default sharding strategy: + +- An outage on some Cortex cluster instances/nodes will only affect a subset of tenants. +- A misbehaving tenant will affect only its shard instances. Due to the low overlapping of instances between different tenants, it's probabilistically likely any other tenant will run on different instances or only a subset of instances will match the affected ones. + +### Low overlapping instances probability + +For example, given a Cortex cluster running **50 ingesters** and assigning **each tenant 4** out of 50 ingesters, shuffling instances between each tenant, we get **230K possible combinations**. + +Randomly picking two different tenants we have the: + +- 71% of chances they will not share any instance +- 26% of chances they will share only 1 instance +- 2.7% of chances they will share 2 instances +- 0.08% of chances they will share 3 instances +- Only the 0.0004% of chances their instances will fully overlap + +![Shuffle sharding probability](/images/guides/shuffle-sharding-probability.png) + + +## Cortex shuffle sharding + +Cortex currently supports shuffle sharding in the following services: + +- [Ingesters](#ingesters-shuffle-sharding) +- [Query-frontend](#query-frontend-shuffle-sharding) +- [Store-gateway](#store-gateway-shuffle-sharding) + +Shuffle sharding is **disabled by default** and need to be explicitly enabled in the configuration. + +### Guaranteed properties + +The Cortex shuffle sharding implementation guarantees the following properties: + +- **Stability**
+ Given a consistent state of the hash ring, the shuffle sharding algorithm always select the same instances for a given tenant, even across different machines. +- **Consistency**
+ Adding or removing 1 instance from the hash ring leads to only 1 instance changed at most, in each tenant's shard. +- **Shuffling**
+ Probabilistically and for a large enough cluster, it ensures every tenant gets a different set of instances, with a reduced number of overlapping instances between two tenants to improve failure isolation. +- **Zone-awareness**
+ When [zone-aware replication](./zone-replication.md) is enabled, the subset of instances selected for each tenant contains a balanced number of instances for each availability zone. + +### Ingesters shuffle sharding + +The Cortex distributor, by default, spreads the received series across all running ingesters. + +When shuffle sharding is **enabled** via `-distributor.sharding-strategy=shuffle-sharding` (or its respective YAML config option), the distributor spreads each tenant series across `-distributor.ingestion-tenant-shard-size` number of ingesters. + +_The shard size can be overridden on a per-tenant basis in the limits overrides configuration._ + +### Query-frontend shuffle sharding + +The Cortex query-frontend, by default, executes the received queries across all running queriers. + +When shuffle sharding is **enabled** setting `-frontend.max-queriers-per-user` (or its respective YAML config option) to a value higher than 0 and lower than the available queriers, the query-frontend will execute each tenant queries across a subset of queriers. + +_The maximum number of queriers can be overridden on a per-tenant basis in the limits overrides configuration._ + +### Store-gateway shuffle sharding + +The Cortex store-gateway -- used by the [blocks storage](../blocks-storage/_index.md) -- by default spreads each tenant blocks across all running store-gateways. + +When shuffle sharding is **enabled** via `-store-gateway.sharding-strategy=shuffle-sharding` (or its respective YAML config option), each tenant blocks will be sharded across a subset of `-store-gateway.tenant-shard-size` store-gateway instances. + +_The shard size can be overridden on a per-tenant basis setting `store_gateway_tenant_shard_size` in the limits overrides configuration._ + +_Please check out the [store-gateway documentation](../blocks-storage/store-gateway.md) for more information about how it works._ diff --git a/website/static/images/guides/shuffle-sharding-probability.png b/website/static/images/guides/shuffle-sharding-probability.png new file mode 100644 index 0000000000000000000000000000000000000000..658b1c928e9e152260952f6fffb2ef8a4ff0238b GIT binary patch literal 32049 zcmeFZbyOVNw=Nn81b2tvAvA8m8+UhTf;JLDaCZmkcqq~=;R*PP${<_c3)mcc+JMSb+>5r&+sr23;r@G6fU z!I>jJ0nQu~qvr$vNLorrsLDx5P^miFy|T16fAk0xXKHMWE5{7#g+h&udk2{qP#xXW zLqj9gjs4oc^-;Bb>!#}b7MrTCzetR|_*k?3QNCJRtrdDC)m0^HFLRYk$X&3tP}S)* z(vO;}F><=pZM8=_@^2hCWA7e!7WE-YiJnF>h}JlI$iaoq!I?$i!;vGJ4Z~H8Q}1F> zx#9-;AQV4;)=Sdc0%wNy%;FjRps_cc+0HZSutM63+_Qj3wWjv3%f%sqarJa?lksr! z>P+9Diy20$x`}C7YI%#u=mnaH!%b#FDK~>sUXdP83^e%d-7zQ zk7|WpoOuuRafhq1J~AhD=l2s5;y?)!VoXwkXM}X^J3HG8u(7$hxv{!&vD!IWuyOG7^RuyYvT<^<0C%uBdDuEb-C1m%X#cL{cRiBk zPNt5Q_Rf}ewp0)GLQU*koP}s;9vb@d_jf`BO@=S3xdB>5>Rr)7 zXOJ&iU$ljJgM=!qHdwv(PA6x5u1nY>wgVt7&*?heg*vY62 z<@8s}@ksX%#ElrtDL;QW73Nv%_%srMe4ai_Z3-Zp!B)g(x=UZ|Msjv})t zP)MoC)6VLSCbzPhEOo&!5`&GW82N7Dm*6S$wK1qf5VANb{wUVPQ_ISmvz}|1F!bD_ z)2p#goAcZ$kqUZNU_bA3ZXi)~5+8<@%;sm_pUk2L!@GnzRXA^rHRpl%X6w0*t?>(w zHF04TY^`3VdKoU*s27WE?+oU2zn{*`8z*gM(Q%Z}N=LE_jK&CGy^gEV3**D@sqRal zpJ~0nwQF3BS11OCsmmdv7^6xRrjh#7&i5o8{GJ74CXqp7w=mnca%;SBM$fL<+#=h@ zb*f*uVxd@<%L3YQg`<);XPkL;cYUz8_zo+2qhM-S<^7AkxqOKbOxK^5h5SrX6$>vE zZ0#5QkpvXxb3-T^bky}fI+F;l#!3|H*E@E3-p8v(M{xUlaU9yUTy`@oID^k!7{V;c z`5jk<{e8ZFQj?~aQU8SyuIRBnTVJJB4B=2gSK(j7_d4n#iVAsm@1y3iz0m5H^`b{W zwp)qWuge27;l)_VOrLhutHI?atsg;y*qa!H9S#xGfFq@Crk7!QY)c~z7s_nJwLx%;*q>JyCVzd45%b7k@Y*Da|^DH zXxf^p$ag|A@^lxuzo{xQYMl>4K`Vwr!*mS2`JlDC9Xl&AB0TQ#yy$yBzUT~tj38!B zimB{v>{d>9A(X!5MH+HRjBf*Zw=mG0mQovgMrd!gf0XRZ#%}a>hT)ken;v7yo#`qhp`Gw(X3b8uBiYXw<2Mmz9LKyFv0+q5;X|+kkWApR7=h<#rVTNQ+h(!ROhsS>X#%|6#Oh=nQ5t&Rel2!h~td;2k1LRuj zE&DdCeC|$v$9C^2+Gsu+m5hor-@`Vf(m~^|@6I9x(i+k$+d%NcOqb_nez*M-ex29N z=krb)D0hb;B6Y46AXw8zT*-`yPS$JUW_Evq+}IJ~{KOGiD%0*r2ND%iQr0H@e7Fyl zALIB>2DtbMa7e=eZ8skd@W@|PMgV{)>mvEtahRE*%ij+!&CY!cN$#Q zrIT{++tbOR&x;|mby)Hvee~H+bv#fTp~T}{P+Qa_T7suo)YpLoCE(m$`$1m)CBESE z1ZYo(48-uYK6awWb(!)eTjO3N3D-c?V&KLNFl`!;jvK7XoHCZGOuA4fWual{XdN{# z;9DG|`b>rWb(UXm_6fFsSVA6-i_kGlwIF2SvKac>L+LlWMm8DBt zt~Nww3l)ssPDR51&cPQ@ zmZjjl7u20$j`FFUc-mo!^e}>9w7aI`)fFta<)t9L_}0J}lb~=Q`@W0yC3_qXZQvX|UjP*TUJbYvx)yesrZ3YgvS3MF#&@t2ubKK`nJ3AS zAOh;}A&FwNm(FJF+zNf>+E031*fM15KMEJlNq7X55Wy(!AG4Agf4#;Zcy5wiyA?h9D(!5gHh_AN{&MXgZ0GumW_m z{(M^;=Mi8{G*Xb8_cE!>W39dO%JkZ=FO4+W68E5C)of2xVJoUT^=;R~r-xC6Q?5mO zsUF3hL!@Krv%SZ3(;+1?L5f(3;n-H{&>-o0o0*y+^>@|qI|$H@Aort469%cxPZ`c* za<-IA`1+fxhXdVAJzK(7efGIa;tn(_W%cW%=QpZ%6dtQMg|lPEZT6cZxylZC8sg2O zG9i@`NN+v;lS2Kj#LzMLWuBd~XOy+3R^tr(2v!W&NBelnlufi_d)lfKj0 zbkt2&S|wGFr8lUSQU3CTS$(}iG|sLrKhi16c@{9@N|mUEeB0gCl01l(nC}=W&%r{! zMkHJhn$H{jNHhE&BmKO;8NQeNdJhpYC&eC!G-z~nk$~46LB=SiY|qaznSk=4k(}+b z*J9k4cg|yZl^5Vhs%Bv`z8W&EDe#{HmNI=f8Ir?fY|p2o;x{)V*celQ*}ar8@@@^v zET1cOsEz3JsS#rC+@K zWaf80vmnhVXXy2fKT$BTvmAJTcZUx&eyEN0*|l%a@Rna7iWfwWu;*c`?l&)YdcYXS z{o7HSH-VLZiuA?n#pCa-S@2Sr5k$)*xz5J*YuQka{o(dlKo(t5(hb8e#q{fP59Ew| zFV~c*moq4bKIoh1cD~{Cy>Vx)CrF(aA2^(+yuXlfqi}Opjcxxhfq1y1BIcsnU_QU_ z4xDo*9x5W9M%`>!{*%V}>UAs*>BWPJ`7yURnQaEsn)1oJZU2zunIV986pnWdwwmYc zb2$jM>O)0k5#M-RitJE8@!_UzKOJSE?&OZ1gcY)*=IY*^`r}(Urwh$+pwS-roUoe6 zh<^@-ghLP2q7h0bSE_9I=K=Z#86JHm&VIZTR1=h>Ux-Bdi;O#)JELFXZXPcHHo@z{ zq4>5~l9e`SF18ew1&80ewKV)!>#`&7&hcLJ_l%$%b- zgJY!_AR^^0BN1YK5>3lEa-i@=Gv8EU4&U=wsUrMK;4LLZU&CXJ`N!%`CVMS)4DWkZq5m7 zRk>;~N(iSvyk6?pEBl0^cVjFkcNEqYG|*fsyH8k6xIr$cJ57GtY8P^};dI(nnyH!~ zBWlNfvkQ-&zC@-2o-~^V%TrVyC(a7KprEU;X#8Q~)7k_3kh#v8LW&deS6Tj>j#3-( zLxfAgbR?{+1!W6GGL|bBh5NZ&K^IqTA%nB?oBW>#b|i34GN}%DJoWI+g#`wz!K|EX zTR*MJT$+8caH7)Y4Qa%fup;dEioWLswFkKK-Zj*|yfCfNeAObB124LiD-I29KR@i) zU)j6%3NFLrC0zx4h9DK9KHnMBnT91g&o0dx#W8c`C%6<(B3!tJ=?Gja+QWo5ycjgW z6$p1A0r1y4F|lK(T-B@_ZyJ0hPo2==$Bbmb)60FIC%znOz6FLVxpm@;_6(DRVV?y@ z_}Y)`h^~Rc6lor2V0f-iu7&me^jp07TX$+&;-XG>H3ivSDOOQ(-oD$sUYQkP;%{`y zuw&K$DWkuOD?@KSQIUuYVoio8X|WL)e(BtlS8qHh>8%3xZauuKT)W?Mxn2r8bZJ#- zh;9Fv#2)|GxC9;qQS)!JA0r;Y@G@BilE0I^s4?`={?{1%F^q)o5K?OR=%eTn-b&zz6XEp=npJU$f!75n;!A&-Pf_HdI5*3 zxgK1;ifjm9WS!aCLu#F5e3_(Z=(?^I+#-^KP_eAglgYX(<&aLgP%9k^I#|=(uR>mE zXDu#dThyUL^)u^ozE;Bb=?%5Rvf1b%WH*6tHu$&c8{i{Z+0=$IPY?Z(DD7cS33!>b zHhnaBmU?f3Aj|#o7o<$6IG#>Wxh|V2qsr^7+E%}*h2W_A=8?mC&2zNMr#KYyxP)NO zwqj56sG^QGH7(8WtMt9FgxY60b#z z^l$V=NKm19#t^s@rb830wO(Oli5@*3SJ|9m|OWuap^w;Lg=_J!&yhPyOcVPCgWE8g6 z)ogu8a;E)0KH712cpM&@O_^vt%OG>3ai;nK>~@en#Cn>WowzqlUv=t&Cow#lopwxE z2|3l~gvvBBeCDJix88evXjD4iS00hc`;%iJb$XCIBht4{xQH!hPq{A zEhMX4KahZ)G+)vQO0150Zp?0)&pdKcs>iDXcX+O3uCNlVx+RVFgT0-8NHe%ICF}=c z*hV~KV?g`q7dKVK_9)&A-L~M7#Sj95vQXpjGXih-*BvS~X1;IjtbM@lyeLt;7|xZi zULQkHM`_5YCx8JV3EN;QYg)co z%9c-@c;O|UT0H!e^!OVZ*vD5@I1aKlwITL-_gugk z5xZO$3zR4b-~8nk)j0Iyysrgho&oNA)c5-DMOivxM_4E4#kwV1Ec+LQ9m7@f)608z zMX0D4xceZ7$h?D@Y{)60Eh?^E$@wtde#GYiBIg62ThFjcFvk%NOUC?OrrSJ;ny)Gi z2>gQC$Md`|gw_B?^b&{x5UVS;O2D%x~Uji^83zN)lU3rJ%g-exzHYCDAcfxOOf*#xzv!2R;H@Vb}| zPh=ReAOk9y=!y?spsG|)*oaTELmzFA*?7tXH{4PyfE}%~o}gqUcEt1vQ@(n@3$0P* zFy752CcE3LuS_RtyF5p-BaLRmFKkeT=C8YsCe32Q z{&Nf_0zv|=@WFtk@wU}m3f6DnBg;Y&@%`9}k_Un)7wvL2CH+jY(K{wCr7_43dX!0@l#Z+%| zk<m(qATAC3Ldj`%M|@HvOSa4+x=+=uQp~G^entq7Z%3#LpPFh(I2@x|6xUri-IeTuR3Vq;#<=sD zVue{7@h6EQt(7=Iovkz>?CMM6LcLrnh0)4rr-btO2P-FKxJ%v;?`fvZSrUt1uHiJ$ z6V%7;EQ)DZ3v4sz3W!0=t*(2xlJ5o#E6pr(_>x>#&Jm@eE~(KUAHBUWH2+GONE<8y za!nc2;R!uUyMal#f4f7HOcZaYjsE2>+w%@fn7GPc$$3F@ znhureic}sYft4eMzJu+jU$H6ax)zfTe+xQF*Jomr%LupVUme&jt4>XZ&$ZbYMUxj# zS~zZ~MlrB9)Ey$Oor9J}`ZS(529NV*2{SW0LGD8r$lYd_RETy6Bbc6697|#`U6A!K z_H2QzgqzH2&goap<+`a4zw8P;}f3@X@{K%U3+{vEu+rxgq1&H9`7fxoG9up zdVGWFOl`C;2ogL8-t?uyMwaNWCb{_;&psn9cct)`lKrqNhiNfxe35WhCSudwZ8`>b ze=%JSBHw9>VUi~$_s@gb{bYxcWwN$oDnR?a`HHP)*wi{)p%*U(PPf-eLs7D0YE`7K=@ezW|gfrD!Q**6{|J1k^L>*;*99|>7B^+>)Yj8eQQi`ii%G_Wk)8Rw?Mw4OgOH>`P4LN z#Tmem;@@&Eg<#-a(%K?Upw0VnTAtdP+5nC~Q>$6OH%-?_NYXaX!{aJ(Q;hBd{yPY5 z6jkDv&kA|Ibs)FJ(s{SbTA1pX%PD8^G^1tuzi^P*y$tg90x&dSi9UhX*QgRFhxfUNXeTfg#Rr_ zYpGtZFOZJ+?@*GIN>1mH2jkU;CzEdQ8H0@xUOui@R~6 zrIRHeL)<8=R3C+IiNH0tp4ImDk?&!i9(mF2_AR1GU!EAS$=mOUO)gV zs!TxSYBRsIGCat1?Q_y!aVcNjLxtjghRl_nQSWDv`0Q_?IJ5(PcXam?Hx}+nkXKE3oRYPq=Uzq-f0qI$rc{!`?GjWo>}5H4EVN#@@_F&ExzSyn`vjV%;-bLU@8Z#%2ltR+0S2=XKf*rb<}30 z95d(=@h9FrM+qvz6_kEA zqq6jrABDC)B+#5niJ|wIwp(0Kw$7>S+lKH6eWy#`xCbm(hqbKQ8u}A8%`6{g&8DMP zsR%;X?V`el1652<*VyVe6R_ULY~O{zR)v|a??niv+liQTZm&)a3~=kQO+zX%6CUEL z#xp>mGF`~4hU3Q<<1gd#4`w-(YZLcm@6?4kGKiivWbfya@f}x_z()vP`NT?~}0+usDY7ziFNd}9f_6#nIXmZcZ-2Da}69XzW5yS#|SiMFDi0DOlA`yNB z084|Gnblg*eXKrd`~tmOB@@4-VevkXW!>Pyx%I!7I3?uHnPWim{iQYfifL2 z?TfHWCN`OM`Z3~@0pp1kfy-Wdp>W=F>9`|8A{g={!>~R;BS^k>rCcLuBZrQbQvz`Y z)}i|F851}1H7*K?tdpaa9>G?2nz9A*J6p48N*LN&Kr~r~tcF;7)i76RyDUlG{GwoD z5~G%5Cf-3df98itH`N=D)`Fq85-}7)H$`=N?s*{U-4n1zOVNxfNaSMiaS9KN(n0bfv!zqHt|_2?n#ypLWEqtbd>&@Sdap3z-ibM6!Q=YvS!t@ zVWD$s4rCCZKg#pT*8Fn0V91?0YR!7N=Vmd_9bRu_kDPuMSbOHDQO zPj%(RRJ*20Vsgt_$fyWWod6laCfnysS063d>1GE@h?`VkzpojG(TGu=k^c^_AncO# zkdoc_?TJEui2BY+#IL)%^Hw()fqeUyvX*U9M@owghKpS4oVc8Jx;qFbZddd#!7ALZ3$Bd zL?M(3qlid=a}`az$z8e66egU^Xyh8ktKC&$DpI>}yYE(t-eoF#`*;|Qt!X3(tqcfu zw(AOdB$toiEJTwFHtc}#Af~t|WL0E$GXNTG$g4z(&i$dnuo(DESu6Magz1+G+^b$Cac9E=T(~CYK ztDG{?qZkqLIt=d!g}3JDIH+>^VAq#t>soH;Din7y={c+@h23pA5;6PndPV`~b0K0`4OoJ~AB$vnnEJj3`)C>Ii5>f(SyKEjl_Sx{H1ZOCmycVXw zW=<%PkqER)zJQ`wr>vBW=6MU`*%rRLVcfTOnf4^8B8^Aei2-%_p9=J_f>#QBa6pz1 zhRG<}aTGg>X*N<2(u~MzI?w$iY?bV_Q4t|RzIXjG4*f4!v-=m+6aG&FpB`O`Y-Bhc zGth(B26A)M^V`Jzm-HcJO)UZRH1a|!W~DtfNmj5l^$ zXUQ&xjM&R8q@%K^y$7nA&A$yg=wY`IP*-j8PCfFR3|rp2?e0hO6PmB_jM`R?d(|5i z9G}I?45B=wS3dlOYTRVcXO!z*YSQOhH|y{;8F;;17ryAQ5wS6cLh8$@RBeYv?!7md z4O>RwJX{Z}Z0nLXA~>a+%0CBnF)?JEM7_N51b1<$@t<#D>GgVpJIo^;NeB;uouF>> zN~smuD(eSpf}Ha#P!Bo|K(qP;x(6p=oE1eh`DJU*=6xb~w&Wbr&wcG23XnitHmPzH zWzb;s=hWG1<=zyr=d0N^N!EiNDv2VC7$2x`33>i+((H8o% z6*LRogOBR)q#I6iT3c&qRB=&0OordR$1PUWTqiPcjuVt9gPp6Uf|U#8h8M7eM$4&} z3_lCTEn`JtC21ebH)pCO;;wVCPj>5Jc+&-)J}KB&@w-Zhz#@R_z_3QcrBqQoY2l6J z?6ktZSv3j9{xpK)IRvSDE&KW@$!FsLzGOyxne||fV%^x|z<4AkW{;K)1<#629M`-T zZ5iFqtqF`Pj}+Ki@kzoNL35*%Lj3{D1bb{(d}>YkbKQbx4q+~tU=wB1wb*XBT9EP! zeUQe{Gqu#C8++lTUpela&f!6kOZ518lZJkS+{TDWUD>{OeGJOz)Eo$rBh;eHU71ZL zkhF3JJoa34b;Eo*U@#K0p_|$M*?w2kh&Vz6WaPGaroq~`lD3HkFm9J8sQ8mjIf?9& zQg?JFp<_w{M+YH9D5d~6iq+bPE$18i_ZUg0?0vKESjmQEk!r&^`tEcw(Zmvd=nRg2 zF#WIgM(ge#N8$LF*fjx{J4MzSMMxFK8rrE@i-Kir2q&5eaRJ+jlCx+USkLiz>v8;> zrE*kZcpt42QJy21+CVWM1yo8RzSGTsg%gNgANaj>l93{>`Ob&MaF_EmM;|~6`Kds! zLn-?=<`ti;WuuZ>EJX5>qLyfn&(6hqbb#6Lyl zO!tX8#9w@67sH`#RmT|skDytG60V~y-#yT( zehf(-2q|a|F(`m|UVI0mK^)wN7*r|}gErke1u1U(UNdPmBzi@wWi?F%d*|nWtCE~H z=caA<6+^#`r9j178kH0GuzfnvN;|(ww3UOoT)!-ghyp#`ogUZq+9JND1MozH^h6UdA?;pUp3{7|D;T<$k(&1?346~ zLUDcGkrtjVPBb@>J~)RRX?$e7V@=GAv*4g8_k}SW5Ul&;ofZp;y+C>on}`tU;gEl% z{d^twQaTWIOvhSN8}z)Gy+feuN9$pAfKD6!>X819}>|F(3@1(Z*Y^W9Ns2#^&NGV5iL~xj4S4l17N2xBE$AN;gJ37%~aBwjwh+S>|wSFGK zV^4R@tQMRuXXTZ+PjGVhydp+S;Ihuf4MTK>oW)LDeA&{VgTYy}`6b6e?#a$$T>cc$ zd{`;YBZp;S-#guBBlvE}Att!deq!g!&JpbwQ3NTH=kn_>p&g3cLZF&N9Ra)|C@v;i^sU^d`Nrpvg2O zB88?dHvKHq`jZXhh-fwDb44O!M`7}aW;(!~`mO~r86T85rZdb`$3BOgF3y{Lybc$I zqTq)eYfj0U<@+&XgA(ovqNjUBEr*2Rp=%5Y8pGv$$)drF?!mX$Yn=Q_;f=_lqQ?)8 zSIUjeD(CyR*`Z`UhCMXpYcQc@d#z$r;8 zOaPs7__q1=|97WE;}}!~U3WtXTGm!qJHf+!l9TOO%k>d%z5sa{dSL4ZaHne=R=T}6 zT*UwFxjMD#GuZg%Xnu+J<#CP2$$D?I#o&AM!Yt3vNvsAxU)I{%tn|dh=#v3}N43@D zkU_`b@mfFaWV_(etPQ|Eeu7SH40tM40EczDYFL=;$Q5u*An80dv-be0Z}xI;rnZqS zn+;MxS z6+6JO&Nc8kov@`W*cnq`8(hhLC%JE1xAf$}SAW1k9(YoXyOsUcV;|&LZvg7?1jC!z zllqM;rzK1Z@D#w|u`7954RfQ+{KJ38_D@7ckTA0Bm!b4Ybow4FElh zsKZLE&Cs*0rwh5(EP2Uo0iaa;Zu1@%L%_wIA?`QdK^6d`7U%pY&ToJc+us=_QAut* z+;O%*f#`a_@xXe0AkCwc`Q_JMawPfa854syV;{n*#98#~Mz$x*C><7Z#2(iUXM7NF z*_IsYajJDcGT|MVEQfZ4>a`nh##}q^&rV;RAK2m1D}SYy2%Z9{am4_;nh<3D-`mB1 zGlgkhpjo7Txj(pi^bJ07!@EE}iMd>HdMHz{#BmK4XO$t7NACeN@@kNDWBz*Ot|iqk#~ht zhxexkr_1Y!gu81Ofexh1u-SUP$z$L17Zx6@!uFf;0tQU((3vlgk7;5Rzp9guNG@2sEt~{VW2CuZ{Dtah( zG8@zFh5JKVETmI|t4P5XvwunmDt>iwC*|3pFWf*b@ zIPO0KQSdUPQ2Y$DJhvH4m9+s5Vn!F5f)+c+00y}U$v-gsLwlkx5ymntDb9!>zO&)= zixY@(;}01Hm|vCyWY-Xq^D;P0yCqA704nc3cru`GmcPx+&Ta33Giux1;l6KA39?{C z*Pax*IqaB^n8A%PCT0(Dr}mFukA7^f&$*6| zg8$_>(L{e$|6dIx5B4`nJJDR$C_xVYSAD(*)NTQa^8BBkR3-pgcXz<8;N|b4R7!w~ z9YdhE`=eH-93e6DvQ+brKL7Ag^ZUhBr9X`eqR4u{hP-t?OZ}dGV-Gbe*&jdut>0rafrl`+ zXknW8(?iRLnxFG|{8?anm4Sz- z{NE9ef2HMBLamUj@lDgP^{-;J%l)ZhS2V%e!AE)xzH!aRb51qZx9V1q<_d$UGxaYG zM%CW?Zu|4*E53J~Ef-rFOgbN%Tx=cVU0%;l)RPH$eTcfhJOTT9e%&<{bla14w*xaT zeEr^X=TWSrdy|>DaE2@R!INvn&`!~N-WS5zCBQA%=UL6iq%tM&&${KWNC!-_$Bwph zHPtotgKTlBw(~91HAwPGmz4xIGo>Xw_EQu4btcoOXZ`OHYd{yRWy32t=e`@aTXr){ zc!uT2<}gIqply2HqTUzhHw1!}`xQJ#DR#RdT_VKLMAjs$Sp9MLq`jeO^7Hw?#@Bn^ z*Vl(Tv$qNB<96ZZCORmy8!e?U_8@rngd)BAxNj%ouWM}6Z`!2S>X#bi2 zX%A7gC#x@G)-6cy4JpXHTBFjhgHGSLB>M>26>I1#^L?!{nDfY)6Id2VZ#0RvnXi~; zO!~6gT(>8~CEc+1*h1L0pNF;LTBK=Q`2A6qMwYgX{!*#Xg5PTjk*A%Fyx|jjw54w6 zt(E;o@J!kOoZpaf()xN1{)_gO=+5N2xR)Dj=4|})^w}}*yAp+}gp;5$yno)rSUH3j z8w?4KOrbjsEMGF0HNEbCx?Z%&Ovj(18KAzT@D7}*;ro;{qpbU)&89;n%IbwfNH&7R zTcbO0_uiyXdCN@c&##Q-d`T4{!u8{-L+hm(OG6=kc?ZqLgDU>-IKbaT}klSGS zdZ!OUcqTlyJU5G^Qy&`&^ypRrcW(`4kFV$1$NnVxTMhX~(o*$n1e;F_Y8bh3>^rSa z+cvuNXxElNcqU<*_1@-gO`bb%0M0Wy-#+TK>m35wU58-nRgtkzOvAT5u3AgCp;BYw z{S@Q<@0^7>bCVlTEB$86XQ7*3DtcL8KsK{Aqscaq^_$!Bi&tdM^VotHZ#~a$5brCd znF^6k%0AD0b1SXU^PIOaS4as9EF}L&8}&-USAxw|OKmWU!Ia8=3xm4bp08DiP8`jh zLi$|y=C`8V5K*z!YczeE0bhyI9##FEP^j1@|E$JkgU4%Wr!eE%@6>NR3nMWrv97s)?=ri*!WOxf+xD+hUhY)7`C#`{AEF))oi1 zhaU|Nm(yJRjB`zTRcQXp+*{tCF2+ zO#GCCy=>F{n!y>~hVu7FP9HvN@_iFdyUb0$xb5HeXoEG{%~jLxtTDbi+V@es#cXRv zn%+#KoQeK+%zanLmeHEOQ|+c#tTpV%-Rrxsq|T)AIZ<#!=h$YtrHgz==AQ+NGfGq% zlI7^=XyO?4mq&0pTc%$>#$!96MSN^9@D?w7_;y5T=vCdIo_i=y_}JTMKj$(EVnHuO z<2Xi}$t?GGdX}}8J<@t4#d&&HJ0FfuVs776^VJpjaM?8}>2c0Qe_rVJUAU{HiX1yT zr*NC`zt+OgUhpvnM2Vr@mHArexuGKXEI`R+T?1ED(S}DIyiE+Y6+_v zdr6WS+S!Ek)5_?O472L9GRt!x_YiXZYUOgyGN&i1I{5#rv{X<0Md276NqBAbmFF5< zPHsgWY#qagJeL#wo$o~&1AI+@A=_}3CO9Ftk})@N;|E9o}!B3NNXFRMSJ zS&Za};oQ3|__!$p$)b+Y0^6T5fUL#F6Ce+?-x7ezhGInSh#hLn_7I#x9e>#9Ca~!W zy6z3Q+3pOn@Xxr}47+9Fk=2y;Pgi^C3|NIJcQ?2(pQqQ@g}c1`?axt}A}7*({5;i#iL7yPHcw zkmy-2ZOoJ$eWsa?p+`XygGP5S2HC4geWxg;48GA|bdu$EFn}!2azJ;1_+O3prK1<% z$JhYberF>RSkplC)DlSYs^Fb>ryUk@{*PqeB3uyR8pG4DG%kymfT?NS8HxiW>aoqV z_%!A+`vB2PTjCPks#n^Lu6yM~wf8_MheUo7NSd8J0&-C;{exhaWk#(n8qvN$o0iyb~zDjT|@}&|A?872?*F zXu)E@FHm-%Vbv&HZz1~xyb5ZH%r>fLj+O>3Fg} z_@mj&&4T~4Mq#vp*Iu>jq=DO-MwaXTtYs3@M#~}2a1!HM3j(f{`o+{i*~_FxurJ{A zoWYpNrhx940*RQl#KS$kJ!mWN0&As zzYSF51v=H@GRAK|$IFaFczgP2pYl6?m~-rBxf=l@YpZ8g6R>M_nXhfrRTeU{9B!ym z#qvpvgvB4ZqMHX&IXZ62vqXB#9M%OntS^9QT0`H~V(cNgSvtciTHQzIo%hs=mUOM*pb0&@F%fx_fUHrE3*yQ9Mt<#apAW)hAi$`b--8>Np+ z98PD6a$ZU0CCUEv0{TbN0mcT(K!de<#1<~M`W@54-SvAA;ZT%14j22aISD;QNjcMlv{t; zXn^0NAoY*Cm-q;9g5*V?gHZpJOM3vz)>o5E7=O!175+gOaA`T9KVJ9$bPABy|40hJ zME*a{DdRFm6<`|_xw!BR*7-{eGM-g5xBn+t)EYK!Ro49KA*cVO)p<*usCKC%L=RXH zi-C<49+*Wx0U*@0n?I|uM6dSeTe!!?z@`+eH^7buJj?f4UVv9+^Rw=rgzHt^$xN+X zZH>*$PaxT!X8-GN{|FenEo>0an&oVr!^6Tw#$h74+A1AI$^&c}(0JI6wzvJI%=LP$ zKcyBR?YuCW#j9^z3MR7#vdnorj?KJQ<75E9leJ1N;W>%?L`m)Z8vv=Top1JUF-)OI6iB(YKv@!If)z2>-pub^|aCSbyAYEnt+hm%I{;gr&B7acGfducVratT_ zVp@mZKWrK!v>rUEYXA2luM^!_t6m^)E8-dN26nUhQS7kaEP^?8epey-urlesC+4~7 z5xIl9%{qhtlVqT@VVe3l+Z2k-#t&7P-~00v*bGb*A8 z>RG0n7DPy;r~d~m_7{?R8&`fPt6h1{LbztRpHc5R+JFn0$rg)fX} zfE?R7Gds#Qz_x4fkgr2SLordWODS&z7DwV9`V9Mk#(~cQ%mz^CF`rYW>awBVt;^1I zRUyVxzL74=t6y7$R)67}=nJ)=Gs<)={9H3bVt`oUy)N~V<8RQcR{*|?(|R%o`uG~y z#bX!P&BN_LTRaR3!a@Y_QDNJpISDMGRYUS`Ul563N!)rMn&01Yb%13Gy+P4lpv`f3?(4apwdW-fHca` zF@O?MQqmv@sC0LC4oEX}NO$+S$KZX>`OaGB`|_d?3!~YKEa_%$2WWwEfQ~QV({R=D8=SYJkSOykPe!r$G=K+ zJN_2$Fym*r?Etv?amTaE*y1bt9aRQxFltU`C;oM7P5DAo6f8sP_eW8kH%g3YqBfWOBtA?CFe!ah-dS4zP=dn9 zaGdBP1N-~X*HQuc(G;(KFzb>sasv!ldvSv(o`0-=8G(QtXh-vx;NBm!~$$nh)jTE zEijr(>)^Xrl;cii_MF7sc%h`K5%=2!n{C&IIU=t~CwAAfOZ_-yAWL;pfOp-%f}CQ5 z1M!nGb}jZOam2nxE|-3V(juK=5TX5-O!S370h>s)MZPitvERt&V-&U$VT(41kBNU- zr5P8|QiJ<@r?H-Z-+B75?Lb&T53q#L+1w2Cvq2oyRI!_}e1-|Gg0tez>!(9m&yL!Q z|B!y>C)t#$<;u4@+D0hDS}!U^dfasO!mthNVsy>N#yD0Zsq9Ax?1FfSGqffu1|zez zW=@L*7Ja42h}@5w-PBLLRM@m=F+_%xD$};^+6zrowZO>@>D?$SjSJTeIRf3ggR&9b zs#ZFt4QbW-?jrLOQp10E6{%qGbMKb6sYT)+>bKoVe@LlP6FA;~X6ZZ~%e3ZYN^h|W zr-rQbAo)N4z6kAe14?Y1_}-gFQA09Y3jC(!<}ci{62(Vk?ao-`>z$6Xq-qnZ}KS)1IptKi%u2G(DL)DXYyLPKhyO}Ej|=5 zx^meUx(YAQLr%V8XEr^`+Ua4@Di0X9T|Voszwf^H;)KoGr$1+~$D&5}unIvqI8Hrg zo=)96BtZG23xG1C49^q&UZ=${NR~mF|6sEUx%fCsVBx~>*-3&@}*_cHoiMP`*6h#d)1^^g(KZ`-^|JU=G42+bMbPpX!6%esqRwieTsF`P-!nt z{DbcktXuXiaF z9HimC(eKoKF!VDgdSQ%mU`jnhz zHpL~IupBnp+&j73o3~M3o-ZHqF1H+vT`b<(&?-86J*-=fbK@sd#4j#zIu~$(c0}cT zYr`h2UJ7uTCJEgVo zyO8M~_Wjtw1%9K24WV^h#tb#P^FebR^2{hmuT$Yep6M#gaDB*W!dRx7bxv63;Q52K zYP^kHuZ%%VRWvB{U?BKcEE9D?m6=BkO;9_!%Z{>8Wi4rgrk&j2YFF#bAmHXl*Mg6@ zdh?CXSS3W~DZY?&>?i$oQfA&%&l;1FLf4dv=H+sekx~m+PMa#3MknHGs+Ec96}LoZ zwYzF<)UR28#W+xv#6kS3ZqmrnXxhwZ7S1%5Nwmw(i}L$3P1}n+t^kCz2_FR$l2R1- z%@W|pdc0pVc+;!#H{1b139=|D$0oq?H*f+$2?($LU5}0b;Sgw$6;Q6c#V>#3G*G!P zf@sc58vl>eA1n$S6-|xFVV41y4%{z1efY)Oe=$7j8>=sP%Z$qm< z>(WODBvS_(f2vyD7rOImwuM~|tpQaSk3HM}T(JmXqzXAVI= zt~g6QufAyu2wMuwvdfw!1IVV&(Q$5vbFA+yz)O(g@a*d2*M>DfJHt*vF=5?+!|S|n z9xLdWFSOlC4;?Z3H1b6{>_+PHpq5n;=sC~=Qq;BoL6oA55iZ|klW#ecsWD3&zmi?i zeOEPek@(hWa{umPZv%@EGJhtQ1gIq%sXNi_*{D9r9>~%fVlQ8$vtRoeYgMceFL29$ z1Q<=}g9eMQKz7o42IMv4?{O(#1Dd!(B};RFMH4s!+F9la2pvv&@xJc=>T=M7N#M)K zX|+c|emY>tYu_}sD0?!u^|>f!lp3OA5tzX@&eWGKflNU~ZSa{%I*8hPf$Y+bVtBGlkdNR*smA1N3cHWR{5^nwq`gWcA7V- z3m`Eh7;QO1O^55JFw)~qBRf|BiS&d|H^nQKzh}(gPI56}rOM~qV+4@%S#FFye5)4j zXHN9v1>vwUxqYe{s)XJKQLPjaZ}T1U6a$rn36~vt2%zGTV5mgoY)O$_LH)fj&Mt<3n1{x-gidyTKSVOSaP2M5?=b%0-Q3M*1d8 z=|^_y4A#m(#wWlx!?lCYuj&clr!=svZW$I}k&C@L7CAqhJ#cDla?3G>{eEj15=8TE zAU?ik6Jl1t6WsIwfQ}Qz0Eve>?v+Rc@%?6fzu)9cLmF?!+&49# zaMo9erykr{h!X66kq%ieZ4f@&eU1c%AJE^V<+}Zwz@$WJN*3fQY?j!lc0BrW<(AnQhcuRBx@Pf+`Hq?Ij8REmr*K1+94{?d!nS(Kd%nk1S|21ehY0 z>DK`A;T92`NiJ%NO=f*2N8|<(bm>QvW}(USd-OczEgk0YBG3Wi9sgSQFOU#L4P{hf zYbR;|bVb&i4d{%J{JDE$E5Ts7omt6U+j5`@96UnV0_v_ z!JMhyx;Lb|@qQtyuhDu8$wOq0?&KSFkevJi{}Ybs5IdwN^G$U_TS9*3Y_kgc4d|PX>K31*90snuiHS0&x(9mLcz#MEsS`K@n`Y;y`sH~v zfDF%~{u^Y-7a`gtd{fi*SAZCiPEuzIU=nMOZNYvPw+TnxRQD$unmKB5Rkx|X_a>G5 zkquq;?*t0KAz0`ywnN6!_9o9i!GRg+0Q+v1Y4Y+&*N_C^BGRZd^mh;fcu(meIKHu3 z6ss@8MScJ{_1QYC8NdVn8#P{Lk(XMSm$$%`H6o{Fvj2t@(P<<#OCfS}iCJ5@!Iksb z8X{i&T@kFF05s=5bld(%n`4B8c^dCo%PyDrjvi3T)~9n?fA3Dy0j}Jhe?!goa*2O| zOv&ZlBf*u!jc#SnT?QuNAn-Nnx5BJ0@6HCU%+kUAqU|#1{Xg8Gc9QN27v%obCeY_X zj7zE>%-=k5o3>Z2dWHiz4`CntY%^3!pk8hI&Gl?YJvRRk0;ySJfJ*JX>R$WuZg$P< z7QOjOP#3iRGt(%G&t5lC^H>DWYAl-WXs~SB=y8!F6PjFV_leNTJny`ZIczll!oXlU;X<8BeP$q$Xsi zJ<+26G$(6@B_dP43VQUT>c4_LOk$3g+cQ>cb-^3Zleok(^E3-YURU?S%0$RPlVZx6 zAN*ozpEUvjw>Z)$GK~g;0(4TbivS-oIUKeJzw4;|_A?d&m}^Y!Af*4N&7 z%Uv7y6l}_o0*@8tUjmO=p4X$CB~{fY_XaBmr=d$33y95c;_=&sKzE^FBgS7*wlEG= z?}{WyW94Okb7D+2Fw|}F=INYE(ERY|y_HP&zUSua?HVvKzKQ0}TSk4KPW9C6s%d|V zQ%{gZjuQo)y9{)>ibTv>X_Z2Uu}qoIIF;Wwne0WXuvZPR4t0LA3{UEU<*NYXd}6R} za^lkc1iJr}RQ-dOCStTJBmU(204`=WwO-LdjztF_8(yZOmido*%=OVa^!~hs79ssW zGv2)FEO?3L3AH(^7p>^aAD)=J7H!04?ov!EP!q)rGWB!;DqByXqoammi1DXAfyxr0 zivgO7(pjTocl>Gz5PT-V^K3fV@jO)ubzd3k69K!TENMJ1`3Rx4(zNokMpVHYMNF^H0J$P z%sVp`QQV)e<+;HTk!xDE8AtFr7jt zblzN|*n~(99VFeHYXu5Dg;sMZ&uCFJ#P!A_A3zbCHKnv=8J@Id>R|;#=$0XGg8AXy z0^2CgTe0b@XQuR?(o=?Ojjx1O5u!`_>-`5ca<+qXY#Qo?gXj%bt&~;kC5#VZ5R(&S zVc(aazgT2by@|5&JnXj5NU&^8Nsb%U!n$+Q z3TbIBJOBAzGsKib2~wj9bM*qpgeyI8C?ROYLIJh=LaisK1A|uMdZdNqdhs!F6g=Za z-znz0bB^qMvzOfl+=O(D9^4(sKRFh(lpW+K)E(?BaCdX6d5|^Uubrcj8hKW^%%oN2 zsFHLeT4g-v`}2gs>+Qn)azg;NA^E~Z6z-Ofo>i{EOjK-CpjoEBNQjySYyT6K=r%W7vz z?ng;Y4zrs(o7U#X#7fkd9R#_!u`gK%KKCTb@(i%Qr===D5-HT9|8Sop?yd7r53tRjZU)zva}KX#Z8u;OtySI8M2lY3 z1ol{LSo;O43c;D&sd8{FAr8?*qDswKLfN-n3wSvjG#4LJmWv)>`@8by9xMeMbIm!P{Q^mrmPyADN69@1uf|_6OPrVwqvH`Ei$Pdi6VQ3> zGiW|N=2oE>XLd@CmP1-v0v7@0*cTnoPbA}4`JTkDvB@ODsfd_GPZ0Og$5)8oPdwQz z=$5Y#kR=|%y^;B>Bp#ZG9h{hO&&{x8zmg-jJv^@?G($?BQ&4T0GU zek-4TlbK9~X_XN8={kdbph}2TrI}l*;U~S+*{6F!d1x@(pK1bB&*%^0b=W;_T(Ds6 zQ_I1%jhr4YteE#4mC0-l?KJaq`(;miEG$e<*-_}Xk?c?3_c<_f#Omv*23Wz5adX+@ zV5WH^ZQZ`?Gfz0Fx1%KdKtx!{d+@qU56tmYd-&Y;M4VGuxm@}DeAX(s`k}+kx+{oM zSMh`a+s+T=`F%M1=|tzRV!futRhQqrGjw0?|LMB=<&}_EM=D*86h=i?qid!_rcP$L zt{p;zEMJ20eA}Xy%jz{5?&C`-UzavlOB(86^(MBw<2UB*SbDe}LL!4^RvS@mXr}t> zi2!tiO`I0x*y!ExZzee32kfz#c+It}{Ic^5cGp0p$C6@B5i~pK{EXy2cx#O-F5$ zYyPuJ>A&-){IDyk*Y6nHXA{n|H0F;vqIJM6MsIcoi?v4{Tj`!vnOc>r1XXDr;D~kO z#Hxe@dbQ1Gdb;d)r0-;c6$IvLES0|A!7BNYr=2nz5D|3hej>Y`OKi!lBA8g8&vCIL zYhZsNuwbsH5G4MLF}I(we1CYMtii6!tfVlyBb{HwT!YwSwPRHqW%d*h6xSsGEXx*C z&(F|}tl^0qiEfeUI|P*y%Z4>`w$>1Tv!iD2Pva7!`iIZxqh1894p)s!o^b-4reJYC zX@zyH5^|BV(Fa}gE3MFP2Z``N_mU-WvXGC?9)7v%Jq-9O`b(NW6-7YCQGe& z=?lzkt~vioE*szH<|7x-b3R_9HVNo~={`Hm#_b$&(|1<%&!QxzhNs>oSH9TI#;=2Hc(Uy?rR8*o>MyC=lbQOR=l%x06Hk}*8dVYqF5ZI#_a*L^0- zbIvx|?BrOcC0S3G;m~BymwKYexZ0sHrr&&-L|boZw&3yF?F9UT(Tn)K@U>&R;r%cT z`f>lhKF^Xxdx0;p9YwD3XE~2~CMDaS3*Y%uyv3JKdh$Vdf7~<2>p)xH{u6G_8;8xM zm@!h=do_XW{NAgS7bYAo4~Cns@5HE+xT}1sDf<*MvlY*BVRH@_q_a;hg&giNRbp*# zD-1nWePqku`#@CyYVKn2GIjPt9b%GTCXYkvX|6~nl(ht9yQ9qTT8I8-bwT@s)#YqH z0YN{&GK|uA=4z(iwybf=&DvdgQS2J;dS~k?1Wcsx6G(5UlA8hrui*}^VMvK$Vd!hc zCaKi}V$*S^=a`%AFz!^@h1j?79u0?jVcC)B01C5ux5~nuAZEtVmKfQHMNQD-hO;iq zO6hv+?|Bw9#gOXK!Zn!eBQfMcfi=3Gv6|9%ujBcB0?OY=q>;hHh$Rvj0%r5{^51O2 z6!BAi#901*GIteRqG9PS*8YvYh>Vbm*W47mCw|$FD+28+bMk`~au@@i5cTH$MGqbr zcYuzCqDp6EIq-f?H}OK-p1+ z=RFZ|u8qv>I74x1^V8UB`~p-wbuB9BKOJwq+!~Yq45BkJ1FXux@%EpdEE#kTO~_vV zbBm+xd9FTU_&Dgmu&DL8stQ_=GSqSsgiclq2rm8yE({j>geI1lX))za=qoT>MD!KG z@N9eE>Cw(wJvK)nFnbss<^TgPw4IhxkG?Y4+Be|v4X=(ASZvR={Sqk}ebRmfgp}Oj z-;vE%&B4@G$PTWBgbvro7ht$Vmg~v>;CGb_mxEqF_`yQOX|&iB^k+t*ep(+FaiBy? z=>_8C?DN;3z`|6Ur9+%SY7#{pb1<^yo0D zYYVz7|C6>r>Hln+N|TQzsSez~T&V;N1B&z5wsMt|T`aee%$?Bxq$5B8ca6p)AeR_# zv{BG=EkB(3F#h1D_#a)M{*aDymbOjPO{AVP_TT~mA*G?r5DB_ay?a|Z@?MF7N`Dfa zyhch$lvJA$@lt&fIXzZyxU_6r>WL5+L8v6wM|{H=X78u@%9boFC3gnkVBOB2pu9FMU?hTi29_Q+lle2`K(@ig`;PVH-emeT0jlr&^WxB7ZmkrLd ze!^HkM$!FViDwn^=ilbL2Bm5VJ#@%`pU61drg3`{kr-5Jzjz+VPWl8>7a35uAsjqB z&#kppRU=is(w70}^D?7f@lG)PkZEu7V{X^ux2An*fj6iG)!(G@T8@$N1fYAgiFWpJ z@=1B1{{D!`)6DBv0JEBv=TM-1?M=OwTX?13oS9bsbafvn94i+}};9=K0 z1E-Z}AW(h+#6W4%v%_jYE)y-{UM&=nczKUIA^0!@0>xZ>SqvIAfN5loZpTeJ0(OaH zPVzCkjYs?D9M$%BBsDgK1MdTJgZt?1lWiRT`u{XA10qGt(Y8M6zk4E8i;2PlceG6& zx=#m0IKDH@MX6sXAG)2E&QW#TGaZK}*HmtNup(=%J{&HXpTkrVWPr0Ps|kZMteCdWX`Ega;<&$k{7y&#&vC3y;GPVmPt5Bc8Y&1E_EFdB$!*w;Ai z*IzhJza#bRe#`^8;wC;w5r*NJwyI z-KXA(#q>0(U#$VA)aRj;nA)9TVT>LnH$^#dZr_*6?WWLS4DuxXP(UmM1HFxyk>TIK zkfPzCx1gUjXOa9Bw#*g9tOKAGa)-0!-I1i3|L~IW6%||o?JCXjt|fmx`GC+ec348A&{p3wwdprZKr{ESI)&?v(DEPTR zpBoMgpAxN)!e@%^yJXF7;6=u1R6s2fN))|w(${w}E||?n@*ls)qh1esdbPlNhDGlh z=#n;Q+FP_oldLhzWgaz(zcYmb1Hz$YavYjC{ANArd$2simgU!s^e=|M z{3SqVe`8#tmkXRb18sh3Nqd|3A8ipQgXG=N!UVw?#}G|5flnFIq3-ccm;z-pF+`L8aPIf!{S?c%#383 zcKyATy-`8!oWa2nQZLE#ss?S%$gWco$!Kb7auvSdj+H3IRkWXO`vIy~CnD7F zv(lF+(l$9;3eVU`VczozLe5wT0!m#a!p(KcA^u>hf@85V3 zp)x1LkzK$2=?LjE0~Q; zN&@G!RJ2zKtv}XoPB+X)YFT~+!guwq9YMt)YLWHiZ%l7ap4bt|3s=*8U;uLBS4IWr0Lxjaw@!W`+jSQGpGw~Z#LjzSK>$Y0Ffgjs9gm) zIditzua9YT3#_m$0U>|@@2IwAP zdr_(sJn1kSxD7h`j}{c-9zF#+#&ks6Fgk+0VwHQ34Aun%ekpdKBS!87d3sN-gFh~E z#a)A|fPV8mvg7lEN%szLgnR+z%`^@O>QD+CC|c7KuHljIe-^DoW^dHky9L17!-;L1 z^mqr1FF2Y#0E6LJ4Tq0Tci=1EeDJ7A%;*mr5Z=dJY0MfL@^_5EJ`jPh_N~~1W!0za zXvA;D25(xp7M>C;?62*#0U~;RNh**zx((ZbIc&%PIqx_hq1Fx-zjynb4#r6U9p*qgB62M4rpsmRz2x@>KQY_#h z++3+6b@&8a4mj|oKnq5K|9!cAV0jq&S=^^R+Bo4G@5`6U-ESx*Fh(D1;j`eI0Y&mB zaL+dIfC5i(mllTrauFq$`K67N~vhFAjA;9x+PWF=a5G?`Ra{&0(4i%WYA2wiqhiLNbvI^F7vP31Fs#hQy@cTPD6)A=kcYr5 z{=Ck;hKx34;S@ZHq91UNNh?0r$=B~Q1np}_cJ@z&PtE!xFfpMT!UOWOO4 zL(jGgIsSkgUa(S3kkWoPU(w_QAhX>7rYCSNgXWrZVB^)lV%!-ueE<{>X%lA$lQ}0_ z9lm^U2l%PQ3EFCa-d+nH?`N{Dw6&y3VA2s6RKIJx2?dqfBWmZSz+DxX$ zW)rw(%m(S@bu&L&ObM<88It2Y%|az-8|o=^sOHI2D}a~;>8Eye&^Sm4G03m}6hza# z$VX=gK|Ve6&Q6^YB%eHvydoBcfem{)0*Es{b#}__9qT=b<~G{)zM2Cx6GC_=(*v+{ zv;+bNPiyU$8U@fm%h7WB;Ax1MGjb$i4Z_Wbltt$VSR*vb)f=n`p6&Lf0;T(FF9H?= z?--{%1<2a{q8yO0OON;nveJtVU|+`zUW@Ib^}^W<`CL2u4h4@wX0|zSkEliuPb|I{ zO{WNyc9++&Zr(lgnU1d~8}(e7UV}V+q>kG(hz6Nn(<-w~rFr|F+M9E_&7Zw>ORSZ2 zElHP5*4e$WfXLjkVvV=v?u+1jUu~-RWvVL@_Wp z%}G5Uy(E=kgaJYLV|-O^9a`!u_1vw7d1$m1t+t?VK+>}L43`$>`GQ4mHwuHRj$WFQ zh}{i8Ml;XgU4Mp}B3uZxEbj6glZx>;A14?6%WE4$w~XUH?={EL z5G+3u!WRS(HIk#|^jQ`0(THa;O>xK#AZ#)=BfKwY9u@Kw^hdss-{vjTc?PJ0@lU?4 zCgl5cv{WJ(cr#8I_%4#!ecp|)&GQemGX{EFI6x;hqvk{0!P1(qfl z@wCvb&Df~SFexpZU;T2`OM8*YJLw*VxIQLIuEZhkX9=xb3S#d@yEE(Gv0NdZ&v~RQ5Nd@!j+Sj4bB5t>C+T zoPqDxVKJFE#QG*me2`ZPu4dPi8=1wI;~kk?pJl@I_PZjasDE32{~fB-)E9!<NidnWIoufjp9Sz Date: Thu, 24 Sep 2020 14:11:41 +0200 Subject: [PATCH 02/16] Update docs/guides/shuffle-sharding.md Signed-off-by: Marco Pracucci Co-authored-by: Christian Simon --- docs/guides/shuffle-sharding.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/shuffle-sharding.md b/docs/guides/shuffle-sharding.md index 9cea8770f63..94f95e71b4d 100644 --- a/docs/guides/shuffle-sharding.md +++ b/docs/guides/shuffle-sharding.md @@ -9,7 +9,7 @@ Cortex leverages on sharding techniques to horizontally scale both single and mu ## Background -The **default sharding strategy** employed by Cortex distributes the workload across the entire pool of instances running a given service (eg. ingesters). For example, on the write path each tenant series are sharded across all ingesters, regardless how many active series the tenant has or how many different tenants are in the cluster. +The **default sharding strategy** employed by Cortex distributes the workload across the entire pool of instances running a given service (eg. ingesters). For example, on the write path each tenant's series are sharded across all ingesters, regardless how many active series the tenant has or how many different tenants are in the cluster. The default strategy allows to have a fair balance on the resources consumed by each instance (ie. CPU and memory) and to maximise these resources across the cluster. From 5b420ec7491a0820a9f47d6c15123f93c32339a3 Mon Sep 17 00:00:00 2001 From: Marco Pracucci Date: Thu, 24 Sep 2020 14:11:49 +0200 Subject: [PATCH 03/16] Update docs/guides/shuffle-sharding.md Signed-off-by: Marco Pracucci Co-authored-by: Christian Simon --- docs/guides/shuffle-sharding.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/shuffle-sharding.md b/docs/guides/shuffle-sharding.md index 94f95e71b4d..449c327b4e8 100644 --- a/docs/guides/shuffle-sharding.md +++ b/docs/guides/shuffle-sharding.md @@ -16,7 +16,7 @@ The default strategy allows to have a fair balance on the resources consumed by However, in a **multi-tenant** cluster this approach also introduces some **downsides**: 1. An outage affects all tenants -1. A mishaving tenant (eg. causing out of memory) could affect all other tenants +1. A misbehaving tenant (eg. causing out of memory) could affect all other tenants The goal of **shuffle sharding** is to provide an alternative sharding strategy to reduce the blast radius of an outage and better isolate tenants. From 29fb017c071994c0f39e612db31a9be5baf9c750 Mon Sep 17 00:00:00 2001 From: Marco Pracucci Date: Thu, 24 Sep 2020 14:12:15 +0200 Subject: [PATCH 04/16] Update docs/guides/shuffle-sharding.md Signed-off-by: Marco Pracucci Co-authored-by: Christian Simon --- docs/guides/shuffle-sharding.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/shuffle-sharding.md b/docs/guides/shuffle-sharding.md index 449c327b4e8..1817a7492cd 100644 --- a/docs/guides/shuffle-sharding.md +++ b/docs/guides/shuffle-sharding.md @@ -22,7 +22,7 @@ The goal of **shuffle sharding** is to provide an alternative sharding strategy ## What is shuffle sharding -Shuffle sharding is a technique used to isolate different tenants workload and to give each tenant a single-tenant experience even if they're running in a shared cluster. This technique has been publicly shared and clearly explained by AWS in their [builders' library](https://aws.amazon.com/builders-library/workload-isolation-using-shuffle-sharding/) and a reference implementation has been shown in the [Route53 Infima library](https://github.com/awslabs/route53-infima/blob/master/src/main/java/com/amazonaws/services/route53/infima/SimpleSignatureShuffleSharder.java). +Shuffle sharding is a technique used to isolate different tenant's workloads and to give each tenant a single-tenant experience even if they're running in a shared cluster. This technique has been publicly shared and clearly explained by AWS in their [builders' library](https://aws.amazon.com/builders-library/workload-isolation-using-shuffle-sharding/) and a reference implementation has been shown in the [Route53 Infima library](https://github.com/awslabs/route53-infima/blob/master/src/main/java/com/amazonaws/services/route53/infima/SimpleSignatureShuffleSharder.java). The idea is to assign each tenant a shard composed by a subset of the Cortex service instances, aiming to minimize the overlapping instances between two different tenants. Shuffle sharding brings the following **benefits** over the default sharding strategy: From 429d139467f5d4f74547795138198143437ff40c Mon Sep 17 00:00:00 2001 From: Marco Pracucci Date: Thu, 24 Sep 2020 14:12:25 +0200 Subject: [PATCH 05/16] Update docs/guides/shuffle-sharding.md Signed-off-by: Marco Pracucci Co-authored-by: Christian Simon --- docs/guides/shuffle-sharding.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/shuffle-sharding.md b/docs/guides/shuffle-sharding.md index 1817a7492cd..b13c526efbf 100644 --- a/docs/guides/shuffle-sharding.md +++ b/docs/guides/shuffle-sharding.md @@ -27,7 +27,7 @@ Shuffle sharding is a technique used to isolate different tenant's workloads and The idea is to assign each tenant a shard composed by a subset of the Cortex service instances, aiming to minimize the overlapping instances between two different tenants. Shuffle sharding brings the following **benefits** over the default sharding strategy: - An outage on some Cortex cluster instances/nodes will only affect a subset of tenants. -- A misbehaving tenant will affect only its shard instances. Due to the low overlapping of instances between different tenants, it's probabilistically likely any other tenant will run on different instances or only a subset of instances will match the affected ones. +- A misbehaving tenant will affect only its shard instances. Due to the low overlapping of instances between different tenants, it's statistically quite likely that any other tenant will run on different instances or only a subset of instances will match the affected ones. ### Low overlapping instances probability From 0df64671eccbf0725adfe72089e85f17c06fa799 Mon Sep 17 00:00:00 2001 From: Marco Pracucci Date: Thu, 24 Sep 2020 14:12:32 +0200 Subject: [PATCH 06/16] Update docs/guides/shuffle-sharding.md Signed-off-by: Marco Pracucci Co-authored-by: Christian Simon --- docs/guides/shuffle-sharding.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/shuffle-sharding.md b/docs/guides/shuffle-sharding.md index b13c526efbf..b763d8cf6b0 100644 --- a/docs/guides/shuffle-sharding.md +++ b/docs/guides/shuffle-sharding.md @@ -52,7 +52,7 @@ Cortex currently supports shuffle sharding in the following services: - [Query-frontend](#query-frontend-shuffle-sharding) - [Store-gateway](#store-gateway-shuffle-sharding) -Shuffle sharding is **disabled by default** and need to be explicitly enabled in the configuration. +Shuffle sharding is **disabled by default** and needs to be explicitly enabled in the configuration. ### Guaranteed properties From a0a7fd654a218de8381eb37715ed9d2763b97ec3 Mon Sep 17 00:00:00 2001 From: Marco Pracucci Date: Thu, 24 Sep 2020 14:12:40 +0200 Subject: [PATCH 07/16] Update docs/guides/shuffle-sharding.md Signed-off-by: Marco Pracucci Co-authored-by: Christian Simon --- docs/guides/shuffle-sharding.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/shuffle-sharding.md b/docs/guides/shuffle-sharding.md index b763d8cf6b0..c58b99e35c9 100644 --- a/docs/guides/shuffle-sharding.md +++ b/docs/guides/shuffle-sharding.md @@ -59,7 +59,7 @@ Shuffle sharding is **disabled by default** and needs to be explicitly enabled i The Cortex shuffle sharding implementation guarantees the following properties: - **Stability**
- Given a consistent state of the hash ring, the shuffle sharding algorithm always select the same instances for a given tenant, even across different machines. + Given a consistent state of the hash ring, the shuffle sharding algorithm always selects the same instances for a given tenant, even across different machines. - **Consistency**
Adding or removing 1 instance from the hash ring leads to only 1 instance changed at most, in each tenant's shard. - **Shuffling**
From bf33e308b5a1b7c746b7176f15409cb98eb99798 Mon Sep 17 00:00:00 2001 From: Marco Pracucci Date: Thu, 24 Sep 2020 14:12:47 +0200 Subject: [PATCH 08/16] Update docs/guides/shuffle-sharding.md Signed-off-by: Marco Pracucci Co-authored-by: Christian Simon --- docs/guides/shuffle-sharding.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/shuffle-sharding.md b/docs/guides/shuffle-sharding.md index c58b99e35c9..d734f6d0511 100644 --- a/docs/guides/shuffle-sharding.md +++ b/docs/guides/shuffle-sharding.md @@ -85,7 +85,7 @@ _The maximum number of queriers can be overridden on a per-tenant basis in the l ### Store-gateway shuffle sharding -The Cortex store-gateway -- used by the [blocks storage](../blocks-storage/_index.md) -- by default spreads each tenant blocks across all running store-gateways. +The Cortex store-gateway -- used by the [blocks storage](../blocks-storage/_index.md) -- by default spreads each tenant's blocks across all running store-gateways. When shuffle sharding is **enabled** via `-store-gateway.sharding-strategy=shuffle-sharding` (or its respective YAML config option), each tenant blocks will be sharded across a subset of `-store-gateway.tenant-shard-size` store-gateway instances. From 3e35851cc3a6271ab92d2fce27942ef77b03b294 Mon Sep 17 00:00:00 2001 From: Marco Pracucci Date: Thu, 24 Sep 2020 14:12:56 +0200 Subject: [PATCH 09/16] Update docs/guides/shuffle-sharding.md Signed-off-by: Marco Pracucci Co-authored-by: Christian Simon --- docs/guides/shuffle-sharding.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/shuffle-sharding.md b/docs/guides/shuffle-sharding.md index d734f6d0511..624c87def59 100644 --- a/docs/guides/shuffle-sharding.md +++ b/docs/guides/shuffle-sharding.md @@ -79,7 +79,7 @@ _The shard size can be overridden on a per-tenant basis in the limits overrides The Cortex query-frontend, by default, executes the received queries across all running queriers. -When shuffle sharding is **enabled** setting `-frontend.max-queriers-per-user` (or its respective YAML config option) to a value higher than 0 and lower than the available queriers, the query-frontend will execute each tenant queries across a subset of queriers. +When shuffle sharding is **enabled** by setting `-frontend.max-queriers-per-user` (or its respective YAML config option) to a value higher than 0 and lower than the available queriers, the query-frontend will execute each tenant's queries across a subset of queriers. _The maximum number of queriers can be overridden on a per-tenant basis in the limits overrides configuration._ From 0149e0a64931f03bd732c57cc2fd8bf1953fe684 Mon Sep 17 00:00:00 2001 From: Marco Pracucci Date: Thu, 24 Sep 2020 14:13:46 +0200 Subject: [PATCH 10/16] Re-built compiled doc Signed-off-by: Marco Pracucci --- docs/blocks-storage/store-gateway.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/blocks-storage/store-gateway.md b/docs/blocks-storage/store-gateway.md index 483fff51598..38c15b6437b 100644 --- a/docs/blocks-storage/store-gateway.md +++ b/docs/blocks-storage/store-gateway.md @@ -48,6 +48,8 @@ The **`shuffle-sharding`** strategy spreads the blocks of a tenant across a subs The shuffle sharding strategy can be enabled via `-store-gateway.sharding-strategy=shuffle-sharding` and requires the `-store-gateway.tenant-shard-size` flag (or their respective YAML config options) to be set to the default shard size, which is the default number of store-gateway instances each tenant should be sharded to. The shard size can then be overridden on a per-tenant basis setting the `store_gateway_tenant_shard_size` in the limits overrides. +_Please check out the [shuffle sharding documentation](../guides/shuffle-sharding.md) for more information about how it works._ + ### Auto-forget When a store-gateway instance cleanly shutdowns, it automatically unregisters itself from the ring. However, in the event of a crash or node failure, the instance will not be unregistered from the ring, potentially leaving a spurious entry in the ring forever. From 7b7de93b63fe1c293730257dbb7d2f337874b107 Mon Sep 17 00:00:00 2001 From: Marco Pracucci Date: Fri, 25 Sep 2020 12:31:37 +0200 Subject: [PATCH 11/16] Update docs/guides/shuffle-sharding.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Marco Pracucci Co-authored-by: Peter Štibraný --- docs/guides/shuffle-sharding.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/shuffle-sharding.md b/docs/guides/shuffle-sharding.md index 624c87def59..cfffa384370 100644 --- a/docs/guides/shuffle-sharding.md +++ b/docs/guides/shuffle-sharding.md @@ -27,7 +27,7 @@ Shuffle sharding is a technique used to isolate different tenant's workloads and The idea is to assign each tenant a shard composed by a subset of the Cortex service instances, aiming to minimize the overlapping instances between two different tenants. Shuffle sharding brings the following **benefits** over the default sharding strategy: - An outage on some Cortex cluster instances/nodes will only affect a subset of tenants. -- A misbehaving tenant will affect only its shard instances. Due to the low overlapping of instances between different tenants, it's statistically quite likely that any other tenant will run on different instances or only a subset of instances will match the affected ones. +- A misbehaving tenant will affect only its shard instances. Due to the low overlap of instances between different tenants, it's statistically quite likely that any other tenant will run on different instances or only a subset of instances will match the affected ones. ### Low overlapping instances probability From 556938a8a23131d2bee3438456bf4ca24ae90155 Mon Sep 17 00:00:00 2001 From: Marco Pracucci Date: Fri, 25 Sep 2020 12:32:07 +0200 Subject: [PATCH 12/16] Update docs/guides/shuffle-sharding.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Marco Pracucci Co-authored-by: Peter Štibraný --- docs/guides/shuffle-sharding.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/shuffle-sharding.md b/docs/guides/shuffle-sharding.md index cfffa384370..0d667d98c24 100644 --- a/docs/guides/shuffle-sharding.md +++ b/docs/guides/shuffle-sharding.md @@ -63,7 +63,7 @@ The Cortex shuffle sharding implementation guarantees the following properties: - **Consistency**
Adding or removing 1 instance from the hash ring leads to only 1 instance changed at most, in each tenant's shard. - **Shuffling**
- Probabilistically and for a large enough cluster, it ensures every tenant gets a different set of instances, with a reduced number of overlapping instances between two tenants to improve failure isolation. + Probabilistically and for a large enough cluster, it ensures that every tenant gets a different set of instances, with a reduced number of overlapping instances between two tenants to improve failure isolation. - **Zone-awareness**
When [zone-aware replication](./zone-replication.md) is enabled, the subset of instances selected for each tenant contains a balanced number of instances for each availability zone. From f69197f9f002c4190abf24dd4e1bbdad088e308c Mon Sep 17 00:00:00 2001 From: Marco Pracucci Date: Fri, 25 Sep 2020 12:32:34 +0200 Subject: [PATCH 13/16] Update docs/guides/shuffle-sharding.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Marco Pracucci Co-authored-by: Peter Štibraný --- docs/guides/shuffle-sharding.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/shuffle-sharding.md b/docs/guides/shuffle-sharding.md index 0d667d98c24..0e8e5ccc814 100644 --- a/docs/guides/shuffle-sharding.md +++ b/docs/guides/shuffle-sharding.md @@ -69,7 +69,7 @@ The Cortex shuffle sharding implementation guarantees the following properties: ### Ingesters shuffle sharding -The Cortex distributor, by default, spreads the received series across all running ingesters. +By default the Cortex distributor spreads the received series across all running ingesters. When shuffle sharding is **enabled** via `-distributor.sharding-strategy=shuffle-sharding` (or its respective YAML config option), the distributor spreads each tenant series across `-distributor.ingestion-tenant-shard-size` number of ingesters. From 17b3126f9d5b66ebb75bea3c34eddf203677068d Mon Sep 17 00:00:00 2001 From: Marco Pracucci Date: Fri, 25 Sep 2020 12:32:43 +0200 Subject: [PATCH 14/16] Update docs/guides/shuffle-sharding.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Marco Pracucci Co-authored-by: Peter Štibraný --- docs/guides/shuffle-sharding.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/shuffle-sharding.md b/docs/guides/shuffle-sharding.md index 0e8e5ccc814..6bf4040fc30 100644 --- a/docs/guides/shuffle-sharding.md +++ b/docs/guides/shuffle-sharding.md @@ -77,7 +77,7 @@ _The shard size can be overridden on a per-tenant basis in the limits overrides ### Query-frontend shuffle sharding -The Cortex query-frontend, by default, executes the received queries across all running queriers. +By default all Cortex queriers can execute received queries for given tenant. When shuffle sharding is **enabled** by setting `-frontend.max-queriers-per-user` (or its respective YAML config option) to a value higher than 0 and lower than the available queriers, the query-frontend will execute each tenant's queries across a subset of queriers. From be9294bc83a23a1e1dab212090df1cfe2132bc90 Mon Sep 17 00:00:00 2001 From: Marco Pracucci Date: Fri, 25 Sep 2020 12:33:07 +0200 Subject: [PATCH 15/16] Update docs/guides/shuffle-sharding.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Marco Pracucci Co-authored-by: Peter Štibraný --- docs/guides/shuffle-sharding.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/shuffle-sharding.md b/docs/guides/shuffle-sharding.md index 6bf4040fc30..a276176c921 100644 --- a/docs/guides/shuffle-sharding.md +++ b/docs/guides/shuffle-sharding.md @@ -79,7 +79,7 @@ _The shard size can be overridden on a per-tenant basis in the limits overrides By default all Cortex queriers can execute received queries for given tenant. -When shuffle sharding is **enabled** by setting `-frontend.max-queriers-per-user` (or its respective YAML config option) to a value higher than 0 and lower than the available queriers, the query-frontend will execute each tenant's queries across a subset of queriers. +When shuffle sharding is **enabled** by setting `-frontend.max-queriers-per-user` (or its respective YAML config option) to a value higher than 0 and lower than the number of available queriers, only specified number of queriers will execute queries for single tenant. Note that this distribution happens in query-frontend. When not using query-frontend, this option is not available. _The maximum number of queriers can be overridden on a per-tenant basis in the limits overrides configuration._ From c8a5dd9bca83f8714340535e23cfc6df9a53e32b Mon Sep 17 00:00:00 2001 From: Marco Pracucci Date: Fri, 25 Sep 2020 12:50:36 +0200 Subject: [PATCH 16/16] Update docs/guides/shuffle-sharding.md Signed-off-by: Marco Pracucci Co-authored-by: Jack Baldry --- docs/guides/shuffle-sharding.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/guides/shuffle-sharding.md b/docs/guides/shuffle-sharding.md index a276176c921..beab40a8b5e 100644 --- a/docs/guides/shuffle-sharding.md +++ b/docs/guides/shuffle-sharding.md @@ -35,11 +35,11 @@ For example, given a Cortex cluster running **50 ingesters** and assigning **eac Randomly picking two different tenants we have the: -- 71% of chances they will not share any instance -- 26% of chances they will share only 1 instance -- 2.7% of chances they will share 2 instances -- 0.08% of chances they will share 3 instances -- Only the 0.0004% of chances their instances will fully overlap +- 71% chance that they will not share any instance +- 26% chance that they will share only 1 instance +- 2.7% chance that they will share 2 instances +- 0.08% chance that they will share 3 instances +- Only a 0.0004% chance that their instances will fully overlap ![Shuffle sharding probability](/images/guides/shuffle-sharding-probability.png)