From 4acf581b8fbc1d6c2b3deb6d45d70bfbc67b84c8 Mon Sep 17 00:00:00 2001 From: Alban Desmaison Date: Fri, 11 Dec 2020 15:31:07 -0500 Subject: [PATCH 1/2] First draft --- RFC-0007-assets/bidirectional_lens.png | Bin 0 -> 49560 bytes RFC-0007-forward-AD.md | 308 +++++++++++++++++++++++++ 2 files changed, 308 insertions(+) create mode 100644 RFC-0007-assets/bidirectional_lens.png create mode 100644 RFC-0007-forward-AD.md diff --git a/RFC-0007-assets/bidirectional_lens.png b/RFC-0007-assets/bidirectional_lens.png new file mode 100644 index 0000000000000000000000000000000000000000..cda842a36fb3dbcf3ff4c3c712f90597147c8236 GIT binary patch literal 49560 zcmeFZ^;?u*7d{FIGN?$2fOIP%0@5*br?emqqafWmfTY9_Qj$uCbayGu2uOD~NDVc^ zoQL;)Kc91*bDbZ~AMpKwi|cxZdG_9Guf5i~?{#k?G}PqrA3S@2fq{Xqs37wO0|WCG z_!$G;zk7mN^d19)4nt8!Qri=AKLdAK`{zVjmLYzyLO&Pjdq{VX{PVj{zjm6XcF z`#7WF51z5Nl40iI1t}!3wJK`~kHszeaE-w)kS~R9MvUE;s*VoZe2y}`GH>P=3?a9? z9=DJYA(#9VeID#bn9?1Dudp8lmP%&pPk#CHzy?H)6&gTq3j8wfaftrUhj)L|$ABGm zSzTE5qws&9h6WI|*Z%L@n9_8Sgs)cN-v{cxKl|tFa>&X1|Gu6>sv{g6$u{@-t2g#z z8qw#??gvV!NGW8f-wi=O(#j_fJxx7Uv&Zt`_lJa<-CEh}n8laMdPfhP`hUC+mGA~$ zu1+w5NFro(T<*Xu(ksuLckIT0Q)Q|6Lbo*K374_a%ht0uGj(=`;mVow-w&q!G@kj@ zDh{+)soY+hBM-pQQv*>rVCIVrirX3#?qkJ?&AnWu_EdV zfwi_%1LspFM{#1oM@#(n^C^7#Px7-F3-;r27mefCS8hTA6ZfhiXR!M{CbyQzBh&$+ zOFM0WQUfs1y|+$D2SXCe-@<4<$D$}8>Gb8Th_Bj3Z)64i#}rv3h@gw-uTAG;p1*0n z%CkE0A3PuX@LQLH&nnrmO>;}t8BQq_)R{nAI&jlT-()_}lJ)(S(*2#&C@*EZ9@_8L z@kigukGCk)RPDA{3&NR6syh&+GxU&gNRf9x5^6zRRG$-3(VycRB51`g#wBOk z_Lhk*PD(%F{)o0V6#3pFhm_UdvnV)DdtGM!4uIk)EH%$wO-Th^NkT=b34*+L=_ zN7evOw1WTcTv@*)umQM{HR!R?htyd!vty5;-%Kw5b`!{lLr>QGL;Q{F#7sL+-Z!7m zzZUr_cK$~z?mA7_%_e(gsRbnwe72)*Q!B6(FQNDTGp|EWmU)1S53)7UiE=e_MrG%rKDViDerI4+1spG~?b6T;BkRB76+FG0L+rBVw z+n-iluGGq%faA71o}W9V;m0{F9}4IyH2E=f$sn^X7Kl5ESJmqgxsIHjhiU>?2I@%M z$5fA>1_jT%ZS!~fOscX^9wayv9L9?;*(XO-do&j?v^!PofbGLxAGH}2*DugTi#Wu8 zBXJ+dY#l`mEP$!M@KIj)P#&$_v+}7J80d2vDW7}d>tlhPPKfT~I(ArS(n;blShe%V z?ZBtx?<>^Iv79KrG~P9=`xqB^4_9fS$xY1X>|zO;#aeuWIEZX<>5ZX|+Y_*xWwPjv zInNZZ_!ZUXbb%`w{`930d1I&`PLaa7uzi&8Jzrd3iyf5^`dgV@rbH{B>K(HJc}Gh+ zNkS&&Pzh1&s;`$mQPVFy+LG2ZOJ$I6GWg!02cu6eeS~7<(dXv|3u$=>oDZEZ?FYkqm z-|QET+WvT3%P{4(`=?)Is?tKf@p3hivA5ZE$5{q*b0mX(>YYahavuVYG%9eyYN(fo z(a6u|;Mz=;Q8yN-rtt^iH~U_l#93UY4|25|U0)m>EreG5t}v4p+W4aNy^N83G(lbb zg%5(sV>g){Id(hPS~KM0E=!qItlj9mp&*7rZh>KJ1(R8a7l&eFNZk}+MV-{&UmCQi zO-id-bjqu?La_r&nf&Dz%PcaoT-Q+0!hr#WTHdeaqc7!kD2{0ISwArB>bC45d!2l% zzIcHo;+4KO9#HVvI!Cm-?M z$8qS}t>a7~m(9L|`X!o+XNN1kJS5}8hvD>|r<}`)Y7L7=)`x1v6s6`rKQ|_g{`{qm z4Ed2@X}%@?@JH?(G$4&xnoa<_D*kd$m`zUZxR5-#baAU_OEY0dUc0ehIJMCchHUUH zum4aHnT4Qv0(w`@esRL{VbZP|qGwoVo#i`g;j&IOrGm=33Xou;#{HNG_m<0Yn6Zs? zC9mMCa>8c;DW`^Y1mQf)^*-NSbRQAgl2KS6m8XrS+G?>acaKLB9HV=g3DiqSIOnFmi4w8epY`r>b9HN6eRijaH{T7QMDI#@hUR9DMdG zd%tIXoBHxXOK#EUNNi`Ox*(fRaIdubds&*eUsD^yi<`hT4^KuvRFe4t`gZdH#f`M{ zBv`^C=iFl@giLNgj(Xb(!TGi(oXv!1muWXp5&6kr^rv0l#pp3p{3V-@wgO?F@G9+* zXZzxL)<)HPtFykZKPAo>i&3)sFSrS7jZf>Rk_Xd&$`?x*=*yIM<}T-dICc5%NE)J* zkVbZsA&6BL4V|tqbE16`dC83{v3c9`{50s{%Wy(sCMtHxWu?hpYLE1xGy$$%9`G^v za`}^H7ly6%C|7Ccr|?Ch&B}BH9BiPJ|c=qNc|oB{>mk7 zB)L>r(Ry(K!rkRjDRs}ZpRf-ct>ceqytMlMsSS?oY z(+qL&EQLn7hlQ;G+Pj(tfylt!QnBcI(fj5Vk;0ctb&-SDo&7I$%`0h{bJ)uoq?}z! zn~&OV7^ZZKwO{Nyy*yrHtoS%nRd|YANok8jk*JG0_3%-70!C$+YQ5eq;Vyr%FPT27 zvA6a&Oi{I6(gQ{y!Bfz>nS7a`!y5AQ`66oijFgp9p+~wurWs^OecbG_HRT{Q7SX5E z-TTZ;+opp5B+e|0ir)z`gnt_7AI0h01C#i+HH(;8oeV=gCd7&$2Gd+0$J;`V7F&H< z23yv&@>L=TooA}7ex9>CM$qDR>f^ceXv$NQddX#p`5y2Y3JHp7qABYML6bqUKOP`I^8C*% zh`DG(gw$00e};;2-*8f3dC|w}^nfA#nT15{H=WxKSl3Z0>AX@Nn6A=M?vzh|w0V>d z9tiOJ`Ay;eh;gMEIxrSVHn`gvof6$>Ly&H`i_QfJt6G!~Co7!|vme!Eo!#Jv1TzMI z0k(HMWLfX$f z;isf+>9^~ZEwR=M#c{V&B#`(EPM+fkCDE>KdG!}31g&#|r-<2%aEbPvLTfBR>$riX z^tV>FF5e9+GcSOu3};I$|Hy0)ZcFXhvH$ux>-tMc*>DVUu1b4P4*D+qA4P$Ft76?T z5Pqy8I{&FpG<>(eqAM*FU)3X9PDdgM(c|i1Y8>5sVrZJU;d<(BiItuBWRB@dF`4a_ zijIY9q@74Y!GL=Nv?Bmh^Rf{dufuxCn{e!NGN0~AM+ml!20*~cCy7ZyuuHIxaM`$7 z&c}K04oa^lme^gN^{yWwR5(A^lHY%bBAkIcz;ef(kI??QF^nVoKYfB5&`Cj`B-p~C zfa^P-^s~x?G79|!QyM>994@TBd?j5Y)^1_ULJY2cPvd#xm`#&E!Q%}RxQ0YW>g{c* zWNb7a!>gRQ71u8qVWPyHsg2k649+W?!(nR(CN%1oa6K=%TAoFBQ6djR}SHY~wVG&0_4sIEHfIXIx2@^P|F! z`1xUzt@Yu*?!SVxoBCM-RwUe>DA5a!tyszOzuw7A_6NX7u3Cn$W-(-h1ih5S6Lj_MZudce@|Y)4&9{}+^Od7jXIxV6s&$lbXoNdiaK9(@<0V!{!y$tMIROj z9ZS42;jz6tV6=DBJf5E#VFCMSpHxa6S6SM(^qC*JToIxeJ%{iP`EHjkS{oU_Zq`$l zAgm$=|IfWauY!N;Y_;n9JYD=SS!$s28kh%n{S^)bEwGG+VRQ^pdK`R8!BJbL3;bff zQrv6of3;Cq@U%|tJ55qB_>uM#uKRR%?#&QO+Oy=PpcZa~(m1KUfK8KX_n~w6<>g4% zQhlSMOjmynmfc>(U93dU{Vh@DE+E_$vCHxTHS~bJ&$b&VEQzR@NkBsF5jajX-rL^_ zx4lIRWFm;Wn%8jVSFWy@Z^!bKeY`CAC0lkrap>nlQNvP)E9wxnGz|cPRyKXhfZHCRfF_zNkfVfu&XIf`Bj+8#u>F}fG}_+OLx|z z&f!<@B1sL?T1Yy;wY(|&lvzzEGolS<*k3rmKIuCzM59JgMy5`O-TW|q%o|2`#q{eF zB>*Fy=075Kv%R3vf^UTA)n&1kfjhnau4eJueSP(lV>{v~ScePG3w0%6C<=46Zdn#M zm2LmA*yx%V#~gtN6a6s8G*ih4t-jx-&$`n!w+MuQ;LOVf46SWpJU_5To@=4m94x zo^e?9J9`A`6(1IROubU(b;)A$NxX@{@x1(}MxoU;dQ; zMeo7XK(TvUp;w#;|L=-A2*bkyh@hgD`wjmZK?(r8()vZZO#e=R21^ck99y~e@A^PQ zuT}}n^4`Y%O9Xg=$d7xaf5rT7AaX2v836eOC)&vVOGd2HoA%HWP>9(7Lq!A=Vlv(m z>K2n0Z4(N5{>|Q>tu5K+F?D#%hjoLXjlUX1D4{Dwaz9L|j%AOte3-C}Jfm)XmsGY$ zwp&G3;~1*r+Y@!UW|1%oxr3yzmg=!Nahq29oYSv{n_U_)_i!I8w0fVfhf> ze#>tZ`7t`ZN}0yfM>ep6fys-XrJD^}Yu}RvPUkr+4qGyy*a2O`=Hi^L1|34@ogaZr zh3<|iXeP_)7sbh}ZMkA1n@;c=Yldq)U}y(CtxeRE58WZmJ{PxM%H6!HTWOAi?x)Iw z*k+C6**|z}241fC=8UARX9_rDe_{pelhU3MW^tFEm^8{EuX@=m%H|u+CA1+G?>82s z`VA#{R=IlP17@DLqY!hT9TT5}+A5c=k-}A4Vb|xo?IpGNkf}2l-eRsn158Ub1}K`IhaE#xcCuF?LJ?QQ5xKkkPs2Br}Aj4 z6Lq9^ueF(A=ve|_htb}U8cS$L&e7)ky3B#^aU5L_89h)VSFJ7=oko)DG``mSb|SHR zZdXD=&g%*vW~%aPY^T%^NAOng0Q~fgm#U^k;y_j2?u;C=NUAOXuJa*-K#YshYH!hy zfm$l>zU{-y{GrBq&R*ym}tWPYwWDAFphc8z4SDdl;VE||)@3cIJ$M6hHi)GHS@u4Ru0e@Bg1bLs73pKN_5 zR8OU%_~0#@UYU`paBsGmvVVJ8rO4RGhoe@xkiidJ^};*izJ|H`XZSm6k_@-sf3YpN zB}83N9JAj0DJ5<(5sHd~Atyf%27gdHCbs#}-kND~LCGyNTSj%%rXq64)tmC6GwlZO zt6AN1uO+JCr#)E%nn#O%VFgcltPUk!E}66tLwT*kv~3!6bxW0F1W>(p{4&?U%}e_9 zI|wp7;$t#C-Q!sq7FxK{>QePZcIiyR(wsZ`i-kZp_jVW#*sJqZ6aE5A_0xlhIXvXL zs73PWBdAsRlJljQt$^%Yzi0VoU|dwpGHq!Uk8}Fzp6>-wu5`Io`7H+f)Iow7iHFh_ z{akKu_`%8AN$XWr>KYxN6T;r%^|@mG@SE0y`&Xq~)^H1Q=MnYqT3^M{tXIy zxg!b{Zluj$euD~zIz*Sr+KVnm6JTvmM|T=L-BcR{yaCR zWou(aByk#AE_Vb1<=s2@<~8bKX$!Br&F_ZcXugsWGTEE2nr68_UAcH_J(NOpDuhk{ zw!zVC{n_f1C$ioETLR&m6~f}iG~NotF(6>RdPP1#9@w} z)%+=DXyG<=9ZY!SEVs1=*^e=rYBN!Tth8}W0?dl@A*G-=>ro*Fu3x>hC$#itTaco^ zM`>_!|5V}Hz1Mac{U|f?)fdCB!bhxL$W|&=CN%G*x;UM^K;wpI;jYrfmy0wtnzXVT z(RA8KVf<&W6_gU$HtghS{aj@x;7C6)f5Z$#KKi%KL@{Y&nO1??Vq*-o$iPoB6Dpy_ zJmqA@&-@;TMvaF}d+QMJjlsHbG)YIRnD?3BDvg*1P+0Ye+0E7v?bmD!Cc~)4w~Tm}z3B{)zcu|l}>_*)gp6Pp{I8)8)H@<%9KL5goYtkS`R(YmvtB(p3*$O|A zzkRtg^|7Abq|4p^!Sdt#m-h$m2%KA4k(B=uIFz?na45~h{TjPH|D#1loa?Tdsje?Y z&nwte{CQUvfn?T1{T=TobFzLr1baPcAFzGt2XaC6|3nC8PKIqL5a#=^rIlz4IFC4|+;UzX2qJs-m<(&_HsxS}z7up!vRM6%`Yuob*=Ox;nB?Cny8W@F$ z$IRhsJ8qfw#$n_MgSen}z>ttf03XTq6*>yUs~?e3F-PtI6n*XrrDnC61gM6XkO3nqVaV;ghG5Dla0=8yA!Pk=`JG?u3uo{*YDQ_x89|dDkJT;`W@uvN!hexXR0Ccp9NjFfA_gG zZYCMp$8~+7uI5;(X4-l#$mXB1bJ+v-J)~b}%u7{vY16uRK-* zeLhF6tYSWhCpGsW8E(3N+{O#kr}T1N)eW8J$m~Sk70p#p^)`8cqg4S{(?k18E9L+B zPHK^Rw7QH&T?pv)P1DA&(SG7ro)B2$_Qmrz&X0UlgpZwD{N?r4y7pZKaQ1Mb=uBuoVo$ zBaD-hNeJ5T4nb+6*E&$b$5K&g?L=yewdZDsbS-(x8s0H+!9HMwj{x_=Z!wq{c+!QC z&{I8`ezfO+3zt0B6IQaw?Ac$hl`vk^c%?q+F!c-kak`=b7A=cRxku*$HD_|`#37Rr zey1R)8`P0l3KKaLOcii2{$u683jN!O3oB&QZU(@fDDF*rnr^u4NkL3O^5uvI(7 zXpftt(J@GEig>st@QQW&v+jzYVgje^N(f4wAao>xRYq#|}GI7wkn&VKwVe!@5* z?&5nF8-y4~$$?wKx=kdBVywnW6muAh`}4%Ntf7^bc1aw%r^?8W{q7b75~|4-D2(Wi|MHbSgUWPjh~{&zcho1)E$I4Dy!d97D>hteE%a54_0c!B z(OHF`pkaxYTZuufSU>*=u)#qso!8*O?Dr0~dZguUV=1G*f(AJ0W+?^?P8wyiac+G| z+LG_Sw64Ny9@3(ZS0J3+cfVIt$9!qhqvjytG)5opsuSzw7HZ$aD{xeqxd;CB2y>Xk zC;Y9}0hcq9QD2ZL^EBI-*7hjrUUej8U)Y7y(9t9EJt0{?4$akGX|S>L!Tv(C2h1bY z!*}4z3nHnPhIOS4{x|@!M@P>7El+5-FoyxY4Y+*iV9$hpHbT|y`>A_-YYX&gMS_qOf2u%r8@h5&JNw=zD1D+HL8TK{T^A{cETkEE>`pnt3F*~ok$ z?XhOkey$i}K?9FW+~kO%$>1`i^zr8k`V{|BG$BxYDWhD8###C0akCf9gJAQa+%&f8 zhe^)fBaCsZVy}1;*(1G`UY|O2gW+S&=4SX=Tw4+=(JJ_Ki1!?ka-*d##wXR|f8L6| z-?G1uio>o4)Y!m!NC>&8U52>hE&vvlebefEZwh9V@OTvxM_n)0bNVeJb><0`YpwHo zq<>DK!V)LVCp`lwOI~s1*w(5lXNT{fXZBc~x%5##aUAd<4V3oPOiE4{T)&7gv z6n6I33~#bAc%b>V8VR~#o4Q;g0=yIMmhpI0xJQ|=X0La7^WBlzX9k3aHAh3t=dX~3 zW-ZbP+w5L1f8)yT^LMwchL8O+o6E=P5h~bdm(SPGe|S6o{~x^F4vRnfjKkyE$5Le8 zHTfv_0^-U$rxx!+)0;STflRQJ_kZcUyImPM5Gv|otM{-5{EgvOwN2*$speuod}hTu&r&K-C;JLb0;PcdT;%+UVn+H>b}D8CN6raBZaCh6=7_ZV;nJ zOOe!}$mMX2i0Yvzq)#MnoeTF!BHT2N#!sO7`J{u|g71dR3H4s4OgQ>S{o?2I=SR~3 z55U+1!;o0cY9Ure7gq%Anj|PAIY_Cv8hhI9-|f49wO_sHb)aid)6Cm^ywhB*`U=cs zzNzFc1WL(U8U`;$(qb299}VA;bgfTcLnDl`*h-1}l{J?Vbb8=5WGIDclG>NX#k)h* zkYP_%Y1YlcDY{bqPQM0(6GZ84nqZyJ$Lp_Io-FtL%E9(tr7yg_IXmUg8toma6wuKu zLe>w;I`uHhedW-F)2Sl`7gZj$IEP`UZ@Z^+q=%(jTHC2q$Yd}Wy0UH-Hy3!J_(3w^ z?Ld`toPDe-`TxNyI;}q9IR>Abjkv0*=*lJ^ks=|dn63WMx96_OJ|cy z5np43)kt-B`h2RE;&gflF!$tx&S;Z(D0(Lr`^KiEQ&aN`ZL2% z2EXAj<=e|yCfjwm%Yze+VAuSXU4e+dnWbL<%JN;mO*zi62aB1`?-oecIY1K^&Yv@y z$|aoMpa31iX|n7bTn-!&)DVv94|}L?}8hvrJ1?&vm`=W7qvi{o+WQ zuzgPw_6)lAYuYaAYJ4lh!#Mczj-)GG#ZA0qg-6@pQz=sR7>GXO<9d0;Sn)Kt;+*1E z>|O$-xcn))iRO0iQe8&U{L}V@R92#jxHbE=aq{+Ib$TSvb8KuE6J8rbPo%< z13j$(a>_|^cLq!0RDjR4VJZf7gt`Wk=Las0}d>i@Mo(gZTw$rE=^N;&w^WQ+IIix|^q!Nfm1VHmG(9UDbiGY=NCXyw+1Amx9S9iP9Ont{{S&kx@~ z#P}Zu1{rbrkMt5N8P(D;y8kJ3t(oRR#Xa!4e|602(KEs~ZORSx9Yz0jnWw-45rL?_ z&O86T)u*PUnHSZo*T?!SUS4*w&{H?O-VqOKzgi@ACBrrA(X#K-=p2sScc>ZX zb-881lnGR9?7l%wL|v~tK-T*7F4Qu|j~YLWN2_!f(|N(a*j9&L(!WP#pUg@tm1(6J z8u)JsqO0dS7}F!k_+Ukjw>}H#i`G;UqrNN;aDVefoX6^VuM&;{c{t?PiE2mpY&6kw z+Z;Nr&)VL3FtCD{{Tpea1}Cc#50-}g!@M4Exu%%B&zJq+$V@^hy^IU<0|U!IJwu?q5tdHCoA_Qq~HCt8*9bgy%1O!VeRV0%6=Cwj% zh?7As`PQn61HN4|FJwQkkA3!XnIq3uJu*%z#&afNq_@P)GiS7GCBt;m+6T8VB1+1= znVcXGyNg@F3ww6Q0_4sqolgGmMuZa;`b_kj3irc?W6&8@l%Cvo#+UYuaBW!Uu?>F_yMrgBJr85h5F>4t1U^LMk7F*MF`g zy6h-%OsQ;d1Ms6ihPsk3;;`F>KHMk5DY=Gr$6C*p8go{WD>v0N@E$_6+C0q_4&u z>1#7=(Dv^PvG4R-7_XcEof(J?V2UMw2o3&AsmEdfPw)b)>;GM^q%1%T7xF=_`2QK4 z_)evT`FsEAzw;{u*ks(=O^&612gd@Quwr~&F#l(M*rWh?oIRPoG54=oFn}l664i$P zlG51>0OS1UYqjsctTY~-X-^i$d-wnG?f~Wge~j~gjHB>h&w04VNy+~)7N7@}(6=Zn zc&3;cXMW<~oqq+E$^kxFQUY%JOTwZOyE{>$9Q};Xa;eR4l=UF%e@&d{z;Pgu-yWcBY?0=_&ZtYG=*}(Rq=~3@mg-;Fz(8L{-=Q<5DEoFg*gj*9sK~I-4$UpVxrApsX%+6lkwCS3yFs*S#HqioacK!p%;gya+t38rjjKtw#y$JW53WO z`>wb4^4TLQ!B=`_^t+OX-v7RyP8|p=r@_xfUXqKrm&&&Q%>8FVPR7H{;WT;B6_75- zhA^`H5C-eh0J%}{$0f-(T-MGdHyY0%v zL{kazIR8NXrwP|V_?!x`R7f}1YmdF~W?=4dJUiTSi@cDLC@T4POMu7>mO{yA^F(L_ zw$dFY4{G*DUGq3Bw)An>43QEDY;u~yx^{%s)_%V)w*B~RKRR&%IA-KfG4JR zlmSi|3x=K4T%BDWK&bM!{>B;mZ{tk^1eoO^rm?=gY+m_#3F8egBiR@L=_)_v&?l@N zPT^*BrlAi6STOoK4Xg)MbO0D|uK|%TmJZeT6lW+v36~f_!a8t^x>->Er(Y=%e;0m< zVvZn@5f{sx^9MLLG`L-%Rcx(%k$c{|p4w&cB|RDf6? zmyQ!-^Vh!NG^~pP5ww`762v_`2>YM^e*@;j^~86jfr3()DbgI3_6@Ym!<~h?= z>NwoUCwUnc|Jk}2E))|w+v4RC{CjICbuDAl-wA}3AJG5H zG7n;aJk-hkaDtkUlivfB6xYu3=a?!|c2)4^#fxT_lhMbHEj|~A*C~CAZu@ghZY3T@ z1;^Y27pzz~J0iM2-fn>H{&^MgT}x7OxyzYS6Y<4#p~&N~MwHnD>%i1wYVY`-2A3~F zSRFdw;3VI{YuQJ5B<^uddwP0qumnZ>t<&V6>}Yp2x$XUOJ7YXD@jVXly*a3ts-O!d z|Hqh(j6iM?5&s5`{Q9ncN@p&|m~}fQQ~JF8^gt-|G=nmU3>Vy*6x((yV@fkZk}gkD zjprG&7^V?0+2IEVw6^Cbj0i%ptD}YH`eksy=647@GqW7iEC}eN|Hm(>A_9oeaVeG) zMAUA*59056xgXBYMAj6ntGuF3M0tO_VtLRXt={=s&r)3KK}79}736l8E@ki_d^i z@Jpwe?1WT`j&>3hkZXbW7{^ci*~4@YAVb{~dSK3TdA(O|1=Le@m+F$8HG9CG(P;w_ zCahTa$E!+03Ce)qK}VQsk?-$0KDT_kaz`!rKd3w${$;WJ)4Css8C}4dpn`i7N{J7# z-jg|6?k}~s3)o_!ankV&lqDck>&`xfYA^b$Z?Pz6sXfNm_RnWEFik_av8(QS#~h`u z?9Pv|VrPJsL$Co(4|hl9UqClQYE94aw&uB2esnrjE}G0rS1N2B4>qLoxz*>wn}tk; z^CL)sYK{5&Tq9LLmIzlqpPM-N z`8M8|dq0hHFD({im|Dc$^Oj)f(9~iQr#F14kI$$C3nS>$UCSQC0tBbdQh`CYuY`2E z9l>}c!EX1%wE|vA1WP^T48s2O<;xk7J^85gCZWhU&kNSCFo5V+IDf{s4j4hy1+>RS z_tz9ll28KVh=Y;&;m$}wA%Kx8b_bmegFzB!!4%dgj36?zr*!k&L73R7MNEakFeTct zuoikJ5N#ENA14?Vu@=P(|1wzCVl=Se)ts)KAN7hd$VE`zne)PtOX?ncHPr^q)m2=n z2jqSU%)4WYq3>Qqzs!JvSM|Vbs_b2>K5h?Hol&~H8*f_Z_aAtTgKQ24O*d_d6)GQ_VT~x0u=XIwlBWw;NDmK@d1e`5mjB3oe_HbwjFt;p{V)Afo4hrHw-f)OZk!%M26U!SS;*U} z!IoO0aHvGC3^v|CaSl`q#rnp=9Y!B(48;2J$of{!heW1KQA!yT1O{jfJB-MOl}y~l zE@%+NSh?L^4ezVjV{%E9P3UA zVTC8$`f5uVt~ir0s06g$T6SVU#KqV09gfuK2PL(A4vh!Gfa5(U2OJNld$5cj`$05@ zySuKarNJtoOjQKJcAk2cIFIclhXXeL&R%G22M1Pg*gSDA!KibF@ZTbvtT>?@-`9B1 z2BdT2W!2K)w%7QH>ZKC0$pI;A`$DrSSG&w;<`mNLF~Br%H>>KKRYe@szSe5%QQ&u= zm~Vf7Twf0n!iwcHKFyJfjt8_A-ofyXn9TkN6~>;m%s8LY^RLjpuc9q6znBv1`cIH4koux0cFB!*%Y9SOZo6 z&RwqZ)1OOw6evn$I%A%Eb}qpFlmR1~)G;AmJjXX9nP8`+_ZFyxHY%Z_ELLhS*-g&& zG~ai5;Dn@7CJ(CC&vR^#J*Q04R36WVpD;ycKT?dOgC!Pq zQ9Vh27eIEVh;@MF4xQQ|IMJJcY6l*`faLfD57-)6tmf7~go4DFa)Ysc2eY-U%MwaR z&^6Ge{|(rA zfJ;3Nkkk5T;?qM_@P(I_`p|vXb&R1Bnt;N_M7wX!2OpCbi%_80OKvZg#rULBbWJEf z^F}4-lTjm?k9ajkiqz$x$?e!_2Sla}J7d>x;w`Q6PJ-IWcZ#Y-%qypUTJ$~2s~fh3 z+hajwMm8 zmZf}mc_1X3P_B>m%5~y)^KbH4zBz#aw(}iY)brfWrH`cmUCLi8U904z2{>fFwwwLK zh(h=^lfBqsznc2Er~oub)_$^ToLs<=M3&E!?_#k67hw0DpL2`WpeN*k`nl>IR9%}= z*TQ4)Cu4ditiRCZP?RkNk7YO{)N~!jg;h z1Sao62biG}Cqw~%De$B}dv>q&ZRs8DDS$~7O>i)hE>SlY1cN`5jTn~vE;7huf8(Fr zyV~o^;xPntQ-Oc(N(%3&1N=~lK%#)NG#s&K;bYb3EA?_*TFvg{=)@JK@1rtS+ z3uT1;@hTqjI27z{`(*i`T+-il-C^BinZ;ay91HL6MXy}lbLeBAVUu6+jD?+5hOFTO zCx($$(pf1>`|VfP_Zi2zX6qF1F}O4@nEoCy;h9 zLQ3zv`-_EC;qYw5%` zv4|8FHNj(TYaw#9nZ4(Fc}vim@&?be9Y4b`B_&V_YT);TLhxc zc6qilzTx1pz^rHFQ_C^m%I%orrzcpXUt5*#v)IUEKQHJY{QdsrvHIyl*}S*(j{+av zRfhN2e^!0*0~PYp)*ap@_>z6Id%aJ^Eq*=nTKP$~amEXerJdx&d%eSmQQ_Z_Wq(yN z|E8@4fMpYDYeyc4>Xh4=3w-^E@jF4?bw!;31r|4vqav-Lz+(rFAA{EjKu&(5yDL(8 z7Qy3;Sno(^i$Ox{ja~wA&$URqD4un?VZ`4w`{DKUllBzVUyq+*h1}gy()c?w94fIw z=qPsSHk)~WJCzp_Q;+Qs5zm}^8r${_Ff&?G>zCp&F>|ip4I(q3yEypELhS|P6N)Rc zvgnzG&;VLYMt?8^fB+R3v23RFAmXBzm0l#DJ&>2j&WB!G{gkwAl)}JyfV?mnP%wy4 z@Ax8{jv89b`X%?id_Wj5adp|ikB-zJ3`T>GGi88v?4{<^()d+re77YVuVGVO{7%dR z6$ipkG7L09=pG{85p;E_eIBCu9SRMfST7jsrl&<|DZ6OcI z1^^gH2jCTw{cDn>5&VPqCLt+A)hu3YX!nCzFDpGCG$7Dv(fIjvp4D-9jwgKus4XF$ zI0lHWFpS*txT}gii*fgitLyVUR@Ouz#kcNgo-@FQO=>sgofogR`ll-`LYBp+Z#1t+ zYBlsu?g>y{g&p}+5a$6lxjnP6cNj9q{?hVVfuJY!9qk-H|KT0TnwT8J2ST948}w$(6X7(AUk@1%GC+c9!9YFkPi2W$T2UCvritO;J+jPD`uTQP;IqNoP76(47rvEF{ZM*C?^w`zkHRgow}B$F30gE`g5Q zTQeR~Ot~9jH-a=inNa++t82n}Ubr}W@;k2(4seB7Nd)c5(anOwT>-GX-uDRiHG?47 z#*|j;3Sz@v?~8*2*f9Igmkr+EpjWf7f>ktt1@b&2R(VXvQc79<`0VD-f1{k?Vi$dP zMKCB-5WI@rzbc!5LU2}KnhM7NJc~D*nB(D!TnF{RjSdpPS*^&Od~iPrT<6soxP~{V z^HR2Y9Vo_=x!Nwf9$L$2FzT}XX+$;oIy4x@B#sa&-~;i!sB;1cOP-)Y>8$&^nxtst z)u|XMG(*^JJ(tHSSk~6H#`dNgtHT9d^c?6RwEA8wc<{JvsyNt#E0IK*uJ-M!N7V$w zgwF+mcj5F79lpXoi4G7xCTVUj&uG@=alpkcfC@h^xovmL{doA- z^1uNwMh8IuZp|^dKO;%d*DUKPon2tymHr#v28Ke2Gz@7Om5$VKBi^^ z)C3p#7xXe_!j<9vI5tV{mGsL%UEzS=NvObsB|Hn;_#O7637}8jG4K*82AL-O5rI2MB#8PI=DSh&i_=1p`*JT{8V3#U9&rq^t9-! zA`%08JqA&!Nb=2LELZX5_U34C2a-t-nJy10W=m73$wv4txuW7zPC=Rl&)0U>+ zgE4`x8}u&kyTId$bxOP+#|tX$?)sI5L7%?nW&cjE@f`5}hd5aaA3^JM%d`j0;}>lx zgrU#{@-!C#QMIedqC1wpb$jd2a96Sxh?)H<|K>X}2`^mAM{Lw6-dKTpq_X?MvtU{R z!Yb%ref@PqogF6Nw;q%Io4&d=dQW3VN88kvx=BzuF!6Yk4DVu&>kkP;Q2KD_%3<`I z&Go9g3onJ5?X%aNS>7NE=XN3g=B3>8#ME&;n|BT8-40L`sTHN_!{cV0`xpUcF}#$- z!ZeK^1-Bb0X)EnyBLfX-k&{j4Q@U!lrU5Y<;Z_4L{PMd6C|v{pyr2B7 z&`y{O56?P?CRj?eEo+pMi+)C0dA_DhRHxbQuiki`dh0)aJ^1~AixlmNV(|-}b10UG zJ@jYK>nX?AiwMNBmByi?$_9vz7p3YW&p!avnkHqS`bYM+kC1LsVwxs@D9_~sp6$ka z=V@w9&88HgKPr4laQ<}9ll%J5sQAMn?Gjy8|EoQdnfGA4T!Lfxe1%F3aH|}+nOYql z)()L9-eoL2fcdh|XgzLY4-ZECyvRqfEPX#ulbr+qS}T~pRz2V?(jvtkTY%(8T`v4m zUZK1H;E~yHeauIA8CLvw?TU~_fDSE;eh;Z56}xZWs-gae>|73IH&gYqM4c47hPxR{ zV!A&Imc(C{i4>mA;<5THFpFN0ZBKxu{iz=Hj}IrKc3}QT3qX6qx@&fIvVpUNXRX3v z)Sy9rK2Ea(A-B}vl2m4sY5K%8OWMV6DRp&;(BFu27+rFB3pF@HF}p1zQjz(Yg}l2l z!nrStw03oI)y`6Pi1oGPkZhGwmrovW*Je{chp2`4gLaw1_L`#G ztgEb)p$4@!se0`o40~za6S#UWS&yxGG$Pfc>=eP(bZLH-ZDz-aVv28@y>@6nviI8l z7S{4UoWn)MRYvzv8bg-wlf);UT|LZr(LD*gHvv@7$BT6yTFy-xJDWna_IU3tLTE&P zE4UvZl_twjoh635Dg1-Wv#pVwS&+ou73F-V)pk2p3rLt!}y8w3WMlLFXrj;#hPU!P)8uRG5rwzoHX>Le%?! zpey%)KK7f_cawp284-r8LT@5{fZsCzO=Gukofqe?a5L<8^R>S#0^*P4I=8+l zzS*D#;o08Fq4({;9q~+j?=$2M&@N%{?`u~YhOMK^n$_f@$ms>Oh<}G^^4P&1zn)+$ z=z8!(SYJvcF!WI(7j||o*y9kJk|U%6p1Wk!<{LX@U*vL46g;VItDK;OR4B8*CwC%u z9`PXNYG|&$_Vp@(7h@(bB$#!bexLq~+gs6%@#V*@v_90V8RPmJxq;8ospqS`Q51}D zFuKLar~4c*6O&&!bhXOMd5F7L0f?_H0pgN|En)7tpO~=~UR!})zK_u#c3KH)n}me6 zz>V75)>fK~8-8oujNgC%u((nwq>*i>2M!9?q#>^7H>GmW_<1nBIRDJ4kMfuRe0Hk6 z5Y?g`lqZ(^)-4g_MYc&p>P(??@@=&}NY{3?CxQnct*;ImxB6I3>iAaGxq$?N_E@78 zlZ*@y7p_$gQOlT3F52~H_NPAJ)$ii?j3I6KRiHfWJeB^4cj|}T-bC$#A47{0r#W^4 zIT1_LnBR&WDPN7->1*1~O{$j?g|!f=fBnNttdywP-cAIu!8IbJmk_aKa-J2o>S^f& z;bYc2t^IQDieP=PVnxp~_a;*EQ(e};(aR)%`HA)vFj<%ppP%v+s-Y913i=eNlt#! zpKhTK*N!SJnq`8Dms4^Py0bWl$ybEn3_hzs?Y|b^qE5Hq$^8keK6RUh4$Z2!sFPDX zlG~n*)7EWQmRIZQrYX&CwNxJSPGiM7>=ZzoYHK}=Zd3Sb1r`^fh>|hKPx?)znlUyK z4qoqRcm4^%s%df-*bZ&zs^%+^sncYc`G>_X_qzPc=>l_tQw*|)6y8hw;C(3 zs+l9~&B$ffbLqKU@qPDJyYR=@!Z%yUa2Q_Wd(Xw=wO*ft7S9M`o{ztRgCo!vQ}0-R zekM&gArPl~7kYl0l*30)=Rv>hKrqir06qz4_2hmlM<`7v{YvW_@Az!y<0OrpCVz$$ zIU-YX4%HOX@yjkj3FpBYUjvu&vLct7&|A+ZmjO&Z9bW)R3a2C_yhbVvHjG8Eb@$&H zl3%{FJk9?lhoon{HS3=BIW4A_^zcp;F0tJ0V{h-eM@yIEcTk$zwRL~rYa>x_y{e>4 zA?1wz(suCwqUx=~qWYrlaY{-$q)SRfK)M;}1}Q;O0i{7odIl5)RJtVwq(e%SX6Wvg z8d5rj9AKFFUOw;p{+{Re+`nPYIrp4%_g;Igwc(Guvr!sf;C7sx=Cr#OB?q;U@>vt_c2-_l4bU2<=G_Ue~ ziW0;k%a$%9i$JXA;85O4jwIZH3@eC}bqrR(+p?QblD>TPz}{Y6oiw2SfVo}GJj>GQtbCh6RWKf2Sdyp z^&9t-j;OC5=%8OnWi2f--4&?!F3aG-V?-jht1i>OK9IQB{VrolU9y+?ofux6PYGM# zc=*NzPn0mn?`U}f@X`5OpPO6qQ>o(5?(C6~kbgAO^KKK1H}l11>|K@MC&h1l8%t%* zS9`qxUygn|eax>h73Nf=3Dzr&o~d-_$Gv&xZ_!;ioh!<>dV|cj-lae8=*EQeL#7o8k2!HT|QnWfi zgJO(-nu|NXCAy^?@h+)JZM3(JgKMF7>FyyWv(u>N0p-f5oQ!S#mFlOKQOo+A_~5I} zRO>9GOD$A|1Ehwj#JADlQz{W)68!eQnr{1l8 zcR}qUnifY(dD5;8q!1wqYPdN zVeimDmHwO62szv)nPwg-<4qAfh<6c*{5jTDiGy{si@RCeWRtW8yN=HEMQ|Jbbiz$< z33y|Omt>zL{&&<2{HiI5&*B9@T;0=+eaOf-->YMM$%Odt*&)b2bLx7{4K_I~;~Vs~ z{t3tvP;03$o%u+alCDanHr~X7IevkBPFuH#Mm-5$f za;W3Z>_+&}QfwoL7TbW>c45{f8X1jLRzxvQso)wEM`QLXVG>M`6haB`+U2O`aFU*> zEF85zonbSLYGfxj@*pe=9BsMe>CUj}7v%nLQuCb!bGMb+QIn(bHd!Gt#w8}Yg_Ty! zi~`RlW860U=x#^6Q%-^|%Njv{M+~o=)X^?|xP=5K8_w}VJy0v0Fq>(86q75eHp=+- zY`)AT!2QB|OgeiXQ3tpsNtAm~d@&F({O(pQyuH+v4rKS@0}TVDYLqH#4m>z4wM>~> zSyjia3b#X|;eDQ3i=UQ?GLa^P>{4S)2OITFhlpA%Nb8OJj#N(SlLeT7(;L1g526-h zDZJ0d)q{-kBRQ4W`NYIZ4@Z-p8G_elKV`FngK1ikYxfxxo&XH@TYdj`QK~5PZ=M8! zVnzg06t77n5fdUW=Dh5xnqkj)`hkFXpJy+d(Eayr8jLRsqwj728;k|t#kQ>B$)a}t zPzfyF6khfi0PS?m)2TR)I~KjikA&hSm;E~}cvXs$ou9Iu9~QZP6=6U`L^R*$jATGC z5%DI6-=CzXZjt=fRmqR2MkamUHW9}FiU4j&cz}f=QH@3CT`*g@ghcob4Fz~trK;I%Bnfv5Umq(XA$fb)7XANFg zmphbgqOTTH%xTp$lkj%fS!2CNJl}Wk`@~NdfpnSQ75noH%g@wabE3_h!B!t}#4!ah zOh~TX>5XP}UzE+KP_^~^IU_dh{;w4&hP)W1Y_vW%gS>}1J6>p|I%$GUrq>s`ZgOAH z>ts>4B|=Ux@SrK;(7Pt*o4-b#?<7y&OS`j`f}-GJI^=lBlb_G#BH<>dE{~k2P~ks# z=4Zx1AoIcs>m*~5A^J5_?9kgA`OU351@7J~;}?Pr=-7#kuQQtB9>`D74z#+m!4DN8 zr-}TiqQl;CGgJA&q(|VtdMGAIvA*BXp`UC&t(pQIul1P_1ZgNbl$pNs%r~B3G~`5+ z)M@VNb-l#)(*#IbBpXvz*}q=R_?N`xUFu`mrZyoaf-P(0K>3PLi1TjFJkPuyxCl_EytN8*%ww5!jGOHyde#;9@IL*JC-P}lt#7>vB@=ANQa@~)TamtIzSg# z$LPn<-k}B|qK=++T+#xQ!OVDYo|?G*{8VEN#3n`ya#`_N1Z>GtVj-zcSZrF^$UF!2 z)-9*>8H@Q8gOqdXHSfe^wW1iS&(^zg-44?>-8?b(8jNeVfP!` z-4}lX&V|g)Y(A5egmlf6nhZxi8*2?2<{a@JEl*z^Pn(Yf5ph}4s5HFst{oswhBOD4 zn^}V7siJ_%M#s+46kjcI`-4!&%2(%3MTOyu6RM~!pG@xc z7e{f10!7-w+=z}t7t&#iZ56GaexlQTTX)h<(ln{9e#XPa>6&rUYTlp1Yp=a}@UySf$R>!~bNKj6f zes|L{Y7VhALJIrx_{GP{=ef<6p53%w2ft?r*f2=-hMmHUjei*)fEKYq)xGIJq5+V^ zZ^pGejZUy7hY4_xwm#QWxFrdw?*MJ~ctJE}+^cujCisOen{P_~ zKh!5hM$b2>dHcuLho#`u{qlvp+ii zu;cPR)J~`h6NQ!*`HU8W(anJ$PHmnfkgQk3(B~)|UoV6VtXu&cT!KT5X<%{a9|bsUvmrESjkoP0{6bG*(ti6aBF|t@MF)Cq1iRV2krSQ$?5c?U-L!M@8u&L z%77pLLgzd4=S=EY-Pu601M~fueRTD_%!f??pnTl<`K>-&5h7))lBEHX3G}HS%0XNQ zF(G)P!q>4gch~W@3?}?810}+nXWjY8>zvGkO9SjM|7O3DXSrMdgn>54wgQeVNsZKVSfAJ ziV&}%BBPe^puN9L_7{aqCqt2MmUD982|p)ZFY~0=zb`6ZZqV0rk?fSUuDFaXVL2#U zhl|(w$3Xl*i2B;SNnHg;Gg@9B(5GRPT=&)>NkuZYo%7?jaI$uH5i+dWF(|`da8AF zN@)M)u5>6@o!)afXBXQqPY!t5Sol7a07ru7z8tVdu}ue;>cFeC=ZBXb4Lg*l(#l?F z6$tY;AN+dBw2^i6qQB*qE`TR&;%DOa+?F#=>G@^hH7#h<*rU{D_(Akn*HU=2Bt|tm z;2(nraUm#Jh+oifwxsrS>&<#^|529Tay*dN6B&{H`r8xhU3-!KI_BN(MFx;ZvV$Mf zOe+9%z}xxArQ4PE-^yjzSVlQ3wNhRG_JFr~4Fl(^S9kXf4FY4= zM>SdqfrqN%G$dB7V;uI0{|2q(ERNZ=8q>EHVqki zUkR2>V9On`@*+9);6%q!z!o%81%C=vTx;D6t+wlQf)V2)T+*xy5!FfjxzU&ugwUdj577h*rSvs3*m}n?dZ+MEcU(eT3O-P zY4O)yHMu>ylwlsJ8|wG!$5;tPFVqbbziV|1e4pYy3epFtINY4-F-c~pSLsI3(v29T zC(r(3!!No%-UVD_{Vf;FK_(s*c+p4Jt1tlAf<9eyu!q?NDZgy`qnJ$=3>vBAC1u(u zuKq4bUZle0v*xhTyXc!s6v2_(>FF(eORxDTCfz89>OBL6ZQ z+>#J1>*qZ1wEt*ZVaAs_bvO}#Cu~znOpS2><86-*uVsmhVi%)XA zb}AYz^q`$<&+H*5T5WPRAhY8U7#^m4GM3X#IT5S99P=vf$ffjsydApE{%BefzAx}M zLyO}PLt2GYz4F+^v(_%aHSp%BB}e*6^|yBsS!`=Xhm+~wkN}7!9xZ6`J?Kdi`(-@3 z83>WSAqmb#XUsJB;59uyzFuxl*duSaE+aikA*9%l{RP2EIkFx^}3 zJCj}&MPa|*#oAoozsBTH!!>Ik8pr}+j1}H-F(+-OrzRd-l5Uv3Y~TiP+{jlgKxct3XOOiV(r*MiYXyj zQEcbny_xZ@LuYIh3_yMh!vnfY*r{PK&K~@7Y?Cxx^puglCJ-=dUu{deb=la(G5d~c z8=_2$2Gi-Rp0G)}vHz0zsS$8xsC!bKT}cK8>L-!J8y`Qi>19eG6;V6Jmp2O*5~J%! za~wtF!)`B6%|1rwu!*jbvm++A-|3`_|H`g)%b4`H>Zmj-YLwh%6R(L0mv{9Ti(*R_ z!ZsmX{7CA5eYT?%N9)=tx#<7Vxa7)UAWFEP3-0rSCa5(xseX+|8XZ1y^6C>r6BCuE zEZ!KaiehfsNy#Ws`b2C%2G&j$r2QdULu}W`g8GK^1{6KKM;#mRyXfiY#dS86ap2mQ zw0W$hd`w-e)YUwnx34SK4t}`QB%e`P(_yfUrFaPbO+tr{AhSBp_xmdYgxv%pVXo(E0hEhYl zi=c!M-X(@Lh|4$*H+h1`|Fo&s9`s60ZJz!cK9zZ?W>#!k{qwpmncp;UvTWkO{?qL= zKn1ULN{pjBk@H+r;drN?A>p=BoEMW2urQOK`6N+d%>$K1bK|GW7toK8-3iB2SgqUF{z2AMR0(f@|dqcO^yi{9tZCzRBhv0gIvz z?8{O7Nc?*>SlnTR+2f35knb^V@Ak2`_kPgc&$^Ap`y=}IogREg!Vh?QlWroIO&h;i zf+uyC&pK5>Zd%&Qr^;=rzNBPeAg->ZYu9H%Q5@cp)`!>H+J&-|ilULaeN~jOc<_oQ zR88*fkm(0b<_&#+<3c^*6Nhf$7i@OEViG@!uIlcf<;&ug&@U{st};|vPbI8L7uWsv zgVv&WH)3G$#0Mujkw~U*$+odDA?{I_j-K46>4|OffQb9*Q0+!tab$DOQFr@MJwUd& z6J#7rMz2JqsI#CZ#+Usg02Ju%fjo+YRY`zll+|7VJkJxEEuVPMMTl!NxK|-O1t262 z3TzH)cb6tlD&=4CfIsXv|5c1>*#Qgnjj(yZmMr$bJFP#R`W+ObB&jPf2LW4w+ACCG z2F1j8Lut_CeNzIMk1pUHA^E?L-)HkV3q&Dc$ohP7NU;A%-J4q-sp{jP^F18H zu2hZW2c>2;fs@U~1!|~O*IM+*BcU}R$098=lRmm+*U|*plQp}}e`VT;izB!VqnIOm zFRP#B{D(!$5N~O|_v;@GQKcE1^@h9yE&B3&Aw97g3paL@Z_Q$UA0m=7A*=VY-+ejd zo7AP$OK*+(qb`4Q|6wN2yr9X74~teb@>d?z=SG(k%p^}>9Texf2Cx7Tc?r5EhHL86 zP3}uF96qH@G#)^zH7Suy{^9l%ewai(kd!(J=X{m=ZSra%5`Ope9+vQcW&BMo3bYXbd2Kl3b)I=UcDFmnt7x5whg>|T)ePdl-IwMwZ)%?9jn{eHk419m7Oq}p-H4*>TjZsh}HW*G- zOG$xw$B7=huRwuGwE^!q<=A$oXmm4DY5E<0M4DLbR=9q4cIzfS-6j8_Vjd@Vxj#Pf zb^54BN$6XzkBk~Be7C#gK9dmZ?ObsBKm;kh#W90JHeXi6nfMwx5h4|~M;3rW^tNCV z8C^f!16U4Y7UdD>uQ7jeaGW2$#byD5>@UqL{sm9fL;)O94IQ9YkrA#lfh<1p)?vkB(lRDT=MDElp16!c~uD0T#+11vmZ z#J;m0Jg3Clc}|_qndQ7zv?S6oIpPQC=d#s7SweY&g~lv^PxZ;t8ek2_!GU(LsoYNY zc)HG>b6s@CJ)uX<4fYEmq?PfMvVUDHXw|5ZAt4Y>3pzc${VU$SkqYbr%{$SAj9->+ zx_SBQd`has^aL%G4DWR1A4aKnId|0$^WWDh?+>fgm}kfVp;r&9fd~OZn7&Cd;fE-E zbxk^iXu+5FWjOqQB=BYcmQ4R0NC*(i-*&N4tyQ8maT|8Om#z0?w81%9zMhYz^CR}Y z#HIB-73GLyPkZ$+>uGKnB}8L5-(^-T3CO-w*R;AsVkgu@z9ai{Vlu@DiK)J+fjZkO zt(vZDyd%Uns^T$o)pvlU!nyH!F&~4l3$u^!cdf1$$SoDDL%&8vCdi3)))y~TVXR&k9uOu!Ew;B%W1l~0N~SDn9Q>nBn2NG$9$zP(;*GKq zz^AfHm(I(MhxGN`Ay*THy4001B`)H!SGZc^Y9cAf-;{aZp9OZyGVaidGdkNmkjbT# z1XwBGLkzXmqRTlfBxX}T66oMk7<0Tj%ElLQ&||qOH4*@z&}yK>Hj-#LT^-ALfXq^2`u-*s*!L%7KcI5U*W0nHTKh9? z@An71wgvjYdq8fK=JK$+|7-KA-2}iJ0VsW~K1bHd4}?lNZ!SKUk1oQWlO2=_MO=u* zDWk|3+M70|t6;h1Ha7a7Nh-LZ!G>^v{b5EI-I2%klhaLcI4_JHob^DJx769lt!qwr z{*T+9z*m5CryBtHO|?+cvimc3b;8WeJBM62c2oH~^R*Siu+GM!ziG#?DzS)XBG=G>^Y)l0sE~|aX;)zf&`CTJ5 z%Hp@kfdOv@{5RzfR!q$eC_rkNK5Lz>P&CiyCkSbbWQ$Jx{(Wj>)Yxq5b||uAG+*`W z1H(#d`76M*KOgW&Iq6~Y=qsyieJlh+)m&QPH^9rUM~NTO!5ROOSp}Cd*CSv2_3L+= zxHyU>NFb;k4WBH|0|mOEcn{B*rwWF-lgsTNfToNF&OPk zyLHio8-Rf1R{>YL)@f4hyM)_gz(GG5_}u`UwmVjd2kiEb`lC4F@d?*K5|Uza<7HQ| z#mnU&eFZeq4)9ov;q7|e(H?>8OPd!WBOtf#nWP+IP2Vq`)m3T!_+->rVU5z#`{4J9 zXJR05iXVcfZr?yWYi2(q)-`4qcg#4!nc9?*CPrU!zBLfFGlYewbI)Bb0DfZe5I`dH zNpUiQ4gBa1Kmz)Dbl(VGsePRHmg-K9{B zktuK&jy9a_O3XQQ!evwyp%ic`fBz4ZBDS{GzRRJ@IYUWvPx1TPc4`gmlU@tj-%qI| zaj>e#I4GZ$ShoaDeSY=IV*J(0%stMaKR^o&Z2-!U@3`~LXd!gek?lzOC`2&N!uT{V zBy!5FElQ&T_=X+$7$ENWK-6ZImCmm1bK#S;FGrRWFDDxN+qP@?D=BJC3Cob)NEQQX zggOGSlDj%`&UzR6CO@KWD3mCXIzV}oT=D6N`mfIoG$*fCRM;Q3YF}Dx`o&!j{quMc zt&Nh;E}<{3LGFb$=EfBBGk+?P-HoFso#J~8cD+n&zcGx0!y^qc zUax<`W5?}-#)EcOvgLyU^C0`^Vhn@05?3*3PR<4kh3z^>NK#ClZ@eGv=vpjE$mT0` z9`&4HXEID_Ca!oPl*0_K^FyHzCCDu~b_5iA-8E70W}6%iL{P3>ked*{K@b{2uAh7e z4g3Na_3dc?`7#Yhn-~rB=@ICpX-0AN5yv}fb3oM4gnW2OAH*WRDna{KqK-uRZtaU= zUAno^VU;lIxf&-ZQKfe@0`zUWtV<>c0tzIu;6y@lY{5xmqwQ$7u3`MYKL6 zj5l$WroRs36!9_BJa?=?!Hq2idjGkU?;U;-Msg zm^*7KQXTSKHck%U{heh@(V80o#6Oj}d_5<>I0f{(Vq{6sJeFHnyJ1cwl!Q}wGWYS7 zD31MxzVc`GI4^pRhlreeueu^j>7UQVeZD(uwR&?#|3h3c6Cf7(xQ@_ioov@<+m*nJbE7C42xYQOq4klf6!&{%E77;$DKJzoUS8@onvO+K6_f=Z2vBV=elv}G&WeHL>Z z3-))!t*YF`fA)t~0Nd>{36IFbSz3!7<`MgR-WarLih>L@8>?0P6YE2xVb-SATeuR?A`U)~W? z62kL8w<7l~DX#6VnI`nTI+KR6@-E%u^;>-z1pn1!Tr#TK)KU4Z{UX~-CV8i`*cS~ zsBi2C--j56w~7Rv#atfES_VZIJ#?^FWe8#75%&dbg}wwVqig@p?Gp> zDY5c(ZJrMWD4A)U zyqx;dxPxKp6d$hb{C8({-+buq`&5xN)05fH`&uPz zyc>O;XEZ^nfWw-_jbl@jX=td6B79s6k}+8RbWO9sHzI{6aNTKQ3Y^zCdGKRQjg;^ZKmGs4Cd08Ct=pd+ zSnzA&b4Gbj#^W_!$VU3fvBAT5JNJ%Rya>LxLULCdi6>hYgFuZv4jB~9@E8UJa~^@T z*)vk5Yp$QJ;tj;JL_=r}q?e-p^!GIVbu4mqhcXm`y)XL^o>7p?dd;wnFtd=ti;T1M zDQ-E97H4^vmZH3tf=uL+l4E56xN`Zm z&PBB`HOQRXIBM>_XHR?oS3ZNG&X$8e3s3YQgqmzyAX`!9kh5`XJcGCIwnZkSnD`N* zB6)cuT{QXNhojB^lu!|GhrTkJZCsUOoU}#G8+DwtRrOHmIN_U|qLdoq2s@X~!mn<0 zTY{O*gilnDi!;zDhk)DLQi(Nkjw?*Pot1qq>)UrvKP57-P=LtH0Od+LQuNeC_o*po z>sQ&eRO$QF!Np4{On`=@A=hr}3v-w;C8$)wxIqx%Qg8i`s%D$w{#o!`am;_~)7{SE z1$+sDsu?4Tdua6D`aJ!G8yk=bo!?jaM+U zAyPBcn@-PKDu~kPCIN2qIC{QL1Q;PA?@%aYLUWloL}!Vs^!w+7Pe?>tPne*Q$!*w8 zocG{y4$}Iz=G{wozxf8dJEc2?RFi-_VSWOVVO$Y4=i|Bi0V!@N#GjkxeJDNA>fr~J zN~kb^pkh@Zec;3EF85cDc@LtmX0%XJW3Pq&Bq`xt$9@B?*))oL5*k@nXF@o5ED`+J z0HPDu1}&7o@A}54Uvw^SFuSv`2ZbS(Q;K|_35i>ZndkOrH!gWc)(iv%;9-ko&(Ed$Ix{v$?r zm1_bK9t#2k339;kFj~6BOup(%G9=P}WIwdX?mbbw!^cRI8aQH)90U`8&c9Tij`ybZqF?)VOfXMEF<-!GL^H3%N5=i3PB;Tppcp4mo8RT2j))m zwIvzv2#$>gG=&JhOEVm+Ki3^H1MaXRjNk_wE>FQXZBQu}mQ+7R!4+P=vbM^DKy0vA z%SLbnIh&wErswDKCD>#ryr|9Q@{h#^P>uDLXL6iZ;p1&#cKm1G#-`{i>;@ia!yL2| zg5$dylCU9b<-V57PPL(TZ9OU$UABZGNRT1KsBeChdX7SiESu)<^ z&Ut)fOX3xKfN;)TCoeCj@rVEYqmkRddLCULr+nR0KpFEx;k#r7>&WwvzrvG7_-H#= z`0nHPtg(0`NcoN`8#{fRReZ39B{53ppV3OuP?>*8!hio{HsUPtOY-9ZKv`y)@mB40 zqQyY{^m{KGN61qzEnr0#jwr9GqI?-}(Ou?8^-T*JSde3lWgv$bxa$|jUHQna?Z|Eg zVmJ4U&nCCoWmG5QZ72lDU);rnPMI9HbSxYO5fLxlNHTzz4MlPe(diFOG9B~&YwR7i zinxz-8v4N*kcf*0d~LMTPd0&fM1qo6CrR+uYUSj7U%NeLZ80B@ZNa0b3$lCN&Mh$_%S%ahNdIq(WllQuvsh9%QJMEEqz3ELw@Z;sE{_fTA>HXi?`O zN73%8Zr|rxenWz!_84B@jmf)*x)ZdQmA(-E?=Q2Z)U=YG@Bxh8WIuSwEUDvs)$5NL zQ-sH&ZT(8Ol+WqpKXa3#l09PEx~Wg8fN3-wdV0^U;(!@{J+Chbcks*e1j@y!F|dRZl1gF(X%T zXch*2-v2{syVAJK~5s}#MP zacrir<`)`%=Qg%d0{)h*okdW;?qu$NlfpDab*8jNlwu#%Jf(l?e}Ty3>t zSOI|b6^9e_JLAe7smi$I#jwyZ_+K{pRRvDc&Cs_+co(NKv5ZsALEzz(?*ZItT>aW^EmT+{0b| zy0k&#i}KW~3#_^pq2)OkggFiC=C7{&9uQ>VKqd1{ZE)FYkmD z;GyOlVwoWT_rp~qHnt_ikU6A>Byr-67L;nY4qss>q;vw&@S56)03~#$(c6E(S(F%x zjoHb!!#n^~FcS)GZ1=LPkM&89MYAd`H@`gCy%a`XGD@oqB(*#wLjLr&31_>!g%^+^ zXWe6t=}s}gH3r;Ht&Pg_dpIcOBx_0{jXi$p74pMr%pL87mUe*Q_=X+{rkix$e z3TH&-pYsuq-!K6=|8eE44j1lt3q^=FJgX-Z?x($rVKPhst;D9Paj^>k>xu!|&v6tC zr2Ka=z!dVKNtMqPO5PmxvO!!)gC|bdKdrThZEIHe=wl=T%^y|RZV@H`3mo@sAF{5D z0}nopes*Sb1m#fBnb+1ug2db40M&udps@D8>vAtg!7$hM9xmEZ;RM7&%jc(kvSM@b zHl8g;<#gfdVjdsWYGsar)9(YM8Dcae_8d?iq@aCuvJ9wuH-I6?ZLWBrm~_oGSjOlh zYr`anzsvZR-oducmZsY8nWbN;f9rcSJASlT28#VWL?2Y`U|JG9ohwDNkfZd|JqE}Ki%C#Aed1nG| zuH7M5n{=})4!{?y#r<4i!!a%!NK=6>ojaY{tPV*a0uON z_Sk>tS1-2|<6-E5?ev3FzPoGTUi%#MmLK~Eo-r;n2)u~E*~ZuSm?Q2oBZd1K=;}HF zR91-aaB6PqIvFmWP&Wv;KROB-JyN${syG;V-l<`6v9n9^bJ&gaFa-!|)@fFFb@7fN z06gmc2nd0~&z>DSE-S7-^jYm{{)P>)tq4+r`)ak3+Wu&it2N+XLy;7VlxI_M2$$u1!%Qm1p@c1^GnCV@1vAIlf(l(6#FL* z_9jT5;7_f(Ts8Te6VqK|>=$}`Do1H)2el^)by!d@gH=Mt!Z7gImK6(~;|e>m^S5(h@Sr@`IS>Xl`RS#8rW6OWoEdZYR5QUhH`yQj zxJQB8M~&8v;PRr!Km0{-XplSD$3W}i!7V665U@=y;in9r#sD0}fqQh;kA9g7>Pc47 zA4iZQXTjCo*qATA(EOkfV%i2{^FOY9cc3a3ir=)tr%hi~qc#lV6KDig4B=@3t-o|Y zk9*hSqTySDUtE@stjKSzW=UG24h>pL_L!YG*~{BTbv-{4Ul1S1RrOV}%Dx|ZIUVOH z1^6}><#X2K4Wcnq3AGx1olUH$ioLEz&#KRl7$xf5xBZ22<`VZiU#&z?NV(06G06B7 z(>yVL-ZKhKcYh_kLe6&Fxz9Ta3qJS`tlw3@i3VV5WDivC4lT=G{7e9>P3GDL@46V8f*tzj?GrRFwZIiKyMlZAIyLY7;k*3@S_ z$>~)X$+R4$?$p+@wi@`2g5{KZ zs#naR{fxVXGs8pXztRn?$F>jd zS8RwYE~HQ={Xe1WbDqbt^b@tddoN}e!sJ<Ad(>4A_Wm4?3w^i72FRW6sFC*?Dr)^ENY1;_c>zzW^SK60OPWA6^&Q&{k z?a$FjzxSzO-bDV5NlR%T@Aul@PD*rn@7jnun|vtke_WQYChhvR5*w?&=P@!U3@R8Z zhQLPB37B&MVt&6in~GrRfRJ}KxM=$QBPrhcNr8t2J%aCi%04{g* zaz|Wr7we0$UPm>L6J~@QHdS!yQL8L_1bJnGow`GxXOU=8+Q>KRW-Peg?r@3M*>;oH z_Sjos06|-Fug6}`t-wRa&>YTVr9`!aRU0c5BcEauE;1)O3=I zAooVWUdV`o5v^WrTh3qqTf^}jO;mtlX-q98XaQ5h>yIr~l;c}qnn(%Xis z^D-#Oto-wBA};v)76J$=L;wdE2B6RNRI!2bcM#?qHjF$BGX77WOMN6Wri&;&v4Py! zcWiVuzd~le+5JK&P{6V>=@}+J&*vFXG=Bogq<0LQj0ikUUANmPtGtjZBI1lK_RnW7 zjKjVgN;}rIS$X)#`=ezp$nD=*i7Il*r}%AAaDx5{E7{D6t}$&gRA+(Qr1vuL4JDOa zWWN-5vu`_6UV$PIHwl+(&6;iGC8FkSqt;wQUyDY)ako2VbK}Qi)p^oK&Pqm9Jiw>J zwzHBr)=Vo$(J3`T9dCR--$VB6SBGI^J%c{om4pxdA>Cgm9pYJTx>WxA+r>0tOTT`? zrM-|Kr}Laa=vcJeVQVDI{rM=+M9m(ssJd&e)Nnc?yICmf0mQr<$Fb@1fSO?6(6j|R zoqDAxGqFbR86EdG*015cqvHO0&#yO58x&4)0QH&2NW!S^zfn-ES5ORNa~E$ySh=x3h5SMCUtt*|=Au(@O$ zHNWn}f6a3Pn0+l3&k(TsKdUG)7!HL;w88!M-|?9=CUJTy{R^KWYM@Tw&5YH+2Hj9_ zS2`JddoC9;KnyfUv6Y$Jk~y8VA&`l{{YcHr;P8@#=UB{A1r`^83Ra9H3NNcNZ)Qrn zp1ZY)t=8QK#^N%|MJM=)L3hCjE4fdfORlK)1L%g942PT{MS}GK{^PdM$QNqA6#%bx z!%|lUAe6^}Y=PzTZ40zS08VM|PUu2=cyHo(zegrn=TnlR-pU<=ijTikBB8?Lp;&N$ zEqkfsaBb;;zSf4!3R5jn@89IaGXEbJ;OT3Mo_oi68bqcgb(KtM$2-d{oUYV2!PFjb zg^_59H-F@bli~^jZ&w$)obH7zkK%UwDXrSRS$`v(^9HaJsPK@bUdRStJ6-ALegN`k zJfOgR_{gN9WhP#V!y5+`C#!k2Qv0gv$7rPSr@ec}c4q!r+~Uu69@g7->rAFA%%)?3 z#(6eVUm;6oD{q9DSJ83)DCUxpb|?n*aS zMPpQFfy!i{^*u0&39Hyc8kv#Q_+}FKow3}H8v`K~cP9z8(n4KBVXMV(Q)$IReM+dq$WD|4g&DoYt9k#x@1X=x-^qB9%PytY#iy6c&`bb00Oim+I!p5BT4i_|vJ3CfP z9Tc~yz=G$Q@<8yZbyA)ETr7+YG&36?B|g2uok~Tdq~2ZIJ4zRxgTvS3(%{TnJ6dNU zP|7A*ahW|+5i&`+u5myi!M&(IM>C=>^>em1@TS=&3@Wef{Dp$4tx_vhqQb}bJ|51= z)hL|t_=W`O0uTNO)H>QcrQhiQ-ON(#%LBtemeMaS*xOyMAYdoKqo4g`;m<|g-Ht}}e^;zna`P%M zFsoeU(kh@u(IT&`k11YF)i~@d7Y4l_%Te?KnuHDR$}a#3H$DnL6rKRdJ!L&JAOP`P zBvbmC|48$8W?;ZC>Cz|BR+i5&!RgKFMoZ`7&&Vgk$Ybo%S1&h_rUy;oig_w zJ(Pp!{It7Sg^%=QJbbolXGUPAxp$A{25~uEhR?Nt7F(QgF;|2GW~`{?N3|`@(Bk! zTxZ~;h_F!3!Ol}e>!OPWMO9cZk^KJi5g;6VIh`SC;Q)W+ncr1F+0x*;nX*NC@zeZ@Wsu0uj=TQtd_Ltmqtlr z2a}()^czUw~}Ey_MA#TsIcK9wU86s%FM8+vGI(?+f{gS^@V0I zPDd06Ww)HgbMHP~0*CTvsnC0lzhf9jGYBa{nMqent!eT;@ai+Zy1RC%SIFHTZXR$` z`T?fTkTq4QHCkkpF}Hd}2NeYH;m8MaRR)|AfBb_p@TD6F%(=9@FQhtE0eB&a|VDk*~=$t0-U!^*sxK7CJm2b zM@ymXFz6WLrZIi_#(b>}Af3+LU4VVQtpOSy_HA(puQ3Dc}6eP>~jP*Fp0&8^EFZ=$FL(V ztyyiHp_3e*bjxjz72?WA0i7ja)EDG69`OOOuI>km9|qjHp2!oi&_Bta6R4am;v8utyI01%TZgVB)o zT1!?p;hMuslI$n727kY?e)XqM5d|q_EK%k}8^e%Bv==Sg`j;d9`0tO73jc zfvzqZxgDRvuMT#G{cd)=%UTy73t(ogy=*dVYUua`!fs?Va<&p61VQ(19|fUq(5RO{ z3pg&o9n98nR^iO}*gBf$_}68zM4J7E0(@^MDHPgDw%liA?z;eMfrr8kNjQS)f!)@^ zZPLDUgHO)-WRpLMj_@@Y=eTz9zVFc;r1*HcR3P0un?Jp>NgwBcw~%z-4i7oIe`7f! zv-f0Y;Sx&wNR0KqU$2-+pLL$-+`8n7K8XjL`amcgOUfP#<&hmwkCtZ*1Jy^7D~r7H zpKKxTxdiCt1%P4J#M#b2A_7#tBrEt@vmd;CmC!PkOy#(!iWBl8pY50yl}En|ZS-&5 z81}2a^T1N|ag<25JY_X}7V2GZ4S=J#heeZh_^7*Qb)tqZlWueG9=OA0zC}Mmd%OjJ z5pOG}8mrxvut2y`fid(?QcX*ryc~r+A89Np!L$CJ?pb9z7VA|E4dORxtflDZh>h_h zP0ZFH*Y!zTc!u|A(Tax@Ns1g^mmG@?8groT&FGDl&q2|gI8ivKJtA!HS?JoT?qy9J z{6c#!aBTEhGHBDbq0EyLI5(;GWoFtqv6m^&FyguQg70A-1)>CG<#+)q)6cIPPh-RS z$cBTO*Ps7R61o>8%mtBHk^J9>JW59QMiUF{(d6{lkiF1wa5P*jvIOv7OafFCgjne* zi~i*>zsdHN6T_oJ!<7@Ngo8EhjUn0Wgf^F>PXzn4XUZFh%3Is7Ojv(xt>#Mw+#B}Z z-?^xlt={#Druh5%DH*;^J(0=sl2e<<|2{|4Uno)X{x$hzX(uD;K-=)tyM&_Vu&tt= z0Bm+ZO*fLVhY6bP7ioT6r;#)Ll!yc=h6=4*VZ(`q^bY_?F=*)zpRSAH$(IRt=A*uN z&Nn^Z7&LFkg^`a`3q^>OpECcRTy}H&|Nov9ZPm<4=lcv9rRbNfGSb{}ogA!F@@EG==)7qoJE)#Gu2%}(Ku+Jm7M!xXxjt<)!H%=jEC zEbrVL7)0NG+Z)l~gU%f9Ju;3ASxt-lw=~mu1hn$x>Dkq{S5^VD7TFr&PIp=2412$BG;7~`e<~N< za*Nk*Q(j-b@LOKD*FpBrz=R2}e2Cuos|FI(SFvF9r-0 znK=M9`o&uo{GBe;zlZ#bXDUT|GNb*Ffj4XVZui~wn1P?Z&~Xl|BtzIYi`_r4^Td+o z8G2t&s>a~M0P2qj#zhDE7cl2zg;EZJaC4F5WHE`2q&8umh`Y3ZiCeXQnOCJ8KP1mOm$JE3q=z2JVqQfl%92?(!zdS3&h1y?12MafZa1J2oK%)Tf+06hoN}CX zE)Zn@WYI-Q4*%H|)X*T{qt`~VN<@V!W^lJ&(S`ilV?a8z7$emi|9*Dots>5yFel>o z6Co!2;9F&8QdMxKS!oSr?ceAZU>{gcwe&te_@Q&op`G){WZI1w7EyR3F z@!fZqbjSlB){M(JTKh{5N7RfPA?IZSaUOo_g$0ZCN`8YDnyC{W#CB`f`1Z+P2^Opt z{NrL`LLm`ix}QhOlgXqN+!fy=>RhKB%?LgBJ0-jurexgHGI9Ivot}#QhJTJn;RuV` zKu?~f9jC1AbVKPz0k!?)oV5ww^;dRh!0rsnDah-S6gT%{oI}@WEIDQcwFqnZA>eRe z@~so+z}2fR?35c7w7g4Xfv^l;7Y9xTdk&KX3j(8@;z_pgfK3xwMJ*ne0IlAyTp+KP zO6n8_S{u3GZ{s}O%QfrYigKU+d1w5~)r`bAPih7zx9_?_Nh6Z{#cPd++KE(sF)D6uVW)ndFx(3J#1wyz_~UH{7=^Z`bd{u!=Mn& zE~u1!P1krR;&jLlFXI%*yR&oJ$yF1H4D9>xA~N^y@9&a9D@@}a7}gq(Ma|=F98G%d z{*MRgzbs$laTy1*m#_e=S&6`fpr}c=mOqa=@>H&2bZ;T9)vf0-c}X z=J{KMYF=Yu73fM(P&LZQoPvIWZyfD_|FQ2sC;%WJ{sB(cs$D4}LgDW$1WS*0(z6$O zgct1hBE6O(POx}uzZrw8qJ0z}{K%$)T~9Bca-FwUfk&U74x3cfq2ea|wZZxG#<7;W zH=rfm(HE7N3?j4LpNkD+%B( zx&#DtI~Dwhd`>f(>RHE~Z}LN(Bw-nW)$Jjon}JQQm$&)K_nIVX+?%)g2f1AXJWrHI zgG7A(PA@^u3liV#Al6iheyap$i4%yAw$iK#xb6nx9HU_tYUgCfA{n9r8J){KoSnRN zuFW;EFAjfWmK(+!M$lozQyd)6Fi{bnj}9k-h1bn=3VO^Iqk=F-^}kg{l$p6sGvn!T z3s$3fZ5c*A0d&9_ED&t(k;*S&#)S#_3JjV}Ks-ncw0G>nVuBx&!yU_$uprYDxv{Ak zDd$)2gO91a75mdu0)`t%(=w_Y$5k)?laVY^v%jPdKOtAqW)xYz@jzX-dI84Szg3ySG_h z>VRK?$y1<16F8<${Y(}ZkL2TsqcB2ZBFiuNKElc@F?kUDOFgvDT5g}#r}5hmj3|WD zx`<0~F7fix0mTht{v$4xz_F`=RQToa0;*#5V{QsK1$>qQC98F(S#l$*zebbnDH-nP z8*9(+0iCo-(Cx{p(o4eT52#|GS3S|D6v0Sq&ND-{8zy_-iUjnwl{}B=-wR;oV-TTV z+xOeMXZH7ZL@AA~UyT=fVC~$cBMDaKV%-8myjYM+Mf?Ue`YoO6W@~HvV{~}C(PD6A zGZvNUfq3GBS#;XTLUpGQn<>zDDd!01th;^=!F-5F_XEuk(;pk{vy^s=P%8vw2x2_; zeWI|Jqn3Q0B%i`lC=>VIX~MJN4N80N|8|)WQuKqEg3%F|cSSHodkZThj&FTEJ}mnd zYg30$*y<}FbHq!m^V@eEZH{9q3Di==J^XI|@T1DKQgPFc47tig?W2!B-2}$_`uN86 z9we&f^fIq{t$(moJNU87A&NIHXS5{7X)57wQ#RUM%pZ@9z#hx#?)NhKL#dkX=ExEj zD(aEi5FTjY%n*Te$LPtNTS3Wxvx|j77Jhh zV4sNpgom~9B#_ayfD+xvd-vsiI`3-&zg3#GbFoCw3^1J)vZ9uzYkXIqJoik>S-hhZ z6m!x5L3wQr*(S;Bk{JDRL04bN{Y&_E2k3Ya2 z^1)}bz2hMG?nS(<4A^q;YxEK#C8PXm?Vk9RTXOi#*F4&Lwvjqm>EPu&H80_ZDa-dn zRJ0kRUaHreHlHr`o^oh6oDfr<5CS5nJWjtGuZ(0^a4fBlTCbeku&`|S_Mq{Iz0l*K z0}1s6sAeDbDq}4=)5NR?$=dmWVwdM!IIC`!l3&=K$%K|ucWTeBTrZ&R&ApVI|K-bW zMc^^{%TgBng|X~fF|dpX)rgKW2;SFE+LO00&Z&J-w!RvWL39Jr&O2gdoQCYA_=3lU(lU4_k zjfstg?J}!{M+jf#mFxFZFI>LCpJw?w{SqJ~NvTV{@lY3NPw<&qyu3TuOx(xTZ5OkX zrUJ4yx=*sM_A~WuR!2Q)y;UiryIi0*{9V@Hqyhfb?jhIY7i8phvg?caioU!jlOsA)DpN(^rG z5C*+|+td&fB9~gCMUx}B{k6?2pUr`~<__N`%>Uq7I)Fc93~I4u=>#R?n5LWib1nh> z8?XV-wH#xVL0xMn=FT8R*x4LaJXJ8OT5i;X`_HXsDw)?P2B|Fi{11g5ZG(ghc)@;} zOU3N$->|==bRj(u10a7Ev8nYp?SF$NH=XQm(f7h&Wd$?;G84JdY5AT*CzZ2?|H&V7 zX5TMylqaQc>G^on6bKnRwFZDbE<-vYPM6tzfqUorOl9!v>%WjbNN^^)G8T}-N{Ruf z@Y(%edF82>B78k{F?R8K9(>TK$LW#|i79nezxi=ROh3*FC2WPo?2K?}Oo`<^++FvK$ zc30`N%B`f_4!A?EoE9;9J_>?it*?wV)L6j9cE)n5NGr>yG-yPei4;$o5!Wh2%5^v9#mmawEl<*=>f zf2pQX`De+9`S)b_MZ}wZJb=RKd|MIfL-fduRwqiGy9dKt5Vzd?vIAIw1k#tp?@f80 zZI*r7pC_raqshar;;graEI}n;sdmoqbRPlgA|>5f%UC|;guc4(V16)iakHwh?K?<= zP1ZPT<#Mz?+`QbFLDP&a=#-j{4c*e2c1e^C%Z>1)qIC$7Qh{5q%LJ;HFLns8JSgD{ zxkAA&j!ufXN@hz1uf@!$` z^nVW!+oooR_W8b2uo;aJZ+%?p4qLJ%5!wJKqb`sm$@4;SH&Ve%$trB7b^SgcFL?IW z)Cinje#qZnnL&9{qT9f9Q4{SHbw5Opu+WA3P}HQ@+#QU(1BW(JZCvdU|IhqF|NBXX zpbKumirgE~l7D=9B=z>ahlGPI=>#CY{L>l*_XJ0I-JcuJ1?VtfMK9p)Tos4sCVziq zS-z-fBsuJSJtWqIFbz4+hnYR=KQqp8G%#(G0}aAehBJ~X%VIyW2ENUXo;fJMDG*Kn zvq@S1d>Vt-P+(!m2ZORca2Y*npuKX_!`A-RMt8D8(x&`TP)i7koviz{YqHRUvPg|+ zwRSnHh{*q*yBFUFi<7}9qI+)>13}unhIU^nB264-=PSH03D|)L*{nt`w~CsRoD}JG zR!towTZZ(+Lqcyj?ctLV_o?!?0^W3=?6G!;fgH4mZHP5}u(A+5S)!sqcNlPy{Z;VP zO)D z(96>O)QTQ!GNOg^CE$-sL^!ACNNS!=PKuKupIGsmSUWW{^uIs*OZfpD)yt|*E4Qw% zEdIS}%=h*iMeTLTFb`!b$CTG#V;K`y6N?TqT=CF2R4?l9-;!#e@3 zXUT^H=VIMF0Q$a&9=-$)5(YzOIym5L0hZQ1!Fx&hFJnk21rJAyN2`_S1tk zzQc{tw3fS~Ufd&{#AD_j$;+}5?;g|s5A+Xd-;c5%vr3ZBa`8}}Oqj?hD% zXO<`ekAl}yGGICR*NJ?ky7EKRN)~a`dt-mQYKV;35>9Iw7uA_W^#N3k)ju#}s7*in zVhyju8N=rJL4`=(4noD%3t+O;S}+d!3;52mLYfNAy4Q+)0hU;Ks+jUq=L-uC*rC4# zfVw783<19?>FoMjX{HjAH#-xZxxdsWRN8Q4Zal=@f}g02%-1VUD?hg5MfIsXurqeL zbMfvq#WMfH=Mrcm&gsG?7{L3q>)3|!XI=sY=O=-1NlH8&emz^b4w>s`=HR`z1I3Rwb&f-C*F%mQ21O7Y~b&p*PDwy%p=?4G=F`@iFq{ zSn`&yTUj>-*0?Zjd|&GevU0mB=N{pAW(y(86~rXbiq`ih2=hJ^G_C79`LXl#*54hO zYoJO<@fjU+@0z0P*hSV0u@C#pcT!$Dz1~Qjkp%Iv=~WGZ3%S0_y1xZ5t+@}|r0gW_ zN4%kC{9&+Zhn^+A5@TH!oO;(Dx>640n@#uO;LMv%rIKZKw*J$(diKfC&WpJsnKKai zUDSrBe6x7b1bvDwz#_^L|B#?37ZQLt9!;!uulo6dY;ZG7Yy*Z zyPsEUn7F`huI%rfk|I*q<&HX%Y*?dc@WVxN?`?02>YS!s&(_pw$o=P`siSV(TL8}VH|U?- zxu<%ZRMl@RbduicOPD1Rgp~a&ljd3{yZWm`f?;F+ch@$i zJg0yAo+%ZR01b@WUVX|!MzFI=Y4@3_pNc_?30h$}0KE13cRZc9tg>?CFB4|Se2NJ-@;qx zY>3YbOfm%oj>7BM1^AmvGuSp#4CvDJL;L4-9HU7o;1NpEyf7qr++ zH@7WAY&N5^=)AeSEg$oZyp!1gVB!FF954kZRzae>T5v$r@jO-}!Lvp+6KLNeXu zWLa!({S~gMR=iIPw8JsJmj`J(<=B;J_Goj$6^$grARFyNGT>v`>eQ?Tyx-Jl1nlLT z62C^LnW#MwObvm_8|#|ux1O;{%Ik0PIexl$OQ8H{y{J^T!0?mB&^1n>EQ26J11C)* zQ~#ImUfEaz@|6XKFYrw>P61A`YatriI!iWF2@LIKq5Lrw8V;Gvryk-@M;1)c*a+A- znAjhurhs$|jFtwsa9oj4<`3e^?VCKBA0;0pQINg-(Y*D)!-WL+f2nX}a|fvSmwDG8 ze`kCB7iQ{?y#JMbIqr|*dNM@$C&D!EJ3nmuVTx=Uqk{Q2v#1E33Q}L3dYqgTx(|Xe z{cfrL{NGpJQWT+fGGOK(^S~7cb(b*V)K>ha)KR>gr^dL*JEVsg1BzzH*%Ze4!bBK zBeh~fGtiEg@VwuDB$Ro6ePS=gNBIT9Mt(k=Vq+wBEP>2Ltt0`AMttgg89DB+azV@m zBY=$d6$>jGYuf!1_V8-Kkr1N*W5LMNoJtA*`rJp05ut7CVnA=W^4Tb8m*Z-(UP)`Ugo0a&oCpbg?x(RbF9|$nUz-`q(|dM+s}XrhToy2DWKVbUSqrI z5d*2$JW5;Z85>A>_vD+D8T9DJ{rj2xT|p9yQL*!MBCG0~VIpC>1%(41;m6}?wq)>R z<%|#i4QIR~>_@e2%i~2^aq^3lj1mJ7hA=y=%?{Pka4?w1QX>pa?H}G47+k&ZIdU|- zIdZO@s%dh zNb)1k&rn9)G1~RQGcNS#b=H7S#*()jmUy+8Yz|qrTQ|hai$6#s2%w$y0Xgi|=|h*% zQ(d`*4iWc_QW!&%7SI3ges){;v>z9-y>hiu5>38BWt}VgW4rFz z*}t>c_BJam+nnMz4e*{rlhAN%ZQzO^1@S>JvEynI|7gvM;v?x9m71XXplgP{23GFY z>bci@eEst%3BhDcCUnn_E~$nj^e9)LzzkTo#b(j9-5jC%)TvlzRm@QxnXPMoQHZR7 zEs|_HWZ)f3NC9+PnbQ-!60)G@g1Z1|diUq!9M%08S4TJYCRIoQKAf;s zLCo~)$rrJgw|ce$8iHUC;(VK-e9RZ`6}(M+>@C>Gp$X|_>C0aGJ2sqR#Yr*ySZsuT z$_^F!hV8YL*4tNcl&d*%t>pO3$I85yu0^s#z>Qrw*BPM+ZjCx4gQ9ti>9%D5E8F*v z#e%a8y4j!h7ykSfT&i%U0SVf%S|KD*OU~bE?-JtIDrrD(V zSUzwscU#rJhF;YlNjW@+8Awkc56X)BL54Er^;Dl?llz)+btj5L3vykLLAHwfe0%I_ zm+Gh*VqLM!jkB1S%2FJ?KqhoyRiwiM)ocZ`Rj!A>c=El502tD$JA>}NV@(1{3~7uJ z72b4vX5RRkcV149+|s=V$?=my?z(K3OD_mSXQ_sKv6~N?^=pPUbY`i!?`6J>YX&V! z24~KSQniXK#~bFWMhE%i${X#9U!>lUA%DjGBhZUrL~tAQ&KH8IAMlY))w5HcS7(&1 zKgrPZ3^*ElkS zWRuYi5=g&9hyG{(zfvL7Dx(9Mysk!j#*?Ky#Uv!@R_^S06{0eFV@i582(?#ckhPAh z2Bkiw8=PxF9Iw$v+YtwjuS_b+wxYzY`i_d%$Qb@c@v?9~zv=H@f?TIQJb5+G{%Lf( z0c$1a;keIeeH9k~TK=()H6!(vN_kq!^e(H|J2V`0&gV@Qt7r=5A7;Crp|_CR>ANcK z`63H?AYkjRi!FGv<3W7#AQ(0!=ddD2qoGH>ez*{xxDk1NZEKmD&r<8bkmFC^-TA@A z&URq3j5r_S>&F+Y%pbc|&rz8YzOwvX_0}9>=^EG0R`ukYHR$SepMh5%l~~HXB(>L#Y#$n4Hv-$Z&L&P}iVOE*SDp2XqR)@+|jBr!*EIQ}eSbMRH0z zC>hSuzVa2+@kRln%F4l@C)w;rFn36SYAJ|&VZSlD;x``zP>+rl{28;urGB+&+Lmud=Xb2YJXS2-4@fXPz@_zo-1S(av%1^ViwDIVQ`|j{8YPf=*kv zc3j4OWRq%FsjjCtd7FvHy`L;n)K8DLrpKH#``5=TyRh0R@shl+LaXBL-_`x2hA$jX zIO0xfDg?IZdNSO@UX30fsr#*23 z5R6L)(n6Zf+jB^@B zA1DWfz!N^`>``%G)=r_Uw?d_@YO%0K!T+v}#KM&Wj=Xt^0GU%`egr_&SQ#ASCq*mI z#SUI0mE+ut^P?ZnfC^!jWKf*ddG>juFh}LWiN{BRNvc#NlO~%R*D@e4ObAZJ4!BZ% zf%Z66#({IFo<$A2gNr^g=NAeTPmOVddU*pZakwiu*zr1TXM#w<6gc9>{(*ANw_hK; zh^m63WHxkqVk@v8C=JEH%?@?96NdTYZE2o$K;0jz6O!|swP5)@_a{qvi6mW>k8gVyN51u~)!JFP6II8HQhj8A1K&k43tX3J2p}_SlDSp$6~RRVFk@$>JCx zIMA&~e2fpn*#dTGZLRH==zch!>f}f|ol=bt)Cbi0v~EZ1pevc1HjG3Ujb5N)w;!g$ zWHzVke0*j+Qf@q4Ol%4NE%qkRYV9;4Gvjtu<+Fin&U@UJbm_KDl)T1 z4$R^-_7>z+_^WBzpPKpOZ-DU=y>BAwtzK-lQo?lc@6xV|g1{u^^n_5g+{kCvr_la$ zF$f&YEQ#d)yBmwjFaG7c@N2jUclnp}?UtjvBu%a^xLhG~QbQ+y{7agzqdQ#l3ollp ztD;AvvC*u5IZr^lh8{po@bCO#)vG9Y^7}<2@y+`pfhR|oA@yXzeu)yF~rbH+daK;D44e-b6ef+4WyriertD z({DuXHpD6IMcMZysQbtHOKmqH&Z_DJ+#3Upfn1nrsnHVoWjYPnO7FNvs(3D5WjM%E ziI%#*@N2DR+WJRx3s^};~FnYOAr!73GHh-uvzW`@m4Ec^8bA--%3#U;92<5PE0-q{y^XMWEC@3Hzl?Wb5NwesDDn^A~la9}Q9) ze=g=FhP;aaMWfnBZOPkAWCNW6ZS!&IThFb3JAns+9S@z@`W;CrJw1u#KbJmHiPee+R6AnKFtc9sKoq5Sk{=O5z6YoYg=E%vgCu|| z9_TH%GHy`u3)+EjU*nba6H_@V3+l#^cAvnKj35^%>8S*xVt0n`sQ5Nd45o$1=pN7XI}KcW)#165G)=BKc^E!jnmU30%5CMfU3khSKu}6*k*;sD;g8+EMar@( z5%K#ZRsgwiw=Kda2?|0mydhvaF?xT=>*q-f2ONL`!+HZN1waMsm&@RSz<$Cq&-9(e zH>vek=8X=I%19nS08ESb+a48r=JTIOPYpMe$FMW1ck~3LyZgIRFCGJJ=rCKAf6YV3 z+waO9@;nB*?OT1UHM#Iu!fw*k4BE}qU%T6OVs|hqs;m_dnCLyll$up&{sMr{LH1-R z|BXZX1NExXO?|LU1VT7)~|#3`PVSZ3pOS;u=D#9EEk!q_Zee<7x7$oQK*%5Fw#y8sV6# zt#iG~d$)TKKa)K*Uk z0h_})M7{0Srq;7ef1fTq=z&O~BrNt_#*0{+ne{4*RsFTBkbY`QlnJQPOPK$Y`>gEC z9-z;!!Ki#TqM08nv$K5I0*V7h!jmRcs46Mw8pD5fkdOfEz<$xyRD!gJSOMQjg||In zVR!PSN28t%e6epv{y+eU#idNwm5ypZsTKwTt;&&y5C61{w}r+$X8&t8ByHOnsCnPIg2RjFf}KR$Y9N-_D(nnR#7L zb&@@K`!(=S|3~7YPtKGKQT?iqXd#6CCPVDaTVWz^YWcvr#drRFbnj4@dF>;lijWeG9YArC89WvUFi)A4QgmO`lXblFo_Zu~;uILVF*vL)w(L>Na;Laqph_=e?7OopyG*~L zYj{ zSx~3>x)N8F$&4E!PZNZB<1&D{!NwNMrU^2tK|Rk^g9D_oW8nm0R7TC_8^k%H&JIfC zoY*b?@C&S-$R*CqXwL@FdQc}Ez{Y+sQ_v2fimEcSz5~4moeR3gIu-q2q`ayNBIVk> zs-g@KDTfS&!1CbI$JwpWR(C)WF9yn+ERHkk)h2^bKg9E`B8!0Ud@IxbeV^30;NrXP zAR^BLWn7DAC=e&P)p8@lr*2+N5PPWS0=%vU$%!+u z)>!uQ$E(ml2#f6vS*X;LfjM`eu3r9n%Uo{zHAp3w_%SH&bv7qO3n>|`BK;9j2!N$} z?SSa>elB)9HfYD3ARAa*Xy2P@)&=CS;#5w)28bc&DmDzCOt11yYhG%mjS>Tyn{$N&bsZX|Cf$=8Q7%;&Tibuvhm1CGb}z&I}s2Yw-q| z@&g7(d`d=dxRiW4{3=fx001D@COee*Jmu8=dZ39nRb57bXW&w-#u|Uqny^{yNZ7Vm z+*go$?^k_gba6t_gPQrv$Qnn2*l&&|Lnr3*A%JK{ok!z#xDHPuGbZBfXM+b9&c8C>FzWqO z51aFC*-M^mZh}fY4Ra}qCJUBhJVoA95eUW`vZ_LmFKbDldGILzt%!k2T=i6*#oowjEwNxAx_rAyD2*{Bu7iy4)o7_T^3sG;vFPXp!QHE? z?R|tY_m3aI=3(^rRaMB>GC@8Xg~Tp<6Wy30wy?qI{%naumip?8VqS?_E& zPH5Ax*HWFBjGsFjx^F}esrPNcCKGMXLk@O8bFR2WB;pAu+5x*7Zw#ul+hUnm$z-3X z3WR^zl&mv?k=CeDkux^2pPA?PcF$>R@hBNge0%rb?wACGhZsb{=&Ug~L)cC@@rHcd zg!k9UcJgGQgFH->YNuTjqxni#CX7~JILSdH!C>QB?^cw$GRp%ee^7cz2wNCF6%u2N zVj;uLM~iX(Q2)ml)7QM?i_N7*pT}6mP?e^YQmTjy-=4lPhDxV=Y6|AOtWNf*N1uEx zH8_IU@EZpBI6koaq8no(qa1*o zGfPs90+RuK8Zxn_PhNI|Oa=n+2+pJk<&Fr(qTQ{2fLnWH`mZrfhbM?+2*{~T6Ty(` zWC*&H3KHBK!i!cu*g2W0uD^deC`gv1`i}HV_oa^*1w(!(1cwC97yq0tB+buJuuc=d z4LCY{pk&qtgQTe(dtC%R@xO1QtCaueKS>D#O42CEZ~dD7_J5zD!@Xeq00O`NKEV@0 v3XfEyUsOH+?}DVaDd3Y7|93fU9vVDsk*ZsXXOF-R0e^Jv8fuiN*@XT-F!Fa+ literal 0 HcmV?d00001 diff --git a/RFC-0007-forward-AD.md b/RFC-0007-forward-AD.md new file mode 100644 index 00000000..ecd12745 --- /dev/null +++ b/RFC-0007-forward-AD.md @@ -0,0 +1,308 @@ +# Forward AD in PyTorch + +The goal of this RFC is to discuss the design and choice linked to adding forward AD to PyTorch. +Most of the content here comes from the discussions in the corresponding github [issue](https://github.com/pytorch/pytorch/issues/10223). + +This RFC will cover: +- Timeline +- Goals for this feature +- Quick theoretical introduction +- End user API +- Implementation + - Dual Tensors + - View and inplace handling +- Testing +- Appendix: View semantic definition + + +## Timeline + +- Implement the design below for a single level and with limited number of formulas: https://github.com/pytorch/pytorch/pull/49097 +- Implement remaining formulas, update testing and ensure no major performance regression +- Add more usage in the high level autograd API and make it default +- Decide and implement multi-level version + +## Goal of this feature + +The main goal is to provide a way to compute Jacobian vector products efficiently by leveraging the forward AD approach. +This will be achieved by: +- Use the new engine to compute quantities such as `jvp`, `hvp` and `hessian` within the high level API. +- Provide an easy to use, python centric API for users to compute forward AD quantities. + +## Quick theorerical introduction + +The simple definition for forward AD is that it is computing a Jacobian vector product. +Indeed, if we define backward mode AD for a function `f` with Jacobian `J`, it is computing the quantity `u^T J` for an arbitrary `u` in an efficient manner. +Forward mode will allow to compute the quantity `J v` for an arbitrary `v` in an efficient manner as well. +In both cases, we mean by efficient that the complexity is the same as evaluating the function `f`. + +The AD implementation is done very similarly in both cases where the Jacobian matrix for the full function is re-written as the product of Jacobian matrices for each elementary function used within the function. For example `J = J_3 J_2 J_1` if `f` is composed of 3 component functions: `f(x) = f3(f2(f1(x)))` + +The computation done by forward AD is then `J v = (J_3 (J_2 (J_1 v)))`. As you can see, each subproduct is done in the same order as the forward functions and thus the forward mode AD can run at the same time as the forward pass. + +A good way to model this is to introduce dual numbers where the imaginary dimension represent the partial Jacobian vector product while the real part represent the function value. + +You can find more details about this in [this blog post for example](https://towardsdatascience.com/forward-mode-automatic-differentiation-dual-numbers-8f47351064bf). + +The design choice that was made in the discussion is to build such dual objects based on Tensors and build our forward AD engine on top of that. + +## End user API + +### High level autograd API + +In the end state, this will be used transparently by the high level API to provide best performance. +Since we might not be able to have full support for formulas and multi-level in the beginning, use of this feature will be controlled by a boolean flag. +This flag will allow to fall back to the slow, backward mode AD based, implementation. + +In the final state, the `jvp`, `hvp` and `hessian` functions will have an extra `fw_mode` flag that will be `False` at the beginning and will become `True` once we are satisfied with our coverage. + +### Standalone API + +Since this API is only intended for advanced users, we decided to keep it close to the design principal of using dual numbers and close in spirit to the current autograd API. + +```python +import torch +import torch.autograd.forward_ad as fwAD + +# Create regular Tensors +x = torch.rand(10) +v = torch.rand(10) + +# Create a level that is given context where your dual objects live +with fwAD.dual_level(): + # Create a dual Tensor that will perform forward AD computations + # This API uses the current level by default + dual_x = fwAD.make_dual(x, v) + + # Do computations as usual + res = f(dual_x) + + # Unpack the result to get both the function and the J v value + f_val, jvp = fwAD.unpack_dual(res) + +``` + +There are some advanced features on this API as well: +- This API is completely backward differentiable and so backward mode AD can be used anywhere with this API. +- When exiting a level, all dual Tensors from this level become regular Tensors again. This ensure in particular that, when used within the high level functional API, the intermediary state is never leaked outside of the high level API function: +```python +with fwAD.dual_level(): + dual_x = fwAD.make_dual(x, v) + + _, jvp = fwAD.unpack_dual(dual_x) + # Within the level, dual data is preserved + assert jvp is not None + +_, jvp = fwAD.unpack_dual(dual_x) +# Outside of the given level, jvp is a regular Tensor again +assert jvp is None + +``` +- This API supports nesting levels and providing a specific level kwarg for each function to enable higher order gradients easily. + + +## Implementation + +### Dual Tensor + +The dual Tensor design will follow the prototype in [this colab](https://colab.research.google.com/drive/1hXB5g1ouHXHFYf9pDTzH6pG2q0Wyz92V?usp=sharing0). +The goal is to have a data structure attached to each Tensor that stores the dual values, if they exist, for each level. +Each operation on such Tensor is then responsible for updating the outputs dual values based on the inputs dual values. + +The first part is implemented in PyTorch with an extra field on the `AutogradMeta` associated with Tensors. +It will store a special structure there that will handle storing dual values from different levels. + +The second part is implemented in codegen and manual functions by having special code that ensure that the output is properly updated if any input is a dual Tensor. +This is done within the VariableType kernel for now even though it will most likely move to a different key in the future as needed. + +Finally, to ensure the strict scoping presented above, we introduce a global state tracking the different levels and which Tensor belongs to which level to ensure that we can clear them properly on exit. + +See the PR for more details on these implementations details. + +### View and inplace handling + +View and inplace is a key feature of pytorch that we want to preserve for this new forward AD project. +The semantic that we are looking for here is described in the Appendix of this document and is based on the bidirectional lens idea. + +The practical implication of this semantic in this implementation are the following: +- backward and forward "differentiable" views are two different things: `detach` is backward non-differentiable and forward differentiable while operations like `fwAD.make_dual` is backward differentiable and forward non-differentiable. +- we need to track differentiable views both for backward and forward AD as in general, their base could be different. +- for operations that only involve dual Tensors, the semantic from the appendix is easily achieved by ensuring that: + - view operation forward AD formula generate a dual that is a view of the input's dual + - inplace operation forward AD formula modify the input's dual inplace +- If non-dual Tensors are mixed with dual ones: + - when a tensor with no dual is updated inplace with a tensor that has a dual, this change needs to be properly reflected on its base if the modified tensor is a view. + - when checking if a Tensor has a dual value, we should check its base when it does not have a dual value already + +## Testing + +For testing purposed, we can easily update the current `autograd.gradcheck` function to compare numerically computed Jacobian with the one constructed using forward mode AD. +In a similar way the backward mode AD gradcheck reconstruct the full Jacobian matrix row by row, the forward AD can reconstruct the full Jacobian matrix column by column and we can compare the final result with the numerical Jacobian. + +The overall testing plan is going to be done in 3 phases: +- Test of the core implementation and the view semantic (proposed in https://github.com/pytorch/pytorch/pull/49098) +- Once few formulas are added, enable gradcheck for forward by default in our test suite and allow it to silently fail when encountering functions that are not implemented. This allows to easily check the behavior with other components such as the high level API, complex, Modules, etc +- Once we have most formulas implemented, make gradcheck properly fail when it is not able to verify a forward gradient and audit the test suite to disable forward check for every function that is not supported yet. + + +## Appendix: View semantic definition +This document was written originally by @ezyang to describe inplace updates on views as bidirectional lenses. + +Bidirectional programming [1](https://www.cis.upenn.edu/~bcpierce/papers/lenses-etapsslides.pdf) [2](https://www.cis.upenn.edu/~bcpierce/papers/wagner-thesis.pdf) is a programming discipline where you have two different representations of otherwise similar information, and editing one representation causes the other representation to be updated. The classic example in the bidirectional literature is that you have a database, and you have computed a view on the database which you modify—you want to propagate this modification back to the un-viewed database. However, in PyTorch, we have our own example of bidirectional lenses, which are *tensor views*. When we make a modification to a view of a tensor, we expect the base tensor to also get updated. Inplace updates on views are easy enough to understand in basic PyTorch, but with the addition of conjugate views and dual tensors, the more sophisticated language of bidirectional lenses can help us answer some semantic questions. + +### Bidirectional lenses in a nutshell + +In the semantic universe of lenses, we are responsible for defining a get function and a putback function: +![Basic view](RFC-0007-assets/bidirectional_lens.png) + +The “get” function, in the context of tensors, is the actual view operation, whereas the putback is how we map mutations on that view back to the base tensor. Each view operation (e.g., view, squeeze, etc.) is defined as both a get and a putback function. + +There are some laws that the get and putback function should obey: + +* *Acceptability*: get(putback(t, s)) = t (if we put something into the source, we should get it out again if we get it out later) +* *Stability*: putback(get(s), s) = s (if the target doesn’t change, neither should the source) +* *Forgetfulness*: putback(t2, putback(t1, s)) = putback(t2, s) (each update completely overwrites the effect of the original one) + +It’s easy to see that conventional inplace view updates on tensors satisfy these properties + (in the absence of overlapping strides, anyway!) + +### Dual tensors + +In functional formulation of dual tensors, a dual tensor is a pair of primal and tangent tensors. Correspondingly, we must enhance the meaning of get/putback for view operations to account for this pair of tensors. Here is the most obvious formulation of get/putback functions for dual tensors (getD/putbackD) written in terms of the get/putback functions for regular tensors. + +def getD(s: DualTensor) -> DualTensor: + return (get(s.primal), get(s.tangent)) + +def putbackD(t: DualTensor, s: DualTensor) -> DualTensor: + return (putback(t.primal, s.primal), putback(t.tangent, s.tangent)) + +As dual tensors are simply pairs of ordinary tensors, we simply define the get and putback functions as applying to both components of the pair. It is a simple matter to verify that if get/putback satisfy the lens laws, then getD/putbackD also satisfy the lens laws. + +Dual tensors also support a number of view-like operations which aren’t simply just simple liftings of view operations on plain tensors. For example, we can project out the primal/tangent components of a dual tensor; similarly, we can construct a dual tensor from a pair of primal and tangent tensors. If we maintain a clear distinction between Tensor and DualTensor, the obvious bidirectional semantics for pairs suffices: + +```python +# In fact, make_dual is a bijective lens +def make_dual_get(p: Tensor, t: Tensor) -> DualTensor: + return (p, t) + +def make_dual_putback(p: Tensor, t: Tensor, d: DualTensor) -> Tuple[Tensor, Tensor]: + return (d.primal, d.tangent) + +def primal_get(d: DualTensor) -> Tensor + return d.primal + +def primal_putback(d: DualTensor, p: Tensor) -> DualTensor + return (p, d.tangent) + +# Proceeds similarly for tangent +``` + +However, we may also wish to say that there is only one concept of a tensor in our system; every tensor is a dual tensor (and simply non-dual tensors are those for whom the tangent is zero.) In that case, we need a slightly more complex semantics when defining lifted versions of all our operations: + +```python +@dataclass +class DualTensor: + primal: Tensor + tangent: Optional[Tensor] + +def make_dual_get(p: DualTensor, t: DualTensor) -> DualTensor: + # TODO: I'm not sure if these asserts are necessary + assert p.tangent is None + # I believe this is the semantics implied by + # https://colab.research.google.com/drive/1hXB5g1ouHXHFYf9pDTzH6pG2q0Wyz92V?usp=sharing + # when there is only one level of nesting, but I'm not sure + assert t.tangent is None + return DualTensor(p.primal, t.primal) + +def make_dual_putback(p: DualTensor, t: DualTensor, d: DualTensor) -> Tuple[DualTensor, DualTensor]: + d_tangent = d.tangent + if d_tangent is None: + d_tangent = torch.zeros_like(d.primal) + # Preserve p.tangent and t.tangent! + return (DualTensor(d.primal, p.tangent), DualTensor(d_tangent, t.tangent)) + +def primal_get(d: DualTensor) -> DualTensor + return DualTensor(d.primal, None) + +def primal_putback(d: DualTensor, p: DualTensor) -> DualTensor + return DualTensor(p.primal, d.tangent) # Preserve d.tangent! + +# Proceeds similarly for tangent +``` + +The most important things to observe are in the definitions of putback, we preserve the pre-existing tangents on the original dual tensors (prior to make_dual or primal). This preservation is essential for maintaining stability: if we didn’t preserve the tangent, then primal_putback(primal_get(d), d) != d (the left hand side would have lost the tangent!) + +### Example: putback transfers tangents + +Let’s take this example from https://github.com/albanD/pytorch/pull/1/files + +```python +# dual is a dual Tensor +out = torch.zeros(100) +out[2] = dual +``` + +Intuitively, we think that out should become a dual tensor after this operation. How is this born out by the semantics? To do this, we first have to define a get and putback for indexing location two in a tensor. + +```python +def index2_get(d: DualTensor) -> DualTensor: + return DualTensor(d.primal[2], d.tangent[2]) + +def index2_putback(d: DualTensor, r: DualTensor) -> DualTensor + # We can't conveniently express functional putbacks in PyTorch's + # functional API, so we clone the relevant Tensors and then use + # the mutable API to perform the operations we want + p = d.primal.clone() + p[2] = r.primal + if r.tangent: + if d.tangent is None: + t = torch.zeros_like(p) + else: + t = d.tangent.clone() + t[2] = r.tangent + else: + if d.tangent is None: + t = None + else: + t = d.tangent.clone() + t[2] = 0 + return DualTensor(p, t) +``` + +Take a moment to verify that all of the laws are preserved. It is now easy to see that when we do a putback on a tensor, if the tensor dual (r in the function) we are putting into out (d in the function) has a non-zero tangent, this will force the new version of output to have a non-zero tangent as well. + +### Example: views can have different perturbations + +Desmaison et al have suggested that perturbations should be associated with memory locations. This would make it effectively impossible to have two views on the same memory with different perturbations. In the semantics we have defined in this paper, this situation would not occur. Let’s consider this example: + +```python +x = torch.zeros(100) +x2 = x[2] +y = make_dual(x2, t_y) +z = make_dual(x2, t_z) +y.add_(dual) +``` + +We have multiple dual operations going on in this example: we view on x, make_dual on the view, and then perform our inplace operation. Furthermore, there is another alias on x in the form of z. To understand the semantics of this program, we must apply our semantics in two steps: + +* When we perform an inplace mutation on some view (e.g., y), we must use putback operations to reapply the inplace mutation all the way to the base tensor (e.g., x) +* Once we have done so, for all child views of any tensor which was viewed from the base or any of its children (e.g., z), we must use get operations to recompute what their new value would be after the modification from putback + +So, in the example above, we have to apply the following steps: + +1. Compute y + dual +2. Upstream this update to x2 using make_dual_putback +3. Upstream this update to x using index2_putback +4. Downstream the update to z (from x2) using make_dual_get + +Let’s hand apply these operations: + +```python +y_new = y + dual +x2_new, t_y_new = make_dual_putback(x[2], t_y, y_new) +x_new = make_index2_putback(x, x2_new) +z_new = make_dual_get(x2_new, t_z) +``` + +Notably, in the make_dual_putback, the primal and tangent of y_new are distributed into x2_new and t_y_new. This means that the tangent of x[2] is NOT updated; all of the tangent of dual is doing is modifying t_y! In the end, y has an updated tangent, but z is not updated at all. Although y and z alias the same primal, their tangents are not aliased, and thus inplace updates to tangent affect one but not the other. + From 6c746af7ea39e9984bfc58c6013522a8839855b1 Mon Sep 17 00:00:00 2001 From: albanD Date: Mon, 26 Apr 2021 17:52:03 -0400 Subject: [PATCH 2/2] Update rfc based on current progress --- RFC-0007-forward-AD.md | 37 ++++++++++++++++++++++++------------- 1 file changed, 24 insertions(+), 13 deletions(-) diff --git a/RFC-0007-forward-AD.md b/RFC-0007-forward-AD.md index ecd12745..71d2cedb 100644 --- a/RFC-0007-forward-AD.md +++ b/RFC-0007-forward-AD.md @@ -17,28 +17,36 @@ This RFC will cover: ## Timeline -- Implement the design below for a single level and with limited number of formulas: https://github.com/pytorch/pytorch/pull/49097 -- Implement remaining formulas, update testing and ensure no major performance regression -- Add more usage in the high level autograd API and make it default +- Done: Implementation for single level in core +- Done: Add codegen mechanisms for adding more formulas +- Add gradcheck tools to test both real and complex valued functions +- Add a limited set of formulas that enable basic use cases +- Add it as backend for high level autograd API +- Update OpInfo to natively test forward mode +- Update custom Function API to allows overwriting forward AD + +Follow ups +- Long running issue to add remaining formulas - Decide and implement multi-level version + ## Goal of this feature The main goal is to provide a way to compute Jacobian vector products efficiently by leveraging the forward AD approach. This will be achieved by: - Use the new engine to compute quantities such as `jvp`, `hvp` and `hessian` within the high level API. -- Provide an easy to use, python centric API for users to compute forward AD quantities. +- Provide an easy to use, python-centric API for users to compute forward AD quantities. ## Quick theorerical introduction The simple definition for forward AD is that it is computing a Jacobian vector product. -Indeed, if we define backward mode AD for a function `f` with Jacobian `J`, it is computing the quantity `u^T J` for an arbitrary `u` in an efficient manner. -Forward mode will allow to compute the quantity `J v` for an arbitrary `v` in an efficient manner as well. -In both cases, we mean by efficient that the complexity is the same as evaluating the function `f`. +Indeed, if we define backward mode AD for a function `f` with Jacobian `J`, it is computing the quantity `v^T J` for an arbitrary `v` in an efficient manner. +Forward mode will allow to compute the quantity `J u` for an arbitrary `u` in an efficient manner as well. +In both cases, we mean by efficient that the complexity is the same order as evaluating the function `f`. The AD implementation is done very similarly in both cases where the Jacobian matrix for the full function is re-written as the product of Jacobian matrices for each elementary function used within the function. For example `J = J_3 J_2 J_1` if `f` is composed of 3 component functions: `f(x) = f3(f2(f1(x)))` -The computation done by forward AD is then `J v = (J_3 (J_2 (J_1 v)))`. As you can see, each subproduct is done in the same order as the forward functions and thus the forward mode AD can run at the same time as the forward pass. +The computation done by forward AD is then `J v = (J_3 (J_2 (J_1 u)))`. As you can see, each sub-product is done in the same order as the forward functions and thus the forward mode AD can run at the same time as the forward pass. A good way to model this is to introduce dual numbers where the imaginary dimension represent the partial Jacobian vector product while the real part represent the function value. @@ -117,7 +125,7 @@ This is done within the VariableType kernel for now even though it will most lik Finally, to ensure the strict scoping presented above, we introduce a global state tracking the different levels and which Tensor belongs to which level to ensure that we can clear them properly on exit. -See the PR for more details on these implementations details. +All of these are now part of core pytorch and was added via these two PRs: https://github.com/pytorch/pytorch/pull/49734 and https://github.com/pytorch/pytorch/pull/56083. ### View and inplace handling @@ -125,8 +133,8 @@ View and inplace is a key feature of pytorch that we want to preserve for this n The semantic that we are looking for here is described in the Appendix of this document and is based on the bidirectional lens idea. The practical implication of this semantic in this implementation are the following: -- backward and forward "differentiable" views are two different things: `detach` is backward non-differentiable and forward differentiable while operations like `fwAD.make_dual` is backward differentiable and forward non-differentiable. -- we need to track differentiable views both for backward and forward AD as in general, their base could be different. +- backward and forward "differentiable" views are two different things: `detach` is backward non-differentiable and forward differentiable while operations, like `fwAD.make_dual`, are backward differentiable and forward non-differentiable. +- we need to track differentiable views both for backward and forward AD independently as, in general, their base could be different. - for operations that only involve dual Tensors, the semantic from the appendix is easily achieved by ensuring that: - view operation forward AD formula generate a dual that is a view of the input's dual - inplace operation forward AD formula modify the input's dual inplace @@ -136,11 +144,14 @@ The practical implication of this semantic in this implementation are the follow ## Testing -For testing purposed, we can easily update the current `autograd.gradcheck` function to compare numerically computed Jacobian with the one constructed using forward mode AD. +For testing purposed, we can update the current `autograd.gradcheck` function to compare numerically computed Jacobian with the one constructed using forward mode AD. In a similar way the backward mode AD gradcheck reconstruct the full Jacobian matrix row by row, the forward AD can reconstruct the full Jacobian matrix column by column and we can compare the final result with the numerical Jacobian. +The fast mode will also be done similarly to the backward mode implementation where we will use the forward AD to compute `J_f u` with a single forward AD pass. We then compute the full reduction by doing a dot product with `v`. +For the complex case, we will consider only functions with real-valued inputs and perform the same computation as the finite difference. + The overall testing plan is going to be done in 3 phases: -- Test of the core implementation and the view semantic (proposed in https://github.com/pytorch/pytorch/pull/49098) +- Done: Test of the core implementation and the view semantic (proposed in https://github.com/pytorch/pytorch/pull/49098) - Once few formulas are added, enable gradcheck for forward by default in our test suite and allow it to silently fail when encountering functions that are not implemented. This allows to easily check the behavior with other components such as the high level API, complex, Modules, etc - Once we have most formulas implemented, make gradcheck properly fail when it is not able to verify a forward gradient and audit the test suite to disable forward check for every function that is not supported yet.