-
-
-
-
-
-
-
-
-
-
-Football prediction model in Stan
-Guillem Hurault
-13 April, 2020
- -
-
-
-
-
-
-
-1 Introduction
-This repository contains the code of a personal project where I am implementing a simple “Dixon-Coles” model to predict the outcome of football games with the probabilistic programming language Stan.
-As a disclaimer, I am not a particular fan of football and the presented model is far too simple to accurately model/predict the outcome of games and fortiori to be used for betting (and if the goal was betting, designing models for individual sports where the outcomes are less uncertain, such as darts or horse racing would probably be safer). Having said that, this project was fun and a good way for me to work with “clean” data and learn about Bayesian workflow.
-Notably, I see my main contribution in the quantities that the model can predict (cf. suffix _test in the Stan code) or that can be used for posterior predictive checking (cf. suffix _rep):
-
-
- the number of games won. -
- the number of games lost. -
- the number of games ending in a draw. -
- the total number of goals scored. -
- the goal difference. -
- the number of points. -
- the ranking. -
In this notebook, I present an overview of what I have done in this project and which is directed to an audience with some familiarity in Bayesian modelling.
-Before going into the details of the analysis, let’s first initialise the notebook.
- - - -set.seed(1559354162) # Reproducibility
-library(HuraultMisc) # Personal function library
-library(ggplot2)
-library(cowplot)
-library(ggtext)
-library(rstan)package 㤼㸱rstan㤼㸲 was built under R version 3.6.3package 㤼㸱StanHeaders㤼㸲 was built under R version 3.6.2rstan_options(auto_write = TRUE) # Save compiled model
-options(mc.cores = parallel::detectCores()) # Parallel computing
-source("functions.R") # Utility functions
-
-2 Data
-In this project, I am using publicly available football data of the 2018-2019 English Premier League season.
-We will only focus on the total number of goals scored by the home team (“Full Time Home Goal” or FTHG in the data), the total number of goals scored by the away team (“Full Time Away Goal”, or FTAG in the data) and the results (“Full Time Results” or FTR in the data), which can can be “Home win” (“H” in the data), “Away win” (“A” in the data) and “Draw” (“D” in the data).
-Each of the 20 teams of the Premier League plays the other teams twice, once at home and once away, for a total number of 380 games.
- - - -df0 <- read.csv("Data/PremierLeague1819.csv")
-
-# Processing
-df <- df0[, c("Div", "Date", "HomeTeam", "AwayTeam", "HTHG", "HTAG", "FTHG", "FTAG", "FTR")]
-df$FTR <- factor(df$FTR, levels = c("A", "D", "H"), ordered = TRUE)
-
-# Teams
-teams <- with(df, sort(unique(c(as.character(HomeTeam), as.character(AwayTeam)))))
-
-# Associate a unique ID to each game
-id <- game_id(teams)
-df <- merge(df, id, by = c("HomeTeam", "AwayTeam"))
-
-# Order by date
-df$Date <- as.Date(df$Date, "%d/%m/%Y")
-df <- df[order(df$Date), ]
-
-heatmap_results(df) +
- labs(title = "Full time results of the 2018/2019 English Premier League")
In the English Premier League, a win is worth 3 points, a draw 1 point and no points is awarded for the losing a game. The team with the highest number of points at the end of the season wins the championship. The goal difference (number of goals scored minus number of goals conceded) is used to break ties when teams finish with an equal number of points.
-This season, Manchester City won the Premier League with 98 points, followed very closely by Liverpool with 97 points.
- - - -(fstats <- football_stats(df)) # Football statistics
-
-
-
-
-
-
-
-3 Model
-In our model, we assumed that the number of goals scored by each team follow independent Poisson distributions.
-For each game, if we index the home team by \(h\) and the away team by \(a\), then the rates \(\lambda_h\) and \(\lambda_a\) of the Poisson distribution are given by:
-\[ -\begin{aligned} -\log(\lambda_h) & = b + \mathit{attack_h} - \mathit{defence_a} + \mathit{advtg} \\ -\log(\lambda_a) & = b + \mathit{attack_a} - \mathit{defence_h} -\end{aligned} -\] Where:
--
-
- \(b\) is the intercept, i.e. the logarithm of the average goals rate assuming the attack and defence abilities of the teams cancels out. -
- \(\mathit{attack_k}\) and \(\mathit{defence_k}\) are the latent attack and defence abilities of the \(k\)-th team. -
- \(\mathit{advtg}\) is the home advantage. -
Priors for the parameters were chosen to be weakly informative and to result in reasonable prior predictive distributions, as we will see in the next section:
--
-
- \(b \sim \mathcal{N}(0, 0.5^2)\). This prior can be understood by considering a situation where the two teams have the same underlying attack and defence abilities and there is no home advantage, resulting in average goal rate of \(\exp(b)\). As a rule of thumb, if we consider that \(\mathcal{N}(0, 0.5^2)\) ranges from -1 to 1 (approx. 95% CI), then the average goal rate follows a log-normal distribution ranging from \(\exp(-1) \approx 0.37\) to \(\exp(1) \approx 2.72\). -
- \(\mathit{attack}_k\) and \(\mathit{defence}_k\) follow the hierarchical prior:
-
-
-
- \(\mathit{attack}_k \sim \mathcal{N}(0, \sigma^2)\) -
- \(\mathit{defence}_k \sim \mathcal{N}(0, \sigma^2)\) -
- \(\sigma \sim \mathcal{N}^{+} \Big( 0, \big( \log(5) / 2.3 / \sqrt{2}\big)^2 \Big)\). This prior can be understood by considering that, if \(\mathit{attack}_k\) and \(\mathit{defence}_k\) are independent, then \(\mathit{attack_h} - \mathit{defence_a} \sim \mathcal{N}\big( 0, (\sqrt{2} \sigma)^2 \big)\). If we are at the upper tail of the distribution, for instance at the 99% quantile (\(z = 2.3\)), this means that the home team would score \(\exp(2.3 * \sqrt{2} * \sigma)\) more goals than the global average. Here we consider that the home team could score at most \(5 = \exp(2.3 * \sqrt{2} * \sigma)\) times more goals than the average, hence the value for \(\sigma\). -
- - \(\mathit{advtg} \sim \mathcal{N}(0.5, 0.25^2)\). Here, we assume that the home advantage is positive, meaning the advantage is indeed an advantage in the sense that a team is more likely to score goals, everything else being equal, at home than away, but that the advantage is unlikely to be very big. As a rule of thumb, \(\mathcal{N}(0.5, 0.25^2)\) would range from 0 to 1, meaning that at best, a team would score \(\exp{1} \approx 3\) times more goals at home. -
The model is implemented in Model/DC_model.stan.
-
-4 Prior predictive check
-In this section, I perform prior predictive check to confirm that the choices of our priors result in simulated data that appears reasonable.
-Let’s first prepare the ground to run MCMC.
- - - -compiled_model <- stan_model("Model/DC_model.stan")
-
-# MCMC options
-n_chains <- 4
-n_it <- 2000
-
-# Parameters of interest
-param_pop <- c("b", "home_advantage", "sigma_ability")
-param_rep <- c("win_rep", "draw_rep", "lose_rep",
- "goal_tot_rep", "goal_diff_rep", "point_rep")
-param_ind <- c("attack", "defence", param_rep)
-param_obs <- c("home_goals_rep", "away_goals_rep")
-param <- c(param_pop, param_ind, param_obs)Then, we can simulate data from the prior predictive distribution by running Stan without evaluating the likelihood.
- - - -# Characteristics of the data to generate
-n_teams <- 20
-teams_simu <- LETTERS[1:n_teams]
-id_simu <- game_id(teams_simu)
-
-data_prior <- list(
- N_teams = n_teams,
- N_games = n_teams * (n_teams - 1),
- home_goals = rep(1, n_teams * (n_teams - 1)), # doesn't matter
- away_goals = rep(1, n_teams * (n_teams - 1)), # doesn't matter
- home_id = sapply(id_simu[["HomeTeam"]], function(x) {which(x == teams_simu)}),
- away_id = sapply(id_simu[["AwayTeam"]], function(x) {which(x == teams_simu)}),
- run = 0
-)
-
-fit_prior <- sampling(compiled_model,
- data = data_prior,
- pars = param,
- iter = n_it,
- chains = n_chains)
-par_prior <- extract_parameters(fit_prior, param, param_ind, param_obs, teams_simu, id_simu$Game, data_stan) # Store parameters for later useWe can check the distribution of each individual parameter:
- - - -plot(fit_prior, pars = c(param_pop, paste0(param_ind[1:2], "[1]")), plotfun = "hist")
We can also inspect, for example, the number of goals scored by the home team for a random game (all teams or games are interchangeable as this point).
- - - -goals <- extract(fit_prior, pars = c("home_goals_rep[1]"))[[1]]
-summary(goals) Min. 1st Qu. Median Mean 3rd Qu. Max.
- 0.000 1.000 2.000 3.124 3.000 691.000 hist(goals, breaks = 40)
hist(goals[goals < 20], breaks = 20)
quantile(goals, probs = c(.25, .5 , .75, .9, .99, .999)) 25% 50% 75% 90% 99% 99.9%
- 1.000 2.000 3.000 6.000 21.000 82.004 Although the prior distribution of goals has most of its mass for small values (e.g. \(< 5\)), it has a long tail meaning that, for instance, the probability of the home team scoring more than 20 goals in the Premier League during one game is 0.0125. While this probability is small, the probability that happens at least once during 380 games is 0.992, which might be considered unrealistic. This would suggest making changes to the model but we will continue with it for illustration purposes.
-We can also look at distribution of the number of games won, lost or ending with a draw for a random team, but we do not detect anything unrealistic:
- - - -pl <- lapply(c("win", "lose", "draw"),
- function(x) {
- otc <- extract(fit_prior, pars = paste0(x, "_rep[1]"))[[1]]
- otc <- factor(otc, levels = 0:(2 * (n_teams - 1)))
- otc <- table(otc) / length(otc)
- ggplot(data = data.frame(otc), aes(x = otc, y = Freq)) +
- geom_bar(stat = "identity") +
- scale_x_discrete(breaks = seq(1, 2 * (n_teams - 1), 2)) +
- labs(x = paste0("Number of ", x), y = "Prior probability") +
- theme_bw(base_size = 15)
- })
-plot_grid(plotlist = pl, ncol = 1)
-
-5 Fake data check
-In this section, we evaluate whether the algorithm “works”, i.e. whether we can retrieve the parameters of the model from the data, when we know the parameters of the true data-generating mechanism. To do this, we just sample the prior predictive distribution and fit the model with the simulated data.
- - - -draw <- 2019 # Draw
-
-# True parameters
-true_param_pop <- lapply(extract(fit_prior, pars = param_pop), function(x) {x[draw]})
-true_param_ind <- lapply(extract(fit_prior, pars = param_ind), function(x) {x[draw, ]})
-true_param <- rbind(
- do.call(rbind,
- lapply(1:length(true_param_ind),
- function(i) {
- data.frame(Variable = names(true_param_ind)[i],
- True = true_param_ind[[i]],
- Team = teams_simu)
- })),
- do.call(rbind,
- lapply(1:length(true_param_pop),
- function(i) {
- data.frame(Variable = names(true_param_pop)[i],
- True = true_param_pop[[i]],
- Team = NA)
- }))
-)
-
-# Fake data
-fd <- cbind(id_simu,
- data.frame(FTHG = extract(fit_prior, pars = "home_goals_rep")[[1]][draw, ],
- FTAG = extract(fit_prior, pars = "away_goals_rep")[[1]][draw, ],
- FTR = NA))
-fd$FTR[fd$FTHG == fd$FTAG] <- "D"
-fd$FTR[fd$FTHG > fd$FTAG] <- "H"
-fd$FTR[fd$FTHG < fd$FTAG] <- "A"
-fd$FTR <- factor(fd$FTR, levels = c("A", "D", "H"), ordered = TRUE)We can visualise the outcome of these simulated games:
- - - -heatmap_results(fd)
And we can also compute some statistics about this fake data:
- - - -(fstats_fake <- football_stats(fd))
-
-
-
-
-
-Let’s now fit the model with the fake data:
- - - -data_fake <- list(
- N_teams = n_teams,
- N_games = n_teams * (n_teams - 1),
- home_goals = fd$FTHG,
- away_goals = fd$FTAG,
- home_id = sapply(fd[["HomeTeam"]], function(x) {which(x == teams_simu)}),
- away_id = sapply(fd[["AwayTeam"]], function(x) {which(x == teams_simu)}),
- run = 1
-)
-
-fit_fake <- sampling(compiled_model,
- data = data_fake,
- pars = param,
- iter = n_it,
- chains = n_chains)
-par_fake <- extract_parameters(fit_fake, param, param_ind, param_obs, teams_simu, fd$Game, data_stan)First, we should check the MCMC diagnostics: nothing to worry about.
- - - -check_hmc_diagnostics(fit_fake)
-Divergences:0 of 4000 iterations ended with a divergence.
-Tree depth:0 of 4000 iterations saturated the maximum tree depth of 10.
-Energy:E-BFMI indicated no pathological behavior.pairs(fit_fake, pars = param_pop)
plot(fit_fake, pars = param_pop, plotfun = "trace")
print(fit_fake, pars = param_pop)Inference for Stan model: DC_model.
-4 chains, each with iter=2000; warmup=1000; thin=1;
-post-warmup draws per chain=1000, total post-warmup draws=4000.
-
- mean se_mean sd 2.5% 25% 50% 75% 97.5% n_eff Rhat
-b -0.49 0 0.10 -0.69 -0.55 -0.49 -0.43 -0.31 2835 1
-home_advantage 0.78 0 0.07 0.63 0.73 0.78 0.82 0.92 6926 1
-sigma_ability 0.22 0 0.04 0.15 0.19 0.22 0.25 0.32 1804 1
-
-Samples were drawn using NUTS(diag_e) at Mon Apr 13 15:17:16 2020.
-For each parameter, n_eff is a crude measure of effective sample size,
-and Rhat is the potential scale reduction factor on split chains (at
-convergence, Rhat=1).Then, we can check whether the posterior estimates “close enough” to the true parameters?
- - - -(ce <- check_estimates(par_fake, true_param, param_pop, param_ind[1:2]))[[1]]
-
-[[2]]
-
-[[3]]
-
-$Coverage
-[1] 0.9767442
Visually, they appear so, but we can also quantify it by computing, for example, the 90% coverage probability, i.e. the proportion of parameters falling in the 90% credible interval. Here the coverage is 0.98 which is close enough to what it should be, i.e. 90%.
-Finally, we can perform posterior predictive checks to detect any discrepancies between the observed (here, fake) and the posterior replications. We can investigate several summary statistics such as the number of games won, lost or draws for a random team, as well as the total number of point or even if the final rank. From the plot, we cannot visually identify any issues with the posterior replications.
- - - -PPC_football_stats(fit_fake, "win_rep", fstats_fake, teams_simu)
PPC_football_stats(fit_fake, "lose_rep", fstats_fake, teams_simu)
PPC_football_stats(fit_fake, "goal_tot_rep", fstats_fake, teams_simu)
PPC_football_stats(fit_fake, "point_rep", fstats_fake, teams_simu)
PPC_football_stats(fit_fake, "rank_rep", fstats_fake, teams_simu)
NB: The fake data check can be repeated to make sure the model can estimate different realisations of the prior predictive distribution in a process that is called Simulation Based Calibration.
-
-
-6 Model fitting
-Having confirmed that the model could be fitted in the previous section, in this section, we will train the model with the data from the 2018/2019 season of the English Premier League.
- - - -data_fit <- list(
- N_teams = length(teams),
- N_games = nrow(df),
- home_goals = df[["FTHG"]],
- away_goals = df[["FTAG"]],
- home_id = sapply(df[["HomeTeam"]], function(x) {which(x == teams)}),
- away_id = sapply(df[["AwayTeam"]], function(x) {which(x == teams)}),
- run = 1
-)
-
-fit <- sampling(compiled_model,
- data = data_fit,
- pars = param,
- iter = n_it,
- chains = n_chains)
-par <- extract_parameters(fit, param, param_ind, param_obs, teams, df$Game, data__fit)First, we inspect converge diagnostics: nothing to worry about.
- - - -check_hmc_diagnostics(fit)
-Divergences:0 of 4000 iterations ended with a divergence.
-Tree depth:0 of 4000 iterations saturated the maximum tree depth of 10.
-Energy:E-BFMI indicated no pathological behavior.pairs(fit, pars = param_pop)
plot(fit, pars = param_pop, plotfun = "trace")
Now we can look at the parameter estimates, the population parameters (e.g. parameters that are shared across teams) and the attack and defence abilities for each teams. The coefficent plot for the population parameters reveal that the priors seems weakly informative enough to “include” the posteriors. In addition, we notice that Manchester City and Liverpool have the best a posteriori attack and defence abilities of this season, which is consistent with the fact that they finished first and second respectively.
- - - -HuraultMisc::plot_prior_posterior(par_prior, par, param_pop) +
- labs(title = "<b>Posterior</b> vs <b style='color:#E69F00'>prior</b> estimates (mean and 90% CI)",
- subtitle = "Population parameters",
- y = "") +
- theme(plot.title = element_markdown(),
- plot.title.position = "plot",
- legend.position = "none")
-plot_abilities(par)
We can also look at the posterior predictive distribution. For concision, I am not plotting the posterior probability for the number wins, lose, draws, goals or points, but we will look at the posterior ranks.
-In the following plot, the size of the colour bars represent the probability at the given rank. For instance, the posterior probability for Manchester finishing first is slightly above 50% and around 30% for finishing second. Similarly, we can visually approximate the posterior probability for Liverpool finishing first to be 40% and a similar probability for finishing second.
- - - -stackhist_rank(compute_rank(fit, "rep"), teams)
-
-7 Model validation
-While the fit can help us understand what was going on during the season a posteriori, it is interesting to know to what extent the model is predictive.
-Since we are dealing with time-series data and want to predict the future based on the past, it is not appropriate to use standard cross-validation techniques such as K-fold cross-validation, rather, we will implement forward chaining where the model is trained on the data from the first week and tested on the next, then trained on the data of the first two weeks and tested on the remaining weeks, etc.
- - - -HuraultMisc::illustrate_forward_chaining()
We can evaluate the performance of the model to predict the full time results using the Ranked Probability Score, a proper scoring rule to measure the accuracy of ordinal (cf. Lose < Draw < Win) probabilistic forecast. It is also possible to evaluate the model in its ability to predict the number of goals for instance, but I will not show these results here.
-The following code implements the forward chaining. Considering the task is parallel in nature, it can be convenient to take advantage of multiple cores that might be available.
- - - -n_cluster <- floor(parallel::detectCores() / n_chains)
-
-# Training unit
-df[["WeekNumber"]] <- strftime(df[["Date"]], format = "%Y-%V")
-weeks <- unique(df[["WeekNumber"]])
-
-# Update parameter of interest
-param_test <- c("win_test", "draw_test", "lose_test",
- "goal_tot_test", "goal_diff_test", "point_test")
-param_ind <- c("attack", "defence", param_test)
-param_obs <- c("home_goals_test", "away_goals_test")
-param <- c(param_pop, param_ind, param_obs)
-
-format_stan_data <- function(df) {
- list(
- N_teams = length(teams),
- N_games = nrow(df),
- home_goals = df[["FTHG"]],
- away_goals = df[["FTAG"]],
- home_id = sapply(df[["HomeTeam"]], function(x) {which(x == teams)}),
- away_id = sapply(df[["AwayTeam"]], function(x) {which(x == teams)}),
- run = 1
- )
-}
-
-library(foreach)
-library(doParallel)
-
-duration <- Sys.time()
-cl <- makeCluster(n_cluster)
-registerDoParallel(cl)
-writeLines(c(""), "log.txt")
-
-out <- foreach(w = 1:(length(weeks) - 1)) %dopar% {
-
- source("functions.R")
- library(rstan)
- rstan_options(auto_write = TRUE) # Save compiled model
- options(mc.cores = parallel::detectCores()) # Parallel computing
-
- sink("log.txt", append = TRUE)
- cat(paste("Starting training at week ", w, " \n", sep = ""))
-
- df_train <- df[df$WeekNumber <= weeks[w], ]
- df_test <- df[df[["WeekNumber"]] > weeks[w], ]
-
- data_stan <- format_stan_data(df_train)
-
- fit <- sampling(compiled_model,
- data = data_stan,
- pars = param,
- iter = n_it,
- chains = n_chains)
-
- # Parameters
- par <- extract_parameters(fit, param = c(param_pop, param_ind), param_ind, param_obs, teams, df_train[["Game"]], data_stan)
- par$WeekNumber <- weeks[w]
- par$ProportionGamePlayed <- nrow(df_train) / nrow(df)
-
- # Rank
- rk <- compute_rank(fit, "test")
- rk <- do.call(rbind,
- lapply(1:length(teams),
- function(i) {
- tmp <- table(factor(rk[, i], levels = 1:length(teams))) / nrow(rk)
- data.frame(Team = teams[i],
- Rank = names(tmp),
- Probability = as.numeric(tmp))
- }))
- rk <- HuraultMisc::factor_to_numeric(rk, "Rank")
- rk$WeekNumber <- weeks[w]
- rk$ProportionGamePlayed <- nrow(df_train) / nrow(df)
-
- # Metrics
- pred <- process_predictions(fit, id)
- m <- compute_metrics(pred = pred, act = df, test_game = df_test[["Game"]], var = "FTR")
- m$WeekNumber <- weeks[w]
- m$ProportionGamePlayed <- nrow(df_train) / nrow(df)
-
- list(Performance = m, Parameters = par, Rank = rk)
-}
-stopCluster(cl)
-(duration = Sys.time() - duration)Time difference of 13.77294 minsm <- do.call(rbind, lapply(out, function(x) {x$Performance}))
-par <- do.call(rbind, lapply(out, function(x) {x$Parameters}))
-rk <- do.call(rbind, lapply(out, function(x) {x$Rank}))We can now plot the predictive performance of the model as a function of training week, or as a function of the proportion of game played in the season.
- - - -ggplot(data = subset(m, Metric == "RPS"),
- aes(x = ProportionGamePlayed, y = Mean, ymin = Mean - SE, ymax = Mean + SE)) +
- geom_pointrange() +
- scale_y_continuous(limits = c(0, NA)) +
- labs(y = "RPS", title = "RPS learning curve (lower the better)") +
- theme_bw(base_size = 15)
Although the RPS is slightly improving, it does not seem to be by much, which suggest a limitation of such a simple model. We could investigate this by plotting how the believes in the teams abilities changes with time.
- - - -tmp <- subset(par, Variable %in% c("attack", "defence"))
-pl4 <- lapply(teams, function(x) {
- cbbPalette <- c("#000000", "#E69F00", "#56B4E9", "#009E73", "#F0E442", "#0072B2", "#D55E00", "#CC79A7")
- ggplot(data = subset(tmp, Team == x),
- aes(x = ProportionGamePlayed, y = Mean, ymin = `5%`, ymax = `95%`, colour = Variable, fill = Variable)) +
- geom_line() +
- geom_ribbon(alpha = 0.5) +
- scale_colour_manual(values = cbbPalette) +
- scale_fill_manual(values = cbbPalette) +
- labs(title = x, y = "Ability", colour = "", fill = "") +
- coord_cartesian(ylim = c(-1, 1)) +
- theme_bw(base_size = 15)
-})
-plot_grid(get_legend(pl4[[1]] + theme(legend.position = "top")),
- plot_grid(plotlist = lapply(pl4, function(p) {p + theme(legend.position = "none")}),
- nrow = 5),
- nrow = 2, rel_heights = c(.05, .95))
Even though the abilities are learnt as more data comes in, they remain uncertain, which could explain the previous result.
-It can also be interesting to see how our predictions changes with time, for instance, how the uncertainty over the final ranking is changing as more games are played. The following plot depicts, as an heatmap, the predicted rank at the end of the season as a function of the number of games played, for each team. As we could expect, early in the season, the predictions are quite uncertain but becomes more confident as fewer games remain to be played and as the model has better estimates of its parameters.
- - - -# Add first week ranking
-rk <- rbind(data.frame(expand.grid(Team = teams,
- Rank = 1:length(teams)),
- Probability = 1 / length(teams),
- WeekNumber = strftime(min(df[["Date"]]) - 7, format = "%Y-%V"),
- ProportionGamePlayed = 0), # add first week
- rk)
-
-# Add last week ranking
-tmp <- data.frame(expand.grid(Team = teams, Rank = 1:length(teams)),
- Probability = 0,
- WeekNumber = weeks[length(weeks)],
- ProportionGamePlayed = 1)
-for (i in 1:nrow(fstats)) {
- id <- which((tmp[["Team"]] == fstats[i, "Team"]) & (tmp[["Rank"]] == fstats[i, "rank"]))
- tmp[id, "Probability"] <- 1
-}
-rk <- rbind(rk, tmp)
-
-pl2 <- lapply(teams,
- function(x) {
- ggplot(data = subset(rk, Team == x),
- aes(x = factor(ProportionGamePlayed), y = Rank, fill = Probability)) +
- geom_tile() +
- scale_fill_viridis_c() +
- scale_y_continuous(expand = c(0, 0), breaks = 1:length(teams)) +
- scale_x_discrete(expand = c(0, 0), breaks = c(0, 0.5, 1)) +
- labs(title = x, x = "Proportion of game played*") + # * not exactly but close
- theme_classic(base_size = 15)
- })
-plot_grid(get_legend(pl2[[1]] + theme(legend.position = "top")),
- plot_grid(plotlist = lapply(pl2, function(x) {x + theme(legend.position = "none")}),
- nrow = 5),
- nrow = 2, rel_heights = c(.05, .95))
For example, we can see how the prediction look like at the middle of the season.
- - - -df_train <- df[df[["WeekNumber"]] <= median(weeks), ]
-test_game <- df[df[["WeekNumber"]] > median(weeks), "Game"]
-data_fit <- format_stan_data(df_train)
-fit <- sampling(compiled_model,
- data = data_fit,
- pars = param,
- iter = n_it,
- chains = n_chains)PPC_football_stats(fit, "win_test", fstats, teams)
PPC_football_stats(fit, "lose_test", fstats, teams)
PPC_football_stats(fit, "point_test", fstats, teams)
stackhist_rank(compute_rank(fit, "test"), teams)
The predictions look reasonable with respect to the outcome that is observed at the end of the season, however, there is still a lot of uncertainty at the mid-season. Interestingly, the last plot shows the model predict that Liverpool will win the championship with the probability of approximately 80% when in the end, it is Manchester City that will win!
-
-
-
-8 Conclusion
-The model was successfully fit to the data and can gives valuable insights into the teams abilities.
-However, its predictive performance appears limited and we would need to move away from this simple “Dixon-Coles” model so we can make more accurate and practically useful predictions. Keeping in mind I am far from being a domain expert, I could suggest to:
--
-
- Predict the number of goals for each half-time periods. This would effectively double the data we are using and provide more accurate estimates of the teams abilities, and hopefully, better predictions. -
- Use a zero-inflated model to describe the fact that “no goals scored” happens more often than expected. -
- Assume correlated attack and defence abilities. -
- Model the fact that the latent abilities might change with time. For example, we could use a Random Walk or an Exponential Smoothing model to describe the evolution of abilities. -
- Team dependent home advantage. Some teams might have a bigger home advantage (better fans?) but at the same time, other teams could less sensitive to this effect. -
- If the data is available, model the team abilities as a combination of the abilities of each player. -
- etc. -
Nonetheless, the Bayesian framework can be useful to make complex predictions beyond which team will win a specific game, but also final ranking at the end of the season.
- -LS0tDQp0aXRsZTogIkZvb3RiYWxsIHByZWRpY3Rpb24gbW9kZWwgaW4gU3RhbiINCmF1dGhvcjogIkd1aWxsZW0gSHVyYXVsdCINCmRhdGU6ICJgciBmb3JtYXQoU3lzLnRpbWUoKSwgJyVkICVCLCAlWScpYCINCm91dHB1dDoNCiAgaHRtbF9ub3RlYm9vazoNCiAgICBudW1iZXJfc2VjdGlvbnM6IHllcw0KICAgIHRvYzogeWVzDQotLS0NCg0KIyBJbnRyb2R1Y3Rpb24NCg0KVGhpcyByZXBvc2l0b3J5IGNvbnRhaW5zIHRoZSBjb2RlIG9mIGEgcGVyc29uYWwgcHJvamVjdCB3aGVyZSBJIGFtIGltcGxlbWVudGluZyBhIHNpbXBsZSAiRGl4b24tQ29sZXMiIG1vZGVsIHRvIHByZWRpY3QgdGhlIG91dGNvbWUgb2YgZm9vdGJhbGwgZ2FtZXMgd2l0aCB0aGUgcHJvYmFiaWxpc3RpYyBwcm9ncmFtbWluZyBsYW5ndWFnZSBTdGFuLg0KDQpBcyBhIGRpc2NsYWltZXIsIEkgYW0gbm90IGEgcGFydGljdWxhciBmYW4gb2YgZm9vdGJhbGwgYW5kIHRoZSBwcmVzZW50ZWQgbW9kZWwgaXMgZmFyIHRvbyBzaW1wbGUgdG8gYWNjdXJhdGVseSBtb2RlbC9wcmVkaWN0IHRoZSBvdXRjb21lIG9mIGdhbWVzIGFuZCBmb3J0aW9yaSB0byBiZSB1c2VkIGZvciBiZXR0aW5nIChhbmQgaWYgdGhlIGdvYWwgd2FzIGJldHRpbmcsIGRlc2lnbmluZyBtb2RlbHMgZm9yIGluZGl2aWR1YWwgc3BvcnRzIHdoZXJlIHRoZSBvdXRjb21lcyBhcmUgbGVzcyB1bmNlcnRhaW4sIHN1Y2ggYXMgZGFydHMgb3IgaG9yc2UgcmFjaW5nIHdvdWxkIHByb2JhYmx5IGJlIHNhZmVyKS4NCkhhdmluZyBzYWlkIHRoYXQsIHRoaXMgcHJvamVjdCB3YXMgZnVuIGFuZCBhIGdvb2Qgd2F5IGZvciBtZSB0byB3b3JrIHdpdGggImNsZWFuIiBkYXRhIGFuZCBsZWFybiBhYm91dCBCYXllc2lhbiB3b3JrZmxvdy4NCg0KTm90YWJseSwgSSBzZWUgbXkgbWFpbiBjb250cmlidXRpb24gaW4gdGhlIHF1YW50aXRpZXMgdGhhdCB0aGUgbW9kZWwgY2FuIHByZWRpY3QgKGNmLiBzdWZmaXggYF90ZXN0YCBpbiB0aGUgU3RhbiBjb2RlKSBvciB0aGF0IGNhbiBiZSB1c2VkIGZvciBwb3N0ZXJpb3IgcHJlZGljdGl2ZSBjaGVja2luZyAoY2YuIHN1ZmZpeCBgX3JlcGApOg0KDQotIHRoZSBudW1iZXIgb2YgZ2FtZXMgd29uLg0KLSB0aGUgbnVtYmVyIG9mIGdhbWVzIGxvc3QuDQotIHRoZSBudW1iZXIgb2YgZ2FtZXMgZW5kaW5nIGluIGEgZHJhdy4NCi0gdGhlIHRvdGFsIG51bWJlciBvZiBnb2FscyBzY29yZWQuDQotIHRoZSBnb2FsIGRpZmZlcmVuY2UuDQotIHRoZSBudW1iZXIgb2YgcG9pbnRzLg0KLSB0aGUgcmFua2luZy4NCg0KSW4gdGhpcyBub3RlYm9vaywgSSBwcmVzZW50IGFuIG92ZXJ2aWV3IG9mIHdoYXQgSSBoYXZlIGRvbmUgaW4gdGhpcyBwcm9qZWN0IGFuZCB3aGljaCBpcyBkaXJlY3RlZCB0byBhbiBhdWRpZW5jZSB3aXRoIHNvbWUgZmFtaWxpYXJpdHkgaW4gQmF5ZXNpYW4gbW9kZWxsaW5nLg0KDQpCZWZvcmUgZ29pbmcgaW50byB0aGUgZGV0YWlscyBvZiB0aGUgYW5hbHlzaXMsIGxldCdzIGZpcnN0IGluaXRpYWxpc2UgdGhlIG5vdGVib29rLg0KDQpgYGB7ciBtZXNzYWdlPUZBTFNFfQ0Kc2V0LnNlZWQoMTU1OTM1NDE2MikgIyBSZXByb2R1Y2liaWxpdHkNCmxpYnJhcnkoSHVyYXVsdE1pc2MpICMgUGVyc29uYWwgZnVuY3Rpb24gbGlicmFyeQ0KbGlicmFyeShnZ3Bsb3QyKQ0KbGlicmFyeShjb3dwbG90KQ0KbGlicmFyeShnZ3RleHQpDQpsaWJyYXJ5KHJzdGFuKQ0KcnN0YW5fb3B0aW9ucyhhdXRvX3dyaXRlID0gVFJVRSkgIyBTYXZlIGNvbXBpbGVkIG1vZGVsDQpvcHRpb25zKG1jLmNvcmVzID0gcGFyYWxsZWw6OmRldGVjdENvcmVzKCkpICMgUGFyYWxsZWwgY29tcHV0aW5nDQpzb3VyY2UoImZ1bmN0aW9ucy5SIikgIyBVdGlsaXR5IGZ1bmN0aW9ucw0KYGBgDQoNCiMgRGF0YQ0KDQpJbiB0aGlzIHByb2plY3QsIEkgYW0gdXNpbmcgcHVibGljbHkgYXZhaWxhYmxlIFtmb290YmFsbCBkYXRhXShodHRwOi8vZm9vdGJhbGwtZGF0YS5jby51ay8pIG9mIHRoZSAyMDE4LTIwMTkgRW5nbGlzaCBQcmVtaWVyIExlYWd1ZSBzZWFzb24uDQoNCldlIHdpbGwgb25seSBmb2N1cyBvbiB0aGUgdG90YWwgbnVtYmVyIG9mIGdvYWxzIHNjb3JlZCBieSB0aGUgaG9tZSB0ZWFtICgiRnVsbCBUaW1lIEhvbWUgR29hbCIgb3IgRlRIRyBpbiB0aGUgZGF0YSksIHRoZSB0b3RhbCBudW1iZXIgb2YgZ29hbHMgc2NvcmVkIGJ5IHRoZSBhd2F5IHRlYW0gKCJGdWxsIFRpbWUgQXdheSBHb2FsIiwgb3IgRlRBRyBpbiB0aGUgZGF0YSkgYW5kIHRoZSByZXN1bHRzICgiRnVsbCBUaW1lIFJlc3VsdHMiIG9yIEZUUiBpbiB0aGUgZGF0YSksIHdoaWNoIGNhbiBjYW4gYmUgIkhvbWUgd2luIiAoIkgiIGluIHRoZSBkYXRhKSwgIkF3YXkgd2luIiAoIkEiIGluIHRoZSBkYXRhKSBhbmQgIkRyYXciICgiRCIgaW4gdGhlIGRhdGEpLg0KDQpFYWNoIG9mIHRoZSAyMCB0ZWFtcyBvZiB0aGUgUHJlbWllciBMZWFndWUgcGxheXMgdGhlIG90aGVyIHRlYW1zIHR3aWNlLCBvbmNlIGF0IGhvbWUgYW5kIG9uY2UgYXdheSwgZm9yIGEgdG90YWwgbnVtYmVyIG9mIDM4MCBnYW1lcy4NCg0KYGBge3J9DQpkZjAgPC0gcmVhZC5jc3YoIkRhdGEvUHJlbWllckxlYWd1ZTE4MTkuY3N2IikNCg0KIyBQcm9jZXNzaW5nDQpkZiA8LSBkZjBbLCBjKCJEaXYiLCAiRGF0ZSIsICJIb21lVGVhbSIsICJBd2F5VGVhbSIsICJIVEhHIiwgIkhUQUciLCAiRlRIRyIsICJGVEFHIiwgIkZUUiIpXQ0KZGYkRlRSIDwtIGZhY3RvcihkZiRGVFIsIGxldmVscyA9IGMoIkEiLCAiRCIsICJIIiksIG9yZGVyZWQgPSBUUlVFKQ0KDQojIFRlYW1zDQp0ZWFtcyA8LSB3aXRoKGRmLCBzb3J0KHVuaXF1ZShjKGFzLmNoYXJhY3RlcihIb21lVGVhbSksIGFzLmNoYXJhY3RlcihBd2F5VGVhbSkpKSkpDQoNCiMgQXNzb2NpYXRlIGEgdW5pcXVlIElEIHRvIGVhY2ggZ2FtZQ0KaWQgPC0gZ2FtZV9pZCh0ZWFtcykNCmRmIDwtIG1lcmdlKGRmLCBpZCwgYnkgPSBjKCJIb21lVGVhbSIsICJBd2F5VGVhbSIpKQ0KDQojIE9yZGVyIGJ5IGRhdGUNCmRmJERhdGUgPC0gYXMuRGF0ZShkZiREYXRlLCAiJWQvJW0vJVkiKQ0KZGYgPC0gZGZbb3JkZXIoZGYkRGF0ZSksIF0NCg0KaGVhdG1hcF9yZXN1bHRzKGRmKSArDQogIGxhYnModGl0bGUgPSAiRnVsbCB0aW1lIHJlc3VsdHMgb2YgdGhlIDIwMTgvMjAxOSBFbmdsaXNoIFByZW1pZXIgTGVhZ3VlIikNCmBgYA0KDQoNCkluIHRoZSBFbmdsaXNoIFByZW1pZXIgTGVhZ3VlLCBhIHdpbiBpcyB3b3J0aCAzIHBvaW50cywgYSBkcmF3IDEgcG9pbnQgYW5kIG5vIHBvaW50cyBpcyBhd2FyZGVkIGZvciB0aGUgbG9zaW5nIGEgZ2FtZS4NClRoZSB0ZWFtIHdpdGggdGhlIGhpZ2hlc3QgbnVtYmVyIG9mIHBvaW50cyBhdCB0aGUgZW5kIG9mIHRoZSBzZWFzb24gd2lucyB0aGUgY2hhbXBpb25zaGlwLg0KVGhlIGdvYWwgZGlmZmVyZW5jZSAobnVtYmVyIG9mIGdvYWxzIHNjb3JlZCBtaW51cyBudW1iZXIgb2YgZ29hbHMgY29uY2VkZWQpIGlzIHVzZWQgdG8gYnJlYWsgdGllcyB3aGVuIHRlYW1zIGZpbmlzaCB3aXRoIGFuIGVxdWFsIG51bWJlciBvZiBwb2ludHMuDQoNClRoaXMgc2Vhc29uLCBNYW5jaGVzdGVyIENpdHkgd29uIHRoZSBQcmVtaWVyIExlYWd1ZSB3aXRoIDk4IHBvaW50cywgZm9sbG93ZWQgdmVyeSBjbG9zZWx5IGJ5IExpdmVycG9vbCB3aXRoIDk3IHBvaW50cy4NCg0KYGBge3J9DQooZnN0YXRzIDwtIGZvb3RiYWxsX3N0YXRzKGRmKSkgIyBGb290YmFsbCBzdGF0aXN0aWNzDQpgYGANCg0KIyBNb2RlbA0KDQpJbiBvdXIgbW9kZWwsIHdlIGFzc3VtZWQgdGhhdCB0aGUgbnVtYmVyIG9mIGdvYWxzIHNjb3JlZCBieSBlYWNoIHRlYW0gZm9sbG93IGluZGVwZW5kZW50IFBvaXNzb24gZGlzdHJpYnV0aW9ucy4NCg0KRm9yIGVhY2ggZ2FtZSwgaWYgd2UgaW5kZXggdGhlIGhvbWUgdGVhbSBieSAkaCQgYW5kIHRoZSBhd2F5IHRlYW0gYnkgJGEkLCB0aGVuIHRoZSByYXRlcyAkXGxhbWJkYV9oJCBhbmQgJFxsYW1iZGFfYSQgb2YgdGhlIFBvaXNzb24gZGlzdHJpYnV0aW9uIGFyZSBnaXZlbiBieToNCg0KJCQNClxiZWdpbnthbGlnbmVkfQ0KXGxvZyhcbGFtYmRhX2gpICYgPSBiICsgXG1hdGhpdHthdHRhY2tfaH0gLSBcbWF0aGl0e2RlZmVuY2VfYX0gKyBcbWF0aGl0e2FkdnRnfSBcXA0KXGxvZyhcbGFtYmRhX2EpICYgPSBiICsgXG1hdGhpdHthdHRhY2tfYX0gLSBcbWF0aGl0e2RlZmVuY2VfaH0NClxlbmR7YWxpZ25lZH0NCiQkDQpXaGVyZToNCg0KLSAkYiQgaXMgdGhlIGludGVyY2VwdCwgaS5lLiB0aGUgbG9nYXJpdGhtIG9mIHRoZSBhdmVyYWdlIGdvYWxzIHJhdGUgYXNzdW1pbmcgdGhlIGF0dGFjayBhbmQgZGVmZW5jZSBhYmlsaXRpZXMgb2YgdGhlIHRlYW1zIGNhbmNlbHMgb3V0Lg0KLSAkXG1hdGhpdHthdHRhY2tfa30kIGFuZCAkXG1hdGhpdHtkZWZlbmNlX2t9JCBhcmUgdGhlIGxhdGVudCBhdHRhY2sgYW5kIGRlZmVuY2UgYWJpbGl0aWVzIG9mIHRoZSAkayQtdGggdGVhbS4NCi0gJFxtYXRoaXR7YWR2dGd9JCBpcyB0aGUgaG9tZSBhZHZhbnRhZ2UuDQoNClByaW9ycyBmb3IgdGhlIHBhcmFtZXRlcnMgd2VyZSBjaG9zZW4gdG8gYmUgd2Vha2x5IGluZm9ybWF0aXZlIGFuZCB0byByZXN1bHQgaW4gcmVhc29uYWJsZSBwcmlvciBwcmVkaWN0aXZlIGRpc3RyaWJ1dGlvbnMsIGFzIHdlIHdpbGwgc2VlIGluIHRoZSBuZXh0IHNlY3Rpb246DQoNCi0gJGIgXHNpbSBcbWF0aGNhbHtOfSgwLCAwLjVeMikkLg0KVGhpcyBwcmlvciBjYW4gYmUgdW5kZXJzdG9vZCBieSBjb25zaWRlcmluZyBhIHNpdHVhdGlvbiB3aGVyZSB0aGUgdHdvIHRlYW1zIGhhdmUgdGhlIHNhbWUgdW5kZXJseWluZyBhdHRhY2sgYW5kIGRlZmVuY2UgYWJpbGl0aWVzIGFuZCB0aGVyZSBpcyBubyBob21lIGFkdmFudGFnZSwgcmVzdWx0aW5nIGluIGF2ZXJhZ2UgZ29hbCByYXRlIG9mICRcZXhwKGIpJC4NCkFzIGEgcnVsZSBvZiB0aHVtYiwgaWYgd2UgY29uc2lkZXIgdGhhdCAkXG1hdGhjYWx7Tn0oMCwgMC41XjIpJCByYW5nZXMgZnJvbSAtMSB0byAxIChhcHByb3guIDk1XCUgQ0kpLCB0aGVuIHRoZSBhdmVyYWdlIGdvYWwgcmF0ZSBmb2xsb3dzIGEgbG9nLW5vcm1hbCBkaXN0cmlidXRpb24gcmFuZ2luZyBmcm9tICRcZXhwKC0xKSBcYXBwcm94IDAuMzckIHRvICRcZXhwKDEpIFxhcHByb3ggMi43MiQuDQotICRcbWF0aGl0e2F0dGFja31fayQgYW5kICRcbWF0aGl0e2RlZmVuY2V9X2skIGZvbGxvdyB0aGUgaGllcmFyY2hpY2FsIHByaW9yOg0KICAtICRcbWF0aGl0e2F0dGFja31fayBcc2ltIFxtYXRoY2Fse059KDAsIFxzaWdtYV4yKSQNCiAgLSAkXG1hdGhpdHtkZWZlbmNlfV9rIFxzaW0gXG1hdGhjYWx7Tn0oMCwgXHNpZ21hXjIpJA0KICAtICRcc2lnbWEgXHNpbSBcbWF0aGNhbHtOfV57K30gXEJpZyggMCwgXGJpZyggXGxvZyg1KSAvIDIuMyAvIFxzcXJ0ezJ9XGJpZyleMiBcQmlnKSQuDQpUaGlzIHByaW9yIGNhbiBiZSB1bmRlcnN0b29kIGJ5IGNvbnNpZGVyaW5nIHRoYXQsIGlmICRcbWF0aGl0e2F0dGFja31fayQgYW5kICRcbWF0aGl0e2RlZmVuY2V9X2skIGFyZSBpbmRlcGVuZGVudCwgdGhlbiAkXG1hdGhpdHthdHRhY2tfaH0gLSBcbWF0aGl0e2RlZmVuY2VfYX0gXHNpbSBcbWF0aGNhbHtOfVxiaWcoIDAsIChcc3FydHsyfSBcc2lnbWEpXjIgXGJpZykkLg0KSWYgd2UgYXJlIGF0IHRoZSB1cHBlciB0YWlsIG9mIHRoZSBkaXN0cmlidXRpb24sIGZvciBpbnN0YW5jZSBhdCB0aGUgOTlcJSBxdWFudGlsZSAoJHogPSAyLjMkKSwgdGhpcyBtZWFucyB0aGF0IHRoZSBob21lIHRlYW0gd291bGQgc2NvcmUgJFxleHAoMi4zICogXHNxcnR7Mn0gKiBcc2lnbWEpJCBtb3JlIGdvYWxzIHRoYW4gdGhlIGdsb2JhbCBhdmVyYWdlLg0KSGVyZSB3ZSBjb25zaWRlciB0aGF0IHRoZSBob21lIHRlYW0gY291bGQgc2NvcmUgYXQgbW9zdCAkNSA9IFxleHAoMi4zICogXHNxcnR7Mn0gKiBcc2lnbWEpJCB0aW1lcyBtb3JlIGdvYWxzIHRoYW4gdGhlIGF2ZXJhZ2UsIGhlbmNlIHRoZSB2YWx1ZSBmb3IgJFxzaWdtYSQuDQotICRcbWF0aGl0e2FkdnRnfSBcc2ltIFxtYXRoY2Fse059KDAuNSwgMC4yNV4yKSQuDQpIZXJlLCB3ZSBhc3N1bWUgdGhhdCB0aGUgaG9tZSBhZHZhbnRhZ2UgaXMgcG9zaXRpdmUsIG1lYW5pbmcgdGhlIGFkdmFudGFnZSBpcyBpbmRlZWQgYW4gYWR2YW50YWdlIGluIHRoZSBzZW5zZSB0aGF0IGEgdGVhbSBpcyBtb3JlIGxpa2VseSB0byBzY29yZSBnb2FscywgZXZlcnl0aGluZyBlbHNlIGJlaW5nIGVxdWFsLCBhdCBob21lIHRoYW4gYXdheSwgYnV0IHRoYXQgdGhlIGFkdmFudGFnZSBpcyB1bmxpa2VseSB0byBiZSB2ZXJ5IGJpZy4NCkFzIGEgcnVsZSBvZiB0aHVtYiwgJFxtYXRoY2Fse059KDAuNSwgMC4yNV4yKSQgd291bGQgcmFuZ2UgZnJvbSAwIHRvIDEsIG1lYW5pbmcgdGhhdCBhdCBiZXN0LCBhIHRlYW0gd291bGQgc2NvcmUgJFxleHB7MX0gXGFwcHJveCAzJCB0aW1lcyBtb3JlIGdvYWxzIGF0IGhvbWUuDQoNClRoZSBtb2RlbCBpcyBpbXBsZW1lbnRlZCBpbiBbYE1vZGVsL0RDX21vZGVsLnN0YW5gXShNb2RlbC9EQ19tb2RlbC5zdGFuKS4NCg0KIyBQcmlvciBwcmVkaWN0aXZlIGNoZWNrDQoNCkluIHRoaXMgc2VjdGlvbiwgSSBwZXJmb3JtIHByaW9yIHByZWRpY3RpdmUgY2hlY2sgdG8gY29uZmlybSB0aGF0IHRoZSBjaG9pY2VzIG9mIG91ciBwcmlvcnMgcmVzdWx0IGluIHNpbXVsYXRlZCBkYXRhIHRoYXQgYXBwZWFycyByZWFzb25hYmxlLg0KDQpMZXQncyBmaXJzdCBwcmVwYXJlIHRoZSBncm91bmQgdG8gcnVuIE1DTUMuDQoNCmBgYHtyfQ0KY29tcGlsZWRfbW9kZWwgPC0gc3Rhbl9tb2RlbCgiTW9kZWwvRENfbW9kZWwuc3RhbiIpDQoNCiMgTUNNQyBvcHRpb25zDQpuX2NoYWlucyA8LSA0DQpuX2l0IDwtIDIwMDANCg0KIyBQYXJhbWV0ZXJzIG9mIGludGVyZXN0DQpwYXJhbV9wb3AgPC0gYygiYiIsICJob21lX2FkdmFudGFnZSIsICJzaWdtYV9hYmlsaXR5IikNCnBhcmFtX3JlcCA8LSBjKCJ3aW5fcmVwIiwgImRyYXdfcmVwIiwgImxvc2VfcmVwIiwNCiAgICAgICAgICAgICAgICJnb2FsX3RvdF9yZXAiLCAiZ29hbF9kaWZmX3JlcCIsICJwb2ludF9yZXAiKQ0KcGFyYW1faW5kIDwtIGMoImF0dGFjayIsICJkZWZlbmNlIiwgcGFyYW1fcmVwKQ0KcGFyYW1fb2JzIDwtIGMoImhvbWVfZ29hbHNfcmVwIiwgImF3YXlfZ29hbHNfcmVwIikNCnBhcmFtIDwtIGMocGFyYW1fcG9wLCBwYXJhbV9pbmQsIHBhcmFtX29icykNCmBgYA0KDQpUaGVuLCB3ZSBjYW4gc2ltdWxhdGUgZGF0YSBmcm9tIHRoZSBwcmlvciBwcmVkaWN0aXZlIGRpc3RyaWJ1dGlvbiBieSBydW5uaW5nIFN0YW4gd2l0aG91dCBldmFsdWF0aW5nIHRoZSBsaWtlbGlob29kLg0KDQpgYGB7ciBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQ0KIyBDaGFyYWN0ZXJpc3RpY3Mgb2YgdGhlIGRhdGEgdG8gZ2VuZXJhdGUNCm5fdGVhbXMgPC0gMjANCnRlYW1zX3NpbXUgPC0gTEVUVEVSU1sxOm5fdGVhbXNdDQppZF9zaW11IDwtIGdhbWVfaWQodGVhbXNfc2ltdSkNCg0KZGF0YV9wcmlvciA8LSBsaXN0KA0KICBOX3RlYW1zID0gbl90ZWFtcywNCiAgTl9nYW1lcyA9IG5fdGVhbXMgKiAobl90ZWFtcyAtIDEpLA0KICBob21lX2dvYWxzID0gcmVwKDEsIG5fdGVhbXMgKiAobl90ZWFtcyAtIDEpKSwgIyBkb2Vzbid0IG1hdHRlcg0KICBhd2F5X2dvYWxzID0gcmVwKDEsIG5fdGVhbXMgKiAobl90ZWFtcyAtIDEpKSwgIyBkb2Vzbid0IG1hdHRlcg0KICBob21lX2lkID0gc2FwcGx5KGlkX3NpbXVbWyJIb21lVGVhbSJdXSwgZnVuY3Rpb24oeCkge3doaWNoKHggPT0gdGVhbXNfc2ltdSl9KSwNCiAgYXdheV9pZCA9IHNhcHBseShpZF9zaW11W1siQXdheVRlYW0iXV0sIGZ1bmN0aW9uKHgpIHt3aGljaCh4ID09IHRlYW1zX3NpbXUpfSksDQogIHJ1biA9IDANCikNCg0KZml0X3ByaW9yIDwtIHNhbXBsaW5nKGNvbXBpbGVkX21vZGVsLA0KICAgICAgICAgICAgICAgICAgICAgIGRhdGEgPSBkYXRhX3ByaW9yLA0KICAgICAgICAgICAgICAgICAgICAgIHBhcnMgPSBwYXJhbSwNCiAgICAgICAgICAgICAgICAgICAgICBpdGVyID0gbl9pdCwNCiAgICAgICAgICAgICAgICAgICAgICBjaGFpbnMgPSBuX2NoYWlucykNCnBhcl9wcmlvciA8LSBleHRyYWN0X3BhcmFtZXRlcnMoZml0X3ByaW9yLCBwYXJhbSwgcGFyYW1faW5kLCBwYXJhbV9vYnMsIHRlYW1zX3NpbXUsIGlkX3NpbXUkR2FtZSwgZGF0YV9zdGFuKSAjIFN0b3JlIHBhcmFtZXRlcnMgZm9yIGxhdGVyIHVzZQ0KYGBgDQoNCldlIGNhbiBjaGVjayB0aGUgZGlzdHJpYnV0aW9uIG9mIGVhY2ggaW5kaXZpZHVhbCBwYXJhbWV0ZXI6DQoNCmBgYHtyfQ0KcGxvdChmaXRfcHJpb3IsIHBhcnMgPSBjKHBhcmFtX3BvcCwgcGFzdGUwKHBhcmFtX2luZFsxOjJdLCAiWzFdIikpLCBwbG90ZnVuID0gImhpc3QiKQ0KYGBgDQoNCldlIGNhbiBhbHNvIGluc3BlY3QsIGZvciBleGFtcGxlLCB0aGUgbnVtYmVyIG9mIGdvYWxzIHNjb3JlZCBieSB0aGUgaG9tZSB0ZWFtIGZvciBhIHJhbmRvbSBnYW1lIChhbGwgdGVhbXMgb3IgZ2FtZXMgYXJlIGludGVyY2hhbmdlYWJsZSBhcyB0aGlzIHBvaW50KS4NCg0KYGBge3J9DQpnb2FscyA8LSBleHRyYWN0KGZpdF9wcmlvciwgcGFycyA9IGMoImhvbWVfZ29hbHNfcmVwWzFdIikpW1sxXV0NCnN1bW1hcnkoZ29hbHMpDQpoaXN0KGdvYWxzLCBicmVha3MgPSA0MCkNCmhpc3QoZ29hbHNbZ29hbHMgPCAyMF0sIGJyZWFrcyA9IDIwKQ0KcXVhbnRpbGUoZ29hbHMsIHByb2JzID0gYyguMjUsIC41ICwgLjc1LCAuOSwgLjk5LCAuOTk5KSkNCmBgYA0KDQpBbHRob3VnaCB0aGUgcHJpb3IgZGlzdHJpYnV0aW9uIG9mIGdvYWxzIGhhcyBtb3N0IG9mIGl0cyBtYXNzIGZvciBzbWFsbCB2YWx1ZXMgKGUuZy4gJDwgNSQpLCBpdCBoYXMgYSBsb25nIHRhaWwgbWVhbmluZyB0aGF0LCBmb3IgaW5zdGFuY2UsIHRoZSBwcm9iYWJpbGl0eSBvZiB0aGUgaG9tZSB0ZWFtIHNjb3JpbmcgbW9yZSB0aGFuIDIwIGdvYWxzIGluIHRoZSBQcmVtaWVyIExlYWd1ZSBkdXJpbmcgb25lIGdhbWUgaXMgYHIgc2lnbmlmKG1lYW4oZ29hbHMgPj0gMjApLCAzKWAuDQpXaGlsZSB0aGlzIHByb2JhYmlsaXR5IGlzIHNtYWxsLCB0aGUgcHJvYmFiaWxpdHkgdGhhdCBoYXBwZW5zIGF0IGxlYXN0IG9uY2UgZHVyaW5nIDM4MCBnYW1lcyBpcyBgciBzaWduaWYoMSAtIGRiaW5vbSgwLCAzODAsIG1lYW4oZ29hbHMgPj0gMjApKSwgMylgLCB3aGljaCBtaWdodCBiZSBjb25zaWRlcmVkIHVucmVhbGlzdGljLg0KVGhpcyB3b3VsZCBzdWdnZXN0IG1ha2luZyBjaGFuZ2VzIHRvIHRoZSBtb2RlbCBidXQgd2Ugd2lsbCBjb250aW51ZSB3aXRoIGl0IGZvciBpbGx1c3RyYXRpb24gcHVycG9zZXMuDQoNCldlIGNhbiBhbHNvIGxvb2sgYXQgZGlzdHJpYnV0aW9uIG9mIHRoZSBudW1iZXIgb2YgZ2FtZXMgd29uLCBsb3N0IG9yIGVuZGluZyB3aXRoIGEgZHJhdyBmb3IgYSByYW5kb20gdGVhbSwgYnV0IHdlIGRvIG5vdCBkZXRlY3QgYW55dGhpbmcgdW5yZWFsaXN0aWM6DQoNCmBgYHtyfQ0KcGwgPC0gbGFwcGx5KGMoIndpbiIsICJsb3NlIiwgImRyYXciKSwNCiAgICAgICAgICAgICBmdW5jdGlvbih4KSB7DQogICAgICAgICAgICAgICBvdGMgPC0gZXh0cmFjdChmaXRfcHJpb3IsIHBhcnMgPSBwYXN0ZTAoeCwgIl9yZXBbMV0iKSlbWzFdXQ0KICAgICAgICAgICAgICAgb3RjIDwtIGZhY3RvcihvdGMsIGxldmVscyA9IDA6KDIgKiAobl90ZWFtcyAtIDEpKSkNCiAgICAgICAgICAgICAgIG90YyA8LSB0YWJsZShvdGMpIC8gbGVuZ3RoKG90YykNCiAgICAgICAgICAgICAgIGdncGxvdChkYXRhID0gZGF0YS5mcmFtZShvdGMpLCBhZXMoeCA9IG90YywgeSA9IEZyZXEpKSArDQogICAgICAgICAgICAgICAgIGdlb21fYmFyKHN0YXQgPSAiaWRlbnRpdHkiKSArDQogICAgICAgICAgICAgICAgIHNjYWxlX3hfZGlzY3JldGUoYnJlYWtzID0gc2VxKDEsIDIgKiAobl90ZWFtcyAtIDEpLCAyKSkgKw0KICAgICAgICAgICAgICAgICBsYWJzKHggPSBwYXN0ZTAoIk51bWJlciBvZiAiLCB4KSwgeSA9ICJQcmlvciBwcm9iYWJpbGl0eSIpICsNCiAgICAgICAgICAgICAgICAgdGhlbWVfYncoYmFzZV9zaXplID0gMTUpDQogICAgICAgICAgICAgfSkNCnBsb3RfZ3JpZChwbG90bGlzdCA9IHBsLCBuY29sID0gMSkNCmBgYA0KDQojIEZha2UgZGF0YSBjaGVjaw0KDQpJbiB0aGlzIHNlY3Rpb24sIHdlIGV2YWx1YXRlIHdoZXRoZXIgdGhlIGFsZ29yaXRobSAid29ya3MiLCBpLmUuIHdoZXRoZXIgd2UgY2FuIHJldHJpZXZlIHRoZSBwYXJhbWV0ZXJzIG9mIHRoZSBtb2RlbCBmcm9tIHRoZSBkYXRhLCB3aGVuIHdlIGtub3cgdGhlIHBhcmFtZXRlcnMgb2YgdGhlIHRydWUgZGF0YS1nZW5lcmF0aW5nIG1lY2hhbmlzbS4NClRvIGRvIHRoaXMsIHdlIGp1c3Qgc2FtcGxlIHRoZSBwcmlvciBwcmVkaWN0aXZlIGRpc3RyaWJ1dGlvbiBhbmQgZml0IHRoZSBtb2RlbCB3aXRoIHRoZSBzaW11bGF0ZWQgZGF0YS4NCg0KYGBge3J9DQpkcmF3IDwtIDIwMTkgIyBEcmF3DQoNCiMgVHJ1ZSBwYXJhbWV0ZXJzDQp0cnVlX3BhcmFtX3BvcCA8LSBsYXBwbHkoZXh0cmFjdChmaXRfcHJpb3IsIHBhcnMgPSBwYXJhbV9wb3ApLCBmdW5jdGlvbih4KSB7eFtkcmF3XX0pDQp0cnVlX3BhcmFtX2luZCA8LSBsYXBwbHkoZXh0cmFjdChmaXRfcHJpb3IsIHBhcnMgPSBwYXJhbV9pbmQpLCBmdW5jdGlvbih4KSB7eFtkcmF3LCBdfSkNCnRydWVfcGFyYW0gPC0gcmJpbmQoDQogIGRvLmNhbGwocmJpbmQsDQogICAgICAgICAgbGFwcGx5KDE6bGVuZ3RoKHRydWVfcGFyYW1faW5kKSwNCiAgICAgICAgICAgICAgICAgZnVuY3Rpb24oaSkgew0KICAgICAgICAgICAgICAgICAgIGRhdGEuZnJhbWUoVmFyaWFibGUgPSBuYW1lcyh0cnVlX3BhcmFtX2luZClbaV0sDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICBUcnVlID0gdHJ1ZV9wYXJhbV9pbmRbW2ldXSwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFRlYW0gPSB0ZWFtc19zaW11KQ0KICAgICAgICAgICAgICAgICB9KSksDQogIGRvLmNhbGwocmJpbmQsDQogICAgICAgICAgbGFwcGx5KDE6bGVuZ3RoKHRydWVfcGFyYW1fcG9wKSwNCiAgICAgICAgICAgICAgICAgZnVuY3Rpb24oaSkgew0KICAgICAgICAgICAgICAgICAgIGRhdGEuZnJhbWUoVmFyaWFibGUgPSBuYW1lcyh0cnVlX3BhcmFtX3BvcClbaV0sDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICBUcnVlID0gdHJ1ZV9wYXJhbV9wb3BbW2ldXSwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFRlYW0gPSBOQSkNCiAgICAgICAgICAgICAgICAgfSkpDQopDQoNCiMgRmFrZSBkYXRhDQpmZCA8LSBjYmluZChpZF9zaW11LA0KICAgICAgICAgICAgZGF0YS5mcmFtZShGVEhHID0gZXh0cmFjdChmaXRfcHJpb3IsIHBhcnMgPSAiaG9tZV9nb2Fsc19yZXAiKVtbMV1dW2RyYXcsIF0sDQogICAgICAgICAgICAgICAgICAgICAgIEZUQUcgPSBleHRyYWN0KGZpdF9wcmlvciwgcGFycyA9ICJhd2F5X2dvYWxzX3JlcCIpW1sxXV1bZHJhdywgXSwNCiAgICAgICAgICAgICAgICAgICAgICAgRlRSID0gTkEpKQ0KZmQkRlRSW2ZkJEZUSEcgPT0gZmQkRlRBR10gPC0gIkQiDQpmZCRGVFJbZmQkRlRIRyA+IGZkJEZUQUddIDwtICJIIg0KZmQkRlRSW2ZkJEZUSEcgPCBmZCRGVEFHXSA8LSAiQSINCmZkJEZUUiA8LSBmYWN0b3IoZmQkRlRSLCBsZXZlbHMgPSBjKCJBIiwgIkQiLCAiSCIpLCBvcmRlcmVkID0gVFJVRSkNCmBgYA0KDQpXZSBjYW4gdmlzdWFsaXNlIHRoZSBvdXRjb21lIG9mIHRoZXNlIHNpbXVsYXRlZCBnYW1lczoNCg0KYGBge3J9DQpoZWF0bWFwX3Jlc3VsdHMoZmQpDQpgYGANCg0KQW5kIHdlIGNhbiBhbHNvIGNvbXB1dGUgc29tZSBzdGF0aXN0aWNzIGFib3V0IHRoaXMgZmFrZSBkYXRhOg0KDQpgYGB7cn0NCihmc3RhdHNfZmFrZSA8LSBmb290YmFsbF9zdGF0cyhmZCkpDQpgYGANCg0KTGV0J3Mgbm93IGZpdCB0aGUgbW9kZWwgd2l0aCB0aGUgZmFrZSBkYXRhOg0KDQpgYGB7ciBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQ0KZGF0YV9mYWtlIDwtIGxpc3QoDQogIE5fdGVhbXMgPSBuX3RlYW1zLA0KICBOX2dhbWVzID0gbl90ZWFtcyAqIChuX3RlYW1zIC0gMSksDQogIGhvbWVfZ29hbHMgPSBmZCRGVEhHLA0KICBhd2F5X2dvYWxzID0gZmQkRlRBRywNCiAgaG9tZV9pZCA9IHNhcHBseShmZFtbIkhvbWVUZWFtIl1dLCBmdW5jdGlvbih4KSB7d2hpY2goeCA9PSB0ZWFtc19zaW11KX0pLA0KICBhd2F5X2lkID0gc2FwcGx5KGZkW1siQXdheVRlYW0iXV0sIGZ1bmN0aW9uKHgpIHt3aGljaCh4ID09IHRlYW1zX3NpbXUpfSksDQogIHJ1biA9IDENCikNCg0KZml0X2Zha2UgPC0gc2FtcGxpbmcoY29tcGlsZWRfbW9kZWwsDQogICAgICAgICAgICAgICAgICAgICBkYXRhID0gZGF0YV9mYWtlLA0KICAgICAgICAgICAgICAgICAgICAgcGFycyA9IHBhcmFtLA0KICAgICAgICAgICAgICAgICAgICAgaXRlciA9IG5faXQsDQogICAgICAgICAgICAgICAgICAgICBjaGFpbnMgPSBuX2NoYWlucykNCnBhcl9mYWtlIDwtIGV4dHJhY3RfcGFyYW1ldGVycyhmaXRfZmFrZSwgcGFyYW0sIHBhcmFtX2luZCwgcGFyYW1fb2JzLCB0ZWFtc19zaW11LCBmZCRHYW1lLCBkYXRhX3N0YW4pDQpgYGANCg0KRmlyc3QsIHdlIHNob3VsZCBjaGVjayB0aGUgTUNNQyBkaWFnbm9zdGljczogbm90aGluZyB0byB3b3JyeSBhYm91dC4NCg0KYGBge3J9DQpjaGVja19obWNfZGlhZ25vc3RpY3MoZml0X2Zha2UpDQpwYWlycyhmaXRfZmFrZSwgcGFycyA9IHBhcmFtX3BvcCkNCnBsb3QoZml0X2Zha2UsIHBhcnMgPSBwYXJhbV9wb3AsIHBsb3RmdW4gPSAidHJhY2UiKQ0KcHJpbnQoZml0X2Zha2UsIHBhcnMgPSBwYXJhbV9wb3ApDQpgYGANCg0KVGhlbiwgd2UgY2FuIGNoZWNrIHdoZXRoZXIgdGhlIHBvc3RlcmlvciBlc3RpbWF0ZXMgImNsb3NlIGVub3VnaCIgdG8gdGhlIHRydWUgcGFyYW1ldGVycz8NCg0KYGBge3J9DQooY2UgPC0gY2hlY2tfZXN0aW1hdGVzKHBhcl9mYWtlLCB0cnVlX3BhcmFtLCBwYXJhbV9wb3AsIHBhcmFtX2luZFsxOjJdKSkNCmBgYA0KDQpWaXN1YWxseSwgdGhleSBhcHBlYXIgc28sIGJ1dCB3ZSBjYW4gYWxzbyBxdWFudGlmeSBpdCBieSBjb21wdXRpbmcsIGZvciBleGFtcGxlLCB0aGUgOTAlIGNvdmVyYWdlIHByb2JhYmlsaXR5LCBpLmUuIHRoZSBwcm9wb3J0aW9uIG9mIHBhcmFtZXRlcnMgZmFsbGluZyBpbiB0aGUgOTAlIGNyZWRpYmxlIGludGVydmFsLg0KSGVyZSB0aGUgY292ZXJhZ2UgaXMgYHIgc2lnbmlmKGNlJENvdmVyYWdlLCAyKWAgd2hpY2ggaXMgY2xvc2UgZW5vdWdoIHRvIHdoYXQgaXQgc2hvdWxkIGJlLCBpLmUuIDkwJS4NCg0KRmluYWxseSwgd2UgY2FuIHBlcmZvcm0gcG9zdGVyaW9yIHByZWRpY3RpdmUgY2hlY2tzIHRvIGRldGVjdCBhbnkgZGlzY3JlcGFuY2llcyBiZXR3ZWVuIHRoZSBvYnNlcnZlZCAoaGVyZSwgZmFrZSkgYW5kIHRoZSBwb3N0ZXJpb3IgcmVwbGljYXRpb25zLg0KV2UgY2FuIGludmVzdGlnYXRlIHNldmVyYWwgc3VtbWFyeSBzdGF0aXN0aWNzIHN1Y2ggYXMgdGhlIG51bWJlciBvZiBnYW1lcyB3b24sIGxvc3Qgb3IgZHJhd3MgZm9yIGEgcmFuZG9tIHRlYW0sIGFzIHdlbGwgYXMgdGhlIHRvdGFsIG51bWJlciBvZiBwb2ludCBvciBldmVuIGlmIHRoZSBmaW5hbCByYW5rLg0KRnJvbSB0aGUgcGxvdCwgd2UgY2Fubm90IHZpc3VhbGx5IGlkZW50aWZ5IGFueSBpc3N1ZXMgd2l0aCB0aGUgcG9zdGVyaW9yIHJlcGxpY2F0aW9ucy4NCg0KYGBge3IgZmlnLmhlaWdodCA9IDEwLCBmaWcud2lkdGggPSAxMH0NClBQQ19mb290YmFsbF9zdGF0cyhmaXRfZmFrZSwgIndpbl9yZXAiLCBmc3RhdHNfZmFrZSwgdGVhbXNfc2ltdSkNClBQQ19mb290YmFsbF9zdGF0cyhmaXRfZmFrZSwgImxvc2VfcmVwIiwgZnN0YXRzX2Zha2UsIHRlYW1zX3NpbXUpDQpQUENfZm9vdGJhbGxfc3RhdHMoZml0X2Zha2UsICJnb2FsX3RvdF9yZXAiLCBmc3RhdHNfZmFrZSwgdGVhbXNfc2ltdSkNClBQQ19mb290YmFsbF9zdGF0cyhmaXRfZmFrZSwgInBvaW50X3JlcCIsIGZzdGF0c19mYWtlLCB0ZWFtc19zaW11KQ0KUFBDX2Zvb3RiYWxsX3N0YXRzKGZpdF9mYWtlLCAicmFua19yZXAiLCBmc3RhdHNfZmFrZSwgdGVhbXNfc2ltdSkNCmBgYA0KDQpOQjogVGhlIGZha2UgZGF0YSBjaGVjayBjYW4gYmUgcmVwZWF0ZWQgdG8gbWFrZSBzdXJlIHRoZSBtb2RlbCBjYW4gZXN0aW1hdGUgZGlmZmVyZW50IHJlYWxpc2F0aW9ucyBvZiB0aGUgcHJpb3IgcHJlZGljdGl2ZSBkaXN0cmlidXRpb24gaW4gYSBwcm9jZXNzIHRoYXQgaXMgY2FsbGVkIFNpbXVsYXRpb24gQmFzZWQgQ2FsaWJyYXRpb24uDQoNCiMgTW9kZWwgZml0dGluZw0KDQpIYXZpbmcgY29uZmlybWVkIHRoYXQgdGhlIG1vZGVsIGNvdWxkIGJlIGZpdHRlZCBpbiB0aGUgcHJldmlvdXMgc2VjdGlvbiwgaW4gdGhpcyBzZWN0aW9uLCB3ZSB3aWxsIHRyYWluIHRoZSBtb2RlbCB3aXRoIHRoZSBkYXRhIGZyb20gdGhlIDIwMTgvMjAxOSBzZWFzb24gb2YgdGhlIEVuZ2xpc2ggUHJlbWllciBMZWFndWUuDQoNCmBgYHtyIG1lc3NhZ2U9RkFMU0UsIHdhcm5pbmc9RkFMU0V9DQpkYXRhX2ZpdCA8LSBsaXN0KA0KICBOX3RlYW1zID0gbGVuZ3RoKHRlYW1zKSwNCiAgTl9nYW1lcyA9IG5yb3coZGYpLA0KICBob21lX2dvYWxzID0gZGZbWyJGVEhHIl1dLA0KICBhd2F5X2dvYWxzID0gZGZbWyJGVEFHIl1dLA0KICBob21lX2lkID0gc2FwcGx5KGRmW1siSG9tZVRlYW0iXV0sIGZ1bmN0aW9uKHgpIHt3aGljaCh4ID09IHRlYW1zKX0pLA0KICBhd2F5X2lkID0gc2FwcGx5KGRmW1siQXdheVRlYW0iXV0sIGZ1bmN0aW9uKHgpIHt3aGljaCh4ID09IHRlYW1zKX0pLA0KICBydW4gPSAxDQopDQoNCmZpdCA8LSBzYW1wbGluZyhjb21waWxlZF9tb2RlbCwNCiAgICAgICAgICAgICAgICBkYXRhID0gZGF0YV9maXQsDQogICAgICAgICAgICAgICAgcGFycyA9IHBhcmFtLA0KICAgICAgICAgICAgICAgIGl0ZXIgPSBuX2l0LA0KICAgICAgICAgICAgICAgIGNoYWlucyA9IG5fY2hhaW5zKQ0KcGFyIDwtIGV4dHJhY3RfcGFyYW1ldGVycyhmaXQsIHBhcmFtLCBwYXJhbV9pbmQsIHBhcmFtX29icywgdGVhbXMsIGRmJEdhbWUsIGRhdGFfX2ZpdCkNCmBgYA0KDQpGaXJzdCwgd2UgaW5zcGVjdCBjb252ZXJnZSBkaWFnbm9zdGljczogbm90aGluZyB0byB3b3JyeSBhYm91dC4NCg0KYGBge3J9DQpjaGVja19obWNfZGlhZ25vc3RpY3MoZml0KQ0KcGFpcnMoZml0LCBwYXJzID0gcGFyYW1fcG9wKQ0KcGxvdChmaXQsIHBhcnMgPSBwYXJhbV9wb3AsIHBsb3RmdW4gPSAidHJhY2UiKQ0KYGBgDQoNCk5vdyB3ZSBjYW4gbG9vayBhdCB0aGUgcGFyYW1ldGVyIGVzdGltYXRlcywgdGhlIHBvcHVsYXRpb24gcGFyYW1ldGVycyAoZS5nLiBwYXJhbWV0ZXJzIHRoYXQgYXJlIHNoYXJlZCBhY3Jvc3MgdGVhbXMpIGFuZCB0aGUgYXR0YWNrIGFuZCBkZWZlbmNlIGFiaWxpdGllcyBmb3IgZWFjaCB0ZWFtcy4NClRoZSBjb2VmZmljZW50IHBsb3QgZm9yIHRoZSBwb3B1bGF0aW9uIHBhcmFtZXRlcnMgcmV2ZWFsIHRoYXQgdGhlIHByaW9ycyBzZWVtcyB3ZWFrbHkgaW5mb3JtYXRpdmUgZW5vdWdoIHRvICJpbmNsdWRlIiB0aGUgcG9zdGVyaW9ycy4NCkluIGFkZGl0aW9uLCB3ZSBub3RpY2UgdGhhdCBNYW5jaGVzdGVyIENpdHkgYW5kIExpdmVycG9vbCBoYXZlIHRoZSBiZXN0IGEgcG9zdGVyaW9yaSBhdHRhY2sgYW5kIGRlZmVuY2UgYWJpbGl0aWVzIG9mIHRoaXMgc2Vhc29uLCB3aGljaCBpcyBjb25zaXN0ZW50IHdpdGggdGhlIGZhY3QgdGhhdCB0aGV5IGZpbmlzaGVkIGZpcnN0IGFuZCBzZWNvbmQgcmVzcGVjdGl2ZWx5Lg0KDQpgYGB7cn0NCkh1cmF1bHRNaXNjOjpwbG90X3ByaW9yX3Bvc3RlcmlvcihwYXJfcHJpb3IsIHBhciwgcGFyYW1fcG9wKSArDQogIGxhYnModGl0bGUgPSAiPGI+UG9zdGVyaW9yPC9iPiB2cyA8YiBzdHlsZT0nY29sb3I6I0U2OUYwMCc+cHJpb3I8L2I+IGVzdGltYXRlcyAobWVhbiBhbmQgOTAlIENJKSIsDQogICAgICAgc3VidGl0bGUgPSAiUG9wdWxhdGlvbiBwYXJhbWV0ZXJzIiwNCiAgICAgICB5ID0gIiIpICsNCiAgdGhlbWUocGxvdC50aXRsZSA9IGVsZW1lbnRfbWFya2Rvd24oKSwNCiAgICAgICAgcGxvdC50aXRsZS5wb3NpdGlvbiA9ICJwbG90IiwNCiAgICAgICAgbGVnZW5kLnBvc2l0aW9uID0gIm5vbmUiKQ0KDQpwbG90X2FiaWxpdGllcyhwYXIpDQpgYGANCg0KV2UgY2FuIGFsc28gbG9vayBhdCB0aGUgcG9zdGVyaW9yIHByZWRpY3RpdmUgZGlzdHJpYnV0aW9uLg0KRm9yIGNvbmNpc2lvbiwgSSBhbSBub3QgcGxvdHRpbmcgdGhlIHBvc3RlcmlvciBwcm9iYWJpbGl0eSBmb3IgdGhlIG51bWJlciB3aW5zLCBsb3NlLCBkcmF3cywgZ29hbHMgb3IgcG9pbnRzLCBidXQgd2Ugd2lsbCBsb29rIGF0IHRoZSBwb3N0ZXJpb3IgcmFua3MuDQoNCkluIHRoZSBmb2xsb3dpbmcgcGxvdCwgdGhlIHNpemUgb2YgdGhlIGNvbG91ciBiYXJzIHJlcHJlc2VudCB0aGUgcHJvYmFiaWxpdHkgYXQgdGhlIGdpdmVuIHJhbmsuDQpGb3IgaW5zdGFuY2UsIHRoZSBwb3N0ZXJpb3IgcHJvYmFiaWxpdHkgZm9yIE1hbmNoZXN0ZXIgZmluaXNoaW5nIGZpcnN0IGlzIHNsaWdodGx5IGFib3ZlIDUwJSBhbmQgYXJvdW5kIDMwJSBmb3IgZmluaXNoaW5nIHNlY29uZC4NClNpbWlsYXJseSwgd2UgY2FuIHZpc3VhbGx5IGFwcHJveGltYXRlIHRoZSBwb3N0ZXJpb3IgcHJvYmFiaWxpdHkgZm9yIExpdmVycG9vbCBmaW5pc2hpbmcgZmlyc3QgdG8gYmUgNDAlIGFuZCBhIHNpbWlsYXIgcHJvYmFiaWxpdHkgZm9yIGZpbmlzaGluZyBzZWNvbmQuDQoNCmBgYHtyIHdhcm5pbmc9RkFMU0V9DQpzdGFja2hpc3RfcmFuayhjb21wdXRlX3JhbmsoZml0LCAicmVwIiksIHRlYW1zKQ0KYGBgDQoNCiMgTW9kZWwgdmFsaWRhdGlvbg0KDQpXaGlsZSB0aGUgZml0IGNhbiBoZWxwIHVzIHVuZGVyc3RhbmQgd2hhdCB3YXMgZ29pbmcgb24gZHVyaW5nIHRoZSBzZWFzb24gYSBwb3N0ZXJpb3JpLCBpdCBpcyBpbnRlcmVzdGluZyB0byBrbm93IHRvIHdoYXQgZXh0ZW50IHRoZSBtb2RlbCBpcyBwcmVkaWN0aXZlLg0KDQpTaW5jZSB3ZSBhcmUgZGVhbGluZyB3aXRoIHRpbWUtc2VyaWVzIGRhdGEgYW5kIHdhbnQgdG8gcHJlZGljdCB0aGUgZnV0dXJlIGJhc2VkIG9uIHRoZSBwYXN0LCBpdCBpcyBub3QgYXBwcm9wcmlhdGUgdG8gdXNlIHN0YW5kYXJkIGNyb3NzLXZhbGlkYXRpb24gdGVjaG5pcXVlcyBzdWNoIGFzIEstZm9sZCBjcm9zcy12YWxpZGF0aW9uLCByYXRoZXIsIHdlIHdpbGwgaW1wbGVtZW50IGZvcndhcmQgY2hhaW5pbmcgd2hlcmUgdGhlIG1vZGVsIGlzIHRyYWluZWQgb24gdGhlIGRhdGEgZnJvbSB0aGUgZmlyc3Qgd2VlayBhbmQgdGVzdGVkIG9uIHRoZSBuZXh0LCB0aGVuIHRyYWluZWQgb24gdGhlIGRhdGEgb2YgdGhlIGZpcnN0IHR3byB3ZWVrcyBhbmQgdGVzdGVkIG9uIHRoZSByZW1haW5pbmcgd2Vla3MsIGV0Yy4NCg0KYGBge3J9DQpIdXJhdWx0TWlzYzo6aWxsdXN0cmF0ZV9mb3J3YXJkX2NoYWluaW5nKCkNCmBgYA0KDQpXZSBjYW4gZXZhbHVhdGUgdGhlIHBlcmZvcm1hbmNlIG9mIHRoZSBtb2RlbCB0byBwcmVkaWN0IHRoZSBmdWxsIHRpbWUgcmVzdWx0cyB1c2luZyB0aGUgUmFua2VkIFByb2JhYmlsaXR5IFNjb3JlLCBhIHByb3BlciBzY29yaW5nIHJ1bGUgdG8gbWVhc3VyZSB0aGUgYWNjdXJhY3kgb2Ygb3JkaW5hbCAoY2YuIExvc2UgPCBEcmF3IDwgV2luKSBwcm9iYWJpbGlzdGljIGZvcmVjYXN0Lg0KSXQgaXMgYWxzbyBwb3NzaWJsZSB0byBldmFsdWF0ZSB0aGUgbW9kZWwgaW4gaXRzIGFiaWxpdHkgdG8gcHJlZGljdCB0aGUgbnVtYmVyIG9mIGdvYWxzIGZvciBpbnN0YW5jZSwgYnV0IEkgd2lsbCBub3Qgc2hvdyB0aGVzZSByZXN1bHRzIGhlcmUuDQoNClRoZSBmb2xsb3dpbmcgY29kZSBpbXBsZW1lbnRzIHRoZSBmb3J3YXJkIGNoYWluaW5nLg0KQ29uc2lkZXJpbmcgdGhlIHRhc2sgaXMgcGFyYWxsZWwgaW4gbmF0dXJlLCBpdCBjYW4gYmUgY29udmVuaWVudCB0byB0YWtlIGFkdmFudGFnZSBvZiBtdWx0aXBsZSBjb3JlcyB0aGF0IG1pZ2h0IGJlIGF2YWlsYWJsZS4NCg0KYGBge3IgbWVzc2FnZT1GQUxTRSwgd2FybmluZz1GQUxTRX0NCm5fY2x1c3RlciA8LSBmbG9vcihwYXJhbGxlbDo6ZGV0ZWN0Q29yZXMoKSAvIG5fY2hhaW5zKQ0KDQojIFRyYWluaW5nIHVuaXQNCmRmW1siV2Vla051bWJlciJdXSA8LSBzdHJmdGltZShkZltbIkRhdGUiXV0sIGZvcm1hdCA9ICIlWS0lViIpDQp3ZWVrcyA8LSB1bmlxdWUoZGZbWyJXZWVrTnVtYmVyIl1dKQ0KDQojIFVwZGF0ZSBwYXJhbWV0ZXIgb2YgaW50ZXJlc3QNCnBhcmFtX3Rlc3QgPC0gYygid2luX3Rlc3QiLCAiZHJhd190ZXN0IiwgImxvc2VfdGVzdCIsDQogICAgICAgICAgICAgICAgImdvYWxfdG90X3Rlc3QiLCAiZ29hbF9kaWZmX3Rlc3QiLCAicG9pbnRfdGVzdCIpDQpwYXJhbV9pbmQgPC0gYygiYXR0YWNrIiwgImRlZmVuY2UiLCBwYXJhbV90ZXN0KQ0KcGFyYW1fb2JzIDwtIGMoImhvbWVfZ29hbHNfdGVzdCIsICJhd2F5X2dvYWxzX3Rlc3QiKQ0KcGFyYW0gPC0gYyhwYXJhbV9wb3AsIHBhcmFtX2luZCwgcGFyYW1fb2JzKQ0KDQpmb3JtYXRfc3Rhbl9kYXRhIDwtIGZ1bmN0aW9uKGRmKSB7DQogIGxpc3QoDQogICAgTl90ZWFtcyA9IGxlbmd0aCh0ZWFtcyksDQogICAgTl9nYW1lcyA9IG5yb3coZGYpLA0KICAgIGhvbWVfZ29hbHMgPSBkZltbIkZUSEciXV0sDQogICAgYXdheV9nb2FscyA9IGRmW1siRlRBRyJdXSwNCiAgICBob21lX2lkID0gc2FwcGx5KGRmW1siSG9tZVRlYW0iXV0sIGZ1bmN0aW9uKHgpIHt3aGljaCh4ID09IHRlYW1zKX0pLA0KICAgIGF3YXlfaWQgPSBzYXBwbHkoZGZbWyJBd2F5VGVhbSJdXSwgZnVuY3Rpb24oeCkge3doaWNoKHggPT0gdGVhbXMpfSksDQogICAgcnVuID0gMQ0KICApDQp9DQoNCmxpYnJhcnkoZm9yZWFjaCkNCmxpYnJhcnkoZG9QYXJhbGxlbCkNCg0KZHVyYXRpb24gPC0gU3lzLnRpbWUoKQ0KY2wgPC0gbWFrZUNsdXN0ZXIobl9jbHVzdGVyKQ0KcmVnaXN0ZXJEb1BhcmFsbGVsKGNsKQ0Kd3JpdGVMaW5lcyhjKCIiKSwgImxvZy50eHQiKQ0KDQpvdXQgPC0gZm9yZWFjaCh3ID0gMToobGVuZ3RoKHdlZWtzKSAtIDEpKSAlZG9wYXIlIHsNCiAgDQogIHNvdXJjZSgiZnVuY3Rpb25zLlIiKQ0KICBsaWJyYXJ5KHJzdGFuKQ0KICByc3Rhbl9vcHRpb25zKGF1dG9fd3JpdGUgPSBUUlVFKSAjIFNhdmUgY29tcGlsZWQgbW9kZWwNCiAgb3B0aW9ucyhtYy5jb3JlcyA9IHBhcmFsbGVsOjpkZXRlY3RDb3JlcygpKSAjIFBhcmFsbGVsIGNvbXB1dGluZw0KICANCiAgc2luaygibG9nLnR4dCIsIGFwcGVuZCA9IFRSVUUpDQogIGNhdChwYXN0ZSgiU3RhcnRpbmcgdHJhaW5pbmcgYXQgd2VlayAiLCB3LCAiIFxuIiwgc2VwID0gIiIpKQ0KICANCiAgZGZfdHJhaW4gPC0gZGZbZGYkV2Vla051bWJlciA8PSB3ZWVrc1t3XSwgXQ0KICBkZl90ZXN0IDwtIGRmW2RmW1siV2Vla051bWJlciJdXSA+IHdlZWtzW3ddLCBdDQogIA0KICBkYXRhX3N0YW4gPC0gZm9ybWF0X3N0YW5fZGF0YShkZl90cmFpbikNCiAgDQogIGZpdCA8LSBzYW1wbGluZyhjb21waWxlZF9tb2RlbCwNCiAgICAgICAgICAgICAgICAgIGRhdGEgPSBkYXRhX3N0YW4sDQogICAgICAgICAgICAgICAgICBwYXJzID0gcGFyYW0sDQogICAgICAgICAgICAgICAgICBpdGVyID0gbl9pdCwNCiAgICAgICAgICAgICAgICAgIGNoYWlucyA9IG5fY2hhaW5zKQ0KICANCiAgIyBQYXJhbWV0ZXJzDQogIHBhciA8LSBleHRyYWN0X3BhcmFtZXRlcnMoZml0LCBwYXJhbSA9IGMocGFyYW1fcG9wLCBwYXJhbV9pbmQpLCBwYXJhbV9pbmQsIHBhcmFtX29icywgdGVhbXMsIGRmX3RyYWluW1siR2FtZSJdXSwgZGF0YV9zdGFuKQ0KICBwYXIkV2Vla051bWJlciA8LSB3ZWVrc1t3XQ0KICBwYXIkUHJvcG9ydGlvbkdhbWVQbGF5ZWQgPC0gbnJvdyhkZl90cmFpbikgLyBucm93KGRmKQ0KICANCiAgIyBSYW5rDQogIHJrIDwtIGNvbXB1dGVfcmFuayhmaXQsICJ0ZXN0IikNCiAgcmsgPC0gZG8uY2FsbChyYmluZCwNCiAgICAgICAgICAgICAgICBsYXBwbHkoMTpsZW5ndGgodGVhbXMpLA0KICAgICAgICAgICAgICAgICAgICAgICBmdW5jdGlvbihpKSB7DQogICAgICAgICAgICAgICAgICAgICAgICAgdG1wIDwtIHRhYmxlKGZhY3Rvcihya1ssIGldLCBsZXZlbHMgPSAxOmxlbmd0aCh0ZWFtcykpKSAvIG5yb3cocmspDQogICAgICAgICAgICAgICAgICAgICAgICAgZGF0YS5mcmFtZShUZWFtID0gdGVhbXNbaV0sDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBSYW5rID0gbmFtZXModG1wKSwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFByb2JhYmlsaXR5ID0gYXMubnVtZXJpYyh0bXApKQ0KICAgICAgICAgICAgICAgICAgICAgICB9KSkNCiAgcmsgPC0gSHVyYXVsdE1pc2M6OmZhY3Rvcl90b19udW1lcmljKHJrLCAiUmFuayIpDQogIHJrJFdlZWtOdW1iZXIgPC0gd2Vla3Nbd10NCiAgcmskUHJvcG9ydGlvbkdhbWVQbGF5ZWQgPC0gbnJvdyhkZl90cmFpbikgLyBucm93KGRmKQ0KICANCiAgIyBNZXRyaWNzDQogIHByZWQgPC0gcHJvY2Vzc19wcmVkaWN0aW9ucyhmaXQsIGlkKQ0KICBtIDwtIGNvbXB1dGVfbWV0cmljcyhwcmVkID0gcHJlZCwgYWN0ID0gZGYsIHRlc3RfZ2FtZSA9IGRmX3Rlc3RbWyJHYW1lIl1dLCB2YXIgPSAiRlRSIikNCiAgbSRXZWVrTnVtYmVyIDwtIHdlZWtzW3ddDQogIG0kUHJvcG9ydGlvbkdhbWVQbGF5ZWQgPC0gbnJvdyhkZl90cmFpbikgLyBucm93KGRmKQ0KICANCiAgbGlzdChQZXJmb3JtYW5jZSA9IG0sIFBhcmFtZXRlcnMgPSBwYXIsIFJhbmsgPSByaykNCn0NCnN0b3BDbHVzdGVyKGNsKQ0KKGR1cmF0aW9uID0gU3lzLnRpbWUoKSAtIGR1cmF0aW9uKQ0KDQptIDwtIGRvLmNhbGwocmJpbmQsIGxhcHBseShvdXQsIGZ1bmN0aW9uKHgpIHt4JFBlcmZvcm1hbmNlfSkpDQpwYXIgPC0gZG8uY2FsbChyYmluZCwgbGFwcGx5KG91dCwgZnVuY3Rpb24oeCkge3gkUGFyYW1ldGVyc30pKQ0KcmsgPC0gZG8uY2FsbChyYmluZCwgbGFwcGx5KG91dCwgZnVuY3Rpb24oeCkge3gkUmFua30pKQ0KYGBgDQoNCldlIGNhbiBub3cgcGxvdCB0aGUgcHJlZGljdGl2ZSBwZXJmb3JtYW5jZSBvZiB0aGUgbW9kZWwgYXMgYSBmdW5jdGlvbiBvZiB0cmFpbmluZyB3ZWVrLCBvciBhcyBhIGZ1bmN0aW9uIG9mIHRoZSBwcm9wb3J0aW9uIG9mIGdhbWUgcGxheWVkIGluIHRoZSBzZWFzb24uDQoNCmBgYHtyfQ0KZ2dwbG90KGRhdGEgPSBzdWJzZXQobSwgTWV0cmljID09ICJSUFMiKSwNCiAgICAgICBhZXMoeCA9IFByb3BvcnRpb25HYW1lUGxheWVkLCB5ID0gTWVhbiwgeW1pbiA9IE1lYW4gLSBTRSwgeW1heCA9IE1lYW4gKyBTRSkpICsNCiAgZ2VvbV9wb2ludHJhbmdlKCkgKw0KICBzY2FsZV95X2NvbnRpbnVvdXMobGltaXRzID0gYygwLCBOQSkpICsNCiAgbGFicyh5ID0gIlJQUyIsIHRpdGxlID0gIlJQUyBsZWFybmluZyBjdXJ2ZSAobG93ZXIgdGhlIGJldHRlcikiKSArDQogIHRoZW1lX2J3KGJhc2Vfc2l6ZSA9IDE1KQ0KYGBgDQoNCkFsdGhvdWdoIHRoZSBSUFMgaXMgc2xpZ2h0bHkgaW1wcm92aW5nLCBpdCBkb2VzIG5vdCBzZWVtIHRvIGJlIGJ5IG11Y2gsIHdoaWNoIHN1Z2dlc3QgYSBsaW1pdGF0aW9uIG9mIHN1Y2ggYSBzaW1wbGUgbW9kZWwuDQpXZSBjb3VsZCBpbnZlc3RpZ2F0ZSB0aGlzIGJ5IHBsb3R0aW5nIGhvdyB0aGUgYmVsaWV2ZXMgaW4gdGhlIHRlYW1zIGFiaWxpdGllcyBjaGFuZ2VzIHdpdGggdGltZS4NCg0KYGBge3IgZmlnLndpZHRoPTEwLCBmaWcuaGVpZ2h0PTEwfQ0KdG1wIDwtIHN1YnNldChwYXIsIFZhcmlhYmxlICVpbiUgYygiYXR0YWNrIiwgImRlZmVuY2UiKSkNCnBsNCA8LSBsYXBwbHkodGVhbXMsIGZ1bmN0aW9uKHgpIHsNCiAgY2JiUGFsZXR0ZSA8LSBjKCIjMDAwMDAwIiwgIiNFNjlGMDAiLCAiIzU2QjRFOSIsICIjMDA5RTczIiwgIiNGMEU0NDIiLCAiIzAwNzJCMiIsICIjRDU1RTAwIiwgIiNDQzc5QTciKQ0KICBnZ3Bsb3QoZGF0YSA9IHN1YnNldCh0bXAsIFRlYW0gPT0geCksIA0KICAgICAgICAgYWVzKHggPSBQcm9wb3J0aW9uR2FtZVBsYXllZCwgeSA9IE1lYW4sIHltaW4gPSBgNSVgLCB5bWF4ID0gYDk1JWAsIGNvbG91ciA9IFZhcmlhYmxlLCBmaWxsID0gVmFyaWFibGUpKSArDQogICAgZ2VvbV9saW5lKCkgKw0KICAgIGdlb21fcmliYm9uKGFscGhhID0gMC41KSArDQogICAgc2NhbGVfY29sb3VyX21hbnVhbCh2YWx1ZXMgPSBjYmJQYWxldHRlKSArDQogICAgc2NhbGVfZmlsbF9tYW51YWwodmFsdWVzID0gY2JiUGFsZXR0ZSkgKw0KICAgIGxhYnModGl0bGUgPSB4LCB5ID0gIkFiaWxpdHkiLCBjb2xvdXIgPSAiIiwgZmlsbCA9ICIiKSArDQogICAgY29vcmRfY2FydGVzaWFuKHlsaW0gPSBjKC0xLCAxKSkgKw0KICAgIHRoZW1lX2J3KGJhc2Vfc2l6ZSA9IDE1KQ0KfSkNCnBsb3RfZ3JpZChnZXRfbGVnZW5kKHBsNFtbMV1dICsgdGhlbWUobGVnZW5kLnBvc2l0aW9uID0gInRvcCIpKSwNCiAgICAgICAgICBwbG90X2dyaWQocGxvdGxpc3QgPSBsYXBwbHkocGw0LCBmdW5jdGlvbihwKSB7cCArIHRoZW1lKGxlZ2VuZC5wb3NpdGlvbiA9ICJub25lIil9KSwNCiAgICAgICAgICAgICAgICAgICAgbnJvdyA9IDUpLA0KICAgICAgICAgIG5yb3cgPSAyLCByZWxfaGVpZ2h0cyA9IGMoLjA1LCAuOTUpKQ0KYGBgDQoNCkV2ZW4gdGhvdWdoIHRoZSBhYmlsaXRpZXMgYXJlIGxlYXJudCBhcyBtb3JlIGRhdGEgY29tZXMgaW4sIHRoZXkgcmVtYWluIHVuY2VydGFpbiwgd2hpY2ggY291bGQgZXhwbGFpbiB0aGUgcHJldmlvdXMgcmVzdWx0Lg0KDQpJdCBjYW4gYWxzbyBiZSBpbnRlcmVzdGluZyB0byBzZWUgaG93IG91ciBwcmVkaWN0aW9ucyBjaGFuZ2VzIHdpdGggdGltZSwgZm9yIGluc3RhbmNlLCBob3cgdGhlIHVuY2VydGFpbnR5IG92ZXIgdGhlIGZpbmFsIHJhbmtpbmcgaXMgY2hhbmdpbmcgYXMgbW9yZSBnYW1lcyBhcmUgcGxheWVkLg0KVGhlIGZvbGxvd2luZyBwbG90IGRlcGljdHMsIGFzIGFuIGhlYXRtYXAsIHRoZSBwcmVkaWN0ZWQgcmFuayBhdCB0aGUgZW5kIG9mIHRoZSBzZWFzb24gYXMgYSBmdW5jdGlvbiBvZiB0aGUgbnVtYmVyIG9mIGdhbWVzIHBsYXllZCwgZm9yIGVhY2ggdGVhbS4NCkFzIHdlIGNvdWxkIGV4cGVjdCwgZWFybHkgaW4gdGhlIHNlYXNvbiwgdGhlIHByZWRpY3Rpb25zIGFyZSBxdWl0ZSB1bmNlcnRhaW4gYnV0IGJlY29tZXMgbW9yZSBjb25maWRlbnQgYXMgZmV3ZXIgZ2FtZXMgcmVtYWluIHRvIGJlIHBsYXllZCBhbmQgYXMgdGhlIG1vZGVsIGhhcyBiZXR0ZXIgZXN0aW1hdGVzIG9mIGl0cyBwYXJhbWV0ZXJzLg0KDQpgYGB7ciBmaWcud2lkdGg9MTAsIGZpZy5oZWlnaHQ9MTB9DQojIEFkZCBmaXJzdCB3ZWVrIHJhbmtpbmcNCnJrIDwtIHJiaW5kKGRhdGEuZnJhbWUoZXhwYW5kLmdyaWQoVGVhbSA9IHRlYW1zLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBSYW5rID0gMTpsZW5ndGgodGVhbXMpKSwNCiAgICAgICAgICAgICAgICAgICAgICAgUHJvYmFiaWxpdHkgPSAxIC8gbGVuZ3RoKHRlYW1zKSwNCiAgICAgICAgICAgICAgICAgICAgICAgV2Vla051bWJlciA9IHN0cmZ0aW1lKG1pbihkZltbIkRhdGUiXV0pIC0gNywgZm9ybWF0ID0gIiVZLSVWIiksDQogICAgICAgICAgICAgICAgICAgICAgIFByb3BvcnRpb25HYW1lUGxheWVkID0gMCksICMgYWRkIGZpcnN0IHdlZWsNCiAgICAgICAgICAgIHJrKQ0KDQojIEFkZCBsYXN0IHdlZWsgcmFua2luZw0KdG1wIDwtIGRhdGEuZnJhbWUoZXhwYW5kLmdyaWQoVGVhbSA9IHRlYW1zLCBSYW5rID0gMTpsZW5ndGgodGVhbXMpKSwNCiAgICAgICAgICAgICAgICAgIFByb2JhYmlsaXR5ID0gMCwNCiAgICAgICAgICAgICAgICAgIFdlZWtOdW1iZXIgPSB3ZWVrc1tsZW5ndGgod2Vla3MpXSwNCiAgICAgICAgICAgICAgICAgIFByb3BvcnRpb25HYW1lUGxheWVkID0gMSkNCmZvciAoaSBpbiAxOm5yb3coZnN0YXRzKSkgew0KICBpZCA8LSB3aGljaCgodG1wW1siVGVhbSJdXSA9PSBmc3RhdHNbaSwgIlRlYW0iXSkgJiAodG1wW1siUmFuayJdXSA9PSBmc3RhdHNbaSwgInJhbmsiXSkpDQogIHRtcFtpZCwgIlByb2JhYmlsaXR5Il0gPC0gMQ0KfQ0KcmsgPC0gcmJpbmQocmssIHRtcCkNCg0KcGwyIDwtIGxhcHBseSh0ZWFtcywNCiAgICAgICAgICAgICAgZnVuY3Rpb24oeCkgew0KICAgICAgICAgICAgICAgIGdncGxvdChkYXRhID0gc3Vic2V0KHJrLCBUZWFtID09IHgpLA0KICAgICAgICAgICAgICAgICAgICAgICBhZXMoeCA9IGZhY3RvcihQcm9wb3J0aW9uR2FtZVBsYXllZCksIHkgPSBSYW5rLCBmaWxsID0gUHJvYmFiaWxpdHkpKSArDQogICAgICAgICAgICAgICAgICBnZW9tX3RpbGUoKSArDQogICAgICAgICAgICAgICAgICBzY2FsZV9maWxsX3ZpcmlkaXNfYygpICsNCiAgICAgICAgICAgICAgICAgIHNjYWxlX3lfY29udGludW91cyhleHBhbmQgPSBjKDAsIDApLCBicmVha3MgPSAxOmxlbmd0aCh0ZWFtcykpICsNCiAgICAgICAgICAgICAgICAgIHNjYWxlX3hfZGlzY3JldGUoZXhwYW5kID0gYygwLCAwKSwgYnJlYWtzID0gYygwLCAwLjUsIDEpKSArDQogICAgICAgICAgICAgICAgICBsYWJzKHRpdGxlID0geCwgeCA9ICJQcm9wb3J0aW9uIG9mIGdhbWUgcGxheWVkKiIpICsgIyAqIG5vdCBleGFjdGx5IGJ1dCBjbG9zZQ0KICAgICAgICAgICAgICAgICAgdGhlbWVfY2xhc3NpYyhiYXNlX3NpemUgPSAxNSkNCiAgICAgICAgICAgICAgfSkNCnBsb3RfZ3JpZChnZXRfbGVnZW5kKHBsMltbMV1dICsgdGhlbWUobGVnZW5kLnBvc2l0aW9uID0gInRvcCIpKSwNCiAgICAgICAgICBwbG90X2dyaWQocGxvdGxpc3QgPSBsYXBwbHkocGwyLCBmdW5jdGlvbih4KSB7eCArIHRoZW1lKGxlZ2VuZC5wb3NpdGlvbiA9ICJub25lIil9KSwNCiAgICAgICAgICAgICAgICAgICAgbnJvdyA9IDUpLA0KICAgICAgICAgIG5yb3cgPSAyLCByZWxfaGVpZ2h0cyA9IGMoLjA1LCAuOTUpKQ0KYGBgDQoNCkZvciBleGFtcGxlLCB3ZSBjYW4gc2VlIGhvdyB0aGUgcHJlZGljdGlvbiBsb29rIGxpa2UgYXQgdGhlIG1pZGRsZSBvZiB0aGUgc2Vhc29uLg0KDQpgYGB7ciBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQ0KZGZfdHJhaW4gPC0gZGZbZGZbWyJXZWVrTnVtYmVyIl1dIDw9IG1lZGlhbih3ZWVrcyksIF0NCnRlc3RfZ2FtZSA8LSBkZltkZltbIldlZWtOdW1iZXIiXV0gPiBtZWRpYW4od2Vla3MpLCAiR2FtZSJdDQpkYXRhX2ZpdCA8LSBmb3JtYXRfc3Rhbl9kYXRhKGRmX3RyYWluKQ0KZml0IDwtIHNhbXBsaW5nKGNvbXBpbGVkX21vZGVsLA0KICAgICAgICAgICAgICAgIGRhdGEgPSBkYXRhX2ZpdCwNCiAgICAgICAgICAgICAgICBwYXJzID0gcGFyYW0sDQogICAgICAgICAgICAgICAgaXRlciA9IG5faXQsDQogICAgICAgICAgICAgICAgY2hhaW5zID0gbl9jaGFpbnMpDQpgYGANCg0KYGBge3IgbWVzc2FnZT1GQUxTRSwgd2FybmluZz1GQUxTRSwgZmlnLndpZHRoPTEwLCBmaWcuaGVpZ2h0PTEwfQ0KUFBDX2Zvb3RiYWxsX3N0YXRzKGZpdCwgIndpbl90ZXN0IiwgZnN0YXRzLCB0ZWFtcykNClBQQ19mb290YmFsbF9zdGF0cyhmaXQsICJsb3NlX3Rlc3QiLCBmc3RhdHMsIHRlYW1zKQ0KUFBDX2Zvb3RiYWxsX3N0YXRzKGZpdCwgInBvaW50X3Rlc3QiLCBmc3RhdHMsIHRlYW1zKQ0Kc3RhY2toaXN0X3JhbmsoY29tcHV0ZV9yYW5rKGZpdCwgInRlc3QiKSwgdGVhbXMpDQpgYGANCg0KVGhlIHByZWRpY3Rpb25zIGxvb2sgcmVhc29uYWJsZSB3aXRoIHJlc3BlY3QgdG8gdGhlIG91dGNvbWUgdGhhdCBpcyBvYnNlcnZlZCBhdCB0aGUgZW5kIG9mIHRoZSBzZWFzb24sIGhvd2V2ZXIsIHRoZXJlIGlzIHN0aWxsIGEgbG90IG9mIHVuY2VydGFpbnR5IGF0IHRoZSBtaWQtc2Vhc29uLg0KSW50ZXJlc3RpbmdseSwgdGhlIGxhc3QgcGxvdCBzaG93cyB0aGUgbW9kZWwgcHJlZGljdCB0aGF0IExpdmVycG9vbCB3aWxsIHdpbiB0aGUgY2hhbXBpb25zaGlwIHdpdGggdGhlIHByb2JhYmlsaXR5IG9mIGFwcHJveGltYXRlbHkgODAlIHdoZW4gaW4gdGhlIGVuZCwgaXQgaXMgTWFuY2hlc3RlciBDaXR5IHRoYXQgd2lsbCB3aW4hDQoNCiMgQ29uY2x1c2lvbg0KDQpUaGUgbW9kZWwgd2FzIHN1Y2Nlc3NmdWxseSBmaXQgdG8gdGhlIGRhdGEgYW5kIGNhbiBnaXZlcyB2YWx1YWJsZSBpbnNpZ2h0cyBpbnRvIHRoZSB0ZWFtcyBhYmlsaXRpZXMuDQoNCkhvd2V2ZXIsIGl0cyBwcmVkaWN0aXZlIHBlcmZvcm1hbmNlIGFwcGVhcnMgbGltaXRlZCBhbmQgd2Ugd291bGQgbmVlZCB0byBtb3ZlIGF3YXkgZnJvbSB0aGlzIHNpbXBsZSAiRGl4b24tQ29sZXMiIG1vZGVsIHNvIHdlIGNhbiBtYWtlIG1vcmUgYWNjdXJhdGUgYW5kIHByYWN0aWNhbGx5IHVzZWZ1bCBwcmVkaWN0aW9ucy4NCktlZXBpbmcgaW4gbWluZCBJIGFtIGZhciBmcm9tIGJlaW5nIGEgZG9tYWluIGV4cGVydCwgSSBjb3VsZCBzdWdnZXN0IHRvOg0KDQotIFByZWRpY3QgdGhlIG51bWJlciBvZiBnb2FscyBmb3IgZWFjaCBoYWxmLXRpbWUgcGVyaW9kcy4gVGhpcyB3b3VsZCBlZmZlY3RpdmVseSBkb3VibGUgdGhlIGRhdGEgd2UgYXJlIHVzaW5nIGFuZCBwcm92aWRlIG1vcmUgYWNjdXJhdGUgZXN0aW1hdGVzIG9mIHRoZSB0ZWFtcyBhYmlsaXRpZXMsIGFuZCBob3BlZnVsbHksIGJldHRlciBwcmVkaWN0aW9ucy4NCi0gVXNlIGEgemVyby1pbmZsYXRlZCBtb2RlbCB0byBkZXNjcmliZSB0aGUgZmFjdCB0aGF0ICJubyBnb2FscyBzY29yZWQiIGhhcHBlbnMgbW9yZSBvZnRlbiB0aGFuIGV4cGVjdGVkLg0KLSBBc3N1bWUgY29ycmVsYXRlZCBhdHRhY2sgYW5kIGRlZmVuY2UgYWJpbGl0aWVzLg0KLSBNb2RlbCB0aGUgZmFjdCB0aGF0IHRoZSBsYXRlbnQgYWJpbGl0aWVzIG1pZ2h0IGNoYW5nZSB3aXRoIHRpbWUuIEZvciBleGFtcGxlLCB3ZSBjb3VsZCB1c2UgYSBSYW5kb20gV2FsayBvciBhbiBFeHBvbmVudGlhbCBTbW9vdGhpbmcgbW9kZWwgdG8gZGVzY3JpYmUgdGhlIGV2b2x1dGlvbiBvZiBhYmlsaXRpZXMuDQotIFRlYW0gZGVwZW5kZW50IGhvbWUgYWR2YW50YWdlLiBTb21lIHRlYW1zIG1pZ2h0IGhhdmUgYSBiaWdnZXIgaG9tZSBhZHZhbnRhZ2UgKGJldHRlciBmYW5zPykgYnV0IGF0IHRoZSBzYW1lIHRpbWUsIG90aGVyIHRlYW1zIGNvdWxkIGxlc3Mgc2Vuc2l0aXZlIHRvIHRoaXMgZWZmZWN0Lg0KLSBJZiB0aGUgZGF0YSBpcyBhdmFpbGFibGUsIG1vZGVsIHRoZSB0ZWFtIGFiaWxpdGllcyBhcyBhIGNvbWJpbmF0aW9uIG9mIHRoZSBhYmlsaXRpZXMgb2YgZWFjaCBwbGF5ZXIuDQotIGV0Yy4NCg0KTm9uZXRoZWxlc3MsIHRoZSBCYXllc2lhbiBmcmFtZXdvcmsgY2FuIGJlIHVzZWZ1bCB0byBtYWtlIGNvbXBsZXggcHJlZGljdGlvbnMgYmV5b25kIHdoaWNoIHRlYW0gd2lsbCB3aW4gYSBzcGVjaWZpYyBnYW1lLCBidXQgYWxzbyBmaW5hbCByYW5raW5nIGF0IHRoZSBlbmQgb2YgdGhlIHNlYXNvbi4NCg==
-
-
-
-