Minas Morgul is better than it looks: how to interpret winrates

 

Isn't it great to have data for MESBG?

Over last few months, the community has been getting a gradual dripfeed of game data from the excellent Andrew at Tabletop Admiral. This data involves winrates, number of games played, average percentages at which armies win or lose, and some breakdown by points values.

Having access to this data is a genuinely great thing for the game, and not just because it gives me lots of easy ideas for content (see: this article). Having access to real data on a decent scale allows for broader statistical analysis than has been possible before.

However, there are real risks in drawing the wrong conclusions from this data (as Andrew notes himself). In order to use it, we need to take it with many grains of salt, and remember the factors that influence and skew these kinds of statistics. By taking them into account, we can start to draw sounder conclusions from the available noise.

Sample sizes

The Tabletop Admiral data when I was writing this had 1500 games entered (i.e. 3000 datapoints, two for each game). That seems like a big number, but it’s one that can still be easily skewed. One excellent player going on a tear at a tournament can drag a faction’s winrate up quite a long way when the numbers are small, and same thing for someone’s tragic 0:5 result.

As an example, the 1000-game data dump had a breakdown of list performance by points value. Out of curiosity, I looked up the Eagles to see how they had done above 800 points, only to discover that literally all 8 games logged were played by me. At that point I'm not just skewing the data, I am the data! That makes it less of an indicator of the faction's strength at this points level, and more an indication of how well I'm doing personally.

As it turns out, not many people are logging games above 800

More generally, the smaller the number of games for a particular faction/points value, the less reliable, and the more subject it is to individual players skewing it. One excellent Three Trolls player could take them to a couple of tournaments and entirely skew their winrate at the moment, and the same goes for a new player diligently logging all of their Minas Morgul games as they get pummelled again and again.

As such, it’s best to only pay much attention to these numbers where the sample sizes are on the larger end. 100 games with Minas Morgul is enough that we can start to comment on the trends (subject to the other factors I’ll raise below). Much below 100 games, however, and these numbers are strictly for fun.

Skewing factors

Even once there is sufficient data to determine a winrate for a faction, it's important to understand and take into account the factors that skew these winrates away from a ‘fair’ representation of the strength of an army. These factors may make a list seem artificially stronger or weaker than it really is, and we should remain aware of them when looking at data like this.

In general, these factors are ones that ‘skew’ a faction’s player representation away from the standard. What I mean by ‘standard’ here is that an ordinary, middle-of-the-pack faction will have a bunch of players logging data, of which some will be new players who win maybe 1/3 of their games, some will be average players who win about half, and some will be veterans who win about 2/3 of their games. If you increase the ratios of one of these groups then the faction winrate will rise and fall accordingly. So if all the veterans jump ship to a marginally better faction, then the winrate will drop substantially. And similarly…

The new player effect

Where a list is likely to attract a lot of new players, its winrate will be deceptively lower than it ought. This attraction can be because it's the flavour of the month, because it's an easy army for new players to access (perhaps because you can make it with a Battlehost or starter set easily), or because it's particularly 'iconic' (e.g. Riders of Theoden or Depths of Moria).

An influx of new players to a faction drags down the winrate, both because those players will be new to the faction and not know its tricks, and also because those players will be less experienced players who are more likely to lose games anyway.

This is likely impacting Muster of Isengard and Minas Morgul a lot currently: new players are bringing sub-optimal versions of these lists to match their Mordor or Isengard battlehosts and getting thumped. Both of those lists really are genuinely strong from my playtesting of them, but may be having their winrates dragged down by the influx of new players.

Minas Morgul is probably getting a lot of players from the old Osgiliath and Pelenor Fields startersets, and it shows

The flipside of all this is also that armies that are unusual will tend to have an artificially higher winrate because players running them will be more likely to be faction experts. Last edition there were not many players with Arnor or Far Harad because of the difficulties in collecting them, so they tended to be experienced ones who knew all the factions' tricks. No one is going to the effort of sourcing a full Mahud list and not getting in a heap of games with them, after all!  These players also tended to benefit from many of their games being the first time that their opponents had played into the faction, and thus not knowing how best to counter them.

Skill floor effect

Another key aspect is whether a list is dependent on high player skill to succeed. Something like Angmar or (at high points) Eagles will be powerful in the right hands but fall apart for most players. As such, these lists will tend to have lower winrates, even if they can still win tournaments in the hands of a good player. And on the flipside, something like last edition's Dragon Emperor was excellent in anyone's hands, so would not be substantially dragged down by newer players.

Even the newest of players could still do okay with these guys back in the day

Snowball effect

Winrates and event wins are also often skewed by whether better players are running them. This can mean that a strong list will be further boosted by having lots of good players running it, while a slightly-worse variant will be left in the doldrums.

The classic example of this effect from last edition involved the famous Witch King/Suladân builds.  These lists weren’t really that much better than pure Mordor, because while Suladân was clearly undercosted he wasn’t doing anything unique or gamebreaking. As such, you might have expected pure Mordor builds that replaced him with Shagrat or the Shadow Lord to have done nearly as well at tournaments and had similar winrates.

However, because Mordor/Serpent Horde was clearly better than pure Mordor (not by much, but definitively), better players would never tend to run pure Mordor. As a result, Mordor/Serpent Horde builds won many more events and presumably had a much higher winrate than pure Mordor, despite only being slightly stronger. This cuts both ways as well: pure Mordor's winrate and event wins were likely lower than they ought to have been, because very few strong players tended to run them, so it was mostly left to newer players. 

This list was never that  much better than pure Mordor, but it was a big enough difference to convince competitive players

Conclusion: Log your games!

All of these factors skew winrate data, making it less reflective of the real power of the lists. This can result in a legitimately powerful list ending up with a weirdly low winrate, or an average army being artificially pumped up more than it should be.

As I said at the start, none of this is at all to knock the excellent data being collected on TTA. If anything, I'd encourage you all to log your games diligently to make the most of it. This is a tool that will become progressively more useful the more games people log, and we should all aim to do our part.

But at the same time, be cautious when interpreting this data, and bear in mind the ways in which it can end up pointing you in the wrong directions. I’m sure that there are probably plenty more skewing factors that I haven’t identified here, so please let me know in the comments if you’ve got any great ones to flag.

In any case, keep an eye out for a followup article in which I gleefully ignore all of my own good advice and rhapsodise about the Top 5 Armies in the Game based on Tabletop Admiral data.

And until then, may your factions always have lots of juicy datapoints to analyse!