PA: Is it fit for purpose? A One-Post Experiment

[snootyjim]

Spoiler Warning

I will not be naming players, and I certainly will not be discussing individual players' PA numbers, but it is possible that people may refer to them in the thread. I would encourage you not to, since it isn't necessary or directly relevant, but it may well crop up. I will however be discussing the PA/CA system, so if you would prefer not to know how players' abilities are decided, I would highly recommend that you stop reading here.

The Experiment

If you need a rough guide to how the PA/CA system works, there's some info here: http://footballmanagerwiki.co.uk/ind...ential_Ability - for the purposes of this experiment, I'll assume you know how the PA system works, and in particular the minus system, rating players from -1 to -10.

I've had my doubts about the PA system for a while. It's clear that a researcher can't know exactly how good a player will be when he's only 16, so SI introduced the minus system to give more flexibility. I still have my doubts though. Can a researcher, no matter how good they are, really assess a player's capabilities to within a 30 point range? That works out as an accuracy of 15%.

To assess the problem, I decided to take a best-case scenario for Football Manager. I've looked solely at the old "Big Four" in the Premiership, namely Arsenal, Chelsea, Liverpool and Manchester United. They should really be the best researched clubs in the FM world. I looked at each of their squads in the FM2009 database, and selected the players with assigned minus PA values. I then tracked each player's progress through FM2010, FM2011, and FM2012, using the most recent patch for each. If any player vanished from the system at any point, they were excluded.

The whole concept of CA and PA is that the PA remains fixed, whilst the CA moves. Therefore, players' PA values from FM2009 through to FM2012 should remain constant. If minus values are used at any point, then there should always be crossover: i.e. a player rated at -7 and also at -8 could end up with a PA of 135, which falls within both -7 and -8. That, therefore, would be fine. If however, they ended up with a PA of 115, the -8 value would be wrong, suggesting a minimum of 130. That would represent a variance of 15, suggesting a problem with the PA system.

The question is, am I right? If the PA values do not show variance from FM2009 to FM2012, then it is working as advertised. If the values are changing from game to game, then we have a problem - players are effectively being stifled or over-rated by system and Football Manager as a game can never properly model their potential.

The Results

The information below shows simply the variance in PAs, and the frequency of each. By variance, I mean the largest gap between any two PA values from FM2009 through to FM2012. As mentioned before, in the case of minus values, I assume best-case scenario and take the smallest or largest number in the range to provide the lowest variance.



So, what does this mean? Well, my total sample size represented 160 players, as shown at the bottom. That's not huge, I accept, but I do think this represents best-case scenario for Football Manager. These should be the most closely analysed clubs, and the ones most capable of having accurately modelled youngsters. Of course, that means that the problem will probably be much worse elsewhere in the world.

We can also see that 98 of the 160 players, or 61%, were accurately represented. Their PA might have moved from year to year, but no value ever indicated that one of the others was impossible.

In 39% of players however, there was a problem - their PA varied between some of the years beyond the distances permitted by the minus system. Now, how bad this problem is, I'm not going to dictate - that's your call. Some people might consider this reasonable error, some might consider it unreasonable.

What we can see is that 7 players, or around 4% of the total, saw a change of 30 or more PA points. That is no small margin - it represents the difference between a world superstar at around 185-190 PA points to a very average squad player in a top team at around 155-160 PA points. What this effectively means is that a researcher could consider a player to be decidedly average, with a maximum possible PA of 160, and he would in reality proceed to become a top international superstar - the next Rooney, Ronaldo or Messi. However many times you play that game, that player will never become the player they ought to become in real life - their potential has been capped.

Conclusions

I have a few thoughts on the matter, but I'm interested to see what people think about it. Am I barking up the wrong tree, or is there something to this? And if there is something to it, then what should the solution be?

[Elnaith]

I think these stats are actually very good. They, after all, deal with the same problem that clubs do when scouting, and clubs would kill people to reach the 61% hit mark. What I am interested in though is whether the youths in this sample were more up- or downscaled. My intuition tells me up, due to the clubs in question automatically making one think it should be a good youngster.

[Biscotti]

Excellent and well presented post There are plenty of people who go on and on about how the system is "broken", but this is the first time I remember seeing someone do any experimentation on the subject.

With regards to your findings, I think that the variance isn't too bad, but probably does lie just outside the bounds of acceptable error. I would think maybe a <3% variance would be acceptable to most, considering humans can make mistakes, but there is also always going to be people who think the researchers should be robots and no mistakes should ever happen.

The problem with fixing things is that SI have almost certainly proposed any number of alternative systems among themselves and have yet to find one that provides a more accurate margin and therefore a more effective system. I don't doubt that if they'd thought of something better it would have been implemented.

Oh, and anyone that says "we need non-static PA's" needs to be shot >.>

[glennuk]

I disagree with the idea that this variance is bad.

I mean if a player is out for a year with an Injury, then surely his PA should change in the following edition of FM.
Or if a player is in the first team getting game time, and then has a bad run of form, and gets put in the reserves where he sulks and doesn't get up to much then again his PA is not going to be as high, as he spent time sulking instead of working on his game.

I mean I would be more annoyed if the results were 100% accurate and that each year the PA of the players stayed the same, as in anything there are always circumstances which change.

The experiment while interesting, to me shows that the researches are doing exactly what they should.

[Biscotti]

I disagree with the idea that this variance is bad.

I mean if a player is out for a year with an Injury, then surely his PA should change in the following edition of FM.
Or if a player is in the first team getting game time, and then has a bad run of form, and gets put in the reserves where he sulks and doesn't get up to much then again his PA is not going to be as high, as he spent time sulking instead of working on his game.

I mean I would be more annoyed if the results were 100% accurate and that each year the PA of the players stayed the same, as in anything there are always circumstances which change.

The experiment while interesting, to me shows that the researches are doing exactly what they should.

That would completely go against what PA represents, though. PA represents the absolute best a player can ever become given perfect circumstances, which isn't something that would change just because they'd not played much football. A drop in CA would certainly be reasonable in that situation, but there'd be no good reason for their PA to be altered drastically.

[glennuk]

I understand that, but if for example from FM 11 to FM 12 Player A was injured with a broken leg, then in FM11 he would have a PA that says if everything goes perfect he could reach a PA of 180, however in FM 12 we realise everything DID NOT go perfect, and therefore his year with a broken leg should be taken into account, and that PA changed to 170 (That is just a random example, not saying a broken leg should take off ten points.)

PA is if everything goes perfect, and if everything does not go perfect from one version to the next then that PA should be changed, right?

[snootyjim]

I understand that, but if for example from FM 11 to FM 12 Player A was injured with a broken leg, then in FM11 he would have a PA that says if everything goes perfect he could reach a PA of 180, however in FM 12 we realise everything DID NOT go perfect, and therefore his year with a broken leg should be taken into account, and that PA changed to 170 (That is just a random example, not saying a broken leg should take off ten points.)

PA is if everything goes perfect, and if everything does not go perfect from one version to the next then that PA should be changed, right?

You are correct, in that serious injury should be the only thing to affect PA. That's certainly how it works with regens, as I understand it.

The question is, did 39% of players suffer horrendous injuries? Also, much of the variance was up as well as down - it was split, roughly 50/50.

The issue I have is that if a researcher gave Lionel Messi a potential of -6 when he was 16, no matter how many times you play the game he will never be a top player. I find that a bit disappointing, and think that it devalues the youth teams at big clubs.

[Biscotti]

You are correct, in that serious injury should be the only thing to affect PA. That's certainly how it works with regens, as I understand it.

The question is, did 39% of players suffer horrendous injuries? Also, much of the variance was up as well as down - it was split, roughly 50/50.

The issue I have is that if a researcher gave Lionel Messi a potential of -6 when he was 16, no matter how many times you play the game he will never be a top player. I find that a bit disappointing, and think that it devalues the youth teams at big clubs.

All PA's in the game are static - a regen has a PA when he's generated and that never changes.

Aside from debating what should and shouldn't affect PA, which is a fruitless argument, I definitely agree that the minus system doesn't allow for researchers to undervalue players that will later turn into huge stars and still have it play out in the game the same way as real life. Trouble is, I really can't think of any other way of doing things without removing PA entirely.

I realise the removal of PA might well be a way of solving the problem, but unless there is some limiting factor on every single player then you will end up with a situation where every player that comes through at a top club will be world class. That would just lead to a situation of the best getting better and everyone else going nowhere.

[fyrros]

I realise the removal of PA might well be a way of solving the problem, but unless there is some limiting factor on every single player then you will end up with a situation where every player that comes through at a top club will be world class. That would just lead to a situation of the best getting better and everyone else going nowhere.

One might argue that it's not so different currently. The big clubs always transfer the high-PA players anyway leaving smaller clubs without any young prospects.

[Biscotti]

One might argue that it's not so different currently. The big clubs always transfer the high-PA players anyway leaving smaller clubs without any young prospects.

True, but that then allows the smaller clubs to make a profit on those players, which can be put towards improving the club. Whereas in a limitless PA system, no club would ever have a need to buy another player unless they wanted to buy someone who was already fully developed.

[fyrros]

well this is the CSE forum; not the suggestions so it doesn't matter anyway XD


But I guess that the point proven is that the fm researchers are not consistent enough ^^