Scouting in part and stars rating in general is broken

[mkhk]

I love FM, especially developing youth players into stars.  I think the developers have a done a great job at creating a game that is believable yet undeterministic (otherwise games turn very boring very fast).  I greatly appreciate their efforts to prevent FM turning into a predictable numbers game.  Having said that, the developers went too far when it comes to scouting, in particular the stars rating.

Now, I understand that the stars rating is not billed to be super reliable, that it depends on various factors like the quality of your scouting team, your league, etc etc....  Yet, its very existence has to warrant some level of reliability, otherwise it is a farce and should be removed altogether.  Why even display it so prominently on the team page?  After several frustrations, I decided to play a couple of seasons while running Genie Scout alongside to monitor how the stars ranking correlate with the actual underlying CA & PA ratings.  I started a game managing Liverpool, immediately overhauled my staff with top of the line scouts and coaches.  I took the attached image after two years into the game.  Look at the three highlighted players on my team, sorted in order of declining PA:

Sergej Milinkovic-Savic receives 5-stars for potential and his actual PA is 184.

Virgil van Dijk receives 4.5-stars for potential and his actual PA is 174, so that looks fine.

Gustavo Masiero receives 2-to-3-stars for potential, but his actual PA is 179!  So, instead of coming in at 4.5- or 5-stars, his PA is underrepresented by at least 1.5 stars, possibly by as much as 3 stars.  That is a huge margin of error.  Again, as mentioned above, I have top of the line scouts and coaches, players are in the same league / on the same team, etc...  There is no reasonable explanation for this huge margin except - and I am sorry to say this - that the developers desire to keep the game unpredictable has lead them to buff this part of the game excessively, thereby rendering the stars rating essentially useless.  I firmly believe the stars rating should not be 100% accurate because scouts/coaches are not, but at the very least it should be reliable enough to serve as a first scan / filter.  If a 4.5- / 5-star talent appears near the bottom of your team, it has failed miserably, there is no way about it.

You might argue that this is because one of the players involved is very young, but I have seen many cases where the stars rating has failed with established players as well.  Perhaps not the best example, but there is one example also in the same attached file.  Trent Alexander-Arnold receives 4-stars for potential and Daniel Sturridge receives 2.5-stars; their actual PAs are 170 and 168, respectively; so, they have essentially identical PAs, yet their stars rating differs by 1.5.  I have seen even worse margins, but this should be enough to make the point.

I am not claiming that scouting in general is useless, but it takes a big hit.  I appreciate that scouting still provides value in identifying players' actual attributes, but this does not help much with regards to a players' PA.  Real-life scouts do provide at least somewhat reliable insight into player's potential, but the game does not reflect this well at all.  This is shame because I believe scouting has the potential to be one of the most interesting parts of this game.

FM19 Liverpool July 2020.pdf

[CFuller]

Here we go again with the "broken" hyperbole. 

2 hours ago, mkhk said:

You might argue that this is because one of the players involved is very young, but I have seen many cases where the stars rating has failed with established players as well.  Perhaps not the best example, but there is one example also in the same attached file.  Trent Alexander-Arnold receives 4-stars for potential and Daniel Sturridge receives 2.5-stars; their actual PAs are ### and ###, respectively; so, they have essentially identical PAs, yet their stars rating differs by 1.5.  I have seen even worse margins, but this should be enough to make the point.

The star system works fine here, because it takes the players' ages into account (which you do not in that example).

Alexander-Arnold is 21 years old and of very high ability. Indeed, he might be very, very close to his PA already. That perhaps explains why your assistant rates him as having 4* CA and 4* PA.

By contrast, Sturridge is 30 years old and has already peaked. Although his star rating suggests is still a pretty decent squad player, he has not fulfilled - and is now too old to fulfil - the potential he was once seen to have (primarily due to injuries). That perhaps explains why your assistant only rates him as having 2.5* PA.

[FlorianAlbert9]

Imho. 

it's right in this way. 'cause this margin of error is the only thing to prevent game becoming too predictable and easy. 

(then, in your example, maybe scouts think that player, even with high PA, will not reach that level for various reason like personality)

if i said i hope one day SI will take off PA from game (maybe using only 2-level, one for normal, one for game-changer like Pelè, Maradona, R9 ) and scounting potential will become like IRL when scout judge a player on CA, personality, family... 

[HUNT3R]

14 minutes ago, mkhk said:

Gustavo Masiero receives 2-to-3-stars for potential, but his actual PA is 179!

Scouts often "guess" potential based on current ability for a player's age. That's the "issue" here. His low current ability would make scouts and coaches think he doesn't have very much potential. In a case like this, you could have that unpredictable Vardy example where players "suddenly" burst out of nowhere.

 

I'm not seeing anything "broken" so far. Seems to be just as real life where scouts get some right and some wrong.

[mkhk]

14 minutes ago, CFuller said:

Point well taken.  I will check some other saves to see if I can come up with a better example...

Here we go again with the "broken" hyperbole. 

The star system works fine here, because it takes the players' ages into account (which you do not in that example).

Alexander-Arnold is 21 years old and of very high ability. Indeed, he might be very, very close to his PA already. That perhaps explains why your assistant rates him as having 4* CA and 4* PA.

By contrast, Sturridge is 30 years old and has already peaked. Although his star rating suggests is still a pretty decent squad player, he has not fulfilled - and is now too old to fulfil - the potential he was once seen to have (primarily due to injuries). That perhaps explains why your assistant only rates him as having 2.5* PA.

 

[mkhk]

16 minutes ago, HUNT3R said:

Ok, understand what you are saying.  But lucky me, the very same image just happens to show a great example why ratings failed: Maseiro and Salinas are of similar age, were scouted similarly, brought aboard at about the same time, and have virtual identical PAs of 179 & 180, respectively.  Yet, one is shown to have 2/3-star potential and the other to have 4/5-star potential -- hard to find two more comparable players.  How can one explain this?  I get the argument that "scouts get some right some wrong", but there has to be some way one can do improve on this -- better scouts (which I have), longer scouting (which I did), ...

 

16 minutes ago, HUNT3R said:

Scouts often "guess" potential based on current ability for a player's age. That's the "issue" here. His low current ability would make scouts and coaches think he doesn't have very much potential. In a case like this, you could have that unpredictable Vardy example where players "suddenly" burst out of nowhere.

 

I'm not seeing anything "broken" so far. Seems to be just as real life where scouts get some right and some wrong.

 

[HUNT3R]

13 minutes ago, mkhk said:

Ok, understand what you are saying.  But lucky me, the very same image just happens to show a great example why ratings failed: Maseiro and Salinas are of similar age, were scouted similarly, brought aboard at about the same time, and have virtual identical PAs of 179 & 180, respectively.  Yet, one is shown to have 2/3-star potential and the other to have 4/5-star potential -- hard to find two more comparable players.  How can one explain this?  I get the argument that "scouts get some right some wrong", but there has to be some way one can do improve on this -- better scouts (which I have), longer scouting (which I did), ...

Your screenshot shows that Salinas is a year younger and already rated as having a better CA than Maseiro. That might well be why he's being thought of as having a better potential.

[yolixeya]

I view the stars rating of CA and PA, just as a scouts opinion. They can be wrong (even the world class scouts) , just like in real life, so I wouldn't say that anything is wrong with the current system. And i think they even change their opinion over time.

[XaW]

I'd just like to add that the scouts/coaches doesn't see the actual PA, they have a Perceived PA. PPA is based on a lot of things, PA included, but also age and a lot of other aspects. The better the staff member, then more "correct" the PPA is. However, it's not always correct. Even the best staff members can be wrong. And the more time they spend scouting/training the player in question, the more pinpointed the assertion is.

So if a young player have a PA of 150 in your Liverpool squad. The PPA might be 2 star, or 4 star depending on a lot of things. The scout ratings are not supposed to be taken as a fact, but as a guide to how much risk you are willing to take. Much like real life! How many players have been called "the next Messi/Ronaldo/Maradona/Pele/<name a great player>" only to never reach it? A lot! Maybe they had the potential and just didn't reach it, or perhaps they were over hyped and the talent exaggerated. Who knows! At least they didn't reach it.

Without going into a more philosophical discussion, in real life we don't have this number, so we cannot know for sure. In FM terms it's a "How good the player can be if everything goes as well as they possibly can". And users are not meant to see that number, so claiming it to be broken is just wrong. That seems like applying a cheat/trainer/hack to a game, and then claim it's too easy, unrealistic, or flawed. It might have issues, but you are doing stuff the game was not meant to handle.

In a final note, PA, or CA for that matter, aren't the only thing that decides how good a player is. Attribute distribution is much more vital. Yes, that is limited by CA/PA, but you can have a high CA player with horrible attribute distribution and he will perform much worse than a lower CA player with great attribute distribution.

[mkhk]

2 minutes ago, HUNT3R said:

Your screenshot shows that Salinas is a year younger and already rated as having a better CA than Maseiro. That might well be why he's being thought of as having a better potential.

Salinas, born 16 Feb 2003, CA/PA of 124/180

Masiero, born 22 Feb 2002, CA/PA 111/179

So, your point makes sense to a certain degree, Salinas should perhaps be rated somewhat higher... but does this really warrant a difference of 2 full stars??

[mkhk]

4 minutes ago, yolixeya said:

I view the stars rating of CA and PA, just as a scouts opinion. They can be wrong (even the world class scouts) , just like in real life, so I wouldn't say that anything is wrong with the current system. And i think they even change their opinion over time.

Yes, they do change, seen it many times.  And I am ok with small adjustments.  But there some unwarranted extreme examples.

[optimusprimal82]

This is almost certainly not how the game calculates it but I view all of the ratings with a pinch of salt, exactly how big that pinch is comes down to their judging stat & this the most logical way I can illustrate/explain my general rule of thumb that I apply to all of my scout/coach ratings - I'll reiterate that I'm not for one minute saying this is how the game calculates the ratings, just my own theory/how much stock I may place in a staff members opinion; I even had an instance recently where two staff members with the same judging stat rated a player differently (which you can read here if that interested!);

 

[mkhk]

8 minutes ago, XaW said:

I'd just like to add that the scouts/coaches doesn't see the actual PA, they have a Perceived PA. PPA is based on a lot of things, PA included, but also age and a lot of other aspects. The better the staff member, then more "correct" the PPA is. However, it's not always correct. Even the best staff members can be wrong. And the more time they spend scouting/training the player in question, the more pinpointed the assertion is.

So if a young player have a PA of 150 in your Liverpool squad. The PPA might be 2 star, or 4 star depending on a lot of things. The scout ratings are not supposed to be taken as a fact, but as a guide to how much risk you are willing to take. Much like real life! How many players have been called "the next Messi/Ronaldo/Maradona/Pele/<name a great player>" only to never reach it? A lot! Maybe they had the potential and just didn't reach it, or perhaps they were over hyped and the talent exaggerated. Who knows! At least they didn't reach it.

Without going into a more philosophical discussion, in real life we don't have this number, so we cannot know for sure. In FM terms it's a "How good the player can be if everything goes as well as they possibly can". And users are not meant to see that number, so claiming it to be broken is just wrong. That seems like applying a cheat/trainer/hack to a game, and then claim it's too easy, unrealistic, or flawed. It might have issues, but you are doing stuff the game was not meant to handle.

In a final note, PA, or CA for that matter, aren't the only thing that decides how good a player is. Attribute distribution is much more vital. Yes, that is limited by CA/PA, but you can have a high CA player with horrible attribute distribution and he will perform much worse than a lower CA player with great attribute distribution.

Great post, agree with everything you wrote.  Downloaded Genie Scout just to check on the correlation, would not use it in a regular game.  I guess my argument/frustration is because of this: I totally understand that the stars rating is not meant to be fully reliable; I only use it as a first scan to decide where to dig deeper and where to let go.  In this case, I may have let go on Masiero and lost a future star.  Stars should serve some purpose, and I am ok with changing things, like increasing the black range, for example, ...

Finally, great point about he attribute distribution.  I have players rated 20-30 point lower but a much better fit for the position I need than some other, so, yeah CA/PA is not everything.

[santy001]

I think the big thing many people need to realise is that 2 stars and above is absolutely fine for an option for your team at the level you are at, especially if coupled with good attributes.

If you're wanting a cheap backup then maybe even a stretch as far as a 1.5 star. People are getting too hung up on the minute differences at the top end of the star spectrum and not even realising most of their best players may not even grace that end of the star chart.

I don't mean to use this as a humble brag, because its not, I get a massive advantage on attributes thanks to the massive dossier SI send me every year for research.

There's a thread detailing my save with Stoke over on the FM Career Updates forum if you've got any interest in how I came to put this squad together. Since I'm largely done with FM19 at the moment due to work, I've just bought an editor, mainly because this topic keeps coming up and I need to see figures myself for some better understanding I think and since I won't be playing this save again I can check the players information.Here's the CA's & PA's of my Champions League winning squad:

Spoiler

 

Then look at the squad ratings:

Some massive standout issues, Adam Idah, Mitchel McClellan, Daramy, Eladio, Rogic, their star ratings are all over done but I can understand why. They've played well at the highest level in football and are young, it seems like they should have more room in the tank, plenty in some cases. This is an essential part of what stars do in their inclusion, they are replicating that taking a look at the "thus far" element and forming a prediction that may or may not be accurate for the player.

Iker Pozo played perfectly fine in a champions league final at 96 CA. The star system isn't perfect but it really is only a loose guide if you don't trust yourself to judge a player. 

Relatively speaking, for the competence within a team the game between 2 and 5 stars is of far less impact than the right spread of attributes than the gap between 0.5 and up to 2 stars. If you absolutely find someone bang on at 1.5 even they can perform well as a backup, but its only below this point where you really ought to be thinking "this player likely won't ever be of use to me".

To be considering ditching players because of them having a 3.5 or a 4, or a 4 instead of a 5 is pure folly. It's rough estimations about players with a degree of uncertainty, but one thing you can be certain of is that there is some potential and that if they get the opportunity you should see how they actually perform before deciding whether to eject them or not.

[XaW]

34 minutes ago, santy001 said:

I think the big thing many people need to realise is that 2 stars and above is absolutely fine for an option for your team at the level you are at, especially if coupled with good attributes.

Exactly! After all 3 stats equal average to your current team.

I even had great success in my Kettering save. My all time top goalscorer were a striker with 5 finishing. Scored truckloads of goals from National League North until the Premier League. By the time I sold him due to having vastly better players he still provided the goals as a sub. The last seasons he were rated 1 star both ability and potential and all the staff I had wanted me to get rid of him for years. I have no idea what his actual CA/PA were, but I suspect he was performing WAY above his supposed level.

I also got rid of players with high ability/potential simply because they didn't perform...

[herne79]

There's an awful lot of spoilers in here guys, we should probably be editing posts to hide hidden values .

We should probably also pin a topic saying what the stars actually mean because there is so much misunderstanding.

1) There is no direct correlation between the hidden CA/PA values and the star ratings.  CA/PA certainly form part of the stars, but it is much more than just that.

2) The star rating for Potential is (technically) something called "Perceived Potential Ability" (PPA for short).  As above, the hidden PA value is a factor in there but a big part of it is a scout or coach looking at a player's actual ability and their age, comparing them to other similar players in your squad and rating them accordingly.  So for example, "wow he looks good for his age, must have great potential"; or "he looks rubbish, he'll go nowhere".

3) Because of that, a young player with a very high hidden PA value but a very low hidden CA value could get a relatively low star rating for potential.  It won't be super low because the hidden PA value is still a factor, but it won't be as high as you may expect.

4) Similarly a young player with a high actual ability for his age but a relatively low hidden PA value could get quite a high number of stars for potential even though he might already be near his peak.

5) This comparison to other team members should not be overlooked.  You could have a 5 star player at Stalybridge United, but drop him into Juventus and he'll be lucky to have 1 star.

6) Star ratings can and do change over time, and not just because your scout or coach gets to know the player better.  As players age and develop, coaches constantly reassess their opinions.  But much more than this, as we develop our squads - bringing in new and better players or existing players improve - so the comparison to team mates will alter.  So a player may have 4 stars to begin with but 3 years down the line as you've improved your squad, that same player may now only have a 3 star rating.  Not because he's become a worse player but because he's being compared against better team mates.

Personally I think it's a really good system because it not only allows for coach or scout "mistakes" but because it does actually address comparisons to other players and factors in an assessment of actual ability to try to "predict" a player's potential.

[sporadicsmiles]

13 minutes ago, herne79 said:

5) This comparison to other team members should not be overlooked.  You could have a 5 star player at Stalybridge United, but drop him into Juventus and he'll be lucky to have 1 star.

To further that, with a partial question. Does it take the entire team into account, or is their a weighting towards players who play in the same position? If the latter, then another reason in the example given by the OP would be that the the RB is being compared to a very good RB (TAA), and thus he will be rated down because he probably is not as good a current or future player.

I think people are massively hung up by star ratings in general. They are not gospel. They are a nice tool for sorting which players to look at based on what the game tells you about them. They definitely should not be something you base your purchasing or team selection on.

For buying players, you can sort by star rating. Then you should be looking at the attributes that player has to see if they have good attributes for the position you want to use them in. You should look at age and injury record. What foot do they use (I rarely sign left footed wingers if my team is predominantly right footed on the wings, because they do not fit). These are the defining things as to whether a player is a good signing or not. You do not even need to know or look at CA or PA for this. It muddies the waters. Besides, a high PA does not mean a player will reach it. If you had a 23 year old with 120 CA and 200 PA (made up example), I would not expect him to reach 200 CA. Different story if it is an 18 year old with the same CA/PA. The point with youth players is you are never sure what you are going to get. You are taking a punt on them most of the time. They may come good, or they may never develop well. It is part of the fun to nurture a future superstar.

For selecting teams, I do not even bother with star ratings. You should know what attributes your players need to play in the position you have selected them. Therefore you should know if they can do it. One of the best players I ever signed (in FM17 I believe) was a left winger who never had a star rating of higher than 2.5 stars. However, his attributes, PPMs, etc. were a wonderful fit for how we played. He ended up having one 30 assist and 30 goal season (I have never done that before or since). He was just the perfect player for my side. If I based signings only on star rating, I would never have found or played him. You should focus on what the player does and how they play way before you consider star ratings.

Indeed, for squad management the only time I really use star ratings is for deciding which young players to keep or let go (I plan senior transfers over 24 month periods, so I always know who is going/coming, and I base it on age, contract length, attributes, wage demands). It is still not absolute. I will look at the players and make my choice too. If I think a player can succeed, he can stay and we will see how he develops.

The conclusion is that star ratings should not be taken so seriously. They are useful, but not all knowing (they should not be). If you want to know everything, you clearly have the ability to find the CA/PA, so just use that instead. Asking for perfect star ratings is tantamount to doing this anyway.

[herne79]

19 minutes ago, sporadicsmiles said:

Does it take the entire team into account, or is their a weighting towards players who play in the same position?

 

44 minutes ago, herne79 said:

comparing them to other similar players in your squad

It won't be comparing strikers with central defenders for example, or keepers with anyone else.  Exactly where the cut off is only the SI devs will know, but comparing players who play in similar positions is a solid starting point .

[Filke]

Lol i dont even look at those stars and I dont listen when they tell me who is going to be star or not.

[ajt]

thanks for the thread, great stuff!

[santy001]

9 hours ago, herne79 said:

It won't be comparing strikers with central defenders for example, or keepers with anyone else.  Exactly where the cut off is only the SI devs will know, but comparing players who play in similar positions is a solid starting point .

Would have to get it verified by someone from SI, but I believe another element of the star rating system is what is expected for a club of a certain size/reputation or level in accordance with the more universal element which comes across in scout reports as a "decent X player", or "top X player"  

I might be wrong on that front however, but I seem to recall it covered a number of aspects to provide better context.