[FM22] [Experiment] Influence of selected hidden attributes and Determination on team performance in domestic league

[Orion_]

  

1. Introduction

In this experiment we will check impact of selected hidden attributes and Determination on team performance in a domestic league measured by team's points earned over the course of a season. This experiment is a continuation and improvement of my previous research of that case made in FM21 that can be viewed here.

2. Game setup and testing environment

In this experiment I've decided to load only 1 playable league that was English Premier League. The team of my choice was Leeds United. The idea to take Leeds came from one of the comments about my previous experiment where I've chosen Liverpool FC, that their good and best performance was indistinguishable simply because the margin of error at the high end of the points gained was very little and the team just couldn't perform better. I've checked the multiple simulations that Knap made to check his tactics and it turned out that Leeds is a team that in most cases ends up somewhere mid table. This results in a room for potential positive and negative impact of attributes change to be still visible in results. Other things that were set for the experiment:

• all first team players attributes that were tested (Ambition, Pressure, Professionalism, Temperament, Consistency, Important Matches and Determination) were set using in game editor and then their attributes were locked preventing their changes throughout experiment
• attributes were set to default, all 10's (kind of control group - all the tested attributes, not all the player's attributes), selected attribute set as 1 (low), selected attribute set as 20 (high)
• option preventing player manager to be sacked was enabled
• individual and team training was designated to assistant manager
• match squad selection was left to assistant manager
• all contract and transfer features were designated to player manager so there was no players coming in or out during the test
• first transfer window was disabled to reduce personal shuffle in other teams in the league
• testing time was 1 season (earliest possible starting date in England ; ending date just after the last fixture of the season)
• only 1 playable league was enabled (English Premier League) and all competition that Leeds United was not participate match details was set to 'None' to reduce processing time
• tactic that was used in all simulations was @knap's KASHMIR 451 P104 EC FA (THP93)
• game version was 22.4.1
• database version was default 22.4.0 database update

As our outcome value we will use amount of points that team gathered in domestic league during the whole season. Final result is shown as average number of points for that data set and as additional feature there will be indicator to show the spread of points gained within certain data set.

Every attribute set was simulated 7 times. Attributes sets that were tested:

• Default [Ambition - 12,88 ; Pressure - 12,68 ; Professionalism - 14,08 ; Temperament - 12,36 ; Consistency - 11,48 ; Important Matches - 12 ; Determination 14,16] - default players attribute values (numbers in brackets are first team average)
• All 10 - control group
• Ambition 1
• Ambition 20
• Pressure 1
• Pressure 20
• Professionalism 1
• Professionalism 20
• Temperament 1
• Temperament 20
• Consistency 1
• Consistency 20
• Important Matches 1
• Important Matches 20
• Determination 1
• Determination 20

3. Results

3.1 Overall results in a table

3.2 Average domestic league points (result spread is the lowest and the highest number of points in single simulation for certain attribute value)

3.3 Results difference compared to All 10 (control group)

4. Analysis

The first thing that is clearly visible is almost 50% drop in performance for very low Pressure and Professionalism values. On the other hand positive difference for high end values for those attributes is not that significant (5,6% and 10,18%). In general we can observe that every single attribute produced lower result when set to 1 when compared to control group. This shows that each tested attribute influences player's performance on the pitch. For the high end values the highest team improvement occurred for Determination, Pressure and Consistency (12,27%, 10,18% and 9,16% respectively). High Ambition and Important Matches did not produce significantly better results than control group but when those attributes were set to low the performance drop was visible (-9,41% and -6,36%). The most interesting result was Temperament that when set to high produced significantly lower result than low value (-8,91% compared to -15,52%).

5. Conclusion and free thoughts

Experiment showed that players should have at least medium range values for Pressure and Professionalism otherwise their negative impact on player's performance is significant. The positive impact of all of the tested attributes was much lower for high values than the negative impact for low values (except Temperament). High Temperament value resulting in a worse outcome than low temperament value might indicate that very high value for this attribute is not desired. However this observation requires further investigation because it's requirement for some player's personalities considered as desired and positive like Model Citizen/Professional or Spirited might indicate that this attribute should be considered as positive.

It is unclear whether 10 is an absolute neutral value or it's relative on the league average which was not calculated in this experiment. This also applies to checking if performance drop due to low Pressure/Professionalism would still be this high in a league with different average value for those attributes.

There was no clear tendency for final number of points spread within certain attributes data sets. Very interesting factor was high spread of result between certain seasons within the same data set - for example for 20 Pressure there was final result of 33 points and 76 points in a two consecutive simulations. There were no investigation for any specific reason behind that big gap between results (like for example injury of key player) but it's worth noticing that number of simulations for future tests should be as high as possible due to significant spread of end results within the same datasets.

Some players despite having 1 or 20 in Important Matches and Consistency did not show this as Pros/Cons indicator in scouting/coaching report even after full season. It behaved the same way every time for the same players in both cases, so for high/low value and both for consistency/important matches. Reason is unknown but is worth noticing in terms of scouting for players with certain values of those hidden attributes that cannot be obtained in any other way without the editor or 3rd party tools.

 

I hope this experiment clarified some questions and also revealed some unexpected aspects of attributes. Any questions or comments are welcomed.

Spreadsheet for those interested in detailed numbers

[werogatda]

Good stuff mate, as a fellow lover of spreadsheets I know how much work goes into experiments like these. Thumbs up from me

[Orion_]

Thank you. Single simulation took around 20 minutes so it was good opportunity to read some books while game was running in the background.

[knap]

@Orion_

We carry out this type of Research on our Discord FMHUB.

If interested in joining our discussion I can send a discord invite

Type of research

 

[chewbaccaloveaddiction]

Great experiment, I would have expected consistency to be more important. 

[Orion_]

4 hours ago, chewbaccaloveaddiction said:

Great experiment, I would have expected consistency to be more important. 

Yep. On one hand 10 points difference is something that makes a change, on the other hand group with 10 was almost in the middle between 1 and 20 results. Funny thing is that consistency 20 had statistically almost double the results spread than consistency 1.

[Thebaker]

intesresting analysis. Surprised that poor professionalism had more impact than poor determination. Pressure is the hidden attribute i never consider but clearly should do.

[Jimbokav1971]

On 18/05/2022 at 21:00, Orion_ said:

  

1. Introduction

In this experiment we will check impact of selected hidden attributes and Determination on team performance in a domestic league measured by team's points earned over the course of a season. This experiment is a continuation and improvement of my previous research of that case made in FM21 that can be viewed here.

2. Game setup and testing environment

In this experiment I've decided to load only 1 playable league that was English Premier League. The team of my choice was Leeds United. The idea to take Leeds came from one of the comments about my previous experiment where I've chosen Liverpool FC, that their good and best performance was indistinguishable simply because the margin of error at the high end of the points gained was very little and the team just couldn't perform better. I've checked the multiple simulations that Knap made to check his tactics and it turned out that Leeds is a team that in most cases ends up somewhere mid table. This results in a room for potential positive and negative impact of attributes change to be still visible in results. Other things that were set for the experiment:

• all first team players attributes that were tested (Ambition, Pressure, Professionalism, Temperament, Consistency, Important Matches and Determination) were set using in game editor and then their attributes were locked preventing their changes throughout experiment
• attributes were set to default, all 10's (kind of control group - all the tested attributes, not all the player's attributes), selected attribute set as 1 (low), selected attribute set as 20 (high)
• option preventing player manager to be sacked was enabled
• individual and team training was designated to assistant manager
• match squad selection was left to assistant manager
• all contract and transfer features were designated to player manager so there was no players coming in or out during the test
• first transfer window was disabled to reduce personal shuffle in other teams in the league
• testing time was 1 season (earliest possible starting date in England ; ending date just after the last fixture of the season)
• only 1 playable league was enabled (English Premier League) and all competition that Leeds United was not participate match details was set to 'None' to reduce processing time
• tactic that was used in all simulations was @knap's KASHMIR 451 P104 EC FA (THP93)
• game version was 22.4.1
• database version was default 22.4.0 database update

As our outcome value we will use amount of points that team gathered in domestic league during the whole season. Final result is shown as average number of points for that data set and as additional feature there will be indicator to show the spread of points gained within certain data set.

Every attribute set was simulated 7 times. Attributes sets that were tested:

• Default [Ambition - 12,88 ; Pressure - 12,68 ; Professionalism - 14,08 ; Temperament - 12,36 ; Consistency - 11,48 ; Important Matches - 12 ; Determination 14,16] - default players attribute values (numbers in brackets are first team average)
• All 10 - control group
• Ambition 1
• Ambition 20
• Pressure 1
• Pressure 20
• Professionalism 1
• Professionalism 20
• Temperament 1
• Temperament 20
• Consistency 1
• Consistency 20
• Important Matches 1
• Important Matches 20
• Determination 1
• Determination 20

3. Results

3.1 Overall results in a table

3.2 Average domestic league points (result spread is the lowest and the highest number of points in single simulation for certain attribute value)

3.3 Results difference compared to All 10 (control group)

4. Analysis

The first thing that is clearly visible is almost 50% drop in performance for very low Pressure and Professionalism values. On the other hand positive difference for high end values for those attributes is not that significant (5,6% and 10,18%). In general we can observe that every single attribute produced lower result when set to 1 when compared to control group. This shows that each tested attribute influences player's performance on the pitch. For the high end values the highest team improvement occurred for Determination, Pressure and Consistency (12,27%, 10,18% and 9,16% respectively). High Ambition and Important Matches did not produce significantly better results than control group but when those attributes were set to low the performance drop was visible (-9,41% and -6,36%). The most interesting result was Temperament that when set to high produced significantly lower result than low value (-8,91% compared to -15,52%).

5. Conclusion and free thoughts

Experiment showed that players should have at least medium range values for Pressure and Professionalism otherwise their negative impact on player's performance is significant. The positive impact of all of the tested attributes was much lower for high values than the negative impact for low values (except Temperament). High Temperament value resulting in a worse outcome than low temperament value might indicate that very high value for this attribute is not desired. However this observation requires further investigation because it's requirement for some player's personalities considered as desired and positive like Model Citizen/Professional or Spirited might indicate that this attribute should be considered as positive.

It is unclear whether 10 is an absolute neutral value or it's relative on the league average which was not calculated in this experiment. This also applies to checking if performance drop due to low Pressure/Professionalism would still be this high in a league with different average value for those attributes.

There was no clear tendency for final number of points spread within certain attributes data sets. Very interesting factor was high spread of result between certain seasons within the same data set - for example for 20 Pressure there was final result of 33 points and 76 points in a two consecutive simulations. There were no investigation for any specific reason behind that big gap between results (like for example injury of key player) but it's worth noticing that number of simulations for future tests should be as high as possible due to significant spread of end results within the same datasets.

Some players despite having 1 or 20 in Important Matches and Consistency did not show this as Pros/Cons indicator in scouting/coaching report even after full season. It behaved the same way every time for the same players in both cases, so for high/low value and both for consistency/important matches. Reason is unknown but is worth noticing in terms of scouting for players with certain values of those hidden attributes that cannot be obtained in any other way without the editor or 3rd party tools.

 

I hope this experiment clarified some questions and also revealed some unexpected aspects of attributes. Any questions or comments are welcomed.

Spreadsheet for those interested in detailed numbers

I was linked to this thread by @Thebaker because of the way I look at the penalty taking attribute, (and because of the way I used to go into fine detail with the development of players).

Loving your work so far. 

There are a few things that jump out at me. 

1. Ambition of 1 has a positive difference in relation to the (All 10) control group. 
2. Important Matches of 20 has a negative difference in relation to the (All 10) control group. 
3. Det 20 bossing it as expected. 
4. Pressure 20 is a surprisingly high result for me. 
5. 1 Pro & 1 Press have a significantly bigger negative impact than 1 Det. That surprises me. 
6. If Det 20 has the biggest positive impact, why doesn't Det 1 have the biggest negative impact? 
7. A possible explanation for 5. & 6. is that I notice the "standard deviation" for both 1 Pro and 1 Press is small, (5.6 & 10.0), compared to the 20 Pro & 20 Press (13.4 & 14.9). Not suggesting anything at this stage. More just thinking out loud. 
8. Temp 20 is a strange one, but a little like Dirtiness & Injury Proneness, is 20 a good score or is 1 a good score? Is Temp possibly like that where 20 Temp is actually a bad score and 1 Temp is a good score? (I don't know I'm thinking out loud again). 
9. The Important Matches 20 data also doesn't make sense to me. Relatively small difference between Imp Match 20 & Imp Match 1 groups, and indeed Imp Match 1 out-performed Imp Match 20 on 2 of the 7 samples. 
10. Amb 1 out-performed Amb 20 in 1 of 7 samples. 
11. Temp 1 out-performed Temp 20 in 4 of 7 samples. 
12. Consis 1 out-performed Consis 20 in 1 of 7 samples. 
13. Imp Match 1 out-performed Imp Match 20 in 2 of 7 samples. 
14. Det 1 out-performed Det20 on 1 of 7 samples.   (It's too easy & lazy to blame the small sample size. That might be it, but equally it might not). 

This takes a little concentration and thought so I will try and come back to this. 

I'm struggling to work out the definitions of some of your columns. Difference, Variance, Standard Deviation & Spread definition please? (I'm stupid). 

[Orion_]

6 hours ago, Thebaker said:

intesresting analysis. Surprised that poor professionalism had more impact than poor determination. Pressure is the hidden attribute i never consider but clearly should do.

Low professionalism had biggest negative impact on player's performance in my previous experiment with Liverpool in FM21 (linked in first post) so I expected it to be also crucial here. Not gonna lie Pressure surprised me and probably many other people by a lot.

That's why it's interesting to perform some of those experiments. They sometimes bring amazing, unexpected results.

[chewbaccaloveaddiction]

  

17 hours ago, Orion_ said:

Yep. On one hand 10 points difference is something that makes a change, on the other hand group with 10 was almost in the middle between 1 and 20 results. Funny thing is that consistency 20 had statistically almost double the results spread than consistency 1.

 

You can use a t-test to check if the difference between two samples is significant, it's very easy in Excel or Google sheets, so I did it: https://docs.google.com/spreadsheets/d/1TtHoQ7oMzwfGg-pipPI0Tj8L0FOVXKg_KLfeBdI4SMU/edit?usp=sharing

 

Pressure 1, Professionalism 1 and Temperament 20 all perform significantly worse than all 10. Now if you remove the 3rd result from Pressure 20, the 33 point outlier, p drops from 0.4143 to 0.0662 and the standard deviation from 14.87 to 8.43. Just to be clear, I'm not saying you should remove results. But it shows the impact of one bad season, or one input error in small samples. Consistency is not as bad, but simply removing the worst season cuts STdev in half, so be careful with these interpretations. 

 

 

@Jimbokav1971 

Difference: Average point difference in percent, so Pressure 1: -47.33% roughly half as many points as all attributes at 10.

Standard Deviation: measures how spread out results/samples are. Look at this chart  

Spoiler

Let's assume red is a the speed of cars on a highway with a speed limit so most cars drive roughly as fast. The blue one is another highway, but without a speed limit, some cars will be a lot faster and the results are spread out more. Also because they're so fast they crash often causing traffic jams, so many cars are also much slower. The average is the same, but the samples look very different.

 

Variance: square of the standard deviation, useful for math 

[Orion_]

5 hours ago, chewbaccaloveaddiction said:

Just to be clear, I'm not saying you should remove results.

To be honest I was considering removing both the best and the worst result for each sample but I thought that being left with just 5 measurements is a little bit too small. Maybe next time I'll do it that way.

[Orion_]

20 hours ago, Jimbokav1971 said:

There are a few things that jump out at me. 

1. Ambition of 1 has a positive difference in relation to the (All 10) control group. 
2. Important Matches of 20 has a negative difference in relation to the (All 10) control group. 
3. Det 20 bossing it as expected. 
4. Pressure 20 is a surprisingly high result for me. 
5. 1 Pro & 1 Press have a significantly bigger negative impact than 1 Det. That surprises me. 
6. If Det 20 has the biggest positive impact, why doesn't Det 1 have the biggest negative impact? 
7. A possible explanation for 5. & 6. is that I notice the "standard deviation" for both 1 Pro and 1 Press is small, (5.6 & 10.0), compared to the 20 Pro & 20 Press (13.4 & 14.9). Not suggesting anything at this stage. More just thinking out loud. 
8. Temp 20 is a strange one, but a little like Dirtiness & Injury Proneness, is 20 a good score or is 1 a good score? Is Temp possibly like that where 20 Temp is actually a bad score and 1 Temp is a good score? (I don't know I'm thinking out loud again). 
9. The Important Matches 20 data also doesn't make sense to me. Relatively small difference between Imp Match 20 & Imp Match 1 groups, and indeed Imp Match 1 out-performed Imp Match 20 on 2 of the 7 samples. 
10. Amb 1 out-performed Amb 20 in 1 of 7 samples. 
11. Temp 1 out-performed Temp 20 in 4 of 7 samples. 
12. Consis 1 out-performed Consis 20 in 1 of 7 samples. 
13. Imp Match 1 out-performed Imp Match 20 in 2 of 7 samples. 
14. Det 1 out-performed Det20 on 1 of 7 samples.   (It's too easy & lazy to blame the small sample size. That might be it, but equally it might not). 

For 1, 2, 10, 11, 12, 13, 14 remember that randomness is a big factor. That's why I don't consider some results 'significant' if they don't stand out. As said in the main post - example of Pressure 20 giving on two consecutive simulations 33 and 76 points is like a joke. Imagine a situation where you theoretically set everything properly and should in most cases end the season in the middle of the league but somehow you almost get relegated, and it turns out it's not even your fault at all.

Also adding to the point 2 I think one of the reasons might be that there are not that much 'important matches' in the domestic league beside maybe some derby? It might have bigger impact in cup games/continental competition.

For your question regarding attributes not having a 'symmetrical' impact - so 1 giving significantly worse results while 20 not giving significantly best results - my guess is that either influence of those attributes is like 'logarithmic' so gives higher raise at low end but not that much impact on high end or my second theory is that it might be relative to the rest of the league. So it's not like 'absolute' attribute but relative, so the bigger the gap, the bigger the drop in performance or gain when you have advantage, but I didn't check league average so it's only a guess at this moment.

For 8 - I also had this thought that maybe Temp 20 might be reversed value - so the higher the worse - but as said it's expected to be high in some desired personalities like Model Citizen.

[chewbaccaloveaddiction]

3 hours ago, Orion_ said:

To be honest I was considering removing both the best and the worst result for each sample but I thought that being left with just 5 measurements is a little bit too small. Maybe next time I'll do it that way.

Btw I always use the pre game editor to remove all injuries for my tactics and testing saves so players can't get sick or injured anymore, that might give you better results in the future. But good job regardless

[Orion_]

40 minutes ago, chewbaccaloveaddiction said:

Btw I always use the pre game editor to remove all injuries for my tactics and testing saves so players can't get sick or injured anymore, that might give you better results in the future. But good job regardless

Oh. I knew that you can 'heal' all the players but I didn't know that you can prevent them from having any injuries at all. So it works like locking the attributes? Good to know for future tests.

[chewbaccaloveaddiction]

16 minutes ago, Orion_ said:

Oh. I knew that you can 'heal' all the players but I didn't know that you can prevent them from having any injuries at all. So it works like locking the attributes? Good to know for future tests.

In the pre game editor, you can remove and edit injuries. Like you could add eg long covid, make broken legs much more likely, or make them contagious if you want. Then just load the database with these changes when you start a new save. 

[HurkaDurk69]

On 22/05/2022 at 22:00, Jimbokav1971 said:

Temp 20 is a strange one, but a little like Dirtiness & Injury Proneness, is 20 a good score or is 1 a good score?

Temp 20 is good because you need a high Temp score to get Model Citizen, Controversey 20 is bad

[Jimbokav1971]

12 hours ago, HurkaDurk69 said:

Temp 20 is good because you need a high Temp score to get Model Citizen, Controversey 20 is bad

That sounds spot on, but give me 5 mins and I won't remember this. 

[Mandy42]

What constitutes an "important match" ? 

Games against rivals? 

Knock out football (which wouldn't be covered in your domestic league sample)

Title or relegation battles?

Guess I'm suggesting that the importance of the "important match" stat is only relevant if you play any important matches. Which if the standard finishing position is mid table. Is it possible to go through a season without having enough important games for this stat to make much difference. Even if its at 20? 

[fc.cadoni]

On 26/06/2022 at 15:34, Mandy42 said:

Games against rivals?

Title or relegation battles?

Yes and yes

[Orion_]

On 26/06/2022 at 14:34, Mandy42 said:

What constitutes an "important match" ? 

Games against rivals? 

Knock out football (which wouldn't be covered in your domestic league sample)

Title or relegation battles?

Guess I'm suggesting that the importance of the "important match" stat is only relevant if you play any important matches. Which if the standard finishing position is mid table. Is it possible to go through a season without having enough important games for this stat to make much difference. Even if its at 20? 

I'm not an expert but my bet would be games against rivals and cup games. I agree with your statement. That's one possible explanation why for Leeds Important Matches didn't matter. They we're expected to finish mid table so if they were not fighting against relegation no league games were 'Important' maybe except some historical rivals like United? I haven't checked what rivals Leeds has in the game right now.

 

On 25/06/2022 at 01:19, HurkaDurk69 said:

Temp 20 is good because you need a high Temp score to get Model Citizen, Controversey 20 is bad

I thought about this too but according to what we have in data it looks like high temperament gives worse results. It may be just bad luck due to small sample size but even if we do not count the highest and the lowest results it's still in significant favour towards Temp1 (~51 vs 47 points).

[nie jem frytek]

Isn't Temperament connected to how likely player is to cheat? The lower Temperament the bigger cheater? Players frustrating opposition by kicking them and diving and winning penalties (at a chance of getting booked instead) might lead to results improvement over players who never cheat. And Model Citizen having high Temperament even if it's detrimental to performance on pitch makes sense too, you wouldn't want your Model Citizen to be a cheater.

[Orion_]

8 hours ago, nie jem frytek said:

Isn't Temperament connected to how likely player is to cheat? The lower Temperament the bigger cheater? Players frustrating opposition by kicking them and diving and winning penalties (at a chance of getting booked instead) might lead to results improvement over players who never cheat. And Model Citizen having high Temperament even if it's detrimental to performance on pitch makes sense too, you wouldn't want your Model Citizen to be a cheater.

You're talking about 'Sportsmanship' not 'Temperament' but interesting point of view. I didn't think that way and removed Sportsmanship from the test to reduce processing time.

[One hell of a keeper]

Experiment showed that players should have at least medium range values for Pressure and Professionalism otherwise their negative impact on player's performance is significant. The positive impact of all of the tested attributes was much lower for high values than the negative impact for low values (except Temperament). 

 

These medium range values, is this 15+ for in this part pressure and professionalism? 

[Orion_]

19 hours ago, One hell of a keeper said:

Experiment showed that players should have at least medium range values for Pressure and Professionalism otherwise their negative impact on player's performance is significant. The positive impact of all of the tested attributes was much lower for high values than the negative impact for low values (except Temperament). 

 

These medium range values, is this 15+ for in this part pressure and professionalism? 

It's hard to tell since I haven't checked it for value 15 but as you can see in the graph when comparing the control group of 'all 10s' with Pressure 1 there was almost -50% drop in points and similar value for the 1 Professionalism while 20 Pressure and Professionalism had +10% and +5% respectively impact on points gain so general rule of thumb I'd say that at least for top 5 leagues the 'optimal minimum' would be said 10 and everything above this doesn't have that much of an impact on points gained.

If you play in smaller leagues/countries it would require additional testing but my assumption would be that the bar is lower there - just like with any other attributes. I can't say for sure simply because we are lacking for example a distribution of Pressure and Professionalism among players in different leagues but again if we assume that the distribution of those attributes follows the difference in 'visible attributes' - so in better leagues the hidden attributes have also on average higher values. Again sorry to say that it's all assumptions and 'ifs' but sadly we can't rely on any data in this matter due to lack of testing.

Attributes in general mean anything in relation to your opposition so it all comes to your league level.