Jump to content

FM Data Analytics: a data visualization tool based on R & Shiny - V3 RELEASED!


Recommended Posts

  • Replies 162
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Videos

Setup demo

https://streamable.com/2382x9

App demo

https://streamable.com/6765al

V3 new features

https://streamable.com/0spqup

UPDATE LOG: V3 (2021/02/21)

* "Home" menu added:

  • Info boxes: number of players at the club & players in the shortlist
  • Summary charts: bar chart of the goal contribution distribution, boxplots of the playing time and wages distribution

* "Progress" submenu added in the "Squad Analysis" menu: dates selection, performance and attributes progress between selected dates

* "Shortlist" submenu in the "Scouting" menu updated: now shows min and max asking prices and wage demands, presence of release clause and conditional release clauses instead of replicate table of the "Filters" submenus

* DT tables now used instead of Reactables: styling and formatting improved

* To use the V3 version you need to name your squad output files "squad_yyyymmdd", where "yyyymmdd" is the in-game date. This allows to compute the variations (progress) between given in-game dates. I updated the data folder in my Github to give you an example.

* It is possible thanks to the above feature to store multiple squad data files. Keep in mind that the more there is squad files in your data folder, the longer the app will take to load. You can find out how much time the different tasks (load data & format data for example) take time by paying attention to the command prompt that is launched when you double-click on FMapp.bat.

 

UPDATE LOG: V2 (2020/01/15)

* The app now directly reads the FM outputs to extract the data, thus meaning the Excel tool is obsolete. It is no longer included in the download.

* FFP Contribution and future time playing removed from the views (5 columns) as they are user-specific

* The app now dynamically install required packages if needed. First thing the code does is check if there are missing required packages and installs them. This means the installpackages.bat batch file is now obsolete. It is no longer included in the download.

 

* FMapp.bat now detects its current directory and uses it to launch the app. It is very important that you don't modify the file's structure! Instead of replacing 3 paths in the last version, you only need to replace 1, and it's a one-time process.

* DNA Model subsection was added in the Squad Analysis section. This enables you to select key attributes for your playing style and calculate a "DNA rating". Similarly to the Scouting section, the color code is used in the tables and the plots.

* Added status messages to show the user what's going on before the app is launched in their browser.

* Points in the plots have now a subtle black outline to make them more visible.

Bugs

[FIXED] "Current Season T,T+1" columns produce errors when extracting the html in the FM tool: removed those columns from the extract, as they will be variable depending on which season you are.

[FIXED] installpackages.bat not working: didn't wrote the R commands well, I fixed it with the help of stack overflow (First answer)

[FIXED] Currency issues: replaced the "£" and "$" symbols in the code by their unicode characters.

Known Issues & Frequent errors

Before exporting data from FM, change the language to British English. All columns/statistics/data points are referred to their British English abbreviations. If anyone wants to do a translation, you're free to do it, it's open source.

FAQ (will be updated if I get the same questions over and over):

I noticed I'm getting warnings in the command prompt when my app is running, on certain plots... That's normal, the plots remove the data with missing values (say if a player has NA passes completed) so you'll have a warning that x observations are missing. You're actually missing missing data :lol:

There are no goalkeeper stats? Will be added in a future update, there is not a lot stats for goalkeepers apart from the saves and penalty conceded/faced...

In the Filters subsections, when I deselect all positions I get an error message instead of the table... Will be fixed in a future update but it has literally no impact. Just select at least a position and it'll go back to normal. Unless you really want to do an analysis on players with no positions? :idiot:

THE APP DOES NOT WORK WITH MY SCREEN! I AM NOT A WEB DEVELOPPER!!! :lol: No seriously, I have 0 background in web dev, so I just went with fixed sizes in my code and made sure it was good for my 15inch screen. Just use it in the browser's max size (full screen), don't minimize it and you should be all good. I got some messages from people with experience in web dev, so hopefully I'll have helping hands on that matter. But again, full size in your browser and you should be all good.

Is it compatible with other OS than Windows? Well the batch files/command prompt is exclusive to Windows (I think so?). So you'll have to know a little bit of R, just enough run the packages install by yourself in R, then install RStudio, open the app.R file in RStudio and click on the Run App button. So that's less than basic knowledge of R. However I'm not sure about the compatibility of the Excel tool and the macros as they're written in VBA.

The app was able to run but it seems that the table and plots are empty... Before exporting the shortlist data from FM, make sure to scroll from top to bottom and/or Make sure that your in-game preferences are as following in FM before exporting:

On 09/01/2021 at 11:28, marculatur said:

Preferences - Advanced - Formats

Positive: €1,00 (The Symbol of your currency has to be before the value)
Decimal Symbol: Dot
Digit Grouping Symbol: Comma
 

How can I contribute? Send me a PM ;)

Edited by gam945
V3 update is released (2021/02/21)
Link to post
Share on other sites

1 hour ago, Mike Trusky said:

It looks really nice, I love that kind of stuff about analysis. If you'd like to check, sometime ago I released a tool that I wrote, including for analysis. It looks like this:

presentation.thumb.png.63ab23bacf55a7ceb64981ea06febef6.png

And at this link is an example of using the tool - Statistics Overview Example

Nice, seems to be close to my idea as well! Your app is built on what language? Mine is exclusively in R but I'm curious about yours. From what I see on your website the tool is not free so we probably won't have access to the source code (unlike mine) but I'd be keen to PM you to share ideas ;)

I thought about letting the user define a "DNA attributes model" and use it to filter their shortlist as well but I preferred to go exclusively with in-game match stats. 

Link to post
Share on other sites

9 minutes ago, gam945 said:

Nice, seems to be close to my idea as well! Your app is built on what language? Mine is exclusively in R but I'm curious about yours. From what I see on your website the tool is not free so we probably won't have access to the source code (unlike mine) but I'd be keen to PM you to share ideas ;)

I thought about letting the user define a "DNA attributes model" and use it to filter their shortlist as well but I preferred to go exclusively with in-game match stats. 

My app is written entirely in C++. At the beginning I wanted to use python but I prefer C++, so I wrote everything in it. And yes, I've spent a lot of time on it, that's why it's not for free. (but really cheap :P).

In my app it depends on which tab you are currently using. Some only use in-game match stats, others have "variables" such as DNA, which can be set be each user. 

And similar like you, I am working on exported html views from FM. (which I loaded into the app).

Link to post
Share on other sites

11 minutes ago, Mike Trusky said:

My app is written entirely in C++. At the beginning I wanted to use python but I prefer C++, so I wrote everything in it. And yes, I've spent a lot of time on it, that's why it's not for free. (but really cheap :P).

In my app it depends on which tab you are currently using. Some only use in-game match stats, others have "variables" such as DNA, which can be set be each user. 

And similar like you, I am working on exported html views from FM. (which I loaded into the app).

Yeah from the GUI I suspected C++/C/C#. Yours look more like a software, more professional while mine is more like a dashboard report.

Will have to teach myself C++ at some point, I was already planning on doing so. Re-writing my app in C++ would be a good starting project :lol:

How much lines/hours of code for your app out of curiosity?

Edited by gam945
Link to post
Share on other sites

  • SI Staff
15 minutes ago, Mike Trusky said:

My app is written entirely in C++. At the beginning I wanted to use python but I prefer C++, so I wrote everything in it. And yes, I've spent a lot of time on it, that's why it's not for free. (but really cheap :P).

In my app it depends on which tab you are currently using. Some only use in-game match stats, others have "variables" such as DNA, which can be set be each user. 

And similar like you, I am working on exported html views from FM. (which I loaded into the app).

As you know C++ have you considered applying for a job at SI? https://www.sigames.com/careers

Let me known if you do, we have lots of vacancies.

Link to post
Share on other sites

12 minutes ago, gam945 said:

Yeah from the GUI I suspected C++/C/C#. Yours look more like a software, more professional while mine is more like a dashboard report.

Will have to teach myself C++ at some point, I was already planning on doing so. Re-writing my app in C++ would be a good starting project :lol:

How much lines/hours of code for your app out of curiosity?

C ++ is really great. Especially when everything works as it should. : D
Now as I check it, it comes out about 6000 lines of code :P

 

13 minutes ago, EdL said:

As you know C++ have you considered applying for a job at SI? https://www.sigames.com/careers

Let me known if you do, we have lots of vacancies.

Oh, I haven't looked at offers in SI yet. But now I'm at the end of my studies and I'm just starting to look for a job. Is the work currently on place or also remotely?

Link to post
Share on other sites

  • SI Staff
16 minutes ago, Mike Trusky said:

C ++ is really great. Especially when everything works as it should. : D
Now as I check it, it comes out about 6000 lines of code :P

 

Oh, I haven't looked at offers in SI yet. But now I'm at the end of my studies and I'm just starting to look for a job. Is the work currently on place or also remotely?

Well we are all remote right now, but normally I'd be in the office. If you are a junior I'd expect when things are back to normal we'd probably like to have you in the office at least a 2/3 days a week as it makes mentoring a lot easier.

Still worth applying and asking about possibilites.

Link to post
Share on other sites

19 hours ago, EdL said:

Well we are all remote right now, but normally I'd be in the office. If you are a junior I'd expect when things are back to normal we'd probably like to have you in the office at least a 2/3 days a week as it makes mentoring a lot easier.

Still worth applying and asking about possibilites.

Yeah, I think I'll try. Spending even more time with a game with which I already spend a lot of time, at work, must be very interesting. :D

Link to post
Share on other sites

@gam945 So I have been thinking more about this tool as I am going to start an attribute masked masked run and will just rely on stats for most of my decisions. I do not  know if you have it but a selection for Passes Per  Defensive Action(PPDA) would be amazing. This would allow anyone to measure how effectively a player presses some players in a given game. Would be a wonderful scouting tool. Just a thought!

Link to post
Share on other sites

12 hours ago, Hav3n_The_Hero said:

@gam945 So I have been thinking more about this tool as I am going to start an attribute masked masked run and will just rely on stats for most of my decisions. I do not  know if you have it but a selection for Passes Per  Defensive Action(PPDA) would be amazing. This would allow anyone to measure how effectively a player presses some players in a given game. Would be a wonderful scouting tool. Just a thought!

I am limited by FM's provided stats, but how would you suggest I do that? Passes completed vs (Tackles won + Interceptions) for example?

To give you an idea, I attached a dictionnary of all data points I extract from FM, you can take a look. I'd be more than pleased to add analysis wanted by the community!

Also, following is all the current visualizations I implemented:

image.png.4424283ad1cf38cb6847ef8b8193d30a.pngimage.png.e6a6d6992f156ad26b1d56acc5564b02.pngimage.png.3b4df5f1609bb599abf34d7171edd129.pngimage.png.d1a62471d7032cd9d27186912334edd7.png

image.png.aa2c4bc205ad456e18de211ada79d93c.pngimage.png.cfed452e5c15aab201de8ac2b3d18e3b.png

For more clarifications:

Contribution refers to either Team goals scored or Team goals conceded
Rating consistency: games played vs average rating
Team results consistency: games played vs points per game

All stats are per 90mins

datadictionary.xlsx

Edited by gam945
Link to post
Share on other sites

The definition of PPDA I could find states, "PPDA = Number of Passes made by Attacking Team (opponent) / Number of Defensive Actions  , in the opponent's 60% of the pitch".

The last requirement may be the most difficult to get, but I have suggestion for that.

1. Number of passes made by attacking team. Which means to me number of passes completed not necessarily attempted.

2. Number of defensive actions include Possession-winning duels, Tackles, Interceptions, Fouls. I am not sure how possession winning duels is measured so I would ignore it, but tackles should include successful and failed tackles. Interceptions and fouls made by defensive team are pretty straight forward.

3. In the opponent's 60% of the pitch is very important and potentially difficult to measure.

3a. If you only measure all passes across the whole pitch it can make low block teams against possession based teams look much more effective at pressing then they actually are so it is critical this part can be measured.

3b. A potential solution to this is in the match analysis screen after a match is complete it has a section listed for the defensive half of the pitch and so it will only generate stats for that area of the pitch. I will share screenshots after to show what I mean. 

3c. I do not know if it is easy to pull that data from football manager, but I do know it is generated I hope that helps.

Link to post
Share on other sites

26 minutes ago, Hav3n_The_Hero said:

The definition of PPDA I could find states, "PPDA = Number of Passes made by Attacking Team (opponent) / Number of Defensive Actions  , in the opponent's 60% of the pitch".

The last requirement may be the most difficult to get, but I have suggestion for that.

1. Number of passes made by attacking team. Which means to me number of passes completed not necessarily attempted.

2. Number of defensive actions include Possession-winning duels, Tackles, Interceptions, Fouls. I am not sure how possession winning duels is measured so I would ignore it, but tackles should include successful and failed tackles. Interceptions and fouls made by defensive team are pretty straight forward.

3. In the opponent's 60% of the pitch is very important and potentially difficult to measure.

3a. If you only measure all passes across the whole pitch it can make low block teams against possession based teams look much more effective at pressing then they actually are so it is critical this part can be measured.

3b. A potential solution to this is in the match analysis screen after a match is complete it has a section listed for the defensive half of the pitch and so it will only generate stats for that area of the pitch. I will share screenshots after to show what I mean. 

3c. I do not know if it is easy to pull that data from football manager, but I do know it is generated I hope that helps.

I see what you mean... however I pull the data from the FM views, not from the after game match analysis. In fact I am 99% sure you can't pull data from it. Unfortunately I can't extract data relative to where in the pitch it has been recorded.

For the defensive actions, best I have in mind would be tackles attempted (not necessarily won) and interceptions. Fouls are likely to come from tackles so we'd count the same action twice... then again I don't know how SI records the match stats

Edited by gam945
Link to post
Share on other sites

Looking forward to this. Love using data for my saves so anything that speeds this up and makes it more accessible is great.

You mentioned injury data earlier on and using the squad views to extract. I’ve never been able to get much off screens that weren’t squad views. For a blog I’m doing I wanted to analyse data from the medical centre but print screen didn’t work. I had to do it manually. Are you able to get this data out as well?

Link to post
Share on other sites

9 minutes ago, Peljam said:

Looking forward to this. Love using data for my saves so anything that speeds this up and makes it more accessible is great.

You mentioned injury data earlier on and using the squad views to extract. I’ve never been able to get much off screens that weren’t squad views. For a blog I’m doing I wanted to analyse data from the medical centre but print screen didn’t work. I had to do it manually. Are you able to get this data out as well?

yeah I can...

image.png.c07658e0e3db7984c6e0ef5e80fef383.png

The tool will include an Excel extractor to pull the data from the html files extracted from FM

Link to post
Share on other sites

1 hour ago, gam945 said:

I see what you mean... however I pull the data from the FM views, not from the after game match analysis. In fact I am 99% sure you can't pull data from it. Unfortunately I can't extract data relative to where in the pitch it has been recorded.

For the defensive actions, best I have in mind would be tackles attempted (not necessarily won) and interceptions. Fouls are likely to come from tackles so we'd count the same action twice... then again I don't know how SI records the match stats

Ok, if that is all you can do. That is all you can do. It will still be useful. It just should be used sparingly. 

Link to post
Share on other sites

Please make this open source. I would love to contribute. I have experience working with front ends and while I don’t have programming experience with R, I might be able to help with some of the backend logic too. Are you planning adding this to GitHub?

Edited by milestobudapest
Link to post
Share on other sites

Just now, milestobudapest said:

Please make this open source. I would love to contribute. I have experience working with front ends and while I don’t have programming experience with R, I might be able to help with some of the backend logic too. Are you planning adding this to GitHub?

Of course this will be open source! I will post it on my GitHub and provide the link here for downloads.

I'm rushing to finish the app today (I'm 6 hours late from the UK time) so I can post it before sleeping lol

PM me if you're interested to contribute, help from someone with front end experience would really be appreciated ;) 

Link to post
Share on other sites

This looks great. I’m a frontend web developer so I’d love to contribute to this. I’ve been using a program called tableau myself for data visualisation with FM. Is there a way you could filter by minutes played? I tend to use that metric rather than appearances 

Great work

Link to post
Share on other sites

15 minutes ago, pilif25 said:

Is this going to work with FM21 only, or FM20 too? Also: how much are you planning on charging for it?

There are minor differences from FM20 but yeah after 2-3 modifications it should work.

From the top of my head, In FM20 the # of chances created was named "Ch C" while in FM21 it's "CCC", clear cut chances created. Also "xG" is not a stat in FM20. These modifications should be implemented to avoid errors when running the app. Nothing too complicated to do.

It will be free, and all the code will be open source, meaning that, quoting wiki, it "is released under a license in which the copyright holder grants users the rights to use, study, change, and distribute the software to anyone and for any purpose."

 

Link to post
Share on other sites

  • SI Staff
33 minutes ago, Grifty said:

I can write neatly and use joined up letters if there's any need for that at SI?

I think we've forgotten what pen and paper are I'm afraid :D

Link to post
Share on other sites

Really hope SI fix how stats are recorded so using this is actually worthwhile. Would be an incredible tool.

But as it stands, chances created, key passes, key tackles are all completely pointless metrics the way the game records them (or does not in the case of chances created). I am not sure how much we can trust any of the stats to be honest.

Link to post
Share on other sites

  • gam945 changed the title to A tool based on R Shiny for the FM data driven manager
  • gam945 changed the title to A tool based on R Shiny for the FM data driven manager -RELEASED!

Really excited to dive into this!

Is there any way to dump all the players in the db out to pull into the reports?

Using the shortlist view, I can only add 327 players to a shortlist at a time. So if I wanted to run a report on a league, it can be quite difficult pulling all of the required players to the shortlist. Is it possible at all to run the shortlist export from the scouting screen instead so we could export all known players?

 

Edit:  Ok so I worked out,  I need to page down as FM can only add players to shortlist that it has seen. But this is still limited to 500 at a time.

 

Now, when i'm running the excel program (after exporting squad.html and shortlist.html and moving them to the folder) I am encountering an error "the column season 2021/22 of the table wasn't found"    - run time error 1004

Edited by IWWROCKS
Link to post
Share on other sites

3 hours ago, gam945 said:

App has now been released, ahead of schedule!

I tried to import the squad data and I get this error. I don't know why because I tried with another save from the day of the job and it works fine so I don't know if the problem is because I'm at the end of the season on this particular save.

error.png.10c4bb5c13edbff99b7ba7e7474247f6.png

I went to look at the html file and that section is really missing so do I just edit the file on excel and add that column? 

1470487571_error2.thumb.png.ce6d17b50720f0c2a27e789e68ae0236.png

 

I was also unable to install the app and I don't know if this is related but after I changed the path for the file in the help folder and install it I see this this on command prompt

988711524_error3.png.5b8888ce8857d322541094d782e9aa99.png 

Edited by DarJ
Link to post
Share on other sites

2 hours ago, DarJ said:

I tried to import the squad data and I get this error. I don't know why because I tried with another save from the day of the job and it works fine so I don't know if the problem is because I'm at the end of the season on this particular save.

error.png.10c4bb5c13edbff99b7ba7e7474247f6.png

I went to look at the html file and that section is really missing so do I just edit the file on excel and add that column? 

1470487571_error2.thumb.png.ce6d17b50720f0c2a27e789e68ae0236.png

 

I was also unable to install the app and I don't know if this is related but after I changed the path for the file in the help folder and install it I see this this on command prompt

988711524_error3.png.5b8888ce8857d322541094d782e9aa99.png 

Thanks for raising this issue! the 4 columns Season <T,T+1> were the only one that referred explicitely to a year. This may change depending on the time in-game. Went on and removed those 4 columns, the Excel too is updated in Github. You could either just re-download the Excel tool or the entire folder if you don't feel comfortable.

As for the the install packages batch file, it has also been corrected. Credits to the first answer (https://stackoverflow.com/questions/26274553/install-r-packages-from-windows-cmd). Same as the Excel, you can re-download it from Github. However do not change the structure of the folder to avoid errors! The help folder shouldn't matter as nothing references to it, but to be sure do not change the structure ;)

3 hours ago, IWWROCKS said:

Really excited to dive into this!

Is there any way to dump all the players in the db out to pull into the reports?

Using the shortlist view, I can only add 327 players to a shortlist at a time. So if I wanted to run a report on a league, it can be quite difficult pulling all of the required players to the shortlist. Is it possible at all to run the shortlist export from the scouting screen instead so we could export all known players?

 

Edit:  Ok so I worked out,  I need to page down as FM can only add players to shortlist that it has seen. But this is still limited to 500 at a time.

 

Now, when i'm running the excel program (after exporting squad.html and shortlist.html and moving them to the folder) I am encountering an error "the column season 2021/22 of the table wasn't found"    - run time error 1004

It has now been fixed, please check my answer just above.

Edited by gam945
Link to post
Share on other sites

3 minutes ago, yanloup64 said:

Something went wrong. Double clicking on the batch file a black window pops up then closes immediately. I'm on Windows 7.

Which batch file? there are two. PM me as I just tested the batch files and there should be no errors.

Did you follow the instructions and edited the paths?

7 minutes ago, r70lla said:

Brilliant idea, however, I'm having similar issues as Darj.

Fixed, please see my answer above.

Link to post
Share on other sites

2 hours ago, R0bby said:

Does it work with open office too, because i don´t have excel

Someone will have to test that...but as I mentioned in my first post, the Excel contains macros written in VBA. So maybe the excel will be compatible but someone will have to re-write the macros in the VBA equivalent for Open Office.

Link to post
Share on other sites

If you see a black window opening then immediately closing when you double-click on FMapp.bat, this is because you didn't change the paths! Please follow the instructions when you download the app to avoid this error!

Basically the FMapp.bat file contains the following:

<full path for Rscript.exe> <full path of app.R>

If you do not change the first path, Windows won't know which program (in this case Rscript.exe which comes with R when you install it) you are using.

If you do not change the second path, Windows won't know where is the script (in this case app.R in the main folder) you are trying to run from the Command Prompt.

Edited by gam945
Link to post
Share on other sites

Hi there,

thank you so much for this tool. I was looking for something like this a long time. :)

I just downloaded and tested your recent update. As you can see in the picture, everything runs fine so far. But i can`t see any data in the browser view. The correct club (RC Strasbourg) is displayed correctly, but every sheet is empty and no rows were found. The data in the excel file contains all the data from the views. Any suggestions?

 

What i had to do to keep it running: I had to switch the ingame language to British English, as i come from germany and the translated categories were not able to import. ;)

fm_analytics.jpg

Link to post
Share on other sites

1 minute ago, marculatur said:

Hi there,

thank you so much for this tool. I was looking for something like this a long time. :)

I just downloaded and tested your recent update. As you can see in the picture, everything runs fine so far. But i can`t see any data in the browser view. The correct club (RC Strasbourg) is displayed correctly, but every sheet is empty and no rows were found. The data in the excel file contains all the data from the views. Any suggestions?

 

What i had to do to keep it running: I had to switch the ingame language to British English, as i come from germany and the translated categories were not able to import. ;)

fm_analytics.jpg

Adding this to the known issues, but yes, the data point names are in English, so the game should be in British English before exporting the data. The Excel tool will also not work if the data is not exported in British English.

Before exporting your data, change the game to British English. Then export and you'll be all set.

Link to post
Share on other sites

1 hour ago, gam945 said:

Adding this to the known issues, but yes, the data point names are in English, so the game should be in British English before exporting the data. The Excel tool will also not work if the data is not exported in British English.

Before exporting your data, change the game to British English. Then export and you'll be all set.

I do have my language set to British English but have the same issue with regards to empty data. The excel sheet runs fine and brings in the data I want but when running the app it's not showing any data. All of the players, clubs, divisions appear on the filters, but not in the data subset (with all filter options selected)

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...