Very Concerned About The Testign And patch Policy from SI

Read Blog number 5 today and TBH I'm not very impressed by what Miles has said.

Whilst the last few releases have had bugs and it has still been playable short term anyone who wanted a long term career game in FM06, FM06, FM07 and FM08 had to wait for at least 2 patches and even then there are still issues that can cripple the long term enjoyment of the game.

So when Miles reports:

"We were attempting to get to the next stage of the process after code lock, whereby we present a "gold master candidate" to Sega, at which point it goes into test at their end for 72 hours, and if it passes that, it can go into manufacture."

You would thing that SI have tested their game to the hilt before passing onto Sega's QA team but then Miles reports that there was problems installing and uninstalling the game so whilst a third party fixed that he writes :

"We didn't know at the time on Monday that it would take days, rather than hours, so a bunch of us sat up playing the game...Anyway, the time that the fix took to come in was quite useful, as we found another couple of issues and fixed them"

That does NOT sound as if this product has been tested thouroughly or systematically if a bunch of guys just playing it can find bugs!

Miles, you mention that you are reviewing the patch policy this year but in reality you really should review the internal testing policy as the last few releases have been frustating for many of your buyers.

I expect bugs in any software, that is not SI's fault, but if you have the proper internal testing and do it systematically then most bugs would be very small and insignificant.

I was part of the invited beta team when you released that for FM08 and found quite a few bugs which were reported but of course the guys that released the code out to public destroyed that, I would whole heartedly recommend you enlarge the testing team and review it, what you have written in your blog sounds very, very "amateurish".

I really hope that there are no major bugs in FM09.

Miles, can you tell us more about the patch policy?

maybe he's just preparing us for the 9.0.0 issueses?

"Anyway, the time that the fix took to come in was quite useful, as we found another couple of issues and fixed them". now I'm really concerned, what is the testing team doing?

8.0.0 was a disaster, i really don't expect 9.0.0 to be too...

Read Blog number 5 today and TBH I'm not very impressed by what Miles has said.

Whilst the last few releases have had bugs and it has still been playable short term anyone who wanted a long term career game in FM06, FM06, FM07 and FM08 had to wait for at least 2 patches and even then there are still issues that can cripple the long term enjoyment of the game.

So when Miles reports:

"We were attempting to get to the next stage of the process after code lock, whereby we present a "gold master candidate" to Sega, at which point it goes into test at their end for 72 hours, and if it passes that, it can go into manufacture."

You would thing that SI have tested their game to the hilt before passing onto Sega's QA team but then Miles reports that there was problems installing and uninstalling the game so whilst a third party fixed that he writes :

"We didn't know at the time on Monday that it would take days, rather than hours, so a bunch of us sat up playing the game...Anyway, the time that the fix took to come in was quite useful, as we found another couple of issues and fixed them"

That does NOT sound as if this product has been tested thouroughly or systematically if a bunch of guys just playing it can find bugs!

Miles, you mention that you are reviewing the patch policy this year but in reality you really should review the internal testing policy as the last few releases have been frustating for many of your buyers.

I expect bugs in any software, that is not SI's fault, but if you have the proper internal testing and do it systematically then most bugs would be very small and insignificant.

I was part of the invited beta team when you released that for FM08 and found quite a few bugs which were reported but of course the guys that released the code out to public destroyed that, I would whole heartedly recommend you enlarge the testing team and review it, what you have written in your blog sounds very, very "amateurish".

I really hope that there are no major bugs in FM09.

Miles, can you tell us more about the patch policy?

The thing is, how many people are actually on the beta testing team. That amunt of people can only give so much feedback on any issues. Like you say, you expect a number of bugs in any software release, but I think the patch policy is good. Lets face it, many companies DONT work on their product and release fixes for things. For that SI have to be praised, however, what I would say on the flipside reagarding the patch policy, SI dont keep us up to date enough with how its going. Yes we know there will be a patch and it will be ready when its ready, but I think that they should let the fans know how far on in the progress it is. That way, these forums shouldnt be filled with useless threads banging on about "when is the patch out". If people can see how far on it is and be regularly updated, then this eases a lot of people, and they are far more likely to be welcoming of the patch when it comes out.

SDK

I think, that SEGA is not testing the game functionality in depth, IMHO their Quality Assurance are testing overall product compatibility (installation process, Win registry writing/removing, documentation, uninstallation etc.) but they are not looking for minor bugs or long time playing errors.

I am convinced, that SI Games have done lot of testing (in game playability, bugs etc.).

I think, that SEGA is not testing the game functionality in depth, IMHO their Quality Assurance are testing overall product compatibility (installation process, Win registry writing/removing, documentation, uninstallation etc.) but they are not looking for minor bugs or long time playing errors.

I am convinced, that SI Games have done lot of testing (in game playability, bugs etc.).

If SI have done that then how come firing up FM09 and a bunch of guys playing it find bugs?

What are the testing team doing? How many are on that team?

It really ranks of SI doing this in their bedrooms like the good old days!

SI have an excellent relationship with their gamers and an excellent response to patching issues but i just feel all that could be avoided if they did it right (testing that is) in the first place!

Lots of bugs last year that I reported were acknowledged by SI as having been caught by internal testers but still did not get fixed when the gold disc was produced.

Guys aren't we jumping the gun a little before the game has even been released? Maybe what Miles means is that they will assess any bugs which are reported in the Bugs Forum.

I think we need to give the guys a break, and wait till the demo it out and see for ourselves what bugs there are and report them.

Guys aren't we jumping the gun a little before the game has even been released? Maybe what Miles means is that they will access any bugs which are reported in the Bugs Forum.

I think we need to give the guys a break, and wait till the demo it out and see for ourselves what bugs there are and report them.

maybe you're right but do you remember 8.0.0? it's unbelieveable that smth like closing down didn't work...

SI can test the product as thoroughly as they like, but it doesn't matter if you like it or not, they are very close to their product . so its possible they could miss something out because they 'know how it should work'.it is extremely common for developers to test the work and then send the product out to be tested by someone who is not so closely linked to the programming of the product.

so what would you prefer? for segas QA team to do it or for no one to do it?

If SI have done that then how come firing up FM09 and a bunch of guys playing it find bugs?

What are the testing team doing? How many are on that team?

It really ranks of SI doing this in their bedrooms like the good old days!

SI have an excellent relationship with their gamers and an excellent response to patching issues but i just feel all that could be avoided if they did it right (testing that is) in the first place!

Lots of bugs last year that I reported were acknowledged by SI as having been caught by internal testers but still did not get fixed when the gold disc was produced.

I do not know any details, but IMHO the main purpose of QA of distributor is not to beta test the software of producer/developer but to test overall quality of that SW within their set of quality tests and attributes. Of course, if there is an error which prevents them from starting the game, it will be going back to developer immediately.

IMHO SEGA is not here for beta testing. SI Games is doing it and I have not any details how it was/is going.

If SI Games lads have found any bugs in the last few hours, it is probably for this reason: As Miles has mentioned in one of his blogs, they are playing long term games, so there is high possibility to find any bugs.

I expect bugs in any software, that is not SI's fault, but if you have the proper internal testing and do it systematically then most bugs would be very small and insignificant.

If you expect bugs in any software, then why is it amatuerish for people that work on the game every day to find a couple and fix them? I would be more concerned if they were sitting on their butts doing nothing - that would be amateurish.

In the space of one thread we have someone saying SI should communicate their progress more and someone criticising their processes when they do. Talk about a no-win scenario.

I think, that SEGA is not testing the game functionality in depth, IMHO their Quality Assurance are testing overall product compatibility (installation process, Win registry writing/removing, documentation, uninstallation etc.) but they are not looking for minor bugs or long time playing errors.

I am convinced, that SI Games have done lot of testing (in game playability, bugs etc.).

Guys aren't we jumping the gun a little before the game has even been released? Maybe what Miles means is that they will assess any bugs which are reported in the Bugs Forum.

I think we need to give the guys a break, and wait till the demo it out and see for ourselves what bugs there are and report them.

Sorry, I'm not getting at the guys.

As Miles has said the amount of time and effort put into SI's games is sheer lunancy and they all deserve great credit for touching our lives with an excellent game.

Blogs are great but Miles represented SI as very poor in its testing area, if he could find a few bugs by just playing it with a few guys how many bugs are going to get caught when it is released to literally hundreds of thousands of gamers?

I shall certainly wait and play the demo before giving any constructive criticsim but the point I make is it really does "sound" like a lot of testing has taken place, or rather a systematic approach to it.

Sorry, I'm not getting at the guys.

As Miles has said the amount of time and effort put into SI's games is sheer lunancy and they all deserve great credit for touching our lives with an excellent game.

Blogs are great but Miles represented SI as very poor in its testing area, if he could find a few bugs by just playing it with a few guys how many bugs are going to get caught when it is released to literally hundreds of thousands of gamers?

I shall certainly wait and play the demo before giving any constructive criticsim but the point I make is it really does "sound" like a NOT lot of testing has taken place, or rather a systematic approach to it.

Edited as I missed out a NOT.

My heart sank when I read blog 5 simply because of what Miles said about him booting the game up and finding bugs, does that sound systematic to you?

I write programs for my job and testing plays a big big part in it where you have to cover every angle...

Game hasn't even gone gold yet, if they were playing a finished copy and found bugs, then there would be a worry.

My heart sank when I read blog 5 simply because of what Miles said about him booting the game up and finding bugs, does that sound systematic to you?

I write programs for my job and testing plays a big big part in it where you have to cover every angle...

I have understood it as a bug in installation process. Installation mechanism is often third party addition and for me personally, this is not agitating or alerting issue.

I think, that SEGA is not testing the game functionality in depth, IMHO their Quality Assurance are testing overall product compatibility (installation process, Win registry writing/removing, documentation, uninstallation etc.) but they are not looking for minor bugs or long time playing errors.

I am convinced, that SI Games have done lot of testing (in game playability, bugs etc.).

I do not know any details, but IMHO the main purpose of QA of distributor is not to beta test the software of producer/developer but to test overall quality of that SW within their set of quality tests and attributes. Of course, if there is an error which prevents them from starting the game, it will be going back to developer immediately.

IMHO SEGA is not here for beta testing. SI Games is doing it and I have not any details how it was/is going.

If SI Games lads have found any bugs in the last few hours, it is probably for this reason: As Miles has mentioned in one of his blogs, they are playing long term games, so there is high possibility to find any bugs.

I agree with you - SEGA surely are not experienced testers when it comes to this game so IMO they are only doing what Majkee suggests above.

I also think that those people criticising SI games should go easy on them. I have done some product testing for my Company and when you are so involved in testing is very easy to miss something obvious - I know from experience. What you must also appreciate is the fact that once you fix one bug, you never know what effect it might have on the rest of the game. The bug you fix could create two, three or more bugs else where in the game.

Just be grateful that they are finding these bugs before the game is released. We all know that there are going to be at least two patches - this is the nature of the beast in this type of game. As long as the main game plays good , then I can live with some bugs.

Unfortunately or fortunately, I have to wait until xmas for the game - my wife is buying it for me - god damn!!! So at least by then a patch should be out and a save game editor (hopefully).

Read Blog number 5 today and TBH I'm not very impressed by what Miles has said.

Whilst the last few releases have had bugs and it has still been playable short term anyone who wanted a long term career game in FM06, FM06, FM07 and FM08 had to wait for at least 2 patches and even then there are still issues that can cripple the long term enjoyment of the game.

So when Miles reports:

"We were attempting to get to the next stage of the process after code lock, whereby we present a "gold master candidate" to Sega, at which point it goes into test at their end for 72 hours, and if it passes that, it can go into manufacture."

You would thing that SI have tested their game to the hilt before passing onto Sega's QA team but then Miles reports that there was problems installing and uninstalling the game so whilst a third party fixed that he writes :

"We didn't know at the time on Monday that it would take days, rather than hours, so a bunch of us sat up playing the game...Anyway, the time that the fix took to come in was quite useful, as we found another couple of issues and fixed them"

That does NOT sound as if this product has been tested thouroughly or systematically if a bunch of guys just playing it can find bugs!

Miles, you mention that you are reviewing the patch policy this year but in reality you really should review the internal testing policy as the last few releases have been frustating for many of your buyers.

I expect bugs in any software, that is not SI's fault, but if you have the proper internal testing and do it systematically then most bugs would be very small and insignificant.

I was part of the invited beta team when you released that for FM08 and found quite a few bugs which were reported but of course the guys that released the code out to public destroyed that, I would whole heartedly recommend you enlarge the testing team and review it, what you have written in your blog sounds very, very "amateurish".

I really hope that there are no major bugs in FM09.

Miles, can you tell us more about the patch policy?

Top post - echoes my concerns too. I wonder what Miles/SI have to say about it?

My heart sank when I read blog 5 simply because of what Miles said about him booting the game up and finding bugs, does that sound systematic to you?

I write programs for my job and testing plays a big big part in it where you have to cover every angle...

But you can't cover every angle with something this complex, you can only cover the most likely angles, particularly with something as subjective and unpredictable as a simulation of the football world. Formal structured testing goes down a set path, so will never find every bug. That means you have to have an element of ad-hoc testing to create scenarios that were never envisaged by the coder or tester - and that's from a list of almost infinite possibilities.

Rigorous testing of any software in a systematic manner can often miss out some bugs that occur just in the course of playing the game because gameplay is far less systematic and there are an infinity of different states the game can get into that can't all be tested.

Testing can be an infinite process, but the longer spent testing the less time spent developing when you have a fixed released cycle of once per year - SI just have to get a balance. Obviously more testers might find more bugs, but good testers need to be paid, testers who do it for free are far less likely to find proper bugs in a systematic way because it would involve "playing" the game in a very boring way often.

I work for a software company and we have a constant list of known bugs in our software that gets added to during testing - people fix all those bugs they can and there is a determination as to which are the most important and some bug fixes will cause bugs elsewhere, etc, etc, but at some point you just have to release the software and then fix more bugs to put into a patch. I haven't read the blog, but based on the original post in this thread I have no problems with the process. I expect a game to have bugs and I know that when I pay for it. If I was really that bothered then I'd play the demo before buying it to see what those bugs are.

Most of the "bugs" are not black and white, yes or no issues though - they are all things that need a balance and are interwoven with each other. A lot of the stuff I work on is based around algorithms that produce an answer that is either right or wrong mathematically - if it is wrong then it is a bug and has to be fixed, if it is right then it is fine. Not all bugs and software are like that though, just the easy ones.

As has been said above, it's a bit of a no-win situation.

I'm not going to sit here typing and claim that any game that we've released is bug free - I don't know of a single bit of software released that ever has been.

We do our best to have the best user experience possible with our games, and have dedicated test teams at both SI & Sega, as well as testing going on with our head researchers, some of their assisants, and the "Dream Team" offsite test team. It's the largest QA team that I know of in gaming.

We also have one of the largest and most complicated games in gaming. So some things will slip through the net.

The issues that were found on the night were the install/uninstall problem, a match engine crash that has been in the game for ages and never effected anyone til now (it had not been seen by any tester), and someone else found an issue to do with alt/tabbing whilst the game was windowed, but maximised. Oh - and one machine that had a graphics card incapable of handling 3d, but was trying to, rather than defaulting to 2d.

The first wouldn't have been noticed before, as it was to do with the final installation build, which we hadn't needed to do. The others were incredibly rare, and are fixed, which we're very happy about.

We will, as always, support our games post release if users have issues. But when you're dealing with PC's and Mac's, you are also dealing with infinite amounts of configurations hardware wise, so different specs react differently to others. It's just the nature of what we do.

Appreciated your comments Miles.

I just hope that other bugs are not caught by us gamers in the same way that you just fired up the game and found these issues.

Roll on the demo!!!!

Key word Miles uses: 'Issues'. This, to me, could mean relatively minor data issues, for example when Ian Roper appeared as 'Ia Roper' in one version of FM.

Just seems as though people see 'issues' and read 'bugs; huge, zonking great BUGS'.

It's been said before that the ME is in better shape than ever due to FML, and the 'closing down bug' in FM08 seemed, to me, to be caused by a poor interaction between instructions and player morale. A player low on confidence in a losing team is going to second guess himself, but what happened in the ME was too dependant upon morale, etc. It seemed this way because a player high on confidence closed down with few problems; he just didn't bother tackling at all, and that was the real issue, I found.

But that's the past.

From what has been said, from what I've read, etc, the ME seems like it should be the least of any problems in-game, and any issues relating to finances, etc, can be solved with a workaround, like the Pompey and Spurs issues in FM08.

Like with the 3D view and lack of arrows, it's going to be a case of wait and see. And at least we get a demo version first. Like (I suspect) most others, I've played games before that are buggy as hell, yet receive no after-support at all.

But you can't cover every angle with something this complex, you can only cover the most likely angles, particularly with something as subjective and unpredictable as a simulation of the football world. Formal structured testing goes down a set path, so will never find every bug. That means you have to have an element of ad-hoc testing to create scenarios that were never envisaged by the coder or tester - and that's from a list of almost infinite possibilities.

i dont think its that easy to systematically test fm. way i think about it is that just because you offer a player to clubs does it mean teams will come back with offers?

no. there could be multiple thinngs that could result from that situation, and from that first layer of multiple occurances you could get another multiple occurances for each.

so we start with the users choice. the game chooses one of 5 occurances based on multiple stats within the game. depending on which of those 5 occurances it choses it would then have a further 7 occurances from that point, again depending on in game stats. and from then they could have another 3 occurances for each of those 7 and it goes on.

if the majority of your program runs on user input ("clickt his button and that will happen" "load the page up and this data should show)then a systematic testing structure is much easier. if the majority of your program runs on in game situations where the shear scale of potential outcomes is massive then it isn't so.

Guys aren't we jumping the gun a little before the game has even been released? Maybe what Miles means is that they will assess any bugs which are reported in the Bugs Forum.

I think we need to give the guys a break, and wait till the demo it out and see for ourselves what bugs there are and report them.

Yaeh, I agree.. I don't know how you can get so much out of that little..

When it all adds up, I think that SI knows how to make games.. atleast better that any of us

Miles replied as I was typing, btw.

Miles replied as I was typing, btw.

Same here

It is nice to see Miles to reply almost immediately after discussion has arisen

FM09 will have been tested more than ever due to people who are BETA testing FML

While I understand the concerns, I think too much is being read into Miles' comments. I think that we should see how 09 is and then raise this concern if 09 is buggy as hell. It's too late for them to change anything now anyway.

There will be bugs in 2009 but most-likely not to the extent they were in 2008, they do there best and even though it did worry me I just treat it as three more bugs out of the game!

It's threads like this that remind me just what a thankless job Miles has. The guy makes every effort to stay in touch with the community and keep people informed of their progress, but his every word, turn of phrase and sentence seems to make him nothing but a target for some people.

Why not just wait and judge the quality of the finished product, rather than trying to second guess what we think might be going on? Just a thought.

It's threads like this that remind me just what a thankless job Miles has. The guy makes every effort to stay in touch with the community and keep people informed of their progress, but his every word, turn of phrase and sentence seems to make him nothing but a target for some people.

Why not just wait and judge the quality of the finished product, rather than trying to second guess what we think might be going on? Just a thought.

Welcome to the forums

We do appreciate all the effort Miles puts into Football Manager but sometimes we get frustrated at the bugs, we thank him by forking out the £30 to get the game, we really like this series are grateful the time Miles and co put into the game!

It's threads like this that remind me just what a thankless job Miles has. The guy makes every effort to stay in touch with the community and keep people informed of their progress, but his every word, turn of phrase and sentence seems to make him nothing but a target for some people.

Why not just wait and judge the quality of the finished product, rather than trying to second guess what we think might be going on? Just a thought.

I think you will find that I praised Miles and SI for the sterling work they do/have done.

I am not criticizing but merely expressing concern at the seemingly causal approach to testing.

I think you will find that I praised Miles and SI for the sterling work they do/have done.

I am not criticizing but merely expressing concern at the seemingly causal approach to testing.

What is casual about it? Miles and co put in some additional testing when they could have sat back and waited for the results from Sega's QA. How is doing something above and beyond their planned testing casual? You give the impression that it is the first time they have sat down to look at the game properly.

Would you prefer Miles to adopt a more formal writing style for the next blog, so as to avoid any possible misunderstandings?

I think you will find that I praised Miles and SI for the sterling work they do/have done.

I am not criticizing but merely expressing concern at the seemingly causal approach to testing.

It was only "seemingly casual" because he didn't go into loads of detail in the blog! I'm absolutely certain that Miles could right pages and pages of detail about testing procedures and so-on, but he has better things to do (like actually testing the game for example).

I fail to see how people expect there to be no bugs in a game when the game has so many variables and possibilities that you could compare it to a chess game!

Granted there were a couple of "big" issues in the last release (Closing Down for example), however, they did not make the game unplayable - that's a narking phrase, how is a game "unplayable" if you can boot it and play it! - merely funny from time to time.

I'm a frequent flyer when it comes to football manager (and championship manager when SI made it), and, with the exception of CM4, I've never had a problem with the game upon release. People should relax a little more - I think people get wound up about things simply for the sake of it.

I dont think they're putting enough time and effort into the game. In the blog Miles said he did a '21 hour shift', but what happened to the other 3 hours in the day? Theres no excuse for slacking off.

I think you will find that I praised Miles and SI for the sterling work they do/have done.

I am not criticizing but merely expressing concern at the seemingly causal approach to testing.

Again, the only sensible response to this is, why don't you judge the game based on the quality of its demo, or the full finished product. If it's strewn with bugs, then by all means we have the legitimate right to 'express concerns' about the quality of the testing, but to 'express concerns' about the quality of the testing based on a couple of lines in a blog, when you've not played a demo or seen a finished product seems to me to be of little concrete value. I'm just expressing a concern about your expression of concern.

I find it saddening that many (not all, not by a long shot) people are predicting bugs galore without even seeing the game.

I think I can predict with a high degree of certainty that these forums will fill with gleeful complaints and I-told-you-sos as soon as the demo comes out, followed by various immature "This is a disgrace, rant, rant, blah, blah", no matter how minor the odd issue is.

Let's just wait and see once it comes out, and PLEASE remember this is not a bespoke game designed to please YOU only; it won't be perfect and I'm sure certain aspects won't please everyone. And like Miles and several people have said you HAVE to expect the odd bug here and there. But let's keep it in perspective: report the bugs in the bugs forum, upload saved games, and count to ten before posting.

OMG I CANNOT BELIEVE THEY ARE FIXING BUGS!

Jesus, some people need to get a grip.

I don't think it's fair to develop a view on SIs testing policies based on a couple of lines in a blog Miles has written.