FM 2009 - was it tested thoroughly enough?

Right, there’s clearly been some, shall we say, criticism of the demo, and although I don’t think it’s all that bad, I did wonder about the testing process.

Do we know what the testing process is? Do any of the members here do any testing for the game? I’m curious about this. I understand why SI don’t release a beta demo to select users after the debacle when the game code was leaked (can’t say I blame them for refusing to do beta demos any more, which is a shame).

A couple of years ago I applied to be a tester when it was advertised and was disappointed to be totally ignored given that I'm on the testing team for the database we use at work. Admittedly, it only serves 100 users compared to the hundreds of thousands FM serves, but the testing process should still be the same. Still, SI weren’t interested in me, fair enough, and I haven’t reapplied since.

One of things I want to know is how methodical the testing is. At work, I have a reputation for being a bit of a bug catcher because I always find major bugs and loads of minor bugs. One of the reasons is that I rarely test the database in the way the developers want – I figure the other members of the UAT (User Acceptance Team) will do that, so I go around trying to break the database. I do tasks in ways that are possible, but not the way the developers had in mind – you find loads of bugs that way. I also look for aesthetic bugs, does the mouse pointer do the right thing, do these boxes move correctly, does the right-click function bring up the right menu?

I ask all these things, because there are quite a few bugs in the game that I’m surprised are there. I’m not necessarily talking about 3D (works fine for me and would be hard to catch in testing given the billions of variables on each machine) and I’m not even talking about the injury bug, but the little things like the texxxxxt bug, the way the mouse pointer reacts when dragging the match speed slider and so on.

So, I’m not wholly convinced by the testing process, and I’d be interested to know how it was carried out if any of that information is available?

Does anyone else have any thoughts on the testing processes?

At work, I have a reputation for being a bit of a bug catcher because I always find major bugs and loads of minor bugs. One of the reasons is that I rarely test the database in the way the developers want – I figure the other members of the UAT (User Acceptance Team) will do that, so I go around trying to break the database. I do tasks in ways that are possible, but not the way the developers had in mind – you find loads of bugs that way. I also look for aesthetic bugs, does the mouse pointer do the right thing, do these boxes move correctly, does the right-click function bring up the right menu?

I have a similar reputation for breaking our software in ways that our normal testers don't manage, because I'm pedantic We only have ~10 people working on our software though so our testing has to be short and sweet!

Would be interesting to know more about the testing process, although I doubt we'll hear details!

The thing that annoys me most about a new FM release is the people who come on and blame the testers for not spotting the most obvious of bugs.

I know from my time as a closed beta tester for FM that it was a very frustrating procedure because we would very often highlight bugs in the beta demo, bring them to the attention of SI and then find that they hadn't been fixed prior to release (usually because they weren't deemed gamebreakers).

So the problem almost certainly isn't with the testers missing things - it's with SI's failure to fix them before releasing the game.

You have to remember, the testers (especially those forum members on the closed beta) are people who know they're going to be playing the game themselves for the next year and they're absolutely determined to point out any bugs they can in order to make their gaming experience as smooth as possible.

I too am a Database Administrator where I write programs that hook into various DB's and I have to go through a huge process of testing before our apps are released to my users.

GillsMan, I too asked in a thread about SI's methodology and just how they test (they report to have various labs for testing) but got no reply.

There are 2 ways to test the app and that is the way you know it should work ie systematically and then the other way of "trying" to break it on purpose as you have said.

So I too would be interested in SI's testing procedure.

But GillsMan, the text bug and the mouse cursor issue is not on all systems (my text and mouse are fine) and I suspect that hardware/software issues that SI did not test come into play and you can be more forgiving as they cannot test EVERY config.

A reply from SI would really be worthwhile.

It would be very interesting to hear from SI regarding this.

Although I love 08 and I have no doubt I'll also love 09 I really can't comprehend how the closing down problem last year and now the player condition/forward runs issue have made it into the gold version of the game.

I'm sure SI's testing methodology is fine, and I would be amazed if things like the text input bug weren't reported. I would guess they weren't fixed because they were probably introduced late in the day and the risks of fixing them outweighed the benefits - i.e. it would be significantly worse if they reduced the keyboard sensitivity such that on certain systems it didn't accept keyboard input at all, that's a clear gamebreaker rather than the minor annoyance it is now.

It's a demo, that means everyone is basically testing it.

I think Ando's spot on tbh. I used to games test for Sony some years ago and even when testing console games, which in those days had to be in a very stable and finished condition to have chance of getting through submissions, many issues would be found and logged but not fixed for the final release.

This would be for many reasons:

- The defect is simply deemed too much of a low priority to risk delaying the release of the game by attempting to fix it. Fixing any bug carries the risk that the fix may affect other areas, perhaps causing a more severe problem then previously.

- The defect is in an area of the game that could have a high risk of being affected by code changes. Again the severity of the defect will have to be weighed up against the risks around fixing it.

- A fix was attempted for the defect close to release time but this caused more problems than it fixed. Again the decision will have to be made as to whether the original defect was severe enough to plow forward, fix the new problems and risk slippage in the schedule. Or would it be more sensible to revert the code back to what it was before the fix was attempted.

- The publishers are really pushing to get the game released, delays have already been incurred due to fixing defects and now decisions have to be made as to what can be left in for the time being in order to meet the publishers demands.

These days it is a lot more acceptable to leave defects in games as patches can be brought out. A patch will no doubt be brought out for the full game very close to release day as a result of the feedback on these forums from the demo. It's just a shame that people don't just report the problems they find in the free demo and then see what state the full release is in once that early patch is applied rather than shoutng the odds after 10 minutes of trying the demo.

It would be very interesting to hear from SI regarding this.

Although I love 08 and I have no doubt I'll also love 09 I really can't comprehend how the closing down problem last year and now the player condition/forward runs issue have made it into the gold version of the game.

That'll be the same thing - the risks of fixing it far outweigh the benefits. If you just make forward runs less tiring, then players will be fitter later in the match, they'll probably contribute more to the game, so scorelines will probably go up a bit. So something will have to be done to compensate for this. The whole match engine will need throroughly retesting and possibly rebalancing. It was probably the same with closing down, if you make defensive play better, scorelines will go down and so something else will need to be done to compensate.

It's a demo, that means everyone is basically testing it.

No! The demo is the finished product, what you buy in the shops is the same as the demo.

No! The demo is the finished product, what you buy in the shops is the same as the demo.

Barring the demo finished after 6 months of course!

I too work as a (automated) software tester and am amazed and what gets released from SI. The patches are great but the COTS product is clearly unfinished. Of course they have deadlines to meet and there is a lot of money at stake so from my perspective it is better SI release and stay in the business than delay and struggle as a viable business. Of course they could lose customers which would affect them but I think the fan base is too big for that.

I am not sure if they now take a break or go straight into the next version but it seems every year we relay on patches to get the game as it should be.

I too am interested in what defect tracking and test case management they have in place. I have a feeling a lot of testers are probably happy to be playing the game rather than test.

No! The demo is the finished product, what you buy in the shops is the same as the demo.

If this is true, i'm gonna cry.

No! The demo is the finished product, what you buy in the shops is the same as the demo.

But early patches can be released on the back of feedback from the demo.

But early patches can be released on the back of feedback from the demo.

Correct but GillsMan is asking about how it is intially tested.

If this is true, i'm gonna cry.

It is true, but SI have said they will release a patch on day of release.

Instead of asking how it's tested you should be asking yourselves what's different about it from the applications you test so laudably.

Therein lies the answer to your questions.

The current testing methodology means very few things actually get missed, but no tester sees the whole finished product until it's very close to gold.

That rules out almost all testing for long term imbalances other than soak testing.

Soak testing gives stats but doesn't test realism achieved in game play on the way to those stats.

Testers always complain about injuries:D injury incidence and length is set to reflect real life, actual game play often unbalances that especially for very short periods and those short periods are much more likely to be in the very beginning.

It almost seems we need an "injury amnesty" for the first season.

Imo the biggest handicap to even more effective testing is the paramount need for security, sad but true.

For that and other good reasons code lock has to be as close to release as possible, so all testing is to an extent on the fly.

Also the analogies with testing limited issue software shouldn't even be made, they aren't being developed to run on hundreds of thousands of different hardware configs which also include hundreds of thousands of different possible software conflicts.

That'll be the same thing - the risks of fixing it far outweigh the benefits. If you just make forward runs less tiring, then players will be fitter later in the match, they'll probably contribute more to the game, so scorelines will probably go up a bit. So something will have to be done to compensate for this. The whole match engine will need throroughly retesting and possibly rebalancing. It was probably the same with closing down, if you make defensive play better, scorelines will go down and so something else will need to be done to compensate.

I agree that fixing something may have a knock on effect on something else.

Whilst I'm not going to bleat about the game being unplayable (mainly because that's utter rubbish) these 2 issues have signifficantly spoilt my enjoyment of the initial releases.

Having said that, fair play to SI in saying a release day patch will be available. Fingers crossed this will sort the conditioning problem out and we'll have another splendid edition of FM to enjoy.

It is true, but SI have said they will release a patch on day of release.

I bet SI are working around the clock to get it right so I feel bad offering any criticism their way.

The current testing methodology means very few things actually get missed

Really?

So all the really glaring bugs were caught by the testers?

^^^ And that is what we are asking about, how can obvious bugs make it through....

Really?

So all the really glaring bugs were caught by the testers?

I guess that's something we'll never know.

why cant they release demo or something before the game goes gold so then they can catch those bugs fix before goes too the manufacturers.i know sounds easy doing it im sure its not thats simple but must be way around it.

II too am interested in what defect tracking and test case management they have in place. I have a feeling a lot of testers are probably happy to be playing the game rather than test.

No tester gets to play a single uninterrupted game for long enough to make that a valid comment, I'm amazed that all these people who test software are unable to understand the difference between what they test and a product like FM.

I've tested on other big games and also on big commercial products as well as much smaller specialist software.

The games were always like this but with the important difference that none were ever supported post release like FM is.

I think if there's one point you may consider, that's not entirely in the defense of SI, but merely the nature of the product:

On the first day of release, more play time occurs than throughout the whole test cycle.

While it's easy to spot bugs as you go along, the lack of linearity means that one user may perform a combination of operations that wouldn't have been considered by the test team - that's not to say it's the test team's fault, it could be an irrational combination. However, such means certain bugs may go overlooked and never get logged.

Ultimately the game does get a lot of testing, but there are only so many hours in a day to both log, fix and regress bugs.

No tester gets to play a single uninterrupted game for long enough to make that a valid comment, I'm amazed that all these people who test software are unable to understand the difference between what they test and a product like FM.

I've tested on other big games and also on big commercial products as well as much smaller specialist software.

The games were always like this but with the important difference that none were ever supported post release like FM is.

Kriss,

I understand what you are saying in the difference in what we test and FM BUT that still does not explain glaring bugs that GillsMan brings up and he has not related a few other "obvious" issues.

We would love for SI to come on and say "This is how we test...."

Really?

So all the really glaring bugs were caught by the testers?

List those you would have expect to definitively have been caught.

I've already explained injuries, they're a real annoyance but can be adjusted quickly, as I said testers always complain about them but the proof of that pudding is always in the mass eating.

I do mean bugs as opposed to the graphics card issues, I always foresaw them as I've seen an identical scenario before, then as now it's a case of working through each type with advice and where possible mods to enable their functionality to operate properly.

perhaps we can argue that it wasn't tested thoroughly enough. i mean, the bugs are there. we can't fault SIs dedication though.

i've often said though, that testing a proram which largely runs via user input is much easier than testing something that has alot of things going on under the bonnet which has nothing to do with user input. which is how i see FM as being.

click button,navigate from one page to another, check database updates, attempt to enter bad data and data on the boundaries of what is accepted etc can be done in a very structured way.

testing all potential code path ways when 1 path way may only run when 10 variables all match correctly is not as easy to test in a structured way. when you have potentially hundreds of different outcomes dependant on the combined outcome of multiple variables it gets harder

Kriss,

I understand what you are saying in the difference in what we test and FM BUT that still does not explain glaring bugs that GillsMan brings up and he has not related a few other "obvious" issues.

We would love for SI to come on and say "This is how we test...."

He didn't define any glaring bugs, he made a generalisation.

Knowing how they test won't help you, it's as effective as that method can be and isn't going to change in procedural terms, because the only perfect way is to code lock months before release and test and fix for months/years from then till release, can't happen/won't happen.

List those you would have expect to definitively have been caught.

I've already explained injuries, they're a real annoyance but can be adjusted quickly, as I said testers always complain about them but the proof of that pudding is always in the mass eating.

I do mean bugs as opposed to the graphics card issues, I always foresaw them as I've seen an identical scenario before, then as now it's a case of working through each type with advice and where possible mods to enable their functionality to operate properly.

http://community.sigames.com/forumdisplay.php?f=107

He didn't define any glaring bugs, he made a generalisation.

Knowing how they test won't help you, it's as effective as that method can be and isn't going to change in procedural terms, because the only perfect way is to code lock months before release and test and fix for months/years from then till release, can't happen/won't happen.

BGS/Eidos are doing that....delaying release to make sure it is right...

testing all potential code path ways when 1 path way may only run when 10 variables all match correctly is not as easy to test in a structured way. when you have potentially hundreds of different outcomes dependant on the combined outcome of multiple variables it gets harder

The only disagreement I have with that is your "hundreds of outcomes"

Hundreds of thousands is the phrase you were looking for

BGS/Eidos are doing that....delaying release to make sure it is right...

Errr no, they haven't code locked yet, and they won't till just as close to release as SI do.

They're delayed release will follow the same path FM's has done and CM's have also always done. I.e. it will release with bugs.

testing all potential code path ways when 1 path way may only run when 10 variables all match correctly is not as easy to test in a structured way. when you have potentially hundreds of different outcomes dependant on the combined outcome of multiple variables it gets harder

Doesnt explain everyone wearing pink boots

or why penalties are 23-24

or shots hitting the crossbar 5 times in a game

or players standing doing nothing for 5 seconds

These issues can clearly been seen from everyone starting the demo.

The only disagreement I have with that is your "hundreds of outcomes"

Hundreds of thousands is the phrase you were looking for

i always like to err on the side of caution

Errr no, they haven't code locked yet, and they won't till just as close to release as SI do.

They're delayed release will follow the same path FM's has done and CM's have also always done. I.e. it will release with bugs.

I bet they will code lock and fix bugs for a lot longer than SI have done before releasing it...

how can a tester miss the injury bug if there testing it surely must have noticed it

Doesnt explain everyone wearing pink boots

or why penalties are 23-24

or shots hitting the crossbar 5 times in a game

or players standing doing nothing for 5 seconds

These issues can clearly been seen from everyone starting the demo.

it maydo if these are dependant on the many number of variations in team tactics/player tactics combined with player stats etc.

pink boots i dont know. maybe it leaked out of the code that was designed as an 'easter egg'.

what i will say though is just because you think its easy or it must only be a case of cliking two buttons to get the error because you think thats all you did then you are very much mistaken. there's alot going on in the game that you are not privy too.

Doesnt explain everyone wearing pink boots

or why penalties are 23-24

or shots hitting the crossbar 5 times in a game

or players standing doing nothing for 5 seconds

These issues can clearly been seen from everyone starting the demo.

It's not to say they haven't been spotted.

Ultimately if your 24 hours from gold, you don't want to be messing about with the fundamentals of on-pitch AI.

I'm not saying it's entirely justifed, but you do have to be sympathetic to the situation Dobs is in.

Doesnt explain everyone wearing pink boots

or why penalties are 23-24

or shots hitting the crossbar 5 times in a game

or players standing doing nothing for 5 seconds

These issues can clearly been seen from everyone starting the demo.

So why have I only seen the players rooted one? Which has been the same in 2D for at least two versions, it's just you notice it in 3D.

I'd love some pink boots:D

I've seen two shots hit the crossbar and about 8 hit the post.

how can a tester miss the injury bug if there testing it surely must have noticed it

people complained about an 'injury bug' in fm08. i noticed notihgn of the sort. alot of people didn't either.

i'm not saying there isn't such a bug because there may be. but i do like the fact that some people say

'i've been playing this game for 2 hours and already i've found an injury bug.'

like 2 hours is enough to check whether its their training, their tactics their coaches their physios the players injury proness rating the fact that its pre season etc etc.

how can a tester miss the injury bug if there testing it surely must have noticed it

Go back and read the whole thread mate.

its gonna be fun when full game is out on 14th people play longer than 6months what else will thwy find.

No tester gets to play a single uninterrupted game for long enough to make that a valid comment, I'm amazed that all these people who test software are unable to understand the difference between what they test and a product like FM.

I've tested on other big games and also on big commercial products as well as much smaller specialist software.

The games were always like this but with the important difference that none were ever supported post release like FM is.

I was more referring to beta testers who I presume have complete builds rather than in-house testers but I hope integration of the entire system is carried out extensively. I am aware of differences between FM and the software I test and our system is much larger in scale with less staff.

I'm with Kriss here, the only bug I've noticed has been the text ddoouubbllee entry one, which only occurs for certain setups and is solved by running in a window.

I have a very good physio, and I'm not seeing an unusual level of injuries.

I've certainly not seen an issue with penalties, or penalty shootouts (losing to Aldershot 5-4 btw )

I've also not seen a particularly high number of shots hitting the woodwork.

Things are not always clear-cut.

I'm with Kriss here, the only bug I've noticed has been the text ddoouubbllee entry one, which only occurs for certain setups and is solved by running in a window.

I have a very good physio, and I'm not seeing an unusual level of injuries.

I've certainly not seen an issue with penalties, or penalty shootouts (losing to Aldershot 5-4 btw )

I've also not seen a particularly high number of shots hitting the woodwork.

Things are not always clear-cut.

You haven't noticed players with forward runs set to often becoming tired really really quickly?

List those you would have expect to definitively have been caught.

I've already explained injuries, they're a real annoyance but can be adjusted quickly, as I said testers always complain about them but the proof of that pudding is always in the mass eating.

I do mean bugs as opposed to the graphics card issues, I always foresaw them as I've seen an identical scenario before, then as now it's a case of working through each type with advice and where possible mods to enable their functionality to operate properly.

Kriss - an entire first team squad out injured isn't obvious until the public get their hands on it?

I'm sorry but you're just blindly sticking up for SI here and ignoring the facts.

Kriss - an entire first team squad out injured isn't obvious until the public get their hands on it?

I'm sorry but you're just blindly sticking up for SI here and ignoring the facts.

if it was as simple as that then presumably SI wouldn't need a structured testing process to find it. which can only mean your sayoing they noticed it but ignored it.

Same topics every year :S

Same topics every year :S

Exactly! That is why we were asking about the testing process cos it never seems to improve, in fact it is getting worse.