I've been managing in Holland for three seasons now, but still haven't really got my head around the playoff format.
I've read up on it at wikipedia and eurosport, but FM just doesn't seem to fit into the descriptions, so I can only assume that you've got the playoff system all wrong.
For instance, last season, Feyenoord finished third, yet did not quality for Europe at all, no champs league, uefa cup, not even the intertoto cup.
The league table looked like this:
PSD (champs league)
AJAX (champs league)
FC Twente (Euro Cup)
Vitesse (Euro Cup)
MVV (Intertoto Cup)
I think the main problem could be that in real life Holland seem to be given 3 Uefa Cup places, where as in FM they're only getting two, but it's very annoying to have strange teams qualifying (or not qualifying in Feyenoords case) for Europe each season.
I understand that the playoff system makes it so that just because you finish above another team, doesn't guarentee you european football, but in real life, all the teams placed 2-5 are guarenteed uefa cup, so really it should be that way in FM.