I'm seriously considering going roll-under for my edition-less houseruled D&D. I generally like roll under: it's simple, clean and gives you a good idea about your chances of doing something.
There is precedence for this: older editions of D&D, before there was an unified resolution system offered the same option: roll under your stat to succeed. So this is actually nothing new or unusual. This makes it kinda like a less granular percentile system where the smallest possible modifier to your chances of success is 5% (+/-1). Looking at the Warhammer RPG line that's not unprecedented either as the smallest possible modifier for anything there is 5%, too.
One problem is that characters can start out unusually powerful. Rolling an 18 on a stat with 3d6 is rare (0.46% chance, to be precise), but when it happens it means you have a starting character with a 85% chance of success, which is pretty darn high. It leaves little room for improvement (without the application of penalties).
Another problem is the disparity of systems: while ability checks are roll-under, combat is still roll-over in older editions. I don't like that kind of diversity, I want my game to use the same rolling mechanic, for simplicity's sake if nothing else. The solution would be to make combat roll-under as well.
Without taking into consideration class bonuses, the universal modifier of meelee rolls has always been Strength. So in a roll-under combat system, rolling under your Strength constitutes a basic attack roll. The enemy's armour would thus need to act as a modifier to the attacker's roll.
Again, there is precendence for this. I'm not really inventing anything new, just rationalizing what exists. Under the old THAC0 system that many find confusing, armour did not modify the "difficulty" treshold needed for a hit as in the newer roll-over editions, it modified the attacker's ability. That's why lower, negative armour class was better: it represented a penalty.
Clearing away with the confusion that is THAC0, an enemy's armour class can simply be applied as a penalty to the attacker's base Strength check.
A quick bit of math shows that a base armour of 0 isn't justified in the roll-under system, so let's make it 2 (thus the percentages correspond to the old roll-over probabilities). Or to put it otherwise, all attacker's base attack value is [their strength-2].
I like this. It's clean and concise and simple. I have just three problems with it:
-adding is, in general, easier for people than subtracting
-I have already invested some time into figuring out the specifics of roll-over mechanics
-you lose the excitement of rolling a natural 20
This last one bothers me the most. I really like that special feeling and I've made "critical successes" a big part of my system. Making it roll-under robs me of that. You can make a natural 1 a critical hit but that just doesn't have the same vibe.
So that's the last big decision I have to make: simple clean roll-under or the exhilarating feeling of a natural 20...