There are a lot of complaints about how long combat in 4e takes. There are some horror stories about a lvl 1 fight taking four (4!) hours. I have trouble understanding that. If you have any previous experience with D&D post 2e you shouldn't have much trouble adapting to the new system, and at low levels the fights are a breeze in my experience. At higher levels things obviously get slowed down, but really, was that any different in earlier editions?
Even so, there are a bunch of fixes for making your 4E experience even more action-packed, if you're not afraid to break the system a little. Note that I'm not super-experienced with the system, some of these are mine, some I've borrowed from elsewhere, some were playtested, some not.
1. Preeliminary: everyone at the table should be paying attention, thinking ahead and optimizing their tablespace and dice rolling. Indecisiveness and calculation is and always was the biggest time-sink of D&D play.
2. Use minions instead of normal monsters. They go down faster.
3. Use monsters with only one or two schticks and save more complex ones for special battles. As a DM you're also responsible for making your decisions quickly and having a focused monster makes it a lot easier.
4. Use the monster's bloodied HP value as their base HP and increase their damage output. They'll go down faster without losing edge. I don't have enough experience with 4E math to talk concrete numbers here, but if you google around you can find out more about it because people have been doing it for a while.
5. Use average damage outputs for monsters. Frex: 12 dmg instead of 2d8+3 dmg or use the table at the bottom of this article.
6. Group initiative. Players go together, discrete monster groups (frex: goblin archers, goblins skirmishers A, goblin skirmishers B) go together. When it's the player's turn that means everyone gets to play off their teamwork even harder.
7. Scratch the short rest/extended rest rules. Scratch the Second Wind power. Expand the action point economy. One action point = extra Standard action OR Healing Surge, two action points = refresh Encounter power, three action points = refresh Daily. Keep giving out one action point for every two combats, plus extras as personal quest/motivation rewards. This bypasses the whole "15 minute adventuring day" issue as well as encouraging players to rush into action.
8. Use goals other than "kill". When you have a different win condition than simply killing everything and everyone, or use morale rules, fights can be over even faster and are certainly more interesting.
9. Escalate. The longer the fight lasts, the crazier and more intense things get. Don't let it drag out, pump it up. This works from "berserk" monster powers where their damage output increases as their HP decreases or their comrades get dropped to escalating the whole situation of the fight with environmental dangers and greater overall fallout of the fight.
10. Simplify, optimize or avoid ongoing damage effects and conditions. These are a bitch to track. One way is to just round it up to however much damage it be over the course of a few rounds (count on your combats to last 3-6 rounds). Use tokens, stickers or whatever you can to make tracking easier.