The Avengers are known in Marvel as "Earth's Mightiest Heroes" and there is a saying that you can't be a hero and taken seriously if you've never been an Avenger. The Avengers are the Justice League of Marvel and represent the best of the heroes. But, there are some people that represent everything the Avengers aren't or just shouldn't be Avengers.
Punisher:
For those of you who don't know the Punisher, he is a vigilante war hero and veteran from the Vietnam War. After his family died after getting caught in the crossfire of a mob war, Frank Castle became the Punisher and set out to rid the world of organized crime. Now, unlike a lot of heroes, Frank Castle will kill and have absolutely no remorse about it. The main reason why Punisher would never be an Avenger is because of Tony Stark. There is a possibility that Cap or someone could talk Punisher into using nonlethal ordinance (or just give it to him and tell him to suck it up), but Stark is an arrogant jerkwad and no way he can work with him on any level. It was shocking that him and Dr. Strange lasted as long as they did.
But...:
In a perfect world, there would be an all kill squad of Avengers along with Elektra, Daken, Moon Knight, and Bullseye.
Deadpool:
Well, first off, everyone on the Avengers except Steve Rogers hates him. When the Avengers look at Deadpool they don't see the soldier who does follow orders and is the perfect teammate to have when the chip are down. All they see is a remorseless killing machine who can't stop making jokes. More than that, he wouldn't be accepted by the public and the Avengers are all about their public image. They feel like they should be the shining and unblemished heroes. I say that fully knowing about the Dark Avengers and loving that roster.
But...:
Even though Luke Cage may not like Deadpool, he does believe in second chances. He'd make a perfect addition to Luke's Mighty Avengers. He would humor to the book and Deadpool might grow being around people like Luke Cage and She-Hulk.
Cyclops:
The leader of the of the X-Men for years and maybe their most important member. It's not that Cyclops isn't good enough. He's arguably the best leader in the Marvel Universe. The problem with Cyclops is that he means too much to the mutant community. Seeing Cyclops working with Captain America or the Avengers is one thing, but to have him actually as a member of the Avengers is another. Cyclops is just one of those guys who should only represent the X-Men.
But...:
No buts with Cyclops. Even if the Avengers put together an all mutant team, Cyclops shouldn't lead it. It would almost seem like they were just giving mutants a bone. Even the Uncanny Avengers seemed like Marvel was forcing the issue of making mutants part of the Avengers team. There was even a press conference to announce mutant involvement. Meanwhile, Beast, Wolverine, Quicksilver, Sacrlet Witch, Justice, and Namor were all mutants. So there was no need for the show. Cyclops joining their ranks on any level would be a mistake.
Just add Magneto to this list for the same reasons.
Mystique:
While villains have been members of the Avengers and reformed (Vision, Quicksilver, Scarlet Witch, Wonder Man), there is no way in Hell that Mystique reforms or is put on a leash short enough to let her be an Avenger. She is all about herself and how she can get over. Mystique would let the entire mutant race go extinct if it meant she could survive and make some money out of it.
But...:
For a while, Mystique was working for Professor Xavier as his secret agent for an extended amount of time and she was under Norman Osborn's thumb for a bit as well, so she can be controlled. The problem is, when she gets control back, she is going to get back at everyone who controlled her and she really isn't worth the trouble. No matter how great an asset she could be.
Cable:
Maybe the best soldier in Marvel ever. He has fought wars in the past, present, and the future and lived to fight the next war. Not even losing his powers or death can stop him from completing his mission. Cable doesn't think irrational on the battlefield and doesn't let his feelings for his teammates or loved ones interfere with he knows needs to be done. It's that reason that why Cable would not make a good Avenger. If the world is at stake, Cable will sacrifice teammates so the world keeps spinning. The Avengers always try and find the other way out. Unfortunately, sometimes that means dealing with these threats again because they couldn't make the hard decision.
But...:
Cable is the perfect person to join the Illuminati (New Avengers comic). Iron Man, Mr. Fantastic, Dr. Strange, Black Bolt, Namor, Beast, and Bruce Banner. They deal with problems that could destroy the world and maybe even the universe and sometimes these decisions need to be made immediately. Cable is someone who thrives on moments like that. He just does what needs to be done and worries about the consequences later.