si funciona con not
Código HTML:
<html>
<head>
<title>Untitled</title>
<script src="http://code.jquery.com/jquery-latest.js"></script>
<script type="text/javascript">
$(function() {
alert($('body').find(":not('[id^=select_]')").length);
});
</script>
</head>
<body>
<input id="select_1" value="input select_2">
<input id="xxx1" value="input xxx1">
<div id="select_2">div select_2</div>
<div id="xxx2">div xxx2</div>
</body>
</html>