This post really belongs over at The Philosophy Smoker, but the comment that I started to write there got too long. A lot (OK, a high proportion) of jobs in philosophy these days ask for evidence of excellence in teaching, without specifying what this means. I suspect that's because search committees want whatever you've got. But perhaps it's possible to be more specific than that.
CV: It hasn't mattered to me how someone's CV is organized, as long as I can find what I'm looking for. Personally, I would put research-related material such as publications first, if only because there's usually less of it than of other things (such as graduate-level courses taken), and you don't want it to be hidden.
Teaching awards: I expected these to make a bigger difference than they have ended up doing in my thinking, for several reasons. One is that very few applicants have ever won one, so you can't insist that only those who have will make the shortlist. Secondly, the information provided about them varies and is usually minimal. Some are university-wide awards, with lots of competition, while others are department-wide with who knows how much or how little competition. Some are called "teaching fellowships," but it's hard to know how competitive or prestigious these are. Some were won last year, others several years ago. Some people who have won awards seem by every other measure to be no better than, and perhaps worse than, other applicants. Still, having won a teaching award of any kind, or even having been nominated for one, is a very good sign in my book. Definitely a plus rather than a requirement though.
Student evaluations: These came to seem more useful than I had expected. If your average scores out of 5 for things like "overall teaching effectiveness" are in the 4s then this is a big plus. Not providing any such scores makes it hard to compare you with other candidates, which makes it harder for you to seem better than the competition. Not providing recent evaluations looks odd, as does providing them from only one course (assuming you have taught more than one course). I pretty much ignored seemingly cherry-picked student comments, but read through them all if all were provided. Complaints like "too much reading" and "graded too hard" are good to see (within reason), and can make up for relatively low scores, whereas complaints like "course was too easy" and "the instructor sometimes let the class get out of hand" look bad, and can make good numerical scores less impressive. Of course, there are well known problems with student evaluations, but I don't know of any better way to judge someone's teaching, especially before you've had a chance to meet the person or see them teach.
Reports on teaching by observers: These usually seem helpful, but the more detailed ones are certainly preferable to the brief, sketchy ones (which are not very helpful, and can even seem damning in their apparent inability to find much good to say). I know from experience that people will rave about the teaching of someone they like, and find fault with anything done by people they dislike. And letters about teaching tend to rave in the same kind of way that letters about research do. So I don't put too much weight on these reports. But good reports are certainly better than either bad ones or none at all.
Other evidence: A really well written teaching statement and/or evidence of taking extra steps to improve one's teaching will help (if someone like me is reviewing your file, that is). Some people combine these into a kind of narrative, combining their approach to teaching with the story of how (and why) it has developed. This can be very effective, especially when combined with other evidence that you are now a much better teacher as a result.
Syllabi: As with statements of teaching philosophy, I find myself not caring much about these in most cases. I look at them, and note anything especially good or bad that sticks out. But most seem pretty similar. "Good" here refers to unusual choices that I like, "bad" means elements of a course that I think would not work with our students.
In statements about someone's teaching (one's own or someone else's, that is) I think a good approach is to provide a kind of list of good points about it and then illustrate each one with an example or two. So you might end up with a paragraph on dealing sympathetically with students' problems, another on designing imaginative assignments, another on having infectious enthusiasm, and so on. Ideally these would be put in some sort of non-random order. That can't be easy to do, but I have seen it done. And it always impresses me.
Do candidates really need all this? No. If you sound fantastic enough from a research point of view, you might need only a few of the things I've said it's good to see. And no one has them all. But if your research record is less stellar than some other people's then enough of these good things might get you an interview despite that fact. There is no standard set of materials to provide, so it would be harsh for anyone to hold it against you that you left this or that out. But some things are pretty standard (statement of teaching philosophy, one letter exclusively about how good a teacher you are, some student evaluation data). Including weak material might well hurt you, but not including it might seem like a red flag to some people. And too many red flags will hurt you too.
All I can do is wish you luck.