Executive Hires: The Case for Extreme Referencing

Hiring a strong executive management team is one of the most proven ways to ensure startup success. Yet, hiring executives is a very different undertaking than hiring for other roles in your company and requires a different process, skill-set, and time commitment. There are three reasons for this:

• First, the stakes are higher. The right or wrong executive hire can have an immense and lasting impact on the trajectory of your company.
• Second, executives can be much more highly skilled in their area of expertise than the founder, making it very difficult for the founder to accurately assess a top candidate.
• Third, as a founder you often bring a strong bias toward hiring. You are usually recruiting at a time when you have a great need for that leadership role and you’re likely evaluating someone who comes in with strong (on-the-surface) recommendations and an impeccable resume.

I believe the only way to truly evaluate an executive candidate is through what my partner at Matrix, Josh Hannah, refers to as “extreme referencing.” This process goes beyond the compulsory reference check and instead has the founder spending many hours finding the right people to talk to and asking very tough questions. Josh has written the post below to help founders implement extreme referencing for their executive hiring. The example reference questions he offers are particularly useful. We’d love to hear in the comments any reference questions that have worked well for you.

Enjoy,

David Skok

*****************

Introduction

For CEOs, especially first timers, one of your most difficult tasks will be hiring truly great executives. There is no substitute for experience in hiring. I learned this the hard way when I was a first time entrepreneur with absolutely no hiring expertise. After hiring more than 100 people – and making a lot of mistakes – by far my best executive hire was my last one: David Yu as CEO of Betfair.

The key lesson? If you are a young or first time entrepreneur you need to surround yourself with investors, advisers and executives who are experienced and talented, especially when it comes to hiring other top executives.

When hiring an engineer, they will have a body of work that can be evaluated and tests that can be given. Technical talent can be evaluated for the first 90 days and replaced if they disappoint. However, if you are hiring a CFO, CTO, VP of Business Development or top marketing and sales executives, it’s not so easy. These executives are sophisticated at selling their experience in a compelling way, and once hired it takes time to evaluate their effectiveness. The mis-hires often do parts of the job well, making it more complicated to remove them. And, once the decision is made to remove them, there is a much greater risk of damage to the organization.

But the good news is that top executives’ experience is also more suitable for extreme referencing, as they tend to have had extended professional interaction with a much greater number of people.

In my experience the leading cause of executive hiring mistakes can be traced to waiting to make (perfunctory) reference checks until the final step in the hiring process, perhaps even after the decision has been made.

It’s a tough pitfall to avoid. You invest great effort to get your executives and investors to interview a number of candidates. You laboriously negotiate to pick your favorite and only then do you ask the candidate for a few references to call. After all this time and effort, which you feel is a distraction from your true job of running the company, you call those references with a confirmatory bias, practically begging them to tell you that the interviewee is a great marketer, so you can get her on board and get back to work.

Your real work as CEO is to infuse top talent into the organization. You need to block time on your schedule to hire. If you don’t have the time, it is doubly important to build the team so you can hand off other work to focus on making the best hires.

Why References are So Important

Resumes, CVs and interviews are all sales pitches. The candidates have carefully crafted the attributes they show you in order to seem perfectly suited to the job.

Some typical issues in a cleverly crafted executive resume:

• Omits or minimizes negative experiences and cherry-picks the most compelling successes;
• Blurs a candidate’s role or responsibilities to make it tough to evaluate true contributions to an initiative, campaign or idea;
• Takes credit when they are just part of a rising tide, i.e. everyone who worked at Google from 2000-2005 delivered unbelievable numbers. Not all of them were rockstars.

By all means build your interviewing skills so you become adept at smoking out pretenders through great questioning, but know that as the candidate’s level of sophistication increases at the executive level, so too will your difficulty in spotting the posers.

Why rely on a two-hour chat with a candidate to try and divine their strengths, personality, ambition and drive when you can get a firsthand account of how they performed in the real world? Why not speak to the people who watched them perform and get their evaluation?

When you think about the most critical attributes you are trying to evaluate for a startup, they can be especially hard to divine from a resume or interview. Skills, experience and intelligence are well suited to interviews. But the things that cause someone to thrive at a startup – drive, teamwork, leadership and cultural fit – are much harder to foretell.

But if reference checks are so important, why do many smart founders reference check so poorly? A few reasons:

• They know people given by the candidate as references will have a vested interest in helping the person who referred them;
• They believe many references are reluctant to say anything negative due to policy or liability concerns;
• Checking references is very time consuming.

These concerns are real. The first two can be mitigated with good technique. On the third, I’m sorry but I’m going to make it worse – I need you to spend even more time on references than you do today.

Reference With the Right Mindset

Before we jump into a how-to guide for extreme referencing, it’s important you go in with the right expectations. Here are my top 6 recommendations for how to approach your extreme referencing:

1. Be mentally prepared to eliminate a candidate. Go back to the drawing board if the references are less than stellar. Expect at least a 50 percent chance of eliminating the candidate based on references. If you are not in this mindset, you will only hear what you want to – that you’ve found the perfect person.
2. Allocate 10-15 hours of total time to spend checking each candidate’s references. This includes research such as question preparation and tracking down additional references as well as about 30 mins per call for 5-10 calls. You need to block this time on your calendar. To repeat: if you spend less than 10 hours on extreme referencing a top hire, you need to come up with better questions and find more people to call.
3. Look for negative information. Positive information about the candidate is expected and should largely be devalued. Because of a natural positive bias in reviews, you have to assume that a lot of the positive feedback may be overstated. Hence, if someone with no clear axe to grind with the candidate gives you relevant negative information, you need to take this very seriously.
4. Notice what is NOT being said. Narrowly defined praise can often be used to show you the weaknesses of a candidate. Sometimes when a candidate is “smart”, it may mean he lacks drive or interpersonal skills.
5. Keep making calls until you hear a consistent story. When you start hearing the same weaknesses and strengths repeated about a candidate, it’s a good sign you’ve completed your extreme referencing.
6. You should not outsource reference checking, especially to executive recruiters. The incentives are all wrong. Despite the fact that someone else may be skilled at it, you cannot outsource this critical bit of judgment in shaping your team.

Would you let a recruiter conduct all the interviews, and pick your new CTO by reading the recruiter’s notes? Obviously not, and you need to be doubly reluctant to rely on your recruiter to conduct your reference checks, a task requiring far more nuance and judgment.

A Step-by-Step Guide to Extreme Referencing

There are four basic mistakes people make when checking executive references: 1) they call too few people, 2) they don’t talk to the right people, 3) they don’t ask the right questions, and 4) they don’t have the experience to best interpret the answers they receive. Below is the step-by-step guide I share with founders to help them avoid these mistakes.

I. Build a large list of the right people

Your goal of extreme referencing is to discover new people to talk to until you find someone who has nothing to lose by telling you the truth about the candidate. You may sometimes get an unbiased view from the list of people the candidate gives you to call. This can happen due to skilled questioning, or just be in the nature of the individual. However, one of the main goals of calling the candidate’s references directly is to find out who else the candidate worked with, and in what capacity, so you can track down those people and talk to them.

Step 1: You should begin by setting the candidate’s expectation that you will be referencing them extensively from the list they give you and also referencing anyone else you find who is relevant. This is the point at which you need to clarify with the candidate if there is anyone you should not call. Be sure to take a hard look at the reasons for any exclusions.

Step 2: Make a list of people you are going to call. Start with the names the candidate gives you. You should plan to talk to all their direct supervisors, some peers, and at least a few people who reported to them. If there are any clear omissions given your knowledge of their work history, be sure to ask for those names.

Step 3: As you call candidates on the list, names of other people will surface. Add them to the list. Prioritize the list from most to least recent and most to least relevant. Work down the list until all your questions about the candidate are answered and you have clarity about the hire.

If there are people on your reference list you want to call that the candidate or other references did not give you, there are a number of good sources for finding their contact information. Do not hesitate to cold call them. LinkedIn is a fantastic option for finding extreme referencing sources; this is the primary reason I have built out my full network on the site. When I want to reference a candidate my first step is to punch in their name and see who I know that is connected to them. This is your best chance to get reference gold: someone who knows the candidate well but knows you better. In addition to LinkedIn, other top sources include Facebook, Google, PeopleSearchPro.

Step 4: It is important to qualify the reference, especially to get a sense of how they rank employees or what “great” means to the reference. Are they used to working with really high caliber people? One tactic is to quickly cross-reference the reference when speaking to other references from the same company. Another is to ask yourself – would you hire the reference person?

Once you’ve done all this, you will have a long list of people to call.

II. Ask the right questions

You can and should ask many of the standard questions of a reference call. But, I want to pay special attention to two categories: key questions you might not think to ask, and a list of typical questions that are phrased in a way that’s different and more effective.

Step 5: Here are some key questions that you might not think to ask explicitly, often because you assume you already know the answer:

• Did Joe report directly to you? For what portion of his tenure at the company? Who else did Joe report to?
• To the best of your knowledge, did Joe resign, or was he asked to leave?
• Ask the recruit, bluntly, “What was the biggest mistake you made in the role” – then reference check against his answer.
• (To build your list of references) Everyone I speak with says such great things about Joe. Is there anyone Joe didn’t see eye-to-eye with? Why do you think?
• Why wasn’t Joe further promoted at your company? An interesting answer sometimes is that there wasn’t a spot (i.e. they would have to replace their boss). If that’s the case, what skills/ability does the boss possess that Joe lacks? How do you think Joe would perform in the boss’s role? (This one might be pretty hard to ask if you are talking to their boss).
• This question came from Brian Halligan, CEO of Hubspot: “One of my favorite questions is to give references a net promoter test: On a scale of 1 to 10 (with 10 being the highest), how strong is your referral. If less than 10 (I’ve yet to get a 10 on a blind reference), my next question is a simple “why not a 10?” After they get the first reason out, I ask why again and again until they run out of things to say.”

III. Ask the hard questions

It’s really important to ask hard questions.

You can elicit useful information by asking questions that have no easy way out, even in cases where the reference has a close connection to the candidate.

Step 6: Most people find it tough to ask hard questions for fear of seeming rude. If this is the case, it’s better to mitigate the rudeness by prefacing the question, rather than by softening the question itself. Here are some examples:

• Joe is by far the best candidate I’ve seen. Still, everyone has their short suits. What are Joe’s?
• We’re inclined to make Joe an offer, and I hope he will take the position. If he does, how would you suggest I manage Joe to get the best performance out of him?
• (If the candidate had peers in the role) You say, “Joe was good.” Compared to Tom and Mary was he clearly the best at the job? Or did they each have their strengths and weaknesses? What was Mary better at than Joe?
• Did Joe have setbacks? How did he handle them?
• Would you hire him again? If so, would you rehire him without hesitation? Create a role for him if you didn’t have one? Only hire him in the right role?
• How would you change the role we’re considering him for to best suit his abilities?
• If I were to hire Joe and six months from now you heard he was fired without other context, what would be your best guess as to why he was fired?
• What are Joe’s passions in and out of work?
• How hungry is Joe for results, for example, what is the craziest thing you’ve ever seen him do to achieve a goal?
• Joe mentioned the launch of the new website was one of his big achievements. What was his actual role in the launch? Was he solely responsible, or was it a team effort?

IV. How to evaluate the answers

The rest of this reference guide is brute force: if you work hard at it, I am confident you will be successful. But evaluating what you hear is an exercise in nuance and judgment, which you will develop over time as you do more hiring (and extreme referencing).

Step 7: If you’re still developing your judgment, lean on experienced and talented advisers who can help you piece through the information you collect.

After working with many founders to reference their executive hires, I’ve found these three approaches to be key to evaluating the data you receive from references:

• Startups have a lot of setbacks along the way. I think that digging into times of adversity or failure of the candidate can be enlightening. If they haven’t worked in roles with real setbacks, it is something to consider.
• One consequence of extreme referencing is that you will hear more bad news than you did previously. You need to have tolerance for some faults in the candidate as there is no perfect person. If he or she didn’t scale sufficiently at a company that grew past 300 employees, but you have just eight employees in your startup, it’s a risk you might be willing to take.
• When hiring into a young startup be especially vigilant on cultural issues. If you’re evaluating someone coming from a more established company, or if you are young founders hiring some “adult supervision,” you should work hard to get a good sense of the candidate’s drive and work ethic. Consider the impact of them leaving at 6 p.m. while the rest of the team codes until midnight.

As CEO of a startup, your top job is to infuse as much talent into the organization as possible. Make it a priority and devote energy to extreme referencing, and you will be doing your job to give your business the best chance to succeed.

Thanks to Ben Elowitz, Jack Herrick, Alastair Mitchell, Mark Peters, Thomas, Antonio Rodriguez and David Skok for their help on this post.  Special thanks to Irv Grousbeck, mentor to a whole generation of great entrepreneurs.