Leading News Orgs to Water by Hacking Our Hiring
Session facilitator(s): Tiff Fehr, Ryann Grochowski Jones
Day & Time: Friday, 11:45am-1pm
Slides: Slide deck
Tiff: Hello. Welcome. Please join us. Pull up a chair. We do not have any activities planned, although we will ask you to chat with us as we go. So you don’t need to worry about who you are sitting near and if they look like good activity partners or not. Or who has Post-its and markers and who does not. This is one where if you choose not to speak, you are welcome not to speak. If you would like to speak, we recommend that you please don’t be the person always speaking. If other people would like to speak, you are welcome to wait until they have.
This is being live transcribed live by [pause for Maggie to smile and say her name, while typing it into the record] Maggie! She is awesome. We’d like toremind you this session is being transcribed, so please use on and off the record. If you are going to say something aloud, please, say your name so Maggie can appropriately attribute who is speaking.
Let’s get started because we have a lot to go through. Welcome to “Leading News Orgs to Water by Hacking Our Hiring”. (It is a functional title and it will work.)
Introductions. My name is Tiff Fehr and and this is Ryann Grochowski Jones. Did I say your middle name correctly?
Ryann: You did.
Tiff: We are here to summarize some interesting overlaps in what we have been doing around hiring in our news organizations. There are many colleagues, some here at SRCCON and even in the room, who have been essential to our teams and groups adopting these practices. So we want to give a shout out to our teams and colleagues who have helped spearhead these practices. We’re summarizing on their behalf because they cannot attend or didn’t want to talk in front of a group, which is understandable.
First we would like to know about who you are. Who is here from small newsrooms? [show of hands] Who is from big newsrooms? (I am myself.) [show of hands]
And we also would like to know what do you do, because it will inform how we adapt our material.
Who are editor types? [show of hands] Who are reporter types? [show of hands] Who are both? [show of hands] Who are all the roles? [all the hands]
Excellent. And how many are involved in hiring? [show of hands] How many of you are actually involved in your hiring process within your team? By which I mean you are part of the planning and executing? [show of hands] How many are just chucked into an interview? Meaning: you are parcipating in a one hour slot and that is about it? [show of hands]
Interesting. All right. Okay, thank you.
So every good meeting has an agenda and this is ours. We will talk about goals and tools, we will talk about our processes (and that will include sub-steps), we will do an overview and go fast. We have a lot but don’t worry, you are not meant to remember it. We will cover tactics for minimizing bias. We will talk about structuring and we will hammer on themes. And then break it down in hacks you can use now because obviously it isn’t a matter of going home and importing the process but picking up bits and pieces where it fits. And then at the end we will hopefully have time for discussion and questions. We do encourage questions as we can. Some we may ask to hold to the end because we will cover that area as we go.
So let’s jump in.
There are many goals for improving our hiring efforts. We will go over a few now, but the are many more.
You want to fight biases. We are working on recognizing how that shapes the people hiring and that is not necessarily a good thing.
We want to have a process that is worthy of our candidates. You have great candidates come in and you don’t want a shitty process. (I just swore on the record.)
We want consistency across the field of candidates and you want consistency for each candidate, so you are giving everybody an even playing field. Say you sucked it up one day and they sucked it up on another day day.
You want to include more about what the candidate can bring to the role. You have materials on paper but you want to dive beyond that and find out interesting things about them as a person and why they would be good as an addition to your team and team culture.
Predictable involvement for interviewers. Participating teammates will be eager to understand the expectations of them and understand why they are central to the process.
Process efficiencies can speed up hiring. These might take time to adopt initially but it will make subsequent efforts faster because people have an agreed idea on what is going on. Not necessarily for the candidates, but they might benefit too.
And you want to coach your team in incremental gains. You will iterate on the process and gradually improve as you go.
And you also want — this is a big piece — you want the process to help filter. It isn’t a good idea if you like all your candidates. And all of them advance to phone screens because you can’t decide. And then all advance to in-person — you are not making decisions along the way. You need to narrow the funnel so you are not taking up too much of the candidate’s time and not wasting your team’s time because you can’t make decisions. The whole point is to make sure you know your filter mechanism works strongly to reduce your applying pool to your finalists and you are making good decisions along the way.
There are many, many more goals. These are the ones we thought of. Every team has different goals they would add to this list and we could probably go on for another entire slide, but you get the point.
This may not be something that is part of your paycheck or listed in the job description but you need to make time for it and that is challenging. There is a lot of work with materials, e-mails and people and coordinating. That is before you do the interviewing, look at résumés, look at materials or do any assessments.
Getting all the information to the right people so they do the right things at the right time? That is more time-consuming than you would expect even if you have a very dim view of that already.
There are definite gains to be had in terms of team engagement. Introducing these improvements give people a reason to buy back into a process they maybe felt disconnected from. It means they are a little more involved when it comes to do their one-hour interview block. We will get to why they are more engaged in that in just a bit.
It should be more efficient to repeat the process.
First time through going to be painful. Subsequently, everything is smoother. You can have a team-wide discussion about changes you want to make rather than only have a couple people invested and a bunch of team members who are not part of that committee or effort. Or those who don’t have a strong opinion. It is worth it over all because you get better people, you feel better about the process, and it professionalizes something we take for granted right now.
Now we would like to talk about the tool agenda item.
These are the no-frills tools we use in our process. If you are on the Google suite or happy to use Google products, not all newsrooms are, we use Google Drive, Google Forms, Google Sheets, Pivot Tables.
If you are on the non-Google side of the house, any network drive is similar. You can do template e-mails if Google Forms or Survey Monkey are not your bag or technology you want to adopt. Collect that into a CSV and then you can pipe that through to data analysis of your choosing. (I would not be shocked if someone piped candidate information into R and turned it into fancy graphs.)
No matter what, you will be doing emails, scheduling, meetings, and more. You will need persistent organization, patience and diplomacy. Those are a given, regardless of the technology toolset you have. If you feel confidant with a good chunk of this, you have all the tools you need, in our opinion, to reboot or add new and productive twists to the hiring process without additional software.
There is additional software that can help with this. I think we have both tended to go around it.
Ryann: At ProPublica we use Screendoor which is expensive version of Google Sheets but has more functions. We want to talk about collaborative options that are free. But if your news organization has the budget, Screendoor is a pretty great program to use for hiring purposes.
Tiff: If your organization has a budget and aptitude for masochistic behavior you might have a program like Workday.
Andrew: I am Andrew. If any of you don’t know, I recommend Textio. You put your text into it and it helps you with [rewriting your job description to have more inclusive language].
Tiff: Ah right, that is indeed helpful. Job descriptions can be competitive, in terms of the inclusivity of the language or lack of gender description you are using.
Ryann: Do you have a question?
Tiff: (This is Ariane.)
Ariane: In my previous job we used Greenhouse, which I really enjoyed. I don’t know how much it costs but I did like it. It was an improvement.
Tiff: The Etherpad is attached to this session in the schedule. We have put in links and it includes an area of please add links as you fit. I now see I forgot to talk about job descriptions at all. They come up later but I missed that point. Thank you for your suggestions (and more time at the end as mentioned).
Let’s talk about the evolving processes. This is where it will get deeper.
ProPublica’s processes — want to explain, Ryann?
Ryann: Yes, I want to give credit where credit is due. I have had my hands in the hiring process lately and we are really trying to – there was no process at ProPublica so we are trying to make it a more open process. But my colleague, Sisi, is the editor for the apps team. A lot of her ideas came from her but she is not here.
Tiff: Sisi gave a talk at SRCCON:Work that was live transcribed and shared the slides as well. A lot of our process formed in conversations and we attended those sessions.
Ryann: I want to gave her a shout out because this is a pet project of hers and she has done a great job and she is the best. (Tiff concurs.)
- When ProPublica gets applications, we have — this is specifically, by the way, for the data news app teams. Although, again, as we are trying to standardize hiring processes throughout the organization other job openings are treated very similarly. But going forward I am really going to be talking about what our news apps team does.
So we get our applicants. We have a lot, sometimes in the hundreds, so as you can imagine going through those applications is pretty time consuming.
We try to do and agreed to do is have at least two reviewers review each application. Sometimes it is actually three but at the very least we do two. They are blind assessments. So if I am reviewing something, I don’t know if Tiff was the other reviewer. When you use Screendoor, you can view simultaneously and I don’t know what score Tiff gave to this person.
Before we close the application process, our team gets together and puts together a rubric (we will talk about this more so I won’t go into it too specifically) and we set out certain goals we want in our eventual hire and make a rubric about how we are going to assess each candidate.
So basically we take the scores from our two-three reviewers and take the top applicants who had the highest combined scores. You cannot game the system by trying to get 100% because there is a meeting where we go back and forth and kind of decide who is the best people to screen through phone calls.
Then we go to phone screens. I would say we do about 10 phone screens per job posting.
Tiff: 100-ish applicants down to 10?
Ryann: Yes. It is tough to get to that 10 and sometimes we go to 12-14 because it is hard to cut it down. Phone screens standardized is a big process of this process. We keep each phone screen to half an hour. Some people talk more than others but we want to make sure everybody has a fair assessment. We are pretty strict at cutting it off at half an hour for each phone screening.
We have a standardized set of questions so everybody is asked the same thing. We make sure the questions ahead of time that are being asked can be answered about everybody so it is not about a specific project you worked on. The questions ask for more information about the points we had in our rubrics, e.g. What the person’s reporting approach is like. And we ask 1-2 questions that will let us know what their technical skill level is because the jobs on our team do have some – I can’t think of the word, but you need some sort of coding knowledge so we try to assess that through questions.
Then we take the phone screens and decide probably about 4-5 people we will give a reporting test. We will talk about the reporting test more.
But basically we give you a dataset and we have you assess a story from it but we will talk about that more later. And that goes down to in-person interviewing where we will bring in two people, three at the most and they are casual interviews with editors, other team members, and we make sure they speak to at least one reporter from the greater newsroom who is not part of the data news app team at all but we are a small organization so chances are you will be working with them.
Tiff: On The Times’ side, this is largely for interactive news and my team. Our process has been adopted by the broader Tech organizations at The New York Times So the ~400 person dev staff of The Times is considering specific technical roles and following what I am about to describe with their own caveats and whatnot which is great. They are doing it role-by-role.
With all of our applicants, we do two reviewers. We’d like to do three but can’t at this time. Those assessments are a blind assessments, where we have no idea who is reviewing who. Candidates pass a the basic rubric, which is a points-based thing as well. And then discussion among the initial reviewers.
Then we sort of put together what is the Short List that will go in front of the entire hiring panel. This is where, in our case, we anonymize the material. We did 17 before and then it dropped off to 35, from an initial applicant pool around 177. The hiring panel does the same assessment with the rubric, against the anonymized materials. (The initial reviewers knew the candidate’s information because we cannot avoid it with our software. Maybe we had additional biases and couldn’t negate ourselves.) From the rest of the hiring panel, we get four totally anonymous assessments and look at the overall scores coming from both sets of reviews.
From there, 10-12 people that pass through that process, and again that is a lengthy, nasty meeting debate in addition to having to data guide us to talking about qualitative assessment.
Then we do phone screens. We do two separate screens — one personal and one technical — and assess them against a rubric. But we have been tinkering with how we do the phone screens. We don’t have strong advice in that area yet.
A number of people (usually I think the most we had is 4-5) move on to in-person interviewing. That is semi-scripted, runs against a rubric, and specific themes. We have been trying to craft all the assessments people have been doing, even if only involved in one hour of the day, them describing what their hour was like builds up to the assessment of the candidate overall and all those people are building one giant assessment even if they are only sliding in the information from their one hour.
The idea is each hour overlaps and hits different things. We have more specific demonstrations for how each of these are put together coming up.
- Anonymizing materials. Copy and paste and clean it up. It is damn painful. It is reveal about your own habits. You learn a lot about how your brain works for shorthanding stuff and it is kind of inpolite at times. It is a good demo for helping a team understand bias. Even doing it as an exercise rather than doing is helpful to try to get your hiring group, the people looking at your short list, to get them thinking about what they are not supposed to be considering. And also a heavy pen for redaction might gut it all. You may take so much out of the material there is no signal about who they are as a person. Watch for that. It is not a catch-all. It is a delicate, judgmental thing and we will go into it later on whether it is appropriate. It is an open discussion.
The main goal of doing this is to reduce what we are not screening for.
We do not screen for school or GPA, the statue of the previous companies they worked for like this person worked at Google hire them.
We are not looking for specific technologies. We might explain our tech stack but related ones – you should not have Java and we accept you because you have a key word in your résumé somewhere.
We are definitely not screening for the idea of nebulous culture fit.
We are not screening for friendship aptitude and whether or not you want to be their friend.
We are not screening for confidence level. Allow them to blow an answer and stutter and get flustered and come back. If they can recover the time and come back with the other answers that are good just let the one pocket go.
We are not looking for coding or computer science regurgitation. Citing algorithms isn’t helpful to see if you can do the work.
What we are trying to do instead of making quick snap judgments on these things we will try to hide information that let’s us get into other characteristics.
We will try a couple examples so you can see how this works and can augment or do a good job replacing this information.
I put together fake application materials — just segments. This is a journalistic example. This person says “I took a junior role at the New City Herald three years ago as a front end developer of the Investigation desk. The staff at The Herald has grown significantly and I have taken on multiple news nerd hats. Graphics, editor, database wrangler, news app developer, Herald fellowship director, web designer and developer. I have reported on disasters and elections (not always the same thing), with an ongoing focus in local incarceration reform efforts. Etc, etc.”
What should be redacted in here?
I will lead with one we should probably redact the of the paper.
What about the desk?
Do you think that is overly leading or generic enough to be appropriate? Yeah, it is a fuzzy judgment. You have to just kind of decide.
What about the hats or roles? Any of those roles so specific you would find yourself trying to guess who this candidate was?
What about the project name? The prison information, is that too specific?
What about team size? This is one I debated quite a bit. We have a sense of other papers and size. You can make guesses off that. This one is not overly specific but if you know there is a three person investigation desk or five person, it might be something to consider. There is all kinds of weird hidden stuff in here. Say your name, please.
Paul: Hi, I am Paul. This is interesting if it is a small team and they are wearing a lot of hats that tells something valuable versus a 14-person team with a super defined role.
Tiff: Yes. What we went through with the yes and nos is the process you would do as a redactor.
This is how I would redact this particular thing having done this for 300-something candidates. This is how I swap them out. Not just a black bar but something that is equivalent where I think it fits.
“I took a junior role at strong regional newspaper three years ago. The staff at strong regional newspaper…” which to point out keep it consistent especially if you run into candidates from the same sourcing or background.
How you end up describing things like what is a strong regional newspaper and is not doesn’t matter. They just need to know newspaper and a good one.
Any other senses of how you would redact this?
I swapped out with small for five. You want to aim for consistency but all of this is a judgment call. We have done this a lot and had multiple debates and not come up with anything better.
The question to the group is does this keep up the tenor of the applicants material? Seeing mixed reviews. Ariane?
Ariane: I have a question about swapping the term ‘wearing multiple hats’ which is many people in the room which for better of worse I can’t identify the person but I have a sense of the community they are from. Is that a good thing?
Tiff: That is something I would be considering as I look through.
More examples. It gets worse.
This example is like a tech person. I don’t need to read them all aloud. You can skim as we go.
This one I haven’t done my suggested redactions so this needs to be a discussion we have here.
Would you redact the name of the blog or the URL? What about the languages this person says is their current/past expertise.
I am seeing nods – is this mixed bag enough or describing a stack where you have an idea of where the person works because of the stack?
Ryann: It depends on what you are hiring from, the stack itself –
(Tiff: This is a gibberish stack. I made this up.)
Ryann: The fact someone knows certain languages implies something about their skills. Even though you are not hiring for specific languages but someone dealing with Python maybe they do data or they don’t. Is that something to consider?
Tiff: In another situation say someone says we use the the LAMP stack. A catch-all that means certain environments as opposed to skill sets. You would have decide.
What about the product name, the Waytech Location Mapper? Would you take that out? I see nods.
What about the fact up in the top it mentions there is software that helps augment and write stories. There is not a lot of operations that do that. If it had a brand name I would feel comfortable redacting it because that would be weird. If it didn’t, I feel this is generic enough I will not try to guess. If it was the brand name you would put something like that.
Anything else? Ready for the next example? Okay. I saw one person nod. So next example.
Another example — an unconventional background.
“Mission to make data more friendly to humans…seen in past roles as researcher, program manager and lead developer for science – that is a title. …2017 invited to be part of the Newer Museum’s Artistic Fellowship program, balancing creative portfolio with practical deliverables…there is a URL based on the projects…has graduated from the college of design with honors.”
What are we redacting here? The museum name seems relevant. Probably not a huge technical staff at a museum. What about the fellowship details?
Obviously the URL I would redact. Usually I redact to URL so they know it was provided but we are not including it because it is too much of a tell. And then the college name? Anything else you see in here where you would start to think about it?
Audience member: Take out with honors as well.
Tiff: Lead developer for science seems a very Google-able title. Some colleagues like to turn this into a game.
Ryann: That is the journalist in them.
Tiff: Onward! Another example. This is like an experience fragment. We were looking at parts of cover letters and this is their experience.
”…Worked as a front-end developer. …They worked at Samtouch devices (which is my spin on Samsung). …Did syndication in New York City…” It is all gibberish after that.
What do we redact? The companies? Yup.
What about the dates of this person’s – we have not talked about this in terms of the examples. Are the dates a bit of a tell? [There are nods around the room.]
What about the technology in this case? There is no right – it is a consideration.
What about the geographic references? They mentioned the fact they were working on Capitol Hill or New York City. Would you take them out? I take them out and swap large metro area or government area.
Ryann: I agree with that. I think people get short sold on the fact they worked in a certain metro area.
Tiff: What about a journalistic example: This is someone that works at Bloomsday and they are a News App Blumngineer which goes to your question about titles. That is so specific we would redact that.
Another one! — an unconventional example. This person was freelance. Did SMS data for analyzing ambient light in stages of meals. (I made that up.) …Does volunteer work at a local LGBTQ youth office in a specific location. There is a Slack and weird bitcoin crap. This person is all over the board.
What do we redact? Papers and magazines and try to describe them as a city-oriented magazine. Title?
Not a title just software engineer in general.
Greg: I would take out freelance because some people might be biased against that. (Greg.)
Tiff: That was Greg. What about bitcoin?
Greg: I am biased against it.
Tiff: I am biased against bitcoin and would sit there and think about it. Yes?
Audience member: The location.
Tiff: Would you take out the LGBTQ part?
Audience member: That was my next question. Is there a way to redact that and use underserved group?
Tiff: If you knew this particular organization and you felt like that was a better redaction and swap, yeah. If you didn’t have a good sense of who this was I think the only signal you have to make that judgment is just the location. This is stuff that has come up more or less.
Ryann: I feel like in the spirit of trying to eliminate biases, I feel like it would be something in my job I would try to generalize just to make sure people didn’t approach it with conscious or unconscious bias.
Tiff: Exactly. This is sort of broad demonstration of concerns. I am going to show you one more. Yes?
Audience member: I had a quick question about the last slide which is I noticed on a previous cover letter style example we didn’t redact only-female whatever in a space. So why would we leave in gender identifiers but not queerness or other minority identifiers?
Tiff: The challenge is you might miss that and it won’t occur until later and you won’t remember you did it. Part of it is trying to figure out what the standard is ahead of time.
Knowing you might mentally have to do this is helpful. I don’t know if it will be perfect but it is humans and a lot of material. Having multiple people doing redactions and checking each other’s work but also telling the team we will be reading these redacted versions of materials and we tried hard, please continue the effort as you see fit. But, yeah, I think, if we were to put together a bullet list of the specific things to look for as an activity that would definitely go on there.
Tiff: So the unexpected complexities one. This example is based on a friend who I have seen his résumé. I will show you this and you can draw the conclusions overall. This goes to the point of you don’t know how complex this material is going to be. We are comfortable with the couple we have done but you might into a résumé that has hundreds of roles and trying to anonymize all of them does get on the ridiculous side. Trying to figure out how to describe correctly some of the permutations between the reporting history for this individual or how it feeds back into the materials is going to get really annoying. If you have decided to go through it, you have to go through it all the way. This one is sort of – it can surprise you.
Ryann: A cautionary tale.
Tiff: Crap. How do I compare all the newspapers?
There are many more challenges as we were talking about.
What do you do with portfolio content which are under a vanity domain?
What if they included social media accounts? You know, what if they have a direct link to their GitHub or other network portfolios with their name all over it. What about clips?
Coding examples? Links to write-ups about their projects?
Way more. This is time consuming and really challenging. Helpful but an open debate on whether this is heading to territory that is weird.
As a mental exercise and a workshop it is fantastic. What I do recommend — probably most strongly — is even if you don’t adopt something like this in your process, do it as a workshop with your team so they understand why we are not screening for specific things.
Anonymize your own. Grab sample résumés from another job or parallel team or off LinkedIn and grab those.
Ariane: What would be the negative of asking people when they submit an application to provide both the raw complete information and also provide their idea of like how would you describe your job conditions without –
Tiff: I think explaining that would be challenging. We have a little bit more to get through. That is a good debate I didn’t anticipate having.
This is great. You can put this in front of people but they may still look at the anonymous material and make judgments. They read through redacted materials, then they are given materials that help them make decisions, and help you get consistent information about candidates out of them. Now we will go through the assessment part.
- Anonimizing doesn’t help with what we are screening for.
- Problem solving ability.
- Collaboration skills.
- Coding skills.
- The ability to instigate change.
- How do you build trust? Is that person good at building trust with the people they work with?
This is a huge laundry list of the characteristics we want to be understanding about those applying but if you are going through “went to this school and such” that doesn’t guarantee any of those things.
These tactics are derived from Medium’s hiring efforts. They have a large write-up about their hiring process. Their work with a bunch of corporate advice on applicants turned into their own version.
There is a lot of materials on Etherpad that goes into their themes and why they test those themes. I will show you more in a second about what we do with this things and that might make Medium’s in-depth blog post. There is a link in the Etherpad for Medium’s larger criteria.
- We looked at a couple cover letter examples and you would think the cover letter might talk about their ability to communicate, build trust and enjoy colleagues or whatever else, but they can provide that context but they tend to be formulaic. If you have written a couple, you know that is true.
They are usually not helpful for the panel assessment goals because the people writing them have no idea what your assessment goals are. It is hard to keep the richer context in the team debate. Compared to looking over a couple bullet points, it is hard to remember the entire paragraph about the project they brought. Turning that into something else is more advantageous to show.
People read it once on the interviewing side and never go back to it. Candidates spend lots of time writing it and it doesn’t play a role. We recommend questionnaires and we’ll get to what that looks like in a bit.
- Why do you want to use assessment rubrics? They will evaluate
- what a candidate knows
- what a candidate can do
- and how does a candidate work.
Assessment rubric in a nutshell ought to match up with themes.
Like the list of collaboration and everything else. I have a list here. All these guys.
We want to make sure the rubric is specifically narrowing these things. It is not saying it ought to bubble up on its own but we are actively pursuing that. We anticipated what the answer might be.
- This is an example of one item from a rubric where the question is does this candidate demonstrate these responsibilities and we have: superior, good and acceptable. Or it could be red, yellow, green.
Ryann: We do 1-5 rating.
Tiff: Ahead of time you would say this is the characteristics we want to assess and when you read you would say this person seems good to superior but you are guiding them through the material to answer this question and not their immediate reaction to the material.
Ryann: I would say if you write a good job posting writing the rubric should be easy. You should have written in your job posting exactly what you have looking for in a candidate.
Sometimes it is helpful to think about the rubric while you are writing the actual job posting because it is not fair to judge candidates on things you didn’t make known in the job ad.
Tiff: This is the matrix with here are a handful of themes and this is effectively what we think would be a good, acceptable and superior answer.
You would sketch it out ahead of time drawing from the materials, what you want from the role, and trying to make sure you know you have read into their materials for these themes. None of these have been necessarily overlooked. You get people to pick I think this person had a super, good or acceptable answer. Have a free form comment where they can add more if they felt like it.
What this does is –
Sidebar: We went through anonymizing résumés and what that looks like and the assessment and why you want to put together a rubric that guides the decision and the way people are reading and assessing.
- Another option is something the government used and othering organizations called a crediting plan. That is the same thing but you don’t have to anonmize materials for that. It is a way that is intentional to go through someone’s résumé and how to read it.
Basically you are crediting a candidate for specific abilities. Some of them say look at the school, look at the last place they worked, look at the references from the last place.
They are not asking you to just look and respond but asking you to look and think about it because the next question is does it meet certain criteria, do you think this is acceptable, it guides you through. It is a forced March through someone’s résumé.
It is a way to pack it all together at once if you need to.
I have not used these — The NYT’s tech group adopted this after doing research. I have not been part of that but helped describe the crediting plan they are using. They don’t anonmize because they are dealing with the volume where they can’t. You have to sort of understand what this is building on top of in order to adopt this wholesale, if that makes sense. And it can replace anonmizeing materials.
This is 18F’s crediting plan notes. They have a larger part of their webiste that is a good read overall. There is a link next to it for unconscious biases. That is a good package to take to a team if you want to workshop this within your team or bring up as a discussion.
- Encoding assessments.
We talked about how we turn it into points. You would assess points and assign it consistently.
Superior, good, acceptable this would be a point scheme or green, yellow, red is one we have used in the past.
It effectively let’s you use the qualitative information and has a baseline where the quantitative information is there. You see how candidates are doing. Low in some areas and high in others. But you can focus on the qualitative assessment. Let’s talk about how this candidate did and what people read in this person’s résumé. What do you think about it as opposed to did everyone read it.
Steven: With anonymzing and doing multiple people is there any step in there for where there is two very different scores on the same candidate?
Tiff: We have used z-scores or standard deviation in the past to knock out the scores but mostly we are good. A good rubric guides you into doing a better job. We have gone with more math and walked away from it but we might adopt it again soon.
- The most important thing about asking your team to go through the rubric, fill out their stuff, is to ask about the Next Step. This is the open-ended thing at the end which is effectively like “what do we do next?”.
That goes in a coded piece of information, too. That is data.
- Here is how quantification works. If you do all this in Google forms, it feeds to a spreadsheet and that you can map over and turn in a qualified situation. You get a chart it generatesgenerates. This is the question we asked, “Next Step”. I think this is after the initial screen.
This is what we asked people to assess and this guided a little bit of our decision making after that. Broad strokes.
How many people are not proceeding? How many people are in the good pocket that we would continue with and should talk about?
If you are doing Google Forms it has automated charts and does terrible things which in this case it is the color-coding. It doesn’t let you specify the color to match up. What I ended up doing is something like this.
[chart where the pie-chart slice for ‘green’ candidates are colored orange]
Pivot table off a spreadsheet and using the conditional coloring things to bubble up cohorts of how people are doing.
Everybody gets a candidate number. The columns where it says e-mail addresses I hid the layer of each there. Those are their scores. We average that together or do median. You can pick your methodology, some is open-ended.
Ryann: When we said we were hacking hiring, we meant it.
PHONE SCREENS & IN-PERSON INTERVIEWING
Tiff: On to phone screens and in person! This will go faster because we set the baseline. Use a rubric. Phone screening. It is a familiar pattern. Pick from your themes.
Develop or script out our questions. Make sure they hit those themes. Anticipate types of answers, write them down ahead of time and help people out. Have the interviewer use a rubric or Google Form and ask for the next step recommendations at the end because that is a qualitative judgment and you can have a free form saying what did you think about this candidate and they can describe for days whatever they think and that should be brought to bare if that candidate is proceeding on and then quantify it at the end and that gives you the database and then you could have a debate beyond that with your teammates.
Say the top 10 or whoever. And people can lobby for someone that didn’t make the top 10. It pushes the debate and makes sure everyone did their homework.
- Coding tests.
Interactive News has a strong opinion against them for many reasons which is the next slide.
First slide goes through the pros. Ryann is going to handle them because I don’t like them.
Ryann: I don’t like them either but they are useful. The pros are it can provide a familiarity between you as the interviewer and the candidate.
We give the candidate a dataset and ask them to answer about five questions, mostly centered around what is a potential story you might pursue through this data. We ask them since they are looking at a database document their code because that is a really important part of the job.
Basically everybody comes up with a different story idea. I never had a test where two people hit on the same story idea. So it helps to have that to talk about with the candidate. It helps you assess technical abilities that might be obscured in their résumé. It gives insight into the what the candidate’s work process is like. And it is a standardized way to compare a candidate’s ability to mining a dataset for reporting leads which is pretty much the most important part of the job.
Tiff: Done well and if you are comfortable with the test.
Ryann: It has been the difference when we have a really strong group of phone screens and we are like how are we going to pick? It has made the difference which is why I like it.
Tiff: The Times does a code test as well.
Brian: This is Brian. Is this in-person?
Tiff: No, take home.
Ryann: We send them a dataset and give them about a week which goes into the coding.
Becky: I have been on both sides. Giving a test and getting one. I think giving an estimated time frame for how long you expect the person to work on it is good. If it is too open ended and two days I have seen candidates really overexert themselves and it is hard to compare results.
Ryann: What we are done and this is again why I kind of don’t like them. We give them about a week. We can go to cons.
Tiff: We have more bullets on this one.
Ryann: We want to give them ample time to do and most likely you have another job. We are cogcognizant of that. We assume or estimate this will take about three hours of work. The problem we found with that is that, you know, I as a busy person might spend three hours on it and I am like I did my three hours here is my results whereas somebody who has more time or just wants to spend 12 hours on it and hands in a much more robust end product then it is a little unfair because we said we think this will take about three hours and we don’t know how much time you spent on it. Are we thinking the person who spent 12 hours could do it in three? We keep it vague for that reason but I don’t want people stressing out thinking this should take 12 hours.
One interview I had years ago, they did a programming exercise in person so they got to see you do the work and time bound it.
Tiff: We’ll get to that.
Ryann: I don’t personally like coding and reporting tests but they are useful. It can seem like free work if it is not structured. We always give a dataset we have made a story on and make it clear we are not generating free leads for stories. We are not asking you to clean data we haven’t cleaned yet. This is nothing that we are actually going to be using for actual work. Often, I would say almost always artificial in nature, we want you coming up with a story idea but don’t want you calling people. It is very much an artificial story idea.
You could overpromise about the work with the wrong example.
Some companies might find it too hard. You might say that is a good thing. Some might find it too easy. You really have no idea of how they actually assess the work and whether it was difficult or easy for them because you just have there final product. Some candidates may be able to spend more time on the tests than others and to me that is the biggest con and the main reason why I am not a huge fan. So what I have tried to do is hit a happy medium and make sure that it is not an onerous task to do this. I make it very clear to people if they need an extension it let me know. I don’t want you slaving after you put your kids to bed.
If things comes up, we are open.
Alyssa: I have done testing and saying you have 48 hours but let me know when you want it so we can send it because then people can make the time so they are not having to complete it.
Ryann: That is a great idea.
Philosophically when you use them do you think of it as a skills assessment or is this like a bozo test? Let’s drop the loosers.
Ryann: That does happen, yes, it does. But I see it more because I am convinced the five people I have given this to I could pick their name out of a hat and be totally happy hiring them. It is more seeing who really excels. I don’t think anybody has failed. We know you a good candidate but it is wow, you thought of an innovative way to approach this skill set.
Another thing is seeing their documentation. Because if you are on my team, you are checking on other’s work and others check your work, so if I can see you are not good at dockcumenting your steps that is something I want to know before hiring. Our team primarily uses R but we let you do your assessment in whatever language you prefer. Nobody did it in R last time and I was sad. I muddled through.
Tiff: There are companies that time box it by going through a coding test-thing. There are all annoying startup names that provide that service. But effectively you have an hour to finish the code test via a website.
There are tons of permutations of this. Alyssa, to your point about the tech question, usually in the phone screening we have a technical phone screen that gets into specific questions to plug it out before we get to the point of that test. That is the bozo step. Get them on the phone and see if they are really glossing over their skills. It is usually youngsters. It is so cute.
In-person interviewing: Pick themes, develop a script, anticipate answers, and quantify. This is the point I want to show what it looks like when we apply this. As we are planning out the candidates 5-6 hours of interview, this is for an in-person one day-ish interviewing.
We have been doing this planning for a one hour bloc. This would be a code walk-through. This one is have the candidate walk through a code sample they provided. They are talking you through it. It is something they selected. You ask questions about how the project went, how they worked with other people, who collaborated on it. A bunch of themes at once.
We would say “here is the code walk-through”, so they know the purpose of the event. And had to provide a sample for the walk-throught So in there is collaboration, follow-through, problem solving, those are our themes and baked right in.
And then we write out sample questions that hammer on the themes. First is, “when you get stuck on a problem what steps do you take next?” And here are follow-up questions. This doubles down on one of the themes because we want to know about follow through and how you do it. If you have time before going on to another scripted question in this area: “what are tradeoffs?”
That might go to a minor note of things you are testing. Maybe other parts of the day long interview we are hitting that harder but this could add information in that regard.
Effectively, try to sketch it out. For the couple people guiding that hour and participating in the hour hopefully they understand these are the reasons we are asking these questions. Listen to what they are saying making sure they don’t off theme.
This is an assessment of what we asked them to do, the reason we are asking the question, the themes we thought it would hit, and then here would be the answer types we were hoping to hear out of this hourhour-long session.
This one uses the red, yellow, green because it is an easy short-hand. Total red. That is person is a yellow. And it is fun to assign colors.
This is sort of how we would basically describe how did this person go through this hour-long segment. This is literally those colors are the Google form we ask people. It is a radio button and then an other and they can choose how they see fit and that goes into the spreadsheet and turns into the data.
HACKS YOU CAN USE NOW
Because this is time intensive and elaborate but it doesn’t need to be done wholesale. (I almost said piece-meal and that is the opposite of what I meant.)
You can adopt a lot of these individual pieces. Here are some of our suggestions. Just to get started.
- Offer to handle communication.
(We will go into each one in-depth with more sub-points.)
You can demonstrate rubrics and what they can bring in an academic sensibility.
You can do your own interview and segment planning for the hour you have been assigned to talk to.
You can take on the internship and fellowship process. That is what Sisi talked about.
You can upward manage — that is a life-long still.
The least effective thing is to say, “look at all the reading materials I have”. Although, that is still minimal help.
So, details about handling communication. This is the first thing I strongly advocated for and used to dig into the process.
- Keep candidates in the loop.
- Draft e-mails for each step of the process.
It would like look thank you for applying we got your materials. That is the first e-mail. Anybody who applies gets that first note just so they know – it is one e-mail you send to everyone.
Just volunteering to be the person sending that e-mail and maintaining the list of people who are applying sets up the pattern of good collaboration.
Then you can do more interesting stuff. When you decided who gets a phone screen you have an e-mail you drafted. Here is your expectation, timing, time slots, and whatever else you need to do. And you draft the thank you, we are proceeding on with other candidates and we would like to see you another time. Sending that will keep so many candidates positive about your organization and wanting to reapply. It is effortless to draft one e-mail. The next one is send people to do the code test or the thank you not at this time note. Next is the in-person interview and you probably want to be a little more verbose with you participated a lot and we appreciate it. And same when you send an offer letter. Those are your finalists, your top couple people, people you would love to work with in any other scenario except one person beat out the others. You don’t want to piss them off with a crappy e-mail.
- Volunteering to write the template and send them on behalf of the team is a good way to normalize the relationship with the candidate.
Say you’re the person told 45 minutes before the candidate showed up you have to read a résumé and will be talking with them for an hour. First question to the bosses is: “any sample questions?” to which they will say “no” so you say “okay, I do my own”. You say, “do I want to find out about technical abilities, problem solving skills, or collaboration…?”, or whatever else. They probably won’t have a strong idea.
If the bosses ask you, “can you send your feedback about the candidate?”, they didn’t describe what they wanted. You can send back the rubric and your assessment of this person on these themes and here is how it sports your thumbs up/thumbs down decision in the end.
- Set a standard, even if you have to be a little paggro about it.
It is open-ended and you will feel better about the process and decision and you have a paper trail. Even if none of that comes to bear, you set the standard.
Bosses will get a note back from a different colleague who wrote one word, “yes” or “no”, while yours has evidence of thinking and engagement and that will be helpful.
Ryann: Your manager is more likely to use that rubric the next time they are hiring. It is very passive aggressive but it works.
Tiff: Big fan!
Starting small. This is where, you know, maybe if you can’t do much about the hiring process, volunteer to help out with the interns or fellowships you might have. Any point where you are reviewing – freelancers and contractors. Just having a structured way to talk about comparing people who are applicants. Any other ideas from the room?
I would add I love improving your internship process first because you do it more often. It is good practice for the team.
Tiff: And since we are getting close on time, we’ll have to hustle through the part we care about the most.
Ryann: And it is the one we will rush through but….
Encouraging Diversity. This is something that ProPublica is going through. We sat down and said we might not have been great about attracting the most diverse candidates but this is a time we can do it right and I am happy to say the efforts paid off. Actively look for job boards to post the role, solicit from the newsroom.
Tiff: And then lately it has been a little trend of putting out notes in various places. Social media, within the job description saying if you are from an underrepresented background we encourage you to explore this organization. There are lawyerly issues around that.
Please talk to your legal people about that but there is a way to word it that says we really want to – please think about our role that doesn’t go into the bounds of saying we are not interested in a certain cookie-cutter type of person.
The biggest thing about this one is in terms of if you really want to change and support the idea of more diverse teams and workplaces is even if you can’t do anything about the process of hiring the team, you can recruit like crazy and shape the people applying and that is half the battle, if not more.
Guiding the people into the pool if you know the pool is awesome and the exact type of people you would want to bubble up on their own is a big part. The open outreach and job descriptions and all that.
And then you run through the crazy process with double-blind materials and crazy rubrics.
And you can ask for demographic information in their application. You know, we have internal rules where we try to make sure once you are filtering down for a group there is a meta check of have we unintentionally filtered out candidates. Is there something we ought to correct for? Andrew?
Andrew: Have you ever had to go back and de-anonmize things in order to do that?
Tiff: Usually this comes in towards the end, right? Where everybody knows who the people are because we had a phone screen and are seeing them in person so usually that is apparent.
Next slide. This is good reminder as well. You can be like “no, we failed, do it again”. Sucks but works.
Okay. Using questionnaire. This is also – earlier I alluded to the fact cover letters are not that specific. Examples of questionnaires would be – we will put them all up.
How did you hear about our job posting?
Describe a time when you blah blah and that is effectively a mini-cover letter but drafted into what you care about.
What perspectives can you bring to the role that will make us better? That is a fascinating one to ask people and hear answers. You will get people that know what you are looking for and happy to go to that territory and you will get people that don’t understand.
This is a little edgy in terms of people’s comfort level but we have probably asked 200-something candidate and the answers have been incredibly helpful.
We overweighed on the quality of the answer to get to this one and the next one is piecemeal.
- What did we forget to ask you? That is the cover letter and also like in a couple words or sentence or paragraph please describe. That gives you salient information into your qualitative assessment of candidates that would be hard to get out of a cover letter. You get it consistently for all of them. I highly recommend trying to get this.
Ryann: I put examples from ProPublica’s most recent job posting.
Instead of a cover letter we ask,
- Why do you want to work here?
- What are your big thoughts and visions?
Because in the end that is what we wanted from the cover letter and I hate cover letter so if that means nobody has to write a cover letter for one of your jobs that makes me happy.
We recently starting doing this. We ran this through the legal department. As long as it is optional, we asked for racial and ethnic background. We have free form for gender identity and if you are member of an underrepresented community we have not mentioned. Legal said it had to be optional but we had a 90-95 percent response rate from various job postings.
That was helpful to see and know whether our recruiting efforts were actually diversifying our applicant pool.
Tiff: Both of us have “how did you hear about this job?”. If you put the job description on boards of different affiliates and this is your only way to find out they round-tripped unless you have magic software I have never heard of and that let’s you know whether your outreach is bringing in people from interesting spots or just things you had not known about.
You didn’t know someone cross-listed you on a special place.
Satchi: Satchi here. When do you look at the data? When you are looking at the pool?
Tiff: The pool. These answers, in particularly, these open ended ones is what we would anonymize. If someone mentioned their specific affiliate organization I would redact that and turn it into bracket things which goes into the discussion about the fine judgments about that. Upper managing always do it.
Point to your competitors. You just learned a lot about how The New York Times and ProPublica do a lot of this. Feel free mention this is something these organizations are investing in actively.
If you need to, in terms of upper management, you can rephrase rubric to team efficiencies. “I will try to get it all into one spreadsheet. Just one spreadsheet for you to look at, boss!”
Share reading materials.
Advocate for politeness toward candidates. You hope they will reapply. You hope they care about the organization and really want to apply.
Ryann: And you don’t want them going around telling people your interview process sucked.
Tiff: The Times had a problem with that. The bad reputation of not getting back to people and résumés going into the black holes and stuff like that.
Ryann: This next section is the most useful thing coming out of the process. We identify candidates who might be a little too junior or maybe not quite right for your role but for another one. You don’t want to just give them a “thanks for applying, we hope to never hear from you again”. This is just a little bit of edited down language. We have a subgroup of people that applied but they are not going to the phone screening we will send an e-mail like this.
- Engage with former candidates. If you are going to ask them send me links you are proud of going forward they will do that. Engage with them.
Tiff: And write back. Say “cool, thanks”.
Ryann: Just keep a list.
Google Sheets and Screendoor makes this easy. The next time you have a job opening send out a distribution list.
Tiff: We have two more slides so bear with us.
Those are there links. We will put links to our slides on the Etherpad.
And this is the point where we would ask for questions. We were hoping there would be time during the session but instead we will do it during lunch or right after until the point where we don’t care about the topic anymore.
You are welcome to sit in here in the meantime and ask questions! Come up and find us over the course of the day.
If you are really hungry, that door is a short cut. If you are not hungry and content to keep discussing, stick around.