#178 - People Management Doesn't Quantize Well

Manager, Ph.D. is a newsletter and community which helps people from the world of research reach their full potential managing teams and enacting changes. We’ve already developed the advanced skills to be exceptional managers; we just need help with the basics.

If you’re new to the community, drop me a note!  Some past issues from the archive which you might be interested in include:

• #132, “Hiring: Make an Onboarding Plan First”

• #139,  “Don't Use Process To Avoid Talking About Expectations”

• #143, “Good Meetings Have Purpose(s)”

Plus of course the one-on-one and feedback guides.

And whether you’re new to the community or not, feel free to email me, or even have a quick chat with me about problems you have, or things you’d like to see!

One saying that my grad-student peers sometimes used to overuse to the point that it bugged me a lot was “the plural of ‘anecdote’ isn’t ‘data’”.

The saying isn’t entirely wrong! But it gets deployed an awful lot to downplay the usefulness of qualitative data, and overstate the importance of quantitative data.

For those peers who became managers, I’ve got some news for you…

Our New Jobs Are Mostly Qualitative

As STEM researchers, our individual work tended to focus on improving quantitative metrics. It was important work! And we got pretty comfortable with it, because it was relatively straightforward to understand. We needed to get the S/N ratio below X, or improve performance of this by Y, etc.

That’s not our job any more. It was fun while it lasted. But we don’t get to toil in blissful solitude working on a single task that can be optimized until we’re happy with it any more.

Instead, as managers, we’re entrusted with stewardship of a team of humans, and having that team consistently producing results for the rest of the organization, not just in the short term but also in the longer term.

And that means everything is tradeoffs. What should we personally be working on? What tasks should we be prioritizing right now in this context? How can our teams have most impact on the organization?

Tradeoffs between incommensurate goals are judgement calls, and judgement calls are inherently qualitative in nature.

For Managers, “How Many” Is A Poor, Shallow Substitute for “How” and “Why”

Quantitative metrics — lines of code written, tickets closed, analyses done — tend to be trivially game-able, because they don't actually reflect what we care about.

God knows encouraging extra lines of code never goes well. Sure, a closed ticket is better than an open one, but how could it make sense that time spent reducing the number of tickets that come in for the future should count against you? Does splitting a larger analysis request into three smaller analyses actually improve anything we care about by a factor of three?

Qualitative measures of impact and value are harder to gather and interpret, but also much harder to game. How happy are our internal clients with our results? What’s the impact of our new process our customers’ success with us? Do team members feel like they’re working together better? Those things improving, even in ways that don’t lend themselves to quantification, mean we’re actually doing good work in areas that “count” (hah!).

Qualitative data often reveals the "how" and "why" behind quantitative metrics - understanding not just that productivity dropped, but the contextual factors causing it. And the whole reason we want data is to be able to do something with it.

People Quantize Poorly

Assessing individual performance on a task or in a job in our line of work pretty much always comes down to qualitative assessment, which makes people with our backgrounds very uncomfortable.

Which in turn means a lot of people try to use quantitative measures anyway.

And sure, there might be things we can count or quantitate - N tasks got done with average level of quality Q - but this is a very surface level understanding of how an individual performed in the context of a team.

For instance, the person may have spent a lot of time helping get new hires up to speed, mentoring another team member through a task, and building relationships with other teams.

Did they spend that time effectively? Even if they did, should they have spent less time on one of those things and more on increasing N or Q? How would you know? How would you inform decisions about this?

And when comparing “objective” between team members doing different work within the same team, the same quantitative metric (e.g., time to complete tasks) can mean very different things in their different contexts, or for different levels of task-relevant maturity (#148)

Setting goals and expectations for an individual, understanding a full picture of an individual's performance in the context of a team's work, helping a team member with their career development — these are matters of qualitative assessment, and require understanding the team and organizations context, talking to people, and collecting qualitative data. And the sooner we get comfortable about this, the sooner we can be consistently giving our team members good feedback and guidance.

A Quantitative Team-Wide View Is Still Limited

Attempts to quantify team performance goes a little better — after all, teams perform, individuals contribute (#172) - but still offers a very limited view.

Again, you can try to quantify outputs in some straightforward way, but even there you immediately face the tradeoff of weighting short-term outputs against efforts to improve things in the longer term (removing tech debt, improving processes, building better relationships with stakeholders)

And to quantify anything more complex than team performance is so fraught that people generally don’t even bother, even though these things matter a lot for sustained, meaningful success. Cross-team collaboration effectiveness is famously difficult to measure quantitatively.

The Things That Matter Most To Us Are Qualitative, and Relatively Easy To Collect Qualitative Data About

On the other hand, things we really care about and are important for the long-term success of our team within our organization are pretty much exclusively qualitative.

Individual performance and career development is qualitative

• Finding out how an individual wants to grow their responsibilities and where they want to go in their career, and what are the gaps they need to fill in to get there from here is an exclusively qualitative discussion

• Same with whether the individual is happy in the team, is feeling overworked, or is considering new roles.

Team health and dynamics is qualitative

• Psychological safety - where team members feel that they can do take reasonable measured risks in their work or in voicing their opinion, without fear of undue consequences, is essential for a team to perform at its peak, and requires understanding your team members and for them to understand each other

Leadership & Decision Making is qualitative

• Your own decision making between multiple priorities (#158, #165) is going to be informed by qualitative discussions with stakeholders and your team members

• Making any kind of change in how your team does things requires assessing readiness and identifying challenges ahead of time

• Making changes outside of the team is best done with pre-wiring, (#147) a fairly extensive qualitative-data collecting process

• Identifying future leaders among your team members benefits from qualitatively watching how they take on new challenges and handle ambiguity

Describing your Qualitative Impact Improves your Relationship with your Stakeholders and Manager

• In another venue and for a different audience, I’ve written about how collecting testimonials and success stories is a much more effective way to advocate for the work of your team than simple dashboards

• People react much more strongly to stories than lines on graphs!

You Already Have Most of The Qualitative Data Collection Tools You Need

Let me guess, you already go to a lot of meetings and take a lot of notes, right? Well, the good news is this gives us the opportunity to pretty much all the necessary qualitative data collection - it’s just that no one much taught us to do it systematically.

There’s A Lot Of Guidance On Taking Good Qualitative Data And Understanding It

Qualitative data collection, and gaining understanding from that data, is an entire field of human endeavour that our colleagues in the humanities and social sciences took grad-level courses in.

The good news is that we don't need that much study! But we can absolutely benefit from some Qualitative Methods 101 cheat sheets. An awful lot of them will sound familiar; many of them are consequences of Feynman’s dictum, “The first principle is that you must not fool yourself and you are the easiest person to fool.”

For data collection:

• Have a clear process and structure for gathering observations, not just collecting random anecdotes - otherwise, selection bias will render that data meaningless. Have topics you’re monitoring, and a clear place to write down notes on those topics.

• Multiple sources/triangulation - make sure you’re getting data from different people and contexts to verify and enrich understanding. Again, selection bias isn’t our friend here.

• Document well - take notes during soon after observations or conversations, not relying on memory. And document what was actually said/done, not your interpretation of what happened. (This is the same as when giving feedback). We don’t have to worry about this as much when doing quantitative results (“but what does that 83.7 actually mean?”) but we absolutely do when collecting qualitative data.

• Awareness of bias - Understand and accounting for your own perspectives and preconceptions that could influence data collection - you’ll be biased towards things you already think is true (X is a problem, Y is ok) - this is part of why it’s so important to document observations not interpretation.

• Context matters - Record relevant contextual information about when/where/how observations were made

• Open-ended questions - When actively collecting qualitative data, use questions that don't presuppose answers or lead respondents

• Active listening - Focusing on understanding rather than judging or problem-solving during data collection

• Saturation - You’re probably collecting enough data when you're no longer getting new kinds of information - it’s like when your reference search keeps coming back to the same papers.

• Ethics - Respect confidentiality and be transparent about how information will be used

For interpreting qualitative data:

• Look for patterns - Start to identify themes and connections as a separate step from the data collection

• Maintaining skepticism - don’t seize on to the first possible conclusion you see, and don’t overstate patterns from listed data. Look for counter-examples and other explanations

• Considering context - Interpreting statements and observations within their full situational context

• Evidence strength - Evaluating how much support exists for different interpretations

• Recognizing where more you might need to talk to other people or ask more questions

• Member checking - Validating interpretations where appropriate

Places You Could Add Some Systematically Qualitative Data Collection This Week

Given the meetings and the responsibilities we have, it’s not hard to find opportunities to start getting more consistent about our qualitative data collection. Pick one that lines up with problems you’re working on, and get started:

In one-on-ones:

You’re probably already taking one-on-one notes, maybe even using the Manager, PhD template. The key here is to make sure you’re systematically collecting data in some set of categories that are important to you. Some common ones are

• Current work challenges

• What’s going well

• Team/collaboration dynamics

• Career goals/interests

and review notes monthly to spot patterns.

In Retrospectives

Similarly, you’re probably already taking notes on the content of the meeting, but it’s useful to take notes on the context:

• Who is speaking up

• Who isn’t

• What topics you hear about in one-on-ones that aren’t coming up here

• Who keeps coming back to particular topics

• Who covers a wide range of topics

And review quarterly or so.

During And After Key Meetings With The Broader Organizatio

Make sure you document

• The context behind key decisions

• Participant reactions

• What topics or points each participant kept coming back to

• Who each participant referenced and/or deferred to

and review notes quarterly or so to spot patterns.

Start a "stakeholder feedback" document

• If there’s not regular discussions with stakeholders, individual or as a group, see where you could start one

• Peer managers are stakeholders too!

• Expressly ask for feedback, or insight into how the team’s work affects the stakeholder, every conversation with a stakeholder

• Record verbatim positive/negative comments about team's work and the impact of that work noting the context and date

New Data Types Give New Insights

There’s absolutely places for us to use our quantitive skills in our work, and those skills are valuable! But they only give us a limited view about what’s going on in our teams and in our organizations.

None of qualitative data collection described is enormously hard or difficult, but like a lot of management practices, the value comes from doing it consistently and reviewing it periodically.

When we start systematically collecting data about context, and looking at the data objectively as a whole, we become smarter about the people systems around us and the qualitative challenges we’re facing. It helps us deal with ambiguity, not just uncertainty (#176). It gives us the ability to be more effective and more strategic about how we solve problems, and which problems we choose to solve.

And now, on to the roundup!

Managing Individuals

3 Ways to Make Sure High Performers Feel Valued - Zach Mercurio, HBR

So here’s something that should be easy for us, but I’ve seen a number of ex-STEM researchers struggle with.

We’ve typically always been high performers, so we should know how high performers like to be treated as a rule! While individuals vary, we generally like to be recognized as being high performing, given opportunities to grow, and shown how our work matters.

But.

As (slightly overwhelmed) managers new to this whole manager thing (and why not, no one ever taught us any different), we’re often preoccupied with the problems our team is facing, and worrying about our low-performing team members. Our high-performing team members can easily feel neglected. They’re not a problem, we don’t worry about them, so we don’t think much about them at all… right up to the point where they feel undervalued and leave.

It doesn’t take much to change that dynamic! Here’s what Mercurio says about making high-performing employees feel valued:

1. Notice them intentionally - Schedule regular check-ins focused on them as people, not just work updates. Use tools like "stoplight check-ins" (green/yellow/red) to gauge energy levels and stress.

2. Affirm specifically - Go beyond generic "good job" praise. Describe the situation, observed behaviors, unique strengths shown, and specific impact made. High performers often receive more feedback but of lower quality.

3. Show they're needed - Demonstrate their unique value with metrics and concrete examples. Use phrases like "If it wasn't for you..." to highlight their specific contributions.

The article shared a striking example: An award-winning oncology nurse manager left after 5 years because he felt "replaceable" - despite excellent performance. His leaders never sought his input or had meaningful conversations with him.

Managing Teams

Are your standards too low? In defense of raising the bar - Wes Kao

This article addresses something I see a lot of in PhD-led teams.

We tend to be very driven and have very high standards for our own work. We can have narrowly very high technical-quality standards of our team members work, too.

But, given the professional culture of collegiality (cough conflict avoidance) we come from, and the longer time lines of academic research, we tend to very much not hold team members or the team as a whole to high standards - whether that’s work quality, work pace, or working together.

But what’s the favourite course you ever took? Or professor you ever had? Or boss you ever had? Probably one that made you feel supported, but also pushed you pretty hard, right? One of the reasons you like that course/prof/boss is that it helped you grow your capabilities.

Kao talks about low-standard teams, which to be clear are the default, and how they differ from teams with high standards:

1. The difference between high and low-standard teams:

• High-standard teams innovate constantly and acknowledge gaps between vision and reality

• Low-standard teams make excuses and often don't realize their standards are low

2. Creating a culture of "high standards, high feedback":

• Give specific, actionable feedback on work outputs

• Don't let mediocre work slide - address it promptly

• Explain the logic behind higher standards, not just the vision

3. Handle the transition thoughtfully:

• Some team members will embrace it, some will leave, and some could go either way

• Support the transition with coaching and clear expectations

• Remember that short-term discomfort leads to long-term growth

While raising standards might feel uncomfortable initially, it's one of the most powerful ways to improve team performance, attract top team members, and make those high-performing team members we just talked about feel challenged and exciting.

Managing Within Organizations

An Introvert’s Guide to Visibility in the Workplace - Melody Wilding, HBR

We tend to be introverts, or at least a little quiet. We tend to value thoughtful, behind-the-scenes work. These are all perfectly fine things; but it’s useful to also learn to be able to put our hands up from time to time.

Next issue I’m probably going to talk a little bit about growing one’s influence in an organization. And the first step in that is to start contributing occasionally and visibly.

This doesn’t have to mean showing off, or claiming credit, or craving being centre of attention! There’s nothing particularly sleazy or self-serving about it. You’re a smart person with insights into what’s going on in your organization; being visible just means getting a bit more in the habit of sharing those insights publicly.

Wilding talks about this issue, which is particularly important in a remote or hybrid environment where it’s no longer a given that people will at least notice us sitting in our cubicle. There’s solid advice:

1. Speak early in meetings - aim to be the second or third contributor. This breaks the ice and makes subsequent participation easier.

2. Take pressure off yourself - you don't need groundbreaking insights every time. Build on others' points, ask thoughtful questions (good questions are super valuable), or summarize discussions.

3. Watch your language - replace self-deprecating phrases ("This may be terrible...") with confident ones ("I'd like to propose..."). Your expertise deserves recognition.

4. Use asynchronous communication - leverage written updates, post-meeting emails, or newsletters where you can thoughtfully compose your message.

5. Express gratitude - frame achievements through appreciation ("I'm thankful for the opportunity to contribute...") to balance visibility with humility.

Letting your work go unnoticed can mean missing out on opportunities, recognition, and career advancement. But it also deprives others of what you have to say and what you have to contribute! It may feel uncomfortable at first, but having some additional tools in the toolbox, these visibility skills, is crucial for professional growth and having impact, and doesn’t need to feel weird or phoney.

Managing Your Own Career

Networking For People Who Don't Network - Stay SaaSy

Networking is another thing that seems uncomfortable and transactional if not kind of sleezy to some of us.

I like this post - it has a nice straightforward way to get started thinking about networking, which is just a much-debased word which means “collecting and maintaining good professional relationships with people in your line of work”.

You like your work, yeah? Probably like talking about it? Do you know who else likes talking about it? Other people in your line of work!

Developing a strong network of professional relationships is good for opening up opportunities for us, which is all well and good; but it’s also just kind of nice to have a bunch of colleagues we can email with questions or to catch up with or to see in person at the occasional conference.

Like everything in management, it’s not hard or difficult, it’s just about keeping on doing it:

The author has great and simple recommendations:

• Start with your coworkers - talk to them, even in other parts of the organization (it’s even part of your job - #171)

• Once you’re comfortable with that, find a way you like talking to people about work topics and do more of that outside of your organization - maybe meetups or online forums/slacks or classes (or heck, start a newsletter)

• Once you’re comfortable with that, reach out to individuals when you think of them or read something you think they might find relevant

• Meet up with people in person semi-regularly when possible

• Don’t be weird or demanding

That’s It…

And that’s it for the week! I hope it was useful; Let me know what you thought, or if you have anything you’d like to share with me about how a newsletter or community about management for people like us might be even more valuable. Just email me, leave a comment, reply to this newsletter if you get it in your inbox, or schedule a quick Manager, Ph.D. reader input call.

Have a great weekend, and best of luck in the coming weeks with your team,

Jonathan