Author Archives: Hugh E. Williams

About Hugh E. Williams

Founder, Advisor, Professor, and Investor. Former Tech Exec.

Don’t use a Pie Chart

2 Replies

I don’t like pie charts. Why? It’s almost impossible to compare the relative size of the slices. Worse still, it is actually impossible to compare the slices between two pie charts.

Pie charts in action.

Pie charts in action.

Take the example above. Take a look at Pie Chart A. The blue slice looks the same size as the red or green slice. You might draw the conclusion they’re roughly the same. In fact, take a look at the histogram below — the red is 17 units, blue is 18 units, and green is 20 units. The histogram is informative, useful for comparison, and clear for communication. (There’s still a cardinal sin here though: no labels on the axes; I’ll rant about that some other day.)

Compare pie charts B and C above. It sure looks like there’s the same quantity of blue in B and C, and about the same amount of green. The quantities aren’t the same: the histograms below show that green is 19 and 20 respectively, and blue is 20 and 22.

You might argue that pie charts are useful for comparing relative quantities. I’d argue it’s possible but difficult to interpret. Take a look at three yellow slices in charts A, B, and C. They look to be similar percentages of the pies, but it’s hard to tell: they are oriented slightly differently, making the comparison unclear. What’s the alternative? Use a histogram with the y-axis representing the percentage.

Pie charts get even more ugly when there’s lots of slices, or some of the slices are ridiculously small. If you’re going to use them — I still don’t think it’s a good idea — then use them when there are only a few values and the smallest slices can be labeled. The pie chart below is ridiculous.

A pie chart that's nearly impossible to read. I'm not sure what conclusions you could draw from this one.

A pie chart that’s nearly impossible to read. I’m not sure what conclusions you could draw from this one.

Why do I care? I’m in the business of communicating information, and I spend much of my time reviewing information that people share with me. Clarity is important — decisions are made on data. See you next time.

Learning Lessons from the Apollo program

1 Reply

I’m fascinated by the Apollo program. It’s a triumph of human endeavour that we sent twenty-four men to the moon, and twelve to the surface between 1969 and 1972. It was a program built with reliable 1950s-designed equipment, and computers far less sophisticated than a cheap modern wristwatch. Achieving Kennedy’s audacious vision was made possible through teamwork, planning, hard work, and ingenuity.

Recently, I decided to learn more about the early Apollo missions to understand what work went into getting us to the moon. It’s an incredibly story, and perhaps more interesting than several later missions.

An incremental approach

Apollo 7 was the first manned Apollo mission to fly in space. It was a confidence-builder, and the first time three people had flown together in space around the Earth. After eleven days in 1968, the crew returned having tested the command module and successfully made a live TV appearance.

The Apollo 7 crew during the first live broadcast from space

The Apollo 7 crew during the first live broadcast from space

The critical pieces of Apollo 7 had flown earlier. The earlier unmanned Apollo 4 and 6 had tested launching a similar unmanned command module and returning it to earth, while Apollo 5 had also tested the same Saturn IB rocket that was used to launch Apollo 7. Apollo 7 was focused on testing putting three astronauts in space in what was a now-trusted setup.

Apollo 8 was a shorter, six day mission. Three men flew to the moon, orbited it a few times, and returned to earth. They were the first to fly atop the Saturn V rocket — the smaller Saturn IB rocket used in Apollo 7 wasn’t sufficient to reach the moon. The Saturn V had been tested in Apollo 4 and 6. This was an audacious mission — testing both a manned Saturn V mission, and visiting the moon. Behinds the scenes, it was a tough sell to management — they didn’t like being that aggressive. But it made sense as a mission: the lunar module was behind in development and wasn’t ready to be tested, and there was fear the Russians might get Cosmonauts around the moon in 1968.

The far side of the moon as seen from Apollo 8. The Apollo 8 crew was the first to ever see the dark side of the moon

The far side of the moon as seen from Apollo 8. The Apollo 8 crew was the first to ever see the dark side of the moon

Apollo 9 was a return to an incremental approach. The mission orbited the earth, tested the lunar module in earth orbit (that would later be used to land on the moon), and included a space walk to test the spacesuits. It also included a rendezvous, which was necessary with the command and lunar modules being separated.

Testing the Apollo 9 lunar module Spider in earth orbit

Testing the Apollo 9 lunar module Spider in earth orbit

Apollo 10 was the full dress rehearsal for landing on the moon. It was time to repeat the test of the lunar module, but this time in moon orbit. The lunar module detached from the command module, and the crew descended to within ten miles of the moon’s surface. They then returned to rendezvous with the command module, and journeyed back to earth. In total, the crew spent eight days in space, and the mission was a huge success — so successful that Apollo 11 was the mission that met Kennedy’s goal in July 1969.

The Apollo 10 lunar module Snoopy returns from almost landing on the moon

The Apollo 10 lunar module Snoopy returns from almost landing on the moon

Interestingly, many people at NASA thought early in the Apollo program that Apollo 12 was likely to be the mission that landed first on the moon. Perhaps if Apollo 9 had happened before Apollo 8 (as was originally planned), there might have been two separate missions to test the manned Saturn V and then a manned Saturn V to the moon. Certainly, if something substantial had gone wrong between Apollo 7 and 10, Apollo 11 would have been repeating validation of the space craft, space suits, and processes.

The tortoise beats the hare

The Soviet Union went all-in with Soyuz 1. It was the first flight of the new Soyuz spacecraft and Soyuz rocket, and was planned to be a rendezvous with the three-manned Soyuz 2. The mission had problems from the start — a solar panel failed to deploy, and this delayed the launch of Soyuz 2. The weather turned bad, and Soyuz 2 didn’t launch. This was fortunate, as the parachute on Soyuz 1 didn’t deploy due to a design fault, and the single Cosmonaut died on reentry. If Soyuz 2 had launched, the crew wouldn’t have survived.

Vladimir Komarov, the cosmonaut who died in the ill-fated Soyuz 1

Vladimir Komarov, the cosmonaut who died in the ill-fated Soyuz 1

This was 1967, a year before Apollo 7. The Soviets went for broke, testing rockets, capsules, rendezvous, and more in one mission. On paper, it looked like they were ahead. The result was failure — and an 18-month delay in the program, and ultimately failure to get to the moon before the Americans. Indeed, Soyuz 4 and 5 in early 1969 eventually completed the mission aims of Soyuz 1 and 2.

The tortoise beat the hare. It was a pretty fast tortoise, but you see the point. The pragmatic approach of trying one complex new component in each mission ultimately made the Apollo program successful. Doing everything at once didn’t work. There’s something in that for all of us. See you next time.

Writing a Book

3 Replies

One day I’ll write another book. Perhaps a sports book about people and their stories, or the story of search engines and the people that build them.

I wrote my first book in 2001 with David Lane, and we rewrote it in 2003 for the second edition. I wrote another book with Saied Tahaghoghi in 2004 – the truth is I started it, and he picked up the pieces when I changed careers and countries; he’s a good man. The first book sold over 100,000 copies over the two editions (I still get a royalty check quarterly) and the second modestly (Saied and I earned our advance back). They’re both dated, old books now.

Web Database Applications with PHP and MySQL. My first book in its second English edition.

Web Database Applications with PHP and MySQL. My first book in its second English edition.

It’s thousands of hours of work to write a book. I spent at least 20 hours per week for 18 months on the first edition of the first book – that’s 1500 hours at least. I got out of bed at 5:30am and I did a few hours of writing before work. I’d also squeeze in a little more after work (typically some proof reading), and a longer period of writing on the weekend (where I’d still get out of bed at 5:30am).

Did I get rich? No. Typically, the authors get less than 10% of the wholesale book price — a couple of dollars per book sold at most. I got more than the minimum wage for the first book (roughly dividing the royalties by the number of hours by the number of authors to get an answer). The second book didn’t pay its way.

The longer I worked in a single sitting, the more productive I was. It takes a certain fixed amount of startup time to begin writing – you reread what you’ve written, edit it a little, get the context back, think about the structure of what you want to say next, and then start writing anew. But I can’t write for an extended period – it’s tiring, and I need to stop and take time away to think about what I want to do next. Three or four hour stints are the most productive for me.

When I wrote the first book, I’d count how many words I wrote in a session, and use that as a measure of success. I’d decide that I was going to write 1000 words before I took a break. It turns out, that doesn’t work for me: I’ve learnt that what’s important is sustained output, averaged over a month or so. Some days, I’ve got writer’s block. But I’ve learnt that that’s when I am doing valuable thinking – I’m working through a larger problem, or thinking through structure, or solving something that’s been bugging me for a while; sometimes, this is a subconscious activity. Other days, I’m a machine: I write as fast as I can type, and thousands of words flow. A whole chapter has been known to flow after a writer’s block.

My second book, Learning MySQL in its one-and-only edition.

My second book, Learning MySQL in its one-and-only edition.

Writing slows down as the book takes shape. I’ll be in the middle of a new section, and I’ll want to reference something else I wrote using something like “as you learned in Chapter <x>, the <something>”. Then I have to figure out what chapter it was, and what exactly was that <something> – that takes time. And, as the book gets longer, you repeat yourself – at least, my memory isn’t amazing enough to make sure I only say the same thing once. I’ll find myself waxing lyrical about some great idea, only to discover that it is somewhere else in the book too. Then it’s a case of figuring out where it should be – which is going to lead to editing a was-finished chapter elsewhere in the book or rethinking what I’m writing today.

I worked with a major publisher, O’Reilly Media Inc., and the wonderful editorial skills of Andy Oram helped me on both books. Editors are awesome – they push, prod, ask questions, push for clarity, and say things that make your book better. But they create a ton of work – you’ll get feedback such as “Perhaps you could merge those two chapters?” or “Chapter 4 needs to be broken into two chapters, and you need to really go into much more depth”. That often happens after you think you’re finished. You’ll get asked for extra chapters, rewrites, more or less content, and complete changes in style. It makes your book better. Between the first edition and second edition of my first book, Andy helped me change my style from a formal computer science style to a more conversational, chatty style – the kind that I use in this blog.

Writing a book isn’t a social experience. You need to enjoy being alone with your thoughts. Prepare for one hour of inspiration and conversation to turn into a hundred hours of hard labor and iteration. If it takes a couple of thousand hours to write a book, ten of them are the inspirational ones where you create the fundamental ideas. I wrote much of the second book in my parent’s Winnebago – parked on the grass, a good distance from the house, deliberately without wifi, and with nothing inside to distract me.

So, why do it? I enjoy writing – there’s rewarding impact in sharing knowledge with thousands of people. If you’re lucky, someone’s success or their impact will be because of what you shared. Or maybe you change how someone thinks or sees the world. Or you make their life better. It’s also cool to see your name on the cover, and to feel it in your hands – it’s even cooler when it’s translated into a language you don’t understand. I wonder what joy there is in knowing people are reading a digital copy? The digital sales on the royalty statement have never quite inspired me the same way.

Have fun. See you soon.

Delivering a Tough Message

1 Reply

A colleague of mine was recently disillusioned with a significant change that had affected them. They’d had a 1:1 meeting with their manager, chatted amicably about work for 25 minutes, and then the manager had dropped the bombshell in the last 5 minutes of the meeting. There wasn’t enough time to discuss the change, the person felt betrayed, and the meeting ended on time and with many questions unanswered.

Many people avoid the tough topics, dislike conflict, and don’t want to deliver tough messages. Unfortunately, it’s part of professional life — and here’s what I’ve learnt about how to do it.

Open with the Facts

If you’ve got something important to share, start by sharing it. Anything else you say before it will be ignored, seem trivial in hindsight, and you may even look rude for not getting started with the important topic. Take a deep breath, say “I’ve got something important to share with you”, and launch into the punchline: outline the conclusion or outcome you want to share.

Be very clear about the facts. For example, “Thanks for meeting me today, Bob. Unfortunately, you were not successful in getting the manager role: I’ve decided to promote Jenny into the role of team manager, I’ve let her know, and we will be announcing it tomorrow”. Here’s another example: “Sammy, we will not be launching your team’s Wizzle product. As a leadership team, we have decided to cancel the project, and reassign you and your team to the Zazzle initiative”.

Don’t get interrupted during the initial discussion. Politely tell the person you’re talking to that you’d like to finish. You owe it to them to share the complete outcome before they get a chance to have a conversation about it. If you want to explain how you got to the conclusion, do it after you’ve shared the conclusion — it’s a huge mistake to walk through the blow-by-blow account of the decision making process while holding back what decision you’ve made until later.

Minimize the Surprises

If you can, don’t surprise people. Lay the foundations for an important conversation by discussing what you’re thinking in the weeks or months that lead up to the decision. If you’re canceling the Wizzle product, hopefully you’ve spent weeks with the team talking about how it isn’t going well, sharing your concerns, and being clear that it isn’t meeting expectations. It’ll then be less of a shock when you make a change.

Sometimes, you have to surprise people. If you’re telling your boss you’re leaving the company, you probably haven’t been talking about it to them for months. That’s ok.

Don’t hide behind others

If you made the decision, own it. Don’t say “we” when it’s actually “I”. Don’t blame others, and don’t bring others unnecessarily into the conversation. Have the courage to own what you decided — you might not be loved for what you’ve decided, but you’ll be respected for having the courage and conviction to own your decisions.

Managers often have problems owning performance discussions. I’ve heard the story many times of a manager saying to an employee “I wanted to give you a 4.0 but my boss decided to give you a 3.0, I’m really sorry”. In 95% of cases, the manager really did drive the outcome — they didn’t put the person at the top of their list, they didn’t unreservedly advocate for the employee, and they were honest about one or more performance issues. So, own it: “When I got together with the leadership team and discussed your performance, I decided your performance was what was expected of someone at your grade and I’ve given you a 3.0. We have an amazing team, and you’ll need to work on three things to be a 4.0 at the next review”.

Conclude clearly

Make sure the conversation has been clearly understood. If you can get the person to play it back to you, you’ll be sure that it’s been understood. If you’re worried that it hasn’t been understood, you should follow it up with a written communication (an email is perfect) soon after the meeting. Indeed, this is often a good idea — I do this when I’m worried there’s room for misinterpretation, or that the decision or actions won’t stick how I want them to.

Above all, be courageous

You’re got past the initial discussion, you’ve conveyed the tough information you decided to share. Don’t change your mind. Don’t be intimidated. Don’t lose your cool. Nothing good ever comes of reversing a tough message in the heat of the moment.

Be calm. Courage doesn’t imply sternness or (worse still) yelling or anger. Be serious, but be rational, empathetic, and fair. Make sure you listen and be respectful. Do unto others what you would have them do unto you.

Good luck. See you next week.

Writing a Performance Review: Part Two

2 Replies

I blogged recently on the topic of annual employee performance reviews. This post continues the story and discusses what I’ve learnt about writing performance reviews.

The Basics of a Review

As I discussed last time, the reviews I deliver typically include a few elements:

  • Sharing company-specific performance ratings
  • Explaining what went well
  • Explaining what didn’t go well
  • Sharing expectations of the employee as the manager

Structuring a Review

I write reviews in paragraph form and hand over a printed copy. I follow up with an electronic copy.

I’ve experimented with several different approaches, and have settled on the following general structure:

  • One paragraph summary of the review: I thank the person for their contributions, communicate the company performance rating, and outline the structure of the document. It’s a good idea to disclose the rating early in the document, it gets key information out of the way and allows the person to focus on really reading the rest of the document
  • A section that describes “what went well”: A series of paragraphs that explain the person’s contributions, why they were valuable, and any suggestions you might have as to how they could be even better. I’m typically reflecting on the goals we agreed when I’m writing this section, as well as reflecting on what I’ve seen, heard, and feedback I’ve received; I discussed last time the topic of how to solicit feedback
  • A section that describes “what didn’t go well”: Same deal, but this time focusing on where there are problems, why they are problems, and what you’d like to see done differently
  • A section that describes “areas for focus”: A few, specific things you’d like to see that are areas for growth. I’m often writing things about future challenges I see in the person’s career, and new skills or competencies that need to developed
  • A one paragraph summary and conclusion

I strive to be balanced. I work hard to ensure that neither the “what went well” or the “what didn’t go well” sections dominate. It doesn’t matter whether the overall performance was outstanding or below expectations, there are always constructive, useful things you can say about performance. My observation is that most people want to hear where they can improve and what they can do better.

My typical reviews are around two pages in length. I’ve received much shorter reviews that were valuable and longer ones that weren’t. I’m not sure length matters — but substance does.

Reviews are important. They’re often tied to financial rewards, they’re written (and so they have gravitas), and they’re usually part of a formal process. It’s therefore important to write down honest, important, business relevant feedback.

Many managers struggle with being transparently honest in reviews. It’s hard to be critical, it’s hard to confront performance issues, and it’s hard to tell someone great that they can be better. As a manager, it’s your obligation. It’s a topic for another time, but this process becomes much easier if you’re honest in every 1:1 about how you see the performance of your employees — the review should never be a surprise.

An Example Summary

Here’s an example of the opening section that I might write:

Sam, thank you for contributions in 2012. I have rated you as <performance rating>.
You made positive contributions to building our next generation platform, hiring and leading the enablement team, and developing your skills in software engineering management. Well done! You did not deliver the new enablement engine on time, have not partnered successfully as we need to with the product team, and you need to continue to grow your presentation skills.

That’s it. A short, to-the-point summary of the document. It doesn’t have to include everything you discuss later — just the key points that you want to highlight, and those that were critical to your decision on the performance rating.

An Example What Went Well Section

I start my “What Went Well” section, imaginatively, with the heading: What Went Well.

I write a few paragraphs, at least as many as the highlights I’ve picked out in the summary section I discussed above. Each of the paragraphs explains what I think and why, and likely includes quotes from others (that I gathered through the feedback process) that support what I want to say. When I’m writing this section, I’m looking at the feedback I gathered from the person’s peers, reports, and other key people that the person interacts with. I’m also thinking about goals and results.

Here’s an example:

Project Alpha exceeded its goal of increasing the business by 3.5% in Europe. Several folks commented on this including examples such as “Thanks Sam for driving the European business, it’s been a pleasure partnering with you” and “You’ve really turned around the story in Portugal, you should feel proud”. It was exciting to see you delivering for Europe, particularly given the language complexities. In particular, it was good to see the successes in translation and price normalization. You also shipped some great work in North America (but missed the goal, that’s discussed later in this review).

Here’s another example:

You’ve made excellent progress on communication and meeting skills. We talked in our review last year about being more concise, letting others communicate, and letting others drive meetings. I can see great progress here – thanks for taking this feedback and working on it. People are noticing too: “I enjoyed working with Sam this year, he’s really developed into a solid communicator”.

An Example What Didn’t Go Well Section

Call me imaginative: I start this section with the heading “What Didn’t Go Well”. It’s ok to call a spade a spade, but you could also use a euphemism if you like such as “Opportunities for Improvement”.

I use the same structure and format as the “What Went Well” section. Paragraphs. One per major point. At least as many as the summary points I made in the introduction.

Here’s an example:

We have had great discussions last year about updating the headcount spreadsheets in a timely fashion, and you reflected in your self-assessment that you’re not doing this. I’d suggest you focus on time management and prioritization — I will support you in developing these skills, including helping you find the right courses you can take. I expect that this is the last year we’ll be having this discussion, and that we will have solved this problem before our next review.

And one more:

I saw development in your ability to think strategically this year, but you can continue to grow in this area. A few people reflected in their feedback to me that you could have taken on more strategic initiatives, or ran with some of the more challenging problems that you saw.  In many of the meetings you’re in, you seeing technical problems that remain unsolved and that you could play a part in driving. You should focus on driving one or two strategic, significant changes this year.

Both of these examples end with a clear call to action and clear expectations. Be clear about what it is you want to see happen.

Areas of Focus

My areas of focus section (titled “Areas of Focus” in my reviews) lists things that I think the person should focus on. They’re not necessarily weaknesses, they’re more often areas where I think the person will need to develop or focus for the future. It’s my advice section.

Here’s an example:

  • Continue to focus on presenting material at the right level for the right audience
  • Many folks worry about your work/life balance (and the example it sets for others)
  • Hire folks who complement your skills

The conclusion

I keep this short. I emphasize the key points, and wrap up with best wishes for the next year. Here’s an example:

You will have a challenging year in 2013, Sam. But I am confident you will do great in the new team, new role, and with your new challenges. Stay focused on strategic thinking! Good luck!

See you next week!

Five tips for Delivering a Presentation

2 Replies

I wrote a few weeks ago on writing a presentation. This week, I offer a few thoughts on delivering one – in no particular order. I’m working on my sequel to my post on performance reviews — expect it next week!

Eye Contact

You want to portray confidence. You don’t want to mumble. You want to engage your audience. Here’s my simplest tip to achieve all three: make eye contact with the audience. Pick out a few friendly faces – people you know who want you to succeed or just people who look friendly – and look them in the eye. Move between those folks as you deliver your presentation.

The side effects are you won’t look down and mumble. You will face the audience and not the slides (the slides aren’t that interested in your talk). You won’t look like you’re only trying to impress your boss (it sure freaks me out when someone spends the whole presentation looking at me). You’ll look like you’re in command as you survey the crowd.

Body Language

Stand up, go to the front, take charge of the room. But don’t plant yourself in one spot – plan to move every few minutes; for example, stroll from one side of the projector screen to the other, or move from the lectern to center stage.

Don’t rock. Plant your feet. Don’t freeze your arms. Make a few gestures – you can even plan to do these every minute or two.

Change your facial expressions every now and then. But not as much as a news anchor – don’t raise your eyebrows every second sentence like they do.

Don’t Read Notes (or Memorize)

Don’t write out your speech. You’ll kill the presentation. Please. For the sake of everyone who is listening. If you must, write a phrase per slide on some palm cards.

Memorizing is the same as writing it down. You will kill the audience.

Only newsreaders can do this, and you are not a newsreader.

Don’t Read Slides

Don’t read the slides to the audience. They can read, and that’s why you’ve put the text on the slides. Again, you’ll kill the talk.

Here’s how I see the role of slides: they’re the key material, and your job verbally is to add flavor to what they’re saying. Relate a story, add an extra point that wouldn’t fit on the slide, point out a key fact, or summarize the key message that the slide is conveying.

The worst thing you can do is to read the slides and track the text with a laser pointer. I hate laser pointers.

 It’s (almost) Impossible to Speak Too Slowly

Earlier this week, I was watching the first election debate in Australia, between the Prime Minister and the Leader of the Opposition. It wasn’t a debate per se, more of a press conference. And as I was thinking about writing this blog post, I was watching and listening for their presentation styles. I noticed careful use of body language, eye contact, and saw the Prime Minister use notes in the form of phrases. Whether either is charismatic is in question, but they are certainly practiced speakers.

What I noticed most was how slowly they spoke. Try an experiment: watch this video (or any video of a leader), and count the number of words they speak in a minute. Now, at work or school, count the number of words a presenter speaks in a minute. Compare and contrast people you think are great, and those that aren’t – you’ll quickly see that the ones you like generally speak slowly.

About 150 words per minute is about right. That’s hard to execute when you’re up on stage – so my practical advice is just to slow down. Speak as slowly as you can – nervousness will make sure it actually isn’t too slow, you’ll go a little faster than you intend anyway.

A Word on My Personal Style

I always walk off stage thinking that I made a mistake in one way or another. That causes me to reflect on what didn’t go well – and to try and capture it, and avoid the same mistakes twice. Here are a few things that I’ve learnt along the way:

  • If I work in humor early, the body language of the audience becomes more positive, and I relax (and become confident, and present more effectively). I try to lighten the mood early – but only when it’s appropriate!
  • I always write my own slides. I can’t present other people’s slides with confidence
  • When I’m repeating a talk, the third time is always the best. Before that I am rehearsing, and after that I am going through the motions
  • When I’m nervous, I gesture too much and I touch my face. I think about putting my hands in a position (such as one hand in a pocket) and keeping them there, and allow them to move only occasionally
  • I practice the endings of my talks. Starts and middles I can do, ends tend to drift. One trick I use is to learn who or what is coming next, and to introduce it in some form. For example, I might say “I would love to spend more time with you today, but I know you are all looking forward to Jenny speaking to you about Hadoop internals. I hope you’ve enjoyed the presentation, and I look forward to seeing you all again soon”. Or something like that. I also clearly signal the end of the presentation verbally

I love to run surveys about large meetings that I run to see what I can learn. I also always ask people in person what they thought of my presentations. If there’s a video, I’ll skim that too (which is always painful – I don’t know anyone who likes watching or listening to themselves). I think I am pretty good at spotting my own flaws – most people are their own best critics.

See you next week.

Writing a Performance Review: Part One

3 Replies

Many companies formally review the performance of employees, and most include a written performance review. This post is the first of two posts where I discuss what I’ve learnt about writing performance reviews and offer a few tips.

The Basics of a Review

Reviews typically include a few elements:

  • Sharing company-specific performance ratings
  • Explaining what went well
  • Explaining what didn’t go well
  • Sharing expectations of the employee as the manager

A manager should at least annually deliver a written review and talk it through in a 1:1 meeting that’s focused only on the review. Some managers send the review, say, an hour before the meeting to give the employee time to read it before the discussion. Other managers print and hand out the review at the beginning of the meeting, and give the employee time to read before the discussion begins. I personally prefer the latter approach — it allows me to react in real time to questions and gauge what’s important for the subsequent discussion. The former approach has the advantage that it gives the employee time and space to read the review.

Gathering Data

Before I write a review, I gather content. There’s a few places I go:

  • Notes, recollections, and emails that reflect on the employee’s performance against their agreed goals. I have regular, weekly or bi-weekly 1:1s with my direct reports, and in these we often discuss performance
  • Reflections from the person I’m reviewing. You’ll find it very helpful to ask your direct report to write down their reflections of their performance and send them to you
  • Reflections on the person I’m reviewing from others. I ask all of the person’s direct reports, peers, and selected other people who they interact with to give me their reflections

Start, Stop, and Continue

Here’s how I go about gathering feedback on the person I’m reviewing. I send an email like this to their reports, peers, and selected other folks they interact with:

Hello,
I have found that gathering feedback on the performance of my direct reports is one of the best ways to assess how they are doing. Please provide your candid feedback, which I will summarize and share anonymously.

To keep it simple, I am asking you to email me three things each that <name of person> should:

  • start
  • stop
  • continue doing

I will take all of the feedback and synthesize it into the review so your comments will be kept confidential.

Please provide your feedback by <date I need it> to <my email address>.

Thanks, Hugh.

Synthesizing the Content

Once I’ve got the replies, I look for consistent themes, great anonymizable examples, and points that help me reinforce the messages I want to share and those that align with the personal reflections of the person. I often include near-direct quotes from the responses, I’ve found they are powerful ways to convey messages. I frequently rewrite or rephrase the quotes so that language styles and quirks don’t give away who contributed the feedback.

By the way, it’s also useful to ask the person you’re reviewing to write their reflections in the Start, Stop, Continue model.

Next in the Series

In this follow up post, I share what I do next — how I put together a review and share it with the person I’m reviewing.

The Cassini Search Engine

7 Replies

Cassini has been in the news lately: Wired.com recently featured a story on Cassini and I was interviewed by eCommerceBytes. Given the interest in the topic, this week’s post is more on the inside Cassini story.

The Beginning

At the end of 2010, we decided that we needed a new search platform. We’ve done great work in improving the existing Voyager search engine, and it had contributed substantially to the turnaround of eBay’s Marketplaces business. But it wasn’t a platform for the future: we needed a new, modern platform to innovate for our customers and business.

We agreed as a team to build Cassini, our new search engine platform. We started Cassini in around October 2010, with a team of a few folks lead by industry veteran Nick Whyte.

Voyager

Our Voyager search engine was built in around 2002. It’s delivered impressively and reliably for over ten years. It’s a good old workhorse. But it isn’t up for what we need today: it was architected before many of the modern advances in how search works, having launched before Microsoft began its search effort and before Google’s current generation of search engine.

We’ve innovated on top of Voyager, and John Donahoe has discussed how our work on search has been important. But it hasn’t been easy and we haven’t been able to do everything we’ve wanted – and our teams want a new platform that allows us to innovate more.

Cassini

Since Voyager was named after the late 1970s space probes, we named our new search engine after the 1996 Cassini probe. It’s a symbolic way of saying it’s still a search engine, but a more modern one.

The Cassini-Huygens probe, the namesake of our Cassini search engine.

The Cassini-Huygens probe, the namesake of our Cassini search engine.

In 2010, we made many fundamental decisions about the architecture and design principles of Cassini. In particular, we decided that:

  • It’d support searching over vastly more text; by default, Voyager lets users search over the title of our items and not the description. We decided that Cassini would allow search over the entire document
  • We’d build a data center automation suite to make deployment of the Cassini software much easier than deploying Voyager; we also built a vision around provisioning machines, fault remediation, and more
  • We decided to support sophisticated, modern approaches to search ranking, including being able to process a query in multiple rounds of ranking (where the first iteration was fast and approximate, and latter rounds were intensive and accurate)

Cassini is a true start-from-scratch rewrite of eBay’s search engine. Because of the unique nature of eBay’s search problem, we couldn’t use an existing solution. We also made a clean break from Voyager – we wanted to build a new, modular, layered solution.

Cassini in 2011

January 2011 marked the real beginning of the project. We’d been working for three or four months as a team of five or so, and we’d sketched out some of the key components. In 2011, we set the project in serious motion – adding more folks to the core team to begin to build key functionality, and embarking on the work needed to automate search. We also began to talk about the project externally.

Cassini in 2012

In 2012, we’d grown the team substantially from the handful of folks who began the project at the end of 2010. We’d also made substantial progress by June – Cassini was being used behind-the-scenes for a couple of minor features, and we’d been demoing it internally.

In the second half of 2012, we began to roll out Cassini for customer-facing scenarios. Our goal was to add value for our customers, but also to harden the system and understand how it performs when it’s operating at scale. This gave us practice in operating the system, and forced us to build many of the pieces that are needed to monitor and operate the system.

The two scenarios that we rolled out Cassini for in 2012 were Completed Search worldwide, and null and low search in North America. Mark Carges talked about this recently at eBay’s 2013 Analysts’ Day.

Completed search is the feature that allows our customers to search listings that have ended, sold or not; it’s a key way that our customers do price research, whether for pricing an item they’re selling or to figure out what to pay when they’re buying. Because Cassini is a more scalable technology, we were able to provide search over 90 days of items rather than the 14 days that Voyager has always offered.

When users get no results from a query (or very few), Voyager offered a very basic solution – no results and a few suggested queries that you might try (this is still what you’ll see on, for example, the Australian site today). Cassini could do much more – it could help us try related queries and search more data to find related matches. After customer testing, we rolled out Cassini in North America, the UK, and Germany to support this scenario – a feature that’s been really well received.

Cassini in May 2013

In January 2013, we began testing Cassini for the default search with around 5% of US customers. As I’ve discussed previously, when we make platform changes, we like to aim for parity for the customers so that it’s a seamless transition. That’s our goal in testing – replace Voyager with a better platform, but offer equivalent results for queries. The testing has been going on with 5% of our US customers almost continuously this year. Parity isn’t easy: it’s a different platform, and our Search Science and Engineering teams have done great work to get us there.

We’ve recently launched Cassini in the US market. If you’re querying on ebay.com, you’re using Cassini. It’s been a smooth launch – our hardening of the platform in 2012, and our extensive testing in 2013 have set us on a good path. Cassini doesn’t yet do anything much different to Voyager: for example, it isn’t by default searching over descriptions yet.

Cassini in 2013 and beyond

We’ll begin working on making Cassini the default search engine in other major markets later in the year. We’re beginning to test it in the UK, and we’ll work on Germany soon too. There’s also a ton of logistics to cover in launching the engine: cassini requires thousands of computers in several data centers.

We’ll also begin to innovate even faster in making search better. Now that we’ve got our new platform, our search science team can begin trying new features and work even faster to deliver better results for our customers. As a team, we’ll blog more about those changes.

We have much more work to do in replacing Voyager. Our users run more than 250+ million queries each day, but many more queries come from internal sources such as our selling tools, on site merchandizing, and email generation tools. We need to add features to Cassini to support those scenarios, and move them over from Voyager.

It’ll be a while before Voyager isn’t needed to support a scenario somewhere in eBay. We’ll then turn Voyager off, celebrate a little, and be entirely powered by Cassini.

See you next week.

The size and scale of eBay: 2013 edition

5 Replies

It’s time for an update of the eBay Marketplaces interesting statistics that I shared last year. eBay Marketplaces means we’re the team that builds ebay.com, ebay.co.uk, ebay.de, ebay.com.au, and most of the other worldwide marketplaces under the eBay brand.

eBay Marketplaces sold over US$75.3 billion in merchandise in 2012

eBay Marketplaces sold over US$75.3 billion in merchandise in 2012

Here are some refreshed and new facts:

  • We have over 50 petabytes of data stored in our Hadoop and Teradata clusters
  • We have over 400 million items for sale
  • We process more than 250 million user queries per day
  • We serve over 100,000 pages per second
  • Our users spend over 180 years in total every day looking at items
  • We have over 112 million active users
  • We sold over US$75 billion in merchandize in 2012

eBay’s an exciting place to be — plenty of engineering, business, and commerce challenges that are driven by users, items, traffic, and sales. See you next week.

Selling on eBay

2 Replies

I gave a webinar last week on Search at eBay. Thank you to those who attended and asked great questions. The webinar was recorded and you can listen and view the slides here.

Search at eBay. A tour of search from a recent webinar.

Search at eBay. A tour of search from a recent webinar.

In the webinar, I present a background of some of the facts and figures about eBay (an updated version of this post), a short introduction to how search works, and a tour of our current and future search platforms. The talk concludes with over thirty minutes of Q&A. The middle of the talk is dedicated to explaining how to sell successfully on eBay, and I summarize that advice in this post.

Listing for Success in Search

It’s important to remember that search isn’t the only step in selling on eBay. It’s a three-step process:

  1. Finding an item in search
  2. Choosing an item
  3. Purchasing the item

Visibility in search is therefore important, but it isn’t sufficient to guarantee sales. You must also focus on ensuring buyers click on your item and make the choice to purchase it.

On the first point: it’s true that most successful sales begin with a search on eBay, but many others occur through buyers finding items on another search engine, in an email,  through the merchandizing that we feature, or through another source such as a social network. Searches on eBay also don’t always have a keyword: many buyers browse through our categories to find the item they want.

Visibility in Search

Our search team focuses on three tenets of delivering a great experience for our buyers:

  1. Trust: ensuring our buyers get a retail-like experience from great sellers
  2. Value: ensuring our buyers find great deals, including shipping costs
  3. Relevance: ensuring our buyers see items that match what they want

If you focus on these three tenets, you will be successful in gaining visibility in search.

Delivering Trust, Value, and Relevance

Here is some specific advice on how you can be found in search, have your items chosen in search, and drive sales:

  • List in the correct category with a meaningful title; a meaningful title contains only words that accurately describe the item, it omits “spam words” (off-topic words) and isn’t unnecessarily short or long
  • Use item specifics and item condition; we match queries against the item specifics, and to be found successfully we recommended you adopt item specifics
  • When it makes sense for your business, use a long-duration, multi-quantity Buy It Now format
  • Study what it takes to be a Top Rated Seller on eBay; it is our definition of the key tenets of being a trusted seller on eBay, and we use those trust signals in search
  • Have great, large, clear pictures for your listings; we know from many experiments that pictures matter to our buyers, and you’re much more likely to be chosen in search and have sales if your pictures are high-quality
  • Be price competitive; this changes frequently, and you need to understand the market price for your items and be prepared to make changes
  • Specify shipping costs; our customers want great value, and that includes value in shipping
  • Have a clear, structured, and comprehensive item description. Stay “on topic” and describe your item accurately
  • Offer fast shipping and international options; fast shipping matters to our buyers, and offering international options means you have more exposure to more customers

We don’t offer specific answers to specific questions about how Best Match works on eBay. For example, we don’t comment on how we use each of the item specifics in search or whether having certain keywords in certain places in your titles matters. Why not? We want sellers to focus on the key tenets of Trust, Value, and Relevance, and not on specific features that may change or that might give sellers a short-term unfair advantage over other great sellers. Indeed, if a shortcut works today, it may not work tomorrow — we want a level playing field for all sellers, and we’re continually improving Best Match to use more Trust, Value, and Relevance information.

I encourage you to listen to the webinar for a richer explanation of how to sell successfully on eBay with a search-centric focus. See you next week.