Monthly Archives: June 2012

The size, scale, and numbers of eBay.com

I work in the Marketplaces business at eBay. That’s the part of the company that builds ebay.com, ebay.co.uk, ebay.de, ebay.com.au, and most of the other worldwide marketplaces under the eBay brand. (The other major parts of eBay Inc are PayPal, GSI Commerce, x.commerce, and StubHub.)

I am lucky to have opportunities to speak publicly about eBay, and about the technology we’re building. It’s an exciting time to give a talk – we are in the middle of rewriting our search engine, we’ve improved search substantially, we’re automating our data centers, we’re retooling our user experience development stack, and much more.

At the beginning of most talks, I get the chance to share a few facts about our scale and size. I thought I’d share some with you:

  • We have over 10 petabytes of data stored in our Hadoop and Teradata clusters. Hadoop is primarily used by engineers who use data to build products, and Teradata is primarily used by our finance team to understand our business
  • We have over 300 million items for sale, and over a billion accessible at any time (including, for example, items that are no longer for sale but that are used by customers for price research)
  • We process around 250 million user queries per day (which become many billions of queries behind the scenes – query rewriting implies many calls to search to provide results for a single user query, and many other parts of our system use search for various reasons)
  • We serve over 2 billion pages to customers every day
  • We have over 100 million active users
  • We sold over US$68 billion in merchandize in 2011
  • We make over 75 billion database calls each day (our database tables are denormalized because doing relational joins at our scale is often too slow – and so we precompute and store the results, leading to many more queries that take much less time each)

They’re some pretty large numbers, ones that make our engineering challenges exciting and rewarding to solve.

Any surprises for you?

How are you doing? How to ask for feedback

Listening to feedback can help develop your career. When you know what others think, you can make choices about what and how to improve, and learn and develop competencies that will improve your effectiveness.

Why Get Feedback?

I’d say nine out of ten people have a healthy self-perception. They’re good at things, and they know what they are. They’re bad at things, and they know what they are. The other one in ten fall in two buckets: thinking they’re awesome when they’re not, or thinking they’re terrible when they’re not.

Net, I believe that most people actually know what they’re good at, and what they’re bad at. It’s good to have your list: try writing down your top five strength and weaknesses as you see them. But do they matter? Does anyone realize that weakness? Is it preventing career progress? Are people talking about it? Are you worried about a flaw that no one cares about? Are you great at something that doesn’t matter?

To find out what matters, it’s important to know what others think. You’ll generally need to ask them – good managers will give you feedback at review time, the rest of the time you’ll need to ask. Most other people aren’t going to burst into talking about you. Asking them is a good career move: it shows you’re interested in learning and growing, that you care what others think, and you’re open to feedback. Set up a 1:1 meeting with a few people whose opinions you know you will value, you know can help you create an actionable list, and find out what they think.

Who do I ask?

Who could or should you ask? I’d definitely ask your manager – it’s their job to help you grow, be successful, and have impact. Their opinion also matters – you’re not going to get that key assignment or promotion without their help (there are no exceptions to that rule). I’d ask a couple of peers, and someone who’s junior to you too (which could be someone who reports to you). That’s five people – a good sample. If you want more, try an ex-boss, your boss’s boss (you may find out what the boss says to his boss about you), or someone you admire in another team but that you don’t work with daily.

Receiving Feedback

Never ask for feedback and argue with the giver that they are wrong. It doesn’t matter if they are – you asked for their opinion, which is a subjective view that makes sense in the movie that’s playing in their head. If you argue, you’ll look foolish, you won’t get any more feedback, and you’ve ruined the discussion as a career move. So, listen, write down what they say, say “thanks” frequently, and ask polite inquisitive questions that help you understand their perspective (don’t ask leading questions that show you don’t agree). Basically, don’t talk much.

You may have trouble getting people to be critical of you. It might take a few explicit questions to get them started, and you’ll need to show you’re thankful for the feedback if you want to get more. Try saying things like “I want to have more impact – and I know to do that I’m going to have to learn new skills. What skills do you think I need to develop?” or “If you were me, what’s the number one thing you’d work on changing?” or “What’s the biggest challenge you think I’d face at the next level or in your role?”. Keep your body language positive: no folding your arms, no furrowed brow, no looming over them and pacing around. They’ll be more comfortable in their territory – go to their office if you can. And be very genuine – people can spot a fake from a mile away.

Most of the time, this is going to turn out great. Most people want to help others. And most recipients of feedback know what they’re going to hear – since most recipients do have that healthy self-perception. So, hopefully, you’ll come away with a great list of positives and negatives, and hopefully they’ll be reinforced by the various people you speak to.

One last word of advice: don’t enter into this process lightly. If you ask for feedback, people are going to expect you to do something with it. You’ll score negative points if you ask for feedback and do nothing – people expect you to do something, and they expect to see some signs of life that show you’re making progress.

Next time, I’ll share some thoughts on how to use feedback to create a development plan.

Passwords

Passwords are in the news lately, and particularly associated with ‘leaks’.

In general, passwords are rarely leaked, because they usually aren’t stored. What is usually leaked are the password hashes.

How Passwords Work

When you provide a new password for an account, here’s what typically happens:

  1. The company hashes your password so that it becomes a string of characters that isn’t the password (more in a moment)
  2. The company stores the hash
  3. The company throws away your original password

When you come back to the site, and provide your password, here’s what happens:

  1. You type your password
  2. The password is hashed (using the same algorithm as before)
  3. The hash is compared to what’s stored by the company
  4. If they’re the same, you’re in. If they’re different, your password is wrong, try again

One-Way Hashes

The method that’s used to turn a password into a hash should be one-way. That is, it should be theoretically impossible to reverse the hash into the password. That’s actually pretty easy: most hashing algorithms throw away lots of information, and so the hash is lossy (that is, it has less information in it than the original password — it’s just a very-likely-to-be-unique string that represents the password). So it is actually usually impossible to reverse a password hashing algorithm.

There are many different algorithms for one-way hashing that can’t be reversed. The most popular one-way hash is the 128-bit MD5 hash, though this has been shown to be somewhat insecure (I’ll explain this in a minute). More recent approaches such as SHA-2 are similar in their theory but more secure.

Here’s an example. If you put the string password into an MD5 function, you get 5f4dcc3b5aa765d61d8327deb882cf99 as the output.

How can something that’s lossy be insecure? Since it’s impossible to reverse it, how can it be used by a hacker to find the original password?

The answer is that you can try putting vast numbers of original passwords through the hashing algorithm, and see if you can match the output to what is actually stored. So, for example, you could hash every English word using MD5, and see if you get a string that matches the leaked hashes. If you do, you’re in. And if you find one that matches, chances are you find many more that match too — lots of users have the same password.

That’s the problem with the MD5 hash: computers are sufficiently fast that it’s possible to try very, very large numbers of input strings and compare them to hashes (more later). And so if you’ve got the computing resources, you can effectively “reverse” the hashing.

Salting Passwords

The best way to make it very hard for hackers to figure out the original passwords is to salt them. Salting is a pretty simple idea: instead of hashing the password, hash the password and some other string concatenated to it. That “some other string” is non changing: it could the user’s user ID in the system, the time they created their account, or something else that’s stored but not typically not known to the user.

Here’s an example. Suppose the user supplies the password “magpie”. Instead of hashing this alone, we might append their user ID, say “123456” to the string. So, we’d be hashing “magpie123456″ to get our password hash that’s stored in the system.

How does salting help? Well, it makes it hard to reverse the hashing: a hacker won’t be able to match common passwords (say, English words) against the hashes, and so they’ll effectively have to try vastly more input strings. When they crack one, they typically also won’t crack more, since even if the passwords of two users are the same, the salted strings aren’t — “magpie123456″ and “magpie123457″ produce vastly different hashes.

Of course, if a hacker gains access to the complete database, and the salting algorithm — and so they have all of the input data, and the algorithm for creating the salted strings, you’re in lots of trouble. But if they’ve done that, you’ve got worse things to worry about.

Salting is essential. Don’t build a password system without it. And when you do build the password system, consider a hashing algorithm such as SHA-2. Even the author of MD5 doesn’t recommend using it now — computers have enough resources to brute force crack MD5 using a birthday attack.

I’ll be back next week.

6 secrets for successful small teams

Managing small engineering teams is fun. You’re close to the technology, close to team that’s writing code, and solving problems hands-on. Don’t get me wrong: it’s satisfying, rewarding, and impactful to manage large teams, but there’s nothing like being in the details and seeing a product come together. You lead a small team, by being an example that folks can follow; in contrast, you manage a large team. It’s different.

Here’s six things I’ve learnt along the way about leading small teams. This post was inspired by Greg Brockman’s 6 secrets for building a super team — Greg explained how to hire a small super team, and this is my sequel on how to make it world class.

1. Lead by Example

I’ve seen newly-minted leaders of small teams fail. One of the common reasons is they stop doing what they’re good at: being technical. And when they stop, engineers stop following them. Engineers want their leaders to be driving engineering, working on the hardest problems, and setting an example.

Great leaders of small teams are at least 50% technical, they spend more than half of their time writing code, creating designs, proposing architectures, doing code reviews, solving technical problems, or otherwise getting deep in the details. They know everything that’s going on in their team — they’ve got the right people working on the right problems, they’re driving the people at the right pace, they’ve holding people accountable for doing everything the right way, and they know the teams’ designs and code like they know their own. In the other 50% or less of their time, they’re passionate and focused on process, people, teams, and leadership — they have just the right amount of process to keep the team on track and happy, they’re managing their dependencies, and they’re spending time growing their people and team.

2. Set clear, simple goals, with clear, measurable metrics

You won’t be successful if you don’t know what success is. It’s surprising how many teams build a product without a way of measuring whether they’ve been successful, deciding when they need to be successful, or even without knowing where they’re headed. You should always start with goals and metrics.

Let’s suppose you’re leading the home page team at a major web company. You’re charged with leading a team of seven engineers, who together build all of the components on the page, and deliver everything from the platform pieces through to the pixels. You know who your competitors are, and you’ve got some great ideas on how to build a better home page than they’ve got. You’ve thought about this — you know that owning the home page means you’re the front door to the store, it’s your job to get the customers inside so that they can use all the products and features that the other teams have built.

I’m a fan of having two or three clear goals and objective ways of measuring them. You might decide that success is having the acknowledged leading home page on the web. How might you measure that? Here’s two or three ideas: first, you have lower abandonment than your competitors (lowest percentage of visitors who don’t click anything); second, you win a head-to-head taste test against your competitors at least 2/3rds of the time; and, third, your page is faster to load that your competitor’s. These are all criteria you can measure on a daily or weekly basis, and you can track them and share with the team.

If you want to create drive, fever, and passion in your team, consider setting a Bold Hairy Audacious Goal (BHAG) date. Decide that you’re going to have the best homepage by January 1. Put a graph up on the wall with the months on the x-axis and your objective, measurable success criteria on the y-axis. Every week, update the points for you and your competitors. Huddle the team around the graph, and work together to make success happen. You’ll be surprised how effective this can be in firing up your team, and getting creative ideas into the product.

3. Balance the portfolio

Small successful teams spread their bets. They have many small projects going on, and a couple of large ones. Importantly, the vast majority of the projects are impactful — they’re projects that are pushing the team towards their measurable goals. Good teams avoid doing work that doesn’t matter.

Why are small projects important? Well, first, they mitigate the risk if a large project gets delayed or doesn’t hit paydirt. Second, they give the team a real rhythm: every week there’s something new going out that moves the needle, and the team stays fired up and on track to their goals. Doing only large projects can be a death march — and it’s hard to keep teams fired up on one of those.

4. Meet the team

You have to know your team to be a successful leader. You need to understand the people who work for you, and customize your style to get the most from them. Some folks will need more advice, others will need to be pushed, some will rebel if you push, and some just need to know the goals and be given plenty of space. There’ll always be some conflict brewing, and everyone will have something on their mind that they want you to know — and the only way to find it out is to meet them regularly, really get to know them, build trust, and help them be successful.

I recommend a weekly 1:1 that’s at least 30 minutes, and not focused on the day-to-day. Ask questions in a 1:1, and leave plenty of white space in the conversation for the person you’re meeting to talk to you. Listen to what they say, and ask more questions that explore what they’re saying. Most importantly, be open and honest — say what you think, and it’ll likely work both ways. Conversation is key to being a successful leader.

5. Keep it simple

When the team’s small, you don’t need onerous process, too many meetings, or too much email. I’d recommend a daily stand-up meeting as the basic tool to stay in touch: go around in a circle, and ask each person to say what they did yesterday, what they’re planning on doing today, and anything that’s blocking them from succeeding. Buy some hour glass timers, and make sure everyone stays under two minutes. Don’t allow point-to-point conversation, the meeting is strictly for broadcasting information (the 1:1 or 2:1 conversations can happen after the meeting). Tell the team you will buy them coffee and food if the meeting goes over 20 minutes, and work hard to keep it under 15. Keeping it tight on time and high on information content will ensure you get good attendance and participation.

My hour glass timers. A large one tracks meeting time (20 minutes), and the two smaller ones are for individual speakers (2 minutes each) — one for the current speaker, and one for the next speaker.

When it comes to tracking a project, I recommend simple (and so do most of the people I work with). If it’s six to eight people, you can track projects with sticky notes, a whiteboard, and a camera for recording key information. I’ve also used a simple spreadsheet, where engineers can record the list of tasks that they’re working on and the time they will take, and the sheet automatically burns down the time remaining and computes completion dates. All the engineer has to do is keep the list of tasks in the sheet accurate, and the time estimates up to date, and the rest takes care of itself. In either case, simple is the key — very low overhead for the engineers, and pretty simple for the leader.

6. Keep it small

What I’ve shared applies to teams of 4 to 8 people that work for one leader. Beyond that, the team isn’t small anymore — you’ll find it nearly impossible to be more than 50% technical, and still get in the 1:1s and run an effective team. I’ve met a few people who can do it, but they’re usually super experienced or have super experienced teams.

We’ve all heard the stories of small startups with a few employees doing amazing things (think instagram or pinterest). It’s often because they’re lead by technologists, who’ve set crisp goals, spread their bets, kept it simple, and know their teams well. If you get your small team leadership right, your small team can change the world — and small teams do regularly change the world. Good luck.