Author Archives: Hugh E. Williams

About Hugh E. Williams

Founder, Advisor, Professor, and Investor. Former Tech Exec.

November Update on CS in Schools

Leave a reply

(This is a cross-post from our CS in Schools website.)

CS in Schools is closer to reality. We’re excited about the next two months, which are focused on the final steps in launching our programme in schools for term one.

laptop computer writing board girl technology symbol geometry training student education monitor brand product study illustration diagram mathematics school learn learning document idea pi presentation digits physics cartoon know radius diameter

We remain focused on two priorities for 2019:

  1. Helping teachers become confident and competent in teaching coding in schools
  2. Evaluating our programme to understand what works and how to scale it in 2020 and beyond

Our plan to achieve those priorities has changed over the past six weeks. Our most significant change is that we have moved to a volunteer-driven model that we believe will scale better in 2020 and beyond. Similarly to TEALS, this model places software engineers in the classroom alongside school teachers. We expect our plans to continue to evolve as we learn.

The schools in our pilot are deep into timetabling and assigning teachers to the programme; we work with them almost daily. We are also busy recruiting volunteers, and have pitched the programme to developers at SEEK, MessageMedia, Ento, and CultureAmp. We’re building a two day training programme for volunteers to learn the basics of working in a classroom and teaching, and this will run on 19 and 20 January.

We have successfully raised enough funds for the first six months of the programme, and continue to look for more funding. The most significant major component that remains is curriculum development, which has only just begun.

If you’d like to help us, there’s two three things you can do! First, if you’re an experienced software engineer, please consider volunteering. Second, if you work in the tech industry, invite me to come and speak to your developers about being involved as volunteers. Third, if you want to support our work, please lobby your employer to donate to the programme. Oh, and if you’re awesome at building websites, we need help!

Programme Overview

CS in Schools is part of RMIT‘s Policy and Impact Portfolio. It’s a charity initiative that’s focused on helping teachers confidently teach computer science to high school students in Australia.

Today, most schools struggle to teach coding: there’s a shortage of teachers who feel qualified to teach computer science, and most successful coding classes are run outside of school hours. We believe that today’s teachers can effectively teach coding if they’re supported through in-class professional development.

Many of the important and best paid jobs of this and the next generation will require computational thinking. Even if a student doesn’t study computer science at university, it’s essential they have the basics because just about every job will be changed by technology. We want every student in Australia to have this opportunity.

In 2019, we are piloting a programme with eight schools, and studying how successfully we can help teachers ramp-up their skills. Beyond 2019, we plan to launch this programme broadly.

Our programme continues to evolve as we learn. We’ve refined our priorities to:

  • Helping teachers become confident and competent in teaching coding in schools
  • Evaluating our programme to understand what works and how to scale it in 2020 and beyond

To achieve these priorities, we will trial novel teacher professional development pilot programmes in 2019. We will make this programme free for schools in 2019, supported by generous donations.

Our plans to deliver the programme are as follows:

  • Find volunteers who are experienced software engineers and prepared to work in a classroom for around two hours per week
  • Train the volunteers through a two-day teaching workshop (onJanuary 19 and 20) and through additional out-of-class support
  • Pair a volunteer with a teacher from a school in the pilot
  • From February, the volunteers will work in the classroom with the teachers from the schools:
    • In the first term (or semester) working in a school, the volunteer will deliver the syllabus to the students and the school’s teacher will learn through observation and by providing student support
    • In our second term (or semester), it’s anticipated we’ll switch the roles: the school’s teacher will be the primary driver, with in-class support and training from the volunteer
  • We’re hopeful that beyond the first two iterations, the school’s teacher will be independent, and require limited support from our programme

Our pilot is focused on:

  • Helping Year 7 teachers in Victorian high schools
  • A one-term (or semester) Year 7 computing subject of two (or sometimes three) hours per week
  • Providing everything that’s needed to teach the subject, including lesson plans, assignments, hardware, and software
  • Covering the “harder parts” of the Victorian DigiTech curriculum, with a focus on the coding skills. In total, we’ll together deliver around half of the recommended 40 hours of the curriculum, and expect that the other half is delivered in other subjects

Volunteering

We have pivoted our model to a volunteer-based scheme. We believe this is more scalable than our initial approach when we conceived the programme, where we planned to hire teachers with a computing background. We feel great about our new approach, which is similar to TEALS, In2Science, AppsForGood, and a few others.

The volunteer-based programme is conceptually simple. We approach tech companies, explain the programme to software engineers / developers, and call for applications through our website. We meet the applicants, discuss the programme, and select applicants based on their suitability (where they’re located, the time they have available, the school needs, and our overall assessment of their aptitude). We then assign successful applicants to a class. We then train the volunteers to have basic teaching skills, connect them to their partner teacher in a school, and support them through one or two terms of volunteering in schools. (There’s more detail here.)

We launched this new model around three weeks ago, and we have twelve applicants that we’ve met. We have two more pitches at tech companies, and we’re yet to push out the programme on social media; indeed, this is the first time we’ve shared it broadly. We likely need 15 to 20 volunteers: we will have close to 15 classes to cover, and we know we will need volunteers who can jump in to cover for others on short notice or replace a volunteer.

We’re feeling good about the new model, and we’re thrilled with the response from the software industry.

Volunteerism can scale nationally. If we’re able to find, recruit, and train software engineers, we can scale to 100 schools in 2020, and we’d love to scale to 1000 schools in 2021. The challenges are recruiting, training, and managing the volunteer cohorts, and figuring out models that will work for regional and remote areas. We already know that the administrative cost of working with schools and volunteers is high, and we will need help.

Fundraising

We have raised funds to support our pilot from three generous donors, Leigh Jasper, Martin Hosking, and Adam Lewis. All three see the immediate need to change computing education in Australia, and we’re thrilled to be able to do something on their behalf. We thank them sincerely, they’re amazing people.

We need more money to fund our programme, and we continue to work hard to raise money from individuals and corporate partners. If you’re interested in supporting the programme, or have a connection into a funding source, we’d love to hear from you.

Corporate Support

If you have a connection to a software company, or a company that employees software engineers (or developers or computer scientists or whatever you’d like to call them!), we’d love to hear from you. We need ways to scale volunteerism, and we’re keen to visit tech companies and pitch the programme. Today, we’re only working in Victoria and mostly in Melbourne, so that’s our focus for now. Of course, we plan to expand Australia-wide in 2020 and beyond.

Thanks for reading all the way to here. Cheers, Hugh and Selina.

CS in Schools: September Update

1 Reply

CS in Schools is a not-for-profit initiative at RMIT University that’s focused on helping high school teachers develop their confidence and competence in teaching computer science.

September was our third month out of stealth mode. Since our last update, we’ve formally joined forces with RMIT and advertised for teachers to join our programme. Please share this link with teacher colleagues, friends, and anyone who might want to work with us.

We signed up an eighth school into our free pilot, and we’re at capacity for 2019. We’re now deep in the logistics of timetabling, resourcing, and planning.

Woman Programming on a Notebook

Summary

CS in Schools is part of RMIT‘s Policy and Impact Portfolio. It’s a charity initiative that’s focused on helping teachers confidently teach computer science to high school students in Australia.

Today, most schools struggle to teach coding: there’s a shortage of teachers who feel qualified to teach computer science, and most successful coding classes are run outside of school hours. We believe that today’s teachers can effectively teach coding if they’re supported through in-class professional development. Microsoft’s TEALS programme exists in the US, and we want Australian teachers to have a similar opportunity.

Many of the important and best paid jobs of this and the next generation will require computational thinking. Even if a student doesn’t study computer science at university, it’s essential they have the basics because just about every job will be changed by technology. We want every student in Australia to have this opportunity.

In 2019, we are piloting a programme with eight schools, and studying how successfully we can help teachers ramp-up their skills. Beyond 2019, we plan to launch this programme broadly.

We’ve had an exciting September. In summary, in the past month:

  • We named ourselves CS in Schools, and bought the domain csinschools.io. You can now contact us at hugh@csinschools.io and selina@csinschools.io.
  • We’ve signed-up our eighth school into our 2019 pilot, and we’re at full capacity
  • We’ve become part of RMIT University, giving us access to resources, educators, facilities, and enabling us to become a Tier 1 DGR charity (an important Australian tax status)
  • We’ve advertised for teachers to work with us
  • We’re refining a roadmap for what happens from now until we start the programme in February

To learn more about our programme, see the “Programme” section in our August update.

September Update

September was another terrific month for our programme: it’s all coming together, and building momentum towards our 2019 pilot.

This month, we formally joined RMIT University. We’re now part of the Policy and Impact Portfolio, and have an office inside RMIT’s Activator space. RMIT’s generosity has been amazing: they’re providing offices, finance support, HR and legal support, and support for philanthropic engagement. Of course, being part of RMIT also gives us enormous credibility and opens all kinds of doors. We thank Vice-Chancellor Martin Bean and his team for their kindness and support in helping CS in Schools take a leap forward.

We finalised the schools that we’ll be working with in our pilot in 2019. These are:

It’s likely that somewhere between 10 and 15 teachers will be trained through our programme next year, and over 1,000 Year 7 students will experience our course. We’re excited to be able to study the outcomes of the programme, and hopefully scale it in 2020.

We are working hard on finalising financial support for 2019. After building our draft timetable and revising our budget, we could do with more money. If you have corporate connections or connections to philanthropic donors, we’d love an intro!

We are now hiring teachers. We plan to hire up to three qualified high school teachers who have experience teaching computing. Our job ad is live, and here’s a link to the position description. Please share it broadly with your colleagues, family, and friends who might be interested. Hopefully, in our next update, we can share the news of who we’ve hired, and have made progress on developing our lessons.

The final news from the month is that we’re getting closer to Microsoft’s TEALS programme. Kevin Wang at Microsoft leads this initiative, and he’s been generous in sharing his time to help us learn about onboarding schools, curriculum development, and structure of their engagement with volunteers, schools, and universities. Under an NDA between RMIT and Microsoft, we’ve recently been able to access their materials, and this has helped us not reinvent the wheel from first principles. We hope to further deepen our TEALS engagement over the next month.

Thanks again for making it all the way to the bottom of the update. Let us know if you’d like to learn more about any topic. Cheers, Hugh and Selina.

 

 

Computing in Schools: August update

1 Reply

We are a charity that’s focused on helping high school teachers develop their confidence and competence in teaching computer science.

It’s our second month out of stealth mode, and it’s been a good one for us. We’re exploring becoming part of RMIT University, and SEEK and REA became supporters of our programme. We have committed donations from generous supporters, and the first year of our programme is completely funded. We’ve been learning from a similar programme in the US that’s supported by Microsoft. We’ve refined our plans, with a sharper focus on Year 7 teacher professional development. We’ve signed up seven schools into our free pilot, and we’re in discussion with three more — we are pretty much at capacity for 2019!

Summary

We’ve launched a charity that’s focused on helping teachers confidently teach computer science to high school students in Australia. Today, most schools struggle to teach coding: there’s a shortage of teachers who feel qualified to teach computer science, and most successful coding classes are run outside of school hours. We believe that today’s teachers can effectively teach coding if they’re supported through in-class professional development. A somewhat similar and successful programme exists in the US, and we want Australian teachers to have this opportunity.

Many of the important and best paid jobs of this and the next generation will require computational thinking. Even if a student doesn’t study computer science at university, it’s essential they have the basics because just about every job will be changed by technology. We want every student in Australia to have this opportunity.

In 2019, we are piloting a programme with up to ten schools, and studying how successfully we can help teachers ramp-up their skills. Beyond 2019, we plan to launch this programme broadly.

We’ve had an exciting August. In summary:

  • We’ve signed-up seven schools into our 2019 pilot, and we’re speaking to
    two or three more
  • We have committed donations from generous supporters, and the first year of our programme is completely funded
  • We’re exploring a partnership with RMIT University
  • We’ve made progress on becoming a charity
  • We’ve signed up corporate supporters
  • We’re developing a roadmap for what happens from now until we start the programme in February

Programme

Our programme continues to evolve. We’ve refined our goals to:

  • Trial a novel teacher professional development pilot programme in 2019
  • Work with between five and ten schools in 2019. This means we’ll work with between seven and twelve school teachers who’ll teach over 1,000 students
  • Study how effectively we can develop computer science teaching skills
  • Develop teachers who can continue to independently teach computing in 2020
  • Make this programme free for schools in 2019, supported by generous donations

Our plans to deliver the programme are as follows:

  • We will hire two or three teachers who have a computing background, and they will be paid by generous donations to our charity
  • Our teachers will be paired with teachers from the schools in the pilot, and will meet for background workshops between November and January
  • From February, our teachers will work in the classroom with the teachers from the
    schools in the pilot
  • In our first term (or semester) working in a school, our teacher will deliver the syllabus to the students and the school’s teacher will learn through observation and by providing some student support
  • In our second term (or semester), it’s anticipated we’ll switch the roles: the school’s teacher will be the primary driver, with in-class support and training from our teacher
  • We’re hopeful that beyond the first two iterations, the school’s teacher will be independent, and require limited support from our programme

Our pilot is focused on:

  • Helping Year 7 teachers in Victorian high schools
  • A one-term (or semester) Year 7 computing subject of two or three hours per week
  • Providing everything that’s needed to teach the subject, including lesson plans, assignments, hardware, and software
  • Covering the “harder parts” of the Victorian DigiTech curriculum, especially the coding skills. In total, we’ll together deliver around half of the recommended 40 hours of the curriculum, and expect that the other half is delivered in other subjects

August Update

We have made significant progress in August, it’s been an exciting month.

SEEK Australia and Real Estate Australia (REA) have endorsed our plans.  We’re now able to tell schools that SEEK and REA stand behind what we’re doing, and endorse the industry relevance of our programme. We’re also meeting and learning from other organisations, including Google and Microsoft.  In particular, we’ve been meeting Kevin Wang at Microsoft who developed the
incredible TEALS programme in the US; it’s likely we will visit Kevin later this year.

We are hopeful of joining forces with RMIT University, and have met twice this month with Vice Chancellor Martin Bean and several times with his leadership team. This would be an amazing outcome, helping us in several ways:

  • It’d enhance our credibility, opening doors more easily to schools, academics, and
    government institutions
  • It’d improve the breadth and depth of our pilot, by giving us easier access to relevant academics, policy experts, and education professionals
  • We’d have access to facilities, including an office, meeting rooms, payroll support, and rooms for running teacher workshops
  • We’d be a “Deductible Gift Recipient (DGR) Charity”, a desirable status for receiving donations
  • We are optimistic we will finalise this arrangement and announce it at the end of September.

We have signed up several more schools. With some help from good friends, we’re now working with three schools on the Mornington Peninsula: Toorak College, Mount Erin College, and McClelland College. We are working with two schools in Sale: Gippsland Grammar School and the Sale Catholic College. We have also signed up two suburban schools,  Greensborough College and Haileybury. We are in discussions with three other schools, including two public schools and one large private girls school. We’re excited to have a mix of public, private, Catholic, city, and country schools to work with — we know this’ll help us better understand the effectiveness of our programme and provide a more persuasive argument as we go forward in 2020.

We’ve begun to work on our roadmap from September to February. This includes planning for hiring, syllabus development, and teacher workshops.

We look forward to an exciting September, and sharing even more progress at the end of the month. Thanks for reading all the way down here!

Cheers, Selina and Hugh.

Computing in Schools

1 Reply

This is the first update on our plans to improve computing education in Australian high schools. We look forward to sharing an update every month.

temp

Summary

We are focused on building and delivering a new computing subject for high schools, and helping teachers become successful at independently and confidently teaching the subject. The programme costs nothing for schools or students in 2019 through the generous support of donors.

Our model works like this:

  • We are hiring teachers who are experts in computing
  • We are planning to put a teacher in a classroom with a teacher from a school that we’re helping
    • In the first term, our teacher teaches the students, and the school’s teacher learns in a support role
    • In the second term, the school’s teacher teaches, and our teacher is there in the classroom to provide full time support
  • After two terms, the school continues teaching the subject. We’re available on call to support the school
  • Our teacher moves to a new school and repeats the same process

Our programme is built around a new foundational computing subject:

  • A compulsory Year 7 computing subject of around two teaching hours per week for one term
  • An introduction to coding and other computational-thinking fundamentals
  • Coverage of around half of the hours required by the Victorian Digital Technology Curriculum for Year 7, and focused on the “hard parts” that school’s struggle to teach
  • A fun, engaging programme that shows kids how computing is important in the real-world

In 2019, we will pilot this new subject and teaching model. The goals are to:

  • Teach kids how to write code
  • Help teachers in schools become confident at teaching coding
  • Teach the subject in seven to ten Victorian high schools
  • Teach at least 1,000 students, and help at least 10 teachers
  • Assess the learning outcomes to understand whether we’ve been effective in teaching basic coding and computational thinking skills, and whether we’ve taught teachers how to confidently deliver the materials

After 2019, we plan to roll this programme out across Victoria, expand to other Australian states, and begin work on subjects for Years 8, 9, and 10.

Why are we doing this?

Computer science is as important today as Maths, English, and the (other) Sciences. We believe it should be compulsory at school.  Even if a child isn’t going to become a Software Engineer, they are almost certainly going to work in a field that will be revolutionised by hardware and software. It’s critical to understand how software is built and what’s possible using a computer.

Most Australian kids aren’t learning the fundamentals of computing at school. There’s often a 3D printer or robots at a school, and maybe the kids have been on an excursion where they’ve built something technical. But that’s not even close to what’s needed if we’re going to help our children be ready for the jobs of the future. Our kids need to learn how to code, in the same way they need to learn how to read and write, and understand mathematics fundamentals.

Kids need to know how to break down problems into steps, and how to represent those steps in a programming language so that a computer can solve a problem. Building on these basics, students can learn how to write applications that do everything from play games to automate common tasks to solve real-world problems. More broadly, they need to know how to create software, so that they can be part of creating new jobs, industries, and technologies that are changing our world.

Nine of the ten most valuable companies today are technology companies including Alphabet (Google), Amazon, Apple, Microsoft, and Facebook. These companies are full of computing professionals, and they’re creating the software and hardware that is changing our world. The next generation of companies could be here in Australia, but that’ll only happen if we have the people with the skills and talents in computer science. Fixing computing education in schools is a critical first step.

Our 2019 plans

In 2019, we’ve committed to teaching 1,000 students, training 10 teachers, and working with 7 to 10 schools. Success is not only hitting these goals, but leaving these schools and teachers with the skills and confidence to continue teaching the programme in 2020 and beyond. RMIT University has agreed to independently assess whether we’ve been successful at the conclusion of our 2019 pilot.

When we work with a school, we’re going to provide the full teaching materials, hardware, and software that are needed to succeed. More importantly, we’re also providing a teacher who’ll help in the classroom and who has two goals: make sure the students have fun learning new skills, and make sure the school can confidently continue teaching computing without our hands-on support.

Progress so far

We started this project around three months ago. So far, we have:

  • Signed-up three schools, including Toorak College and two others that we will announce soon
  • Raised money. We have promises of nearly enough money to fund out 2019 pilot, but we’d always love more (because it makes more possible — we’re happy to think bigger)
  • Focused our plans. There were many possible starting points, but we have decided on a Year 7 compulsory subject for our pilot
  • Sought corporate support. We want industry to stand behind what we’re doing, so that schools are confident the programme is helping their students develop real-world skills. We will announce corporate supporters soon
  • Received a commitment from RMIT University to independently assess the educational outcomes of our programme

Focus for August 2018

This month, we are focused on the following next steps:

  • Signing-up more Victorian schools. We are particularly looking for public schools, but we’re open to working with any school. It costs nothing. Can you help introduce us to a school? Send us a note at hugh@hughwilliams.com or selina@selinawilliams.com
  • Establishing our venture as a formal entity. We are exploring becoming a registered charity, which means that donors can give us money and receive a tax deduction. It turns out this isn’t easy, and we’ve enlisted the help of lawyers who specialise in establishing charities
  • Hiring teachers. Now that we’ve signed-up a few schools, we’re ready to hire our first teacher. It’s critical that they’re a great teacher with a computing background. Their role will be helping build materials, as well as teaching and training in 2019. If you have any referrals, send us a note at hugh@hughwilliams.com or selina@selinawilliams.com
  • Wooing corporate supporters. It’ll make a huge difference to everyone if they know that major names believe in our programme. This will help us with funding, signing-up schools, and developing the programme itself

Perhaps surprisingly, the hardest thing so far has been signing-up schools. Many schools don’t even bother to respond to an email. Once they do, it’s a long “sales cycle” from a first meeting to convincing a school to make room in their Year 7 curriculum for computing. We’re pretty surprised about this, but perhaps this will change as we sign up more schools, have more corporate support, and our programme gains a reputation. In any case, if you’d like to help, the most valuable thing you can do today is introduce us to a decision maker in a Victorian public school — we need to sign-up another four to seven schools over the next month or two.

If you’d like to join our mailing list to receive this monthly update in email (and a little more information), send us a note at hugh@hughwilliams.com or selina@selinawilliams.com

Thanks for reading all the way to the bottom. We’ll post another update in a month.

Best wishes, Hugh and Selina.

 

 

 

 

 

 

 

Guillain–Barré syndrome

15 Replies

I suffered from Guillain–Barré Syndrome (GBS) in 2009.

I am now perfectly healthy, and I’m on a mission to raise money for the GBS/CIDP Foundation. I’m going to do what GBS stopped me doing for a while: running! I’m going to run 52 races this year. Please join me in helping others who suffer from these terrible conditions by doing four things:

  1. Liking my fundraising page on Facebook
  2. Following my fundraising on Twitter @fiftytwofives
  3. Reading my fundraising and running blog at http://fiftytwofives.com
  4. By donating at IndieGogo. Every dollar counts — there’s cool perks available!

Now, here’s the story…

The Short Story

One day in the middle of 2009, I woke up to a tingling sensation in my toes, almost like a cramp that made them feel like they were curling. I was a little dizzy.

Over a couple of days, more strange sensations began in my legs and eventually my arms. My heels went numb, then my toes, and then my calves started tingling and went mostly numb too. I felt strange buzzing sensations in my legs. Walking was awkward. My face started feeling a little numb, and talking felt strange. I suddenly felt exhausted just walking to the mailbox. I was freaked out. Very freaked out.

Late in 2009, a few months after I was diagnosed with GBS. I'm out of shape, but on my way to recovery

Late in 2009, a few months after I was diagnosed with GBS. I’m out of shape, but on my way to recovery

My general practitioner didn’t know what was happening, and sent me to a neurologist.

The neurologist tried some reflex tests. A hammer-hit on the knee did nothing. I normally had good reflexes. I had a few other tests, including getting to know the inside of an MRI machine pretty well. Somewhere around this time, doctors started muttering possible diagnoses such as Multiple Sclerosis. I wouldn’t wish that undiagnosed limbo on anyone.

The next step was a visit to the UCSF Medical Center, where I was finally but quickly diagnosed with Guillain-Barre Syndrome (GBS). While I wasn’t happy to be sick, I was very happy to know what the problem actually was, and to learn more about the condition. I’ll come back to that in a moment.

It was a tough time. My family was in Seattle, and I was in California. I’d just started a new job at eBay. Luckily, eBay was very understanding, and I was able to quietly work part-time for a while. My days went like this: wake up at 8am, feeling exhausted; struggle to make breakfast; slowly get ready (lots of sitting and resting); drive to work; pretend you’re ok for a few hours (while sitting, resting, and feeling strange sensations); go home; struggle to make a meal; sleep for 13 to 15 hours; begin the next day.

What is GBS?

GBS is rare and unusual. It occurs in one to two people per 100,000.

It’s thought that after you’ve had an infection (in my case, probably a viral ear infection), your immune system loses the plot and attacks your nervous system. GBS is otherwise known as Acute Inflammatory Demyelinating Polyneuropathy (AIDP) which gives you clues as to what it does. Acute means it’s not as serious as it could be; chronic is worse. Demyelinating means your immune system strips your peripheral nerves of their myelin, leading to strange sensations and loss of function. Polyneuropathy means it affects many nerves at once. It perhaps should be called Peripheral AIDP since it doesn’t affect your central nervous system (brain and spine): it starts at your feet and works its way up, often taking out your arms.

Serious cases leave people on a respirator and can involve months in hospital. My case wasn’t serious. There’s no cure for GBS – you just wait it out. Doctors often prescribe strong steroids, but there’s no evidence I can find that they help. Blood transfusions can ease the symptoms. Luckily, most people are back to normal within a year or two. Not everyone is that lucky though.

People who’ve suffered from GBS have the same chance of getting it again as those who’ve never had it.

For the Australians out there, you may recall that Alistair Clarkson, the coach of the Hawthorn footy club had GBS in May 2014. He recovered quickly and, of course, coached the Hawks to the flag (championship). Other notable folks who’ve had GBS include Franklin D. Roosevelt, Joseph Heller, Andy Griffith, and Lucky Oceans.

Getting Better

Over the next year, I put on 25+ pounds and became horribly out of shape. There’s no way I could exercise, and I wasn’t in the mood to eat right. I often woke in the night because of numb hands and tingling legs. I was phenomenally tired and easily exhausted. My balance was off, and I became dizzy easily. But I steadily got better – and by mid 2010 was thinking about getting gently back into exercise. Fast forward to today and I’m completely fine.

A recent race. I'm back running, and ready to run 52 races in 2015 to fight GBS

A recent race. I’m back running, and ready to run 52 races in 2015 to fight GBS

I think differently post-GBS. I know that health is the most important thing in the world: without it, you can’t look after your family, let alone work. I spend more time looking after myself, and balancing work and life more effectively. I also think more about people less fortunate than me: having GBS was tough, but my case was mild and there are far worse things that happen to people.

Helping People with GBS

I am turning the tables on GBS, and fundraising for the GBS/CIDP Foundation. I am going to run fifty-two races in 2015, and blog about it over at fiftytwofives.com. The name is inspired from running fifty-two races of 5k in distance — but I will count anything that has a timer as one race (one-milers, 5ks, 10ks, ten-milers, half-marathons. Heck, maybe even a marathon). If you’d like to be part of my fundraising, just head over here to IndieGoGo.

The GBS/CIDP Foundation is doing great work. The foundation helps with education for those suffering GBS and its chronic, recurring cousin CIDP and related conditions. They fund research to understand more about this condition. They lobby to ensure people get access to the health care they need (I’m lucky because I can afford access to great healthcare; of course, not everyone can).

A Quirky Afterword

Next time you have a flu vaccination, take note of the questions you’re asked. Down the bottom, you’ll find “Have you ever suffered from GBS?”. If you answer yes, your doctor probably won’t give you the flu vax – in 1976, after the flu vax was issued, there was roughly twice the incidence of GBS in people who’d had the vax compared to the population that hadn’t. It’s not understood why that happened. And even though they think GBS doesn’t recur, they generally won’t give the flu vax to folks who’ve had GBS.

I didn’t get the flu vax for that reason from 2009 to 2013. However, after I got the flu in 2013, I talked my doctor into giving me the flu vax in 2014. I just got this year’s too. Nothing interesting happened. I’m happy about that.

I thank Sean Southey for encouraging me to share my GBS story.

See you next time.

It’s a Marathon, not a Sprint…

4 Replies

What makes a career successful over the long term? How do you sustain (or even increase) your professional impact, while also deriving meaning and enjoyment from your life? This was the question I set about answering in my talk at StretchCon last week. To see my answers, you can watch the presentation. You can also view the prezi. 1966321_313990785474109_1977531760847682083_o

The Backstory

I asked eleven colleagues about their success. I chose colleagues who have made it to the C-suite (whether as a CEO, CTO, or another C-level designation) and that appeared to do it with balance between their professional and personal lives. Ten of the eleven responded, and nine of the ten shared thoughts before my deadline. I thank Chris Caren, Adrian Colyer, John Donahoe, Ken Moss, Satya Nadella, Mike Olson, Christopher Payne, Stephanie Tilenius, and Joe Tucci for their help.

I sent each of these colleagues an email that went something like this:

I am speaking about successful careers being about sustained contribution (and not a series of sprints, all-nighters, or unsustainable peaks). Would you be up for giving me a quote I could use and attribute to you? I admire your ability to work hard and smart, while obviously also having a life outside of work.

Their replies were varied, but as you’ll see in the video, there were themes that repeated in their answers. I shared edited quotes in the talk, and promised that I’d share their complete thoughts in my blog. The remainder of this blog is their complete words.

Chris Caren

Chris is the CEO and Chairman of Turnitin. We worked together at Microsoft, and Chris was (and sometimes still is!) my mentor. Here are his thoughts in response to my questions:

My philosophy:  I do my best work when my life is in balance — family, me, and work.  I need a routine of hard work, but no more than 9-10 hours a day, solid exercise daily, low stress (via self-control), 7-8 hours of sleep at a minimum each day, and the time I want with my family and for myself.  When I maintain this balance, I am maximally effective at work — both in terms of quality of thinking and decision making, and maximum output.  More hours worked actually pull down my impact as a CEO.

Adrian Colyer

Adrian was the CTO of SpringSource, the custodians of the Spring Java programming framework. We worked together at Pivotal, where he was the CTO of the Application Fabric team. Recently, Adrian joined Accel Partners as an Executive-in-Residence. Here are Adrian’s thoughts:

A great topic! Maybe the most counter-intuitive lesson I’ve learned over the years is that I can make a much more valuable contribution when I work* less. So work-life balance isn’t really a trade-off as most people normally present it (I have more ‘life’, but sacrifice ‘work’ to get it), it’s actually a way of being better at both life *and* work!

* ‘work’ in the sense that most people would intuitively think of it – frenetic activity.

When I’ve analysed this, I came to realise that when work crowds everything else out I often end up in a very reactive mode. But the biggest and most impactful things you can do – especially as a leader – don’t come about during that constant fire-fighting mode. The vast majority of my important insights and decisions – the things that made the biggest positive impact on the organisations I was working with at the time – have come in the space I’ve made around the busy-ness of the office to actually allow myself the luxury of thinking! Running, cycling, walking and so on have all been very effective for me over the years. But so is just taking some time out in the evening and not necessarily even consciously thinking about work, the brain seems to be very good at background processing! That time has also given space to allow my natural curiosity and love of learning to be indulged. In turn that creates a broader perspective, exposes you to new ideas, and allows you to make connections and insights that you otherwise might not of. All of this feeds back into the work-related decisions and issues you are wrestling with and helps you to make breakthroughs from time to time.

To the extent I’ve been successful over the years, I attribute much of that not to being smarter than the people around me, nor to working ‘harder’, but to creating the space to think.

John Donahoe

John is the CEO of eBay Inc. John was an enthusiastic sponsor of my work while I was there. When I asked John for his thoughts, he sent me a speech he’d recently given to the graduating class at the Stanford Business School. In it, you’ll find John’s thoughts of his professional and personal journey.

Ken Moss

Ken recently became the CTO of Electronic Arts. Prior to that, Ken and I worked together on, off, and on over a period of nine years. Ken was the GM of Microsoft’s MSN Search when I joined Microsoft, and left to found his own company. I managed to help persuade Ken to come to eBay for a few years. Here are Ken’s thoughts:

Always focus on exceeding expectations in the present, while keeping your tank 100% full of gas for the future. There is no quicker way to stall your career than by burning yourself out. I’ve seen many potentially brilliant careers cut short as someone pushed themselves too far past their limits and became bitter under-performers. It’s always in your control.

Satya Nadella

Satya became the CEO of Microsoft at the beginning of 2014. Satya was the VP of the Bing search team at Microsoft for around half the time I was there, and we have stayed in touch since. Here are Satya’s thoughts:

I would say the thing that I am most focused on is to harmonize my work and life vs trying to find the elusive “balance”. Being present in the lives of my family in the moments I am with them is more important than any quantitative view of balance.

Mike Olson

Mike is the Chairman, Chief Strategy Officer, and former CEO of Cloudera. We have interacted during my time at Pivotal, and also during my time at eBay. Mike was kind enough to invite me to give the keynote at Hadoop World in 2011. Here’s Mike’s thoughts:

I have always tried to optimize for interesting — working on problems that are important to me, with people who blow my hair back. The combination has kept me challenged and inspired, and has guaranteed real happiness in the job.

By corollary, you have to be willing to walk away from a good paycheck and fat equity if the work or the people are wrong. Money is cheaper than meaning. I’ve done that a few times. There’s some short-term angst, but it’s paid off in the long term.

Christopher Payne

Christopher is the SVP of the North America business at eBay. Christopher and I have worked on, off, and on for nine years. Christopher was the founding VP of the search team at Microsoft. He left to found his own company, his company was bought by eBay, he hired me to eBay to help run engineering, and he then moved over to run the US and Canadian business teams. Here are Christopher’s thoughts:

I believe strongly in the need to maintain my energy level in order to have the most impact in my career. To do this I find I have to make the time to recharge. For me this means taking walks during the work day, taking all of my vacation, and not being on email 24/7. With my energy level high I find I can be significantly more creative and productive over the long term.

Stephanie Tilenius

Stephanie recently founded her own company, Vida. While she’s spent parts of her career at Kleiner-Perkins, Google, and other places, we met at eBay where we spent around six months working together. Here are Stephanie’s thoughts:

… my point of view is that you have to do something you love, that will sustain you. You also have to know what drives you, what gets you out of bed, for me it is having an impact (for others it may be making money or playing a sport, etc.) You will always be willing to give it your all and you are more likely to innovate if you love what you are doing and constantly growing, challenging the status quo (stagnation is really out of the question, humans don’t thrive on it!). I am committed to my work and to constant innovation but also to having a family and I could not be great at either without the other. They are symbiotic in my mind, they both make me happy and a better person. I have learned it is about integration not necessarily perfect balance. If you integrate life and work, you are much more likely to be successful. The other day my son was out of school early and our nanny had an issue so I brought him to work and he did code academy and talked to some of our engineers. He enjoyed himself and was inspired.

Joe Tucci

Joe is the Chairman of EMC, VMware, and Pivotal, and the CEO of EMC. I met Joe in the interview process at Pivotal, and have worked with him through board and other meetings over the past year. Here’s Joe’s thoughts:

Being a successful CEO is relatively straight forward… 1st – retain, hire, and develop the best talent, 2nd – get these talented individuals to work together as a team (do not tolerate selfishness), 3rd – get this leadership team to embrace a stretch goal that is bigger then any of them imagine they can attain, and 4th – maniacally focus the leadership team on our customers (always striving to exceed their expectations)

I enjoyed giving the talk at Stretch, and interacting with these colleagues in putting it together. I hope you enjoyed it too. See you next time.

A Whirlwind Tour of a Search Engine

Leave a reply

Thanks to Krishna Gade and Michael Lopp,  I had an opportunity to speak at Pinterest’s new DiscoverPinterest tech talk series.

I spoke for around 45 minutes, taking the audience on a tour of the world of (mostly) web search — skating across the top of everything from ranking, to infrastructure, to inverted indexing, to query alterations, and more. I had a lot of fun.

Here’s the video:

I also had the chance to listen to four of Pinterest’s engineering leaders discuss their work in browsing, content-based image retrieval, infrastructure, and graph processing and relevance. They’re up to some interesting work — particularly if you’re interested in the intersection of using images, human-curated data, and browsing.

On a social note, it was great to see several of the folks I worked with at Bing. Krishna took a selfie with Yatharth Saraf and I. Those were truly the days — we were in many ways ahead of our time.

Shameless advertisement: if you’d like me to present on search (or anything else) at your organization, please feel free to ask. I have a 30 minute, 1 hour, and whole day tutorial on search engines. I’m also available for consulting and advising!

See you next time.

Measuring Search Relevance

1 Reply

How do you know when you’ve improved the relevance of a search engine? There are many ways to understand this, for example running A/B tests on your website or doing qualitative studies in a lab environment with a few customers. This blog post focuses on using large numbers of human judges to assess search performance.

Relevance Judgment

The process of asking many judges to assess search performance is known as relevance judgment: collecting human judgments on the relevance of search results. The basic task goes like this: you present a judge with a search result, and a search engine query, and you ask the judge to assess how relevant the item is to the query on (say) a four-point scale.

Suppose the query you want to assess is ipod nano 16Gb. Imagine that one of the results is a link to Apple’s page that describes the latest Apple iPod nano 16Gb. A judge might decide that this is a “great result” (which might be, say, our top rating on the four-point scale). They’d then click on a radio button to record their vote and move on to the next task. If the result we showed them was a story about a giraffe, the judge might decide this result is “irrelevant” (say the lowest rating on the four point scale). If it were information about an iPhone, it might be “partially relevant” (say the second-to-lowest), and if it were a review of the latest iPod nano, the judge might say “relevant” (it’s not perfect, but it sure is useful information about an Apple iPod).

The human judgment process itself is subjective, and different people will make different choices. You could argue that a review of the latest iPod nano is a “great result” — maybe you think it’s even better than Apple’s page on the topic. You could also argue that the definitive Apple page isn’t terribly useful in making a buying decision, and you might only rate it as relevant. A judge who knows everything about Apple’s products might make a different decision to someone who’s never owned an digital music player. You get the idea. In practice, judging decisions depend on training, experience, context, knowledge, and quality — it’s an art at best.

There are a few different ways to address subjectivity and get meaningful results. First, you can ask multiple judges to assess the same results to get an average score. Second, you can judge thousands of queries, so that you can compute metrics and be confident statistically that the numbers you see represent true differences in performance between algorithms. Last, you can train your judges carefully, and give them information about what you think relevance means.

Choosing the Task and Running the Judging

You have to decide what queries to judge, how many queries to judge, how many answers to show the judges, and what search engines or algorithms you want to compare. One possible approach to choosing queries is to randomly sample queries from your search engine query logs. You might choose to judge hundreds or thousands of queries. For each query, you might choose to judge the first ten results, and you might choose to compare the first ten results from each of (say) Google, Bing, and DuckDuckGo.

Most search companies do their relevance judgment with crowdsourcing. They put tasks in the public domain, and pay independent people to perform the judgments using services such as CrowdFlower. This does create some problems – some people try to game the system by writing software that randomly answers questions, or they answer fast and erroneously. Search companies have to work constantly on detecting problems, and removing both the poor results and judges from the system. To give you a flavor, one thing search folks do is inject questions where they know what the relevance score should be, and then check that the judges answer most of those correctly (this is known as a ringer test). Another thing folks do is look for judges who consistently answer differently from other judges for the same tasks.

Scoring Relevance Judgments

When you’ve got tens of answers for each query, and you’ve completed judging at least a few hundred queries, you’re ready to compute a metric that allows us to compare algorithms.

An industry favorite is NDCG, Normalized Discounted Cumulative Gain. It sounds complicated, but it’s a common-sense measure. Suppose that on our four-point scale, you give a 0 score for an irrelevant result, 1 for a partially relevant, 2 for relevant, and 3 for perfect. Suppose also that a query is judged by one of the judges, and the first four results that the search engine returns are assessed as relevant, irrelevant, perfect, and relevant by the judge. The cumulative gain after four results is the sum of the scores for each result: 2 + 0 + 3 + 2 = 7. That’s shown in the table below: result position or rank in the first column, the judges score or gain in the second column, and a running total or cumulative gain in the third column.

Rank Judgment (Gain)
 Cumulative Gain
1 2 2
2 0 2
3 3 5
4 2 7

Now for the Discounted part in NDCG. Search engine companies know that the first result in the search results is more important than the second, the second more important than the third, and so on. They know this because users click on result one much more than result two, and so on. Moreover, there’s plenty of research that shows users expect search engines to return great results at the top of the page, that they are unlikely to view results low on the page, and that they dislike having to use pagination.

The Discounted part of NDCG adds in a weighting based on position: one simple way to make position one more important than two (and so on) is to sum the score divided by the rank. So, for example, if the third result is ”great”, its contribution is 3 / 3 = 1 (since the score for “great” is 3, and the rank of the result is 3). If “great” were the first result, its contribution would be 3 / 1 = 3. In practice, the score is often divided by the log of the rank, which seems to better match the user perception of relevance. Anyway, for our example and to keep it simple, the Discounted Cumulative Gain (DCG) after four results is 2 / 1 + 0 / 2 + 3 / 3 + 2 / 4 = 3.5. You can see this in the table below: the third column has the discounted gain (the gain divided by the rank), and the fourth column keeps the running total or cumulative gain.

Rank Judgment (Gain)
 Discounted Gain Discounted Cumulative Gain (DCG)
1 2 2/1 2
2 0 0/2 2
3 3 3/3 3
4 2 2/4 3.5

The Normalized part in NDCG allows us to compare DCG values between different queries. It’s not fair to compare DCG values across queries because some queries are easier than others: for example, maybe it’s easy to get four perfect results for the query ipod nano, and much harder to get four perfect results for 1968 Porsche 912 targa soft window. If the search engine gets a high score for the easy query, and a poor score for the hard query, it doesn’t mean it’s worse at hard queries – it might just mean the queries have different degrees of difficulties.

Normalization works like this: you figure out what the best possible score is given the results you’ve seen so far. In our previous example, the results scored 2, 0, 3, and 2. The best arrangement of these same results would have been: 3, 2, 2, 0, that is, if the “great” result had been ranked first, followed by the two “relevant” ones, and then the “irrelevant”. This best ranking would have a DCG score of 3 / 1 + 2 / 2 + 2 / 3 + 0 / 4 = 4.67. This is known as the “ideal DCG,” or iDCG.  Our NDCG is the score we got (3.50) divided by the ideal DCG (4.67), or 3.50 / 4.67 = 0.75. Now we can compare scores across queries, since we’re comparing percentages of the best possible arrangements and not the raw scores.

The table below builds out the whole story. You’ve seen the first four columns before. The fifth and sixth columns show what would have happened if the search engine had ordered the results in the perfect order. The seventh and final column shows a running total of the fourth column (the DCG) divided by the sixth column the (ideal or iDCG), and the overall NDCG for our task is shown as 0.75 in bold in the bottom-right corner.

Rank Judgment (Gain)
 Discounted Gain Discounted Cumulative Gain (DCG)
 Ideal Discounted Gain Ideal Discounted Cumulative Gain (iDCG) Normalized Discounted Cumulative Gain (NDCG)
1 2 2/1 2.0 3/1 3.0 0.67
2 0 0/2 2.0 2/2 4.0 0.5
3 3 3/3 3.0 2/3 4.67 0.64
4 2 2/4 3.5 0/4 4.67 0.75

Comparing Search Systems

Once you’ve computed NDCG values for each query, you can average them across thousands of queries. You can now compare two algorithms or search engines: you take the mean average NDCG values for each system, and check using a statistical test (such as a two sided t-test) whether one algorithm is better than the other, and with what confidence. You might, for example, be able to say with 90% confidence that Google is better than Bing.

As I mentioned at the beginning, this is one important factor you could consider when comparing two algorithms. But there’s more to search engine comparison than comparing NDCG metrics. As I’ve said in previous posts, I’m a huge fan of measuring in many different ways and making decisions with all the data at hand. It takes professional judgment to decide one algorithm is better than another, and that’s part of managing any search engineering effort.

Hope you found this useful, see you next time!

Afterword

I published a first version of this post on eBay’s technical blog in 2010. I owe thanks to Jon Degenhardt for cleaning up a math error, and formatting the tables.

Fireside chat at the DataEdge Conference

1 Reply

The video of my recent conversation with Michael Chui from McKinsey as part of the UC Berkeley DataEdge conference is now online. Here it is:

The discussion is around 30 minutes. I tell a few stories, and most of them are mostly true. We talk about my career in data, search, changing jobs, inventing infinite scroll, eBay, Microsoft, Pivotal, and more.  Enjoy!

Armchair Guide to Data Compression

3 Replies

Data compression is used to represent information in less space than its original representation. This post explains the basics of lossless compression, and hopefully helps you think about what happens when you next compress data.

Similarly to my post on hash tables, it’s aimed at software folks who’ve forgotten about compression, or folks just interested in how software works; if you’re a compression expert, you’re in the wrong place.

A Simple Compression Example

Suppose you have four colored lights: red, green, blue, and yellow. These lights flash in a repeating pattern: red, red, green, red, red, blue, red, red, yellow, yellow (rrgrrbrryy). You think this is neat, and you want to send a message to a friend and share the light flashing sequence.

You know the binary (base 2) numbering system, and you know that you could represent each of the lights in two digits: 00 (for red, 0 in decimal), 01 (for green, 1 in decimal), 10 (for blue, 2 in decimal), and 11 (for yellow, 3 in decimal). To send the message rrgrrbrryy to your friend, you could therefore send them this message: 00 00 01 00 00 10 00 00 11 11 (of course, you don’t send the spaces, I’ve just included them to make the message easy for you to read).

Data compression. It's hard to find a decent picture!

Data compression. It’s hard to find a decent picture!

You also need to send a key or dictionary, so your friend can decode the message. You only need to send this once, even if the lights flash in a new sequence and you want to share that with your friend. The dictionary is: red green blue yellow. Your friend will be smart enough to know that this implies red is 00, green is 01, and so on.

Sending the message takes 20 bits: two bits per light flash and ten flashes in total (plus the one-off cost of sending the dictionary, which I’ll ignore for now). Let’s call that the original representation.

Alright, let’s do some simple compression. Notice that red flashes six times, yellow twice, and the other colors once each. Sending those red flashes is taking twelve bits, four for sending yellow, and the others a total of four for the total of twenty bits. If we’re smart, what we should be doing is sending the code for red in one bit (instead of two) since it’s sent often, and using more than two bits for green and blue since they’re sent once each. If we could send red in one bit, it’d cost us six bits to send the six red flashes, saving a total of six bits over the two-bit version.

How about we try something simple? Let’s represent red as 0. Let’s represent yellow as 10, green as 110, and blue as 1110 (this is a simple unary counting scheme). Why’d I use that scheme? Well, it’s a simple idea: sort the color flashes by decreasing frequency (red, yellow, green, blue), and assign increasingly longer codes to the colors in a very simple way: you can count 1s until you see a 0, and then you have a key you can use to look in the dictionary. When we see just a 0, we can look in the dictionary to find that seeing zero 1s means red. When we see 1110, we can look in the dictionary to find that seeing three 1s means blue.

Here’s what our original twenty-bit sequence would now look like: 0 0 110 0 0 1110 0 0 10 10. That’s a total of 17 bits, a saving of 3 bits — we’ve compressed the data! Of course, we need to send our friend the dictionary too: red yellow green blue.

It turns out we can do better than this using Huffman coding. We could assign 0 to red, 10 to yellow, 110 to blue, and 111 to green. Our message would then be 0 0 111 0 0 110 0 0 10 10. That’s 16 bits, 1 bit better than our simple scheme (and, again, we don’t need to send the spaces). We’d also need to share the dictionary: red <blank> yellow <blank> blue green to show that 0 is red, 1 isn’t anything, yellow is 10, 11 isn’t anything, blue is 110, and green is 111. A slightly more complicated dictionary for better message compression.

Semi-Static Compression

Our examples are two semi-static compression schemes. The dictionary is static, it doesn’t change. However, it’s built from a single-pass over the data to learn about the frequencies — so the dictionary is dependent on the data. For that reason, I’ll call our two schemes semi-static schemes.

Huffman coding (or minimum-redundancy coding) is the most famous example of a semi-static scheme.

Semi-static schemes have at least three interesting properties:

  1. They require two passes over the data: one to build the dictionary, and another to emit the compressed representation
  2. They’re data-dependent, meaning that the dictionary is built based on the symbols and frequencies in the original data. A dictionary that’s derived from one data set isn’t optimal for a different data set (one with different frequencies) — for example, if you figured out the dictionary for Shakespeare’s works, it isn’t going to be optimal for compressing War and Peace
  3. You need to send the dictionary to the recipient, so the message can be decoded; lots of folks forget to include this cost when they share the compression ratios or savings they’re seeing, don’t do that

It doesn’t matter what kind of compression scheme you choose, you need to decide what the symbols are. For example, you could choose letters, words, or even phrases from English text.

Static Compression

Morse code is an example of a (fairly lame) static compression scheme. The dictionary is universally known (and doesn’t need to be communicated), but the dictionary isn’t derived from the input data. This means that the compression ratios you’ll see are at best the same as you’ll see from a semi-static compression scheme, and usually worse.

There aren’t many static compression schemes in widespread use. Two examples are Elias gamma and delta codes, which are used to compress inputs that consist of only integer values.

Adaptive Compression

The drawbacks of semi-static compression schemes are two-fold: you need to process the data twice, and they don’t adapt to local regions in the data where the frequencies of symbols might vary from the overall frequencies. Imagine, for example, that you’re compressing a very large image: you might find a region of blue sky, where there’s only a few blue colors. If you built a dictionary for only that blue section, you’d get a different (and better) dictionary than the one you’d get for the whole image.

Here’s the idea behind adaptive compression. Build the dictionary as you go: process the input, and see if you can find it in the (initially empty) dictionary. If you can’t, add the input to the dictionary. Now emit the compressed code, and keep on processing the input. In this way, your dictionary adapts as you go, and you only have to process the data once to create the dictionary and compress the data. The most famous example is LZW compression, which is used in many of the compression tools you’re probably using: gzip, pkzip, GIF images, and more.

Adaptive schemes have at least three interesting properties:

  1. One pass over the data creates the dictionary and the compressed representation, an advantage over semi-static schemes
  2. Adaptive schemes get better compression with the more input they see: since they don’t approximate the global symbol frequencies until lots of input has been processed, they’re usually much less optimal than semi-static schemes for small inputs
  3. You can’t randomly access the compressed data, since the dictionary is derived from processing the data sequentially from the start. This is one good reason why folks use static and semi-static schemes for some applications

What about lossy compression?

The taxonomy I’ve presented is for lossless compression schemes — those that are used to compress and decompress data such that you get an exact copy of the original input. Lossy compression schemes don’t guarantee that: they’re schemes where some of the input is thrown away by approximation, and the decompressed representation isn’t guaranteed to be the same as the input. A great example is JPEG image compression: it’s an effective way to store an image, by throwing away (hopefully) unimportant data and approximating it instead. The MP3 music file format and the MP4 video format (usually) do the same thing.

In general, lossy compression schemes are more compact than lossless schemes. That’s why, for example, GIF image files are often much larger than JPEG image files.

Hope you learnt something useful. Tell me if you did or didn’t. See you next time.