BC MK-803 product use testing 3-15-12

52
© 2012 Steve Herbst Confidential and Proprietary Steve Herbst Developer of Topical Storm software -a web-based outlining program that offers suggestions to overcome writer’s block This presentation for MK-803: My product development story and what I know about Product Use Testing SM
  • date post

    11-Sep-2014
  • Category

    Technology

  • view

    637
  • download

    0

description

 

Transcript of BC MK-803 product use testing 3-15-12

Page 1: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

Steve Herbst

Developer of Topical Storm software

-a web-based outlining program that offers suggestions to overcome writer’s block

This presentation for MK-803:

My product development story and what I know about Product Use Testing

SM

Page 2: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

Cartoon by Barbara Smaller, The New Yorker 4/26/10

Page 3: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

Names don’t work that well compared to product. Google – we worry about the name … will the world accept … or Yahoo. Or Lotus. How silly. Lotus. Apple. And yet it’s the product. The product really, really matters.

-Dan Bricklin, co-founder of VisiCalc-Interview May 7, 2004

Page 4: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

1998

Groundwork:

Reasoning software runs on a PC

2006Inspiration:

Daughter writescrappy essay

2010Stay on target:

Education marketgoes crazy for

collaboration tools

2010Reality check:

Customersexpect perfection

2011Validation:Field test at

Douglas SchoolIn Acton MA

NEXTSTEPS

My product development history

2011Enlightenment:

Software isShow biz

Page 5: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

1998

Groundwork:

Reasoning software runs on a PC

2006Inspiration:

Daughter writescrappy essay

2010Stay on target:

Education marketgoes crazy for

collaboration tools

2010Reality check:

Customersexpect perfection

2011Validation:Field test at

Douglas SchoolIn Acton MA

NEXTSTEPS

2011Enlightenment:

Software isShow biz

My product development history

Page 6: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary© 2011 Steve Herbst Confidential and Proprietary

Hurricanes start when warm, moist air from the ocean surface begins to rise rapidly, where it encounters cooler air that causes the warm water vapor to condense and to form storm clouds and drops of rain. The condensation also releases latent heat, which warms the cool air above, causing it to rise and make way for more warm humid air from the ocean below.

As this cycle continues, more warm moist air is drawn into the developing storm and more heat is transferred from the surface of the ocean to the atmosphere. This continuing heat exchange creates a wind pattern that spirals around a relatively calm center, or eye, like water swirling down a drain.

Daughter’s 6th grade essay topic:

Warm air causes hurricanes

Using a Google search, she finds:

Done!

Page 7: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

Warm air causes hurricanes

How does warm air cause hurricanes?

Rule

Page 8: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

How does warm air cause hurricanes?

Evidence that warm air causes hurricanes

What causes warm air?

What do hurricanes cause?

What else causes hurricanes?

What else does warm air cause?

Can warm air be used to predict hurricanes?

Can preventing warm air prevent hurricanes?

Do different kinds of warm air cause different kinds of hurricanes?

And so on.

Warm air causes hurricanes

Noticing the pattern “A causes B”, the computer asks:

Page 9: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

Warm air causes hurricanes

Noticing the pattern “A causes B”, the computer asks:

How does warm air cause hurricanes?

Evidence that warm air causes hurricanes

What causes warm air?

What do hurricanes cause?

What else causes hurricanes?

What else does warm air cause?

Can warm air be used to predict hurricanes?

Can preventing warm air prevent hurricanes?

Do different kinds of warm air cause different kinds of hurricanes?

And so on.

Page 10: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

Warm air causes hurricanes

Noticing the pattern “A causes B”, the computer asks:

How does warm air cause hurricanes?

Evidence that warm air causes hurricanes

What causes warm air?

What do hurricanes cause?

What else causes hurricanes?

What else does warm air cause?

Can warm air be used to predict hurricanes?

Can preventing warm air prevent hurricanes?

Do different kinds of warm air cause different kinds of hurricanes?

And so on.

Page 11: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

Warm air causes hurricanes

Noticing the pattern “A causes B”, the computer asks:

How does warm air cause hurricanes?

Evidence that warm air causes hurricanes

What causes warm air?

What do hurricanes cause?

What else causes hurricanes?

What else does warm air cause?

Can warm air be used to predict hurricanes?

Can preventing warm air prevent hurricanes?

Do different kinds of warm air cause different kinds of hurricanes?

And so on.

Page 12: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

Too much television causes ADHD

Noticing the pattern “A causes B”, the computer asks:

How does too much television cause ADHD?

Evidence that too much television causes ADHD

What causes too much television?

What does ADHD cause?

What else causes ADHD?

What else does too much television cause?

Can too much television be used to predict ADHD?

Can preventing too much television prevent ADHD?

Do different kinds of too much television cause different kids of ADHD?

And so on.

Page 13: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

Too much television causes ADHD

Noticing the pattern “A causes B”, the computer asks:

How does too much television cause ADHD?

Evidence that too much television causes ADHD

What causes too much television?

What does ADHD cause?

What else causes ADHD?

What else does too much television cause?

Can too much television be used to predict ADHD?

Can preventing too much television prevent ADHD?

Do different kinds of too much television cause different kids of ADHD?

And so on.

Page 14: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

Too much television causes ADHD

Noticing the pattern “A causes B”, the computer asks:

How does too much television cause ADHD?

Evidence that too much television causes ADHD

What causes too much television?

What does ADHD cause?

What else causes ADHD?

What else does too much television cause?

Can too much television be used to predict ADHD?

Can preventing too much television prevent ADHD?

Do different kinds of too much television cause different kids of ADHD?

And so on.

Page 15: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

Product: An outliner that generates questions:

We should focus on market share What evidence do you have that market share is important?

What other factors are also important?

Are there any unspoken assumptions behind this statement?

What steps can we take to focus on market share?

What is the purpose of market share?

Who knows the most about market share?

Who might disagree with this statement?

How can we research this recommendation further?

Page 16: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

1998

Groundwork:

Reasoning software runs on a PC

2006Inspiration:

Daughter writescrappy essay

2010Stay on target:

Education marketgoes crazy for

collaboration tools

2010Reality check:

Customersexpect perfection

2011Validation:Field test at

Douglas SchoolIn Acton MA

NEXTSTEPS

2011Enlightenment:

Software isShow biz

My product development history

Page 17: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

2010

November Learning (Boston)

Conference: Building Learning Communities

Page 18: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

2010

High energy, very exciting. All these free tools:

Page 19: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

2010

Lesson learned:

Hold steady to the original vision. Don’t rush to change the product in response to trends.

Positive outcome: clarity

Plenty of existing tools for sharing content. But there may be value in helping people create good written content to share.

Page 20: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

1998

Groundwork:

Reasoning software runs on a PC

2006Inspiration:

Daughter writescrappy essay

2010Stay on target:

Education marketgoes crazy for

collaboration tools

2010Reality check:

Customersexpect perfection

2011Validation:Field test at

Douglas SchoolIn Acton MA

NEXTSTEPS

2011Enlightenment:

Software isShow biz

My product development history

Page 21: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

Showed a civics demo to a history professor friend.

Me: Isn’t this technology super cool? My program generates all these ideas with just a few powerful rules!

John: This is bad grammar. And this question is vague. Teachers won’t accept it.

Reality check 2010

Page 22: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

Lesson learned:

Nobody cares how the software works, they want magic.

Positive outcome: user focus

Spare us the fancy “Artificial Intelligence” technology. Make it simple; hard-code the answers. Don’t mention “rules”.

Concentrate on what people want from the product.

Reality check 2010

Page 23: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

1998

Groundwork:

Reasoning software runs on a PC

2006Inspiration:

Daughter writescrappy essay

2010Stay on target:

Education marketgoes crazy for

collaboration tools

2010Reality check:

Customersexpect perfection

2011Validation:Field test at

Douglas SchoolIn Acton MA

NEXTSTEPS

2011Enlightenment:

Software isShow biz

My product development history

Page 24: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

Douglas School Acton MAMay 2011

Page 25: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

May 2011

Survey comments from 6th grade writing students:

Yes I think it is a very successful website. Could be a great tool for kids writing essays.

i really do enjoy filling in all of the checklists things and answering the questions. i really think that topical storm is a very productive and fun way to write an essay.

It helps writers that have writers block and are unable to organize there ideas while trying to write there essay or story.

I would definetly use a program like this again because it really makes you think about what you are writing about.

I would use this program again because it comes up with ideas that i would have not come up with.

I would use this web site agen because never befor have i been able to make a first draft of an essay in a single day before. Thank you!

Page 26: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

Lessons learned:

Sixth graders are awfully perceptive.

They don’t want happy cartoon characters.

They love the scoreboard.

Make it Smartboard ready, touch-based.

Simplify it beyond belief.

Getting people to use the product is priceless.

May 2011

Page 27: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

1998

Groundwork:

Reasoning software runs on a PC

2006Inspiration:

Daughter writescrappy essay

2010Stay on target:

Education marketgoes crazy for

collaboration tools

2010Reality check:

Customersexpect perfection

2011Validation:Field test at

Douglas SchoolIn Acton MA

NEXTSTEPS

2011Enlightenment:

Software isShow biz

My product development history

Page 28: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

Software as Show Business

(SASB)

“I wouldn’t buy this – even for 5 dollars. But…”

Page 29: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

Early morning realization:

Nobody wants this.

December 2011

Page 30: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

“Enlightened persons who know the True Self work for the welfare of the world, unattached, ever helping to point humanity toward dharma (right action, living a truth-based life).”- THE BHAGAVAD GITA, 200 BCE – 200 AD

Page 31: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

Product Use Testing

also called Usability Testing

involves a task, for example:

• make something with a tool

• drive a vehicle

• fill out a form

• chop carrots

• sleep in a tent

Page 32: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

Things to find out:

• How quickly people do the task

• How accurately

• How long they stick with it

• How they feel about the experience afterwards

• Would they buy? Recommend?

• How much they remember later

Page 33: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

Method

Simple:

• Watch people use your product on a realistic task

• See when they get confused or lose interest

• Ask them for suggestions afterwards

• Use this priceless information to improve the product

• Testing is useful at every stage of development

Page 34: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

Conventional view of testing

TESTHERE

FIX

Page 35: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

Reality of testing

SMALL TESTS…

Page 36: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

Test early & often

• Test a mockup

• For tangible goods: model, prototype

• For software, focus test a “wireframe”:

Page 37: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

Testing in large organizations

• Professional usability testers

• Many test subjects

• Fancy equipment – eye tracking, voice stress

• Statistical analysis of results

• Feh

Page 38: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

thinkeyetracking.com

Nielsen Norman Group University of Minnesota

Page 39: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

Simpler non-fanatic approach

• A few test subjects

• Realistic task

• Listen carefully, take notes

• Record the session if possible

• Think, fix

See: Steve Krug, Rocket Surgery Made Easy

Page 40: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

Step by step

1. Define the goal of this test

2. Choose a realistic task

3. Select participants

4. Prepare participants – expectation, motivation

5. Pre-questionnaire? Understand the person

6. Run the test

7. Ask for reactions, suggestions

8. Report results to the participants

Page 41: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

1. Define the goal of this test

Find out how to improve the product:

See where people get stuck

Gauge emotional reactions

Discover the most & least attractive features

Compare several products or concepts

(“Use each of these machines to make espresso”)

Page 42: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

1. Define the goal of this test

Note:

A Product Use Test tests the product,

not the participant

Task should not be overly difficult

Everything the user does is right; the

product is wrong

Page 43: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

2. Choose a realistic task

Realistic: Something that people need to do and that your product needs to facilitate

Examples of tasks for a website:

Create a document

Make a purchase

Find the answer to a question

Sign up (how enticing is the lead-up?)

Page 44: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

3. Select participants

Who:

People

It takes only five users to uncover 80 percent of high-level usability problems.

-Jakob Nielsen

Page 45: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

Why do the test subjects know better than us?

They know very little about our product

(just like all the people who will use it someday)

They’re detached

They’re focused on their own needs and couldn’t care less how much more work we have to do.

They’re in a hurry

(just like all the people …)

Page 46: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

4. Prepare the participants

Set expectations, give motivation:

Credibility: Who is conducting this test?

How long will the test take?

If you will be recording, tell them

Guarantee confidentiality

Promise a reward: cash, iTunes card, free product or subscription

Ask them to sign a consent form

Let them view the test results later!

Page 47: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

5. Questionnaire beforehand

Sample questions:

What’s your profession?

What types of tasks do you do?

(How) do you currently use this type of product?

What difficulties or needs do you have?

What other similar products do you know?

Blue sky: What product would you love to have?

Page 48: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

6. Run the test

While performing the task(s):

Ask them to think aloud:

What they are about to do How they plan to do it What step they are trying now What they think a feature does What surprises them What questions they have Why they want to kill you

Page 49: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

7. Ask for reactions, suggestions

Post-interview or questionnaire:

Summarize this experience

What was difficult?

What was confusing?

What would you change?

You know, stuff like that

Page 50: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

8. Report results to participants

This is one of the strongest incentives to participate

Get contact info or post on a website

Report what most people liked & hated

Report any interesting ideas that came up

Page 51: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

Today’s Product Use Test is different

• Everybody all at once

• We can’t do think-aloud (unfortunately)

• The program will save your work and comments, and I will look at them later

• We’ll discuss when done

• I will post a summary of the results

Page 52: BC MK-803 product use testing 3-15-12

© 2012 Steve Herbst Confidential and Proprietary

Today’s Product Use Test

1. Log in with your user name & password

2. Pretend you want to write a college application essay

3. Start by knowing nothing about the product

3. See how far you get (that’s the goal of this test)

4. Click this button to enter your comments:

5. Whole class discussion at 9PM

My email: [email protected]

Results will be posted at: www.topicalstorm.com/BC/results

This PowerPoint at: www.topicalstorm.com/BC/slides