Unweighted mean scores for two design variants tested.Įven though Design A had a nominally higher mean score (8.58 vs 8.37), using a standard t-test to compare the means, we find no significant difference at the alpha =. The mean, standard deviation, and sample size for both groups on a confidence question are shown in Table 1 below. We assessed comprehension and ease of use using ten-point scales. 857 qualified participants were randomly assigned Design A or Design B. We presented two variants and wanted to see which one was statistically preferred on a number of dimensions, including comprehension and ease. We recently examined how users of an online retail website would react to a different design of product information. You can see the computation notes in the paper by Bland and Kerry. Participants that should account for, say, 60% of the population have scores that are weighted at 60%, even if they make up, say, only 20% of your sample. It adjusts the means and standard deviations based on how much to weight each respondent. However, the weighted version of the t-test does factor in a second variable. This means that you can’t compare participants’ attitudes on Design A vs Design B AND factor in their prior experience (say low experience and high experience) with your product. (We cover it extensively in Chapter 5 of Quantifying the User Experience.) While the t-test is a “workhorse” of statistical analysis, it only considers one variable when determining statistical significance. The t-test works for large and small sample sizes and uneven group sizes, and it’s resilient to non-normal data. When comparing two groups with continuous data, the t-test is the recommended approach. About the Weighted t-TestĪ relatively simple method for handling weighted data is the aptly named weighted t-test. You’ll also generally want the help of a statistician to assist with the setup and analysis of ANOVA results. The ANOVA is more computationally intensive than the t-test and usually requires specialized software, such as SPSS, R, or Minitab, to conduct. More importantly, it enables you to see the effects of multiple variables simultaneously. The Analysis of Variance (ANOVA) is the statistical procedure you use to compare more than two means at once. With unbalanced samples, two approaches can mitigate and control for the effects of prior experience on your outcome measures: a weighted t-test and a Type I ANOVA. When you need to determine which design is preferred, or to make any comparison, you don’t want the decision to be based on the improper composition of your sample. Even though, for example, your data shows that 30% of your mobile website users have not accessed your website in the last year, it may be difficult to find these users to participate in a study. You can’t always weight your sample to match the population. Most of our clients choose this method-matching the sample to the population-because, when you explain it to stakeholders, it makes sense to them. You can then compute confidence intervals and run statistical comparisons (between, say, two design alternatives) and draw conclusions as to which design users perform better on or prefer. If you believe, for example, that 60% of your website visitors use the site weekly and the other 40% use it less, you can recruit participants to match that composition. One way researchers control for prior experience is to match the experience level of the sample with the experience level of the population. Even if you aren’t planning on using this measure, you should add it as it often comes in handy when it’s analysis time. So in any sample of participants in a research study, you’ll want at least to measure participants’ prior experience. In general, the more experience study participants have had, the better their performance on tasks and the more positive their attitudes toward the product or service being tested. We see this in usability tests and surveys measuring brand attitudes. More than gender, age, income, and occupation, prior experience with products, software, and websites has a major impact on customer attitudes and behavior. One of the most common variables that impacts our measurements is prior experience. There are a number of variables that affect how customers think and behave toward products and services. Rarely is a customer population made up of a homogenous group of customers who share the same attributes.Ĭonsequently, our samples contain a mix of customers who may or may not reflect the composition of the customer population.
0 kommentar(er)
