Most liked posts in thread: MINI reliability - better info

Our current stats for the Hatchback, Convertible, and Clubman cover owner experiences through June 30, 2012.

Repair frequencies, in terms of repair trips per 100 cars per year:

2012: 16, very small sample size

2011: 65

2010: 61

2009: 69

2008: 79

2007 (hatch and Clubman): 103

2006: 47

2005: 109

2003: 74, very small sample size

We have reworked the scale, which has tipped the 2005, 2007, 2008, and the 2011 into "worse than average," and the 2009 and 2010 close to it.

The 2012 is looking good so far, but we need more data on more cars for a definitive stat. The Countryman isn't yet in the survey, but the 2012 is just half a dozen cars short of the minimum to include it.

To see how competitors compare:

MINI Cooper reliability ratings and comparisons

 

All reasonable questions, thanks.

The results are intended primarily for our members, but also for other people looking for information on cars.

The goal is as stated in the OP, better, more up-to-date information.

The site is self-funded. Revenue does come primarily from ads, I'm not sure why you didn't notice them, though I don't put them in obnoxious locations. I think most are clicked on by people who enter the site through organic search, and it's not what they were searching for. I don't think we get a substantial amount of revenue from members, but that's okay.

We carefully review all submitted repairs, and exclude those that are clearly due to mods. This isn't often a problem. In a very small number of cases it isn't clear whether or not the problem would have still happened without the mod, and then a judgment call must be made.

 

You make a good point. I hadn't noticed that the situation was quite this extreme, but was aware that for some of these years a common problem is a hood scoop that melts, and this obviously applies only to the S.

The problem is that, without more participants, splitting the results by powertrain could result in no stats for either. I'll see what we can do with the November update.

With more participants we could easily split the stats by powertrain.

 

Which concerns aren't addressed?

Compared to other reliability surveys we:
--post the actual repair frequencies, not just dots
--promptly update four times a year, not just once a year after a long delay
--ask a clear, relatively objective question on the survey
--post all reported repair descriptions (noted in your post)

I don't think we're yet a good way to measure cost of ownership--this would require especially large sample sizes, and we don't have these for most models yet. Trying to fix this!

 

It's not necessary to have a large number of responses for accurate stats. Even 25 is a decent number, though 40+ is much better. We don't yet have 40+ in many cases, but it's not far out of reach.

Owners generally expect the car to start, the power windows to work, and so forth. By asking an objective question--did the car require a repair, and not "did the car have a problem you considered serious"--we manage this variable much better than other surveys.

Similarly, by having a continuing survey that starts covering the car when a member joins, and not a retrospective survey, this analysis is less subject to self-selection than others.

It's very easy to split the results by engine--if we have enough participants. We're doing this for other models where reliability differs by engine.

Yes, a smiley face is much like a dot. It's intended to be. The difference is we have the actual repair frequency next to it. You won't find such numbers elsewhere.

All repairs shown on the repair histories pages are not included in the analysis. For instance, brake pads are excluded after 24,000 miles. I can assure you that most people spending hundreds of dollars to decarbonize an engine don't see it as simply maintenance. Nor are manufacturers telling people they should expect this service much like they expect to have to replace the brake pads. If it truly becomes common with all DI engines, the general public is going to be outraged. While a stuck sunroof doesn't impair the basic operation of the car, it does reflect the reliability of that part of the car, the sunroof. We clearly define "reliability" as the number of repair trips required. It's a closer fit than other possible terms, such as "quality" and "durability."

A repair frequency over 100 per 100 cars does generally mean that most (but rarely more than 80 percent) of the cars required at least one repair during the past year. For many models in the survey the score is under 30, even under 20. A score over 100 DOES imply that at least one repair per year is very likely. (In comparison, a black dot in CR can represent a reported problem frequency as low as 25 per 100 cars, but very few people realize this because they don't post the numbers.) For cars with enough responses, we publish additional "repair odds" stats, that display the percentage of cars with zero repairs and the percentage with 3+ repair trips. The analysis includes an outlier control, so a single car with a very large number of repairs will not distort the result.

You misquote the disclaimer on the repair histories page, partly because we might need to relocate it. It says that the "repair histories," the repair descriptions, should not be used to infer repair frequency. You've substituted "survey" for "repair histories." On these pages we only display cars for which repairs have been reported. Someone merely glancing at these pages might infer that the car is highly unreliable, since every car displayed has required repairs. But there could be ten times as many cars for which no repairs have been reported.

In sum, of your critiques that of sample size is by far the most valid. It's also the most easily fixable. In part by posting here, I'm doing what I can to fix it.

 

Thanks, Nathan.

With mods it is sometimes necessary to make a judgement call on whether the problem would have happened even without the mod. The analysis excludes repairs caused by mods.

 

You seem to have some knowledge of statistics, so I'm surprised that you think that the size of the population is relevant in determining the required sample size. It's not even part of the equation.

We've always posted the number of responses. Neither CR nor JD Power do.

We used to post the survey, but recently switched to a dynamic form that varies based on the questions answered. I'd like to also post this one, but no one has asked for this so other things have been higher priorities.

Essentially the form asks people to report all repairs to the car, even minor ones, but not to report scheduled maintenance or problems due to accidents or mods.

On your sunroof issue, it's always possible to split hairs further. But if you've got even sample sizes as large as we do this sorts itself out. You get a lot more variance if you ask people to report problems "they considered serious." With this other approach, you'll have some people not reporting transmission failures because they were covered under warranty while other people report rattles because they feel a new car shouldn't have them.

Also, DIY repairs can and should be reported. We don't require that a shop perform the repair.

Self-selection doesn't distort the results to the same degree in every case. What we're worried about here is people self-selecting because they 1) have had a lot of problems, and want to report them 2) are a fanboi for the car, and want to boost its score.

For the first, a prospective survey will greatly reduce the distortion, as people usually cannot report the problems that led them to join.

For the second, asking a relatively objective question limits its impact. Asking people to report problems "they considered serious" lets people honestly under-report problems. They really like the car, so the problems they've had with it don't seem serious. This can be a huge source of bias in other surveys. With our survey, people have to be dishonest to under-report, and this is less likely.

The additional metrics are available even to people who don't sign up, just with a higher minimum sample size.

Thanks for the tip on the text on the landing page. I hadn't realized it read that way, and thought you were referring to the more accurate text on the destination pages. Fixed.

It's absolutely impossible to gather or report perfect survey information, just like it's impossible to create a car that does everything the best. There are inherent tradeoffs with cost and people's willingness and ability to objectively respond. We work within these constraints as best we can, and I feel we do a better job of it than people who've been doing it for far longer with much larger budgets.

In the end, our #1 weakness, by far, are the sizes of the samples. I did look into splitting the stats for the Cooper and Cooper S yesterday, and we don't have enough responses to do this yet.

 

We have updated our reliability stats for the MINI based on owner experiences through September 30, 2012.

Repair frequencies, in terms of repair trips per 100 cars per year:

2012: 16, better than average, small sample size

2011: 66, worse than average

2010: 53, about average

2009: 60, about average

2008 (hatch and Clubman): 74, worse than average

2007 (hatch and Clubman): 81, worse than average

2006: 51, about average

2005: 104, worse than average

2003: 72, about average, very small sample size

We'll have further updates in February and May. We'd love to provide more precise stats, cover all model years, and provide separate stats for the S--just a matter of getting more owners involved!

To see how competitors compare, and to sign up to help:

MINI Cooper reliability ratings and comparisons