Wednesday, March 30, 2016

How to check Likert scale summaries for plausibility

Suppose you are reading a paper that uses Likert scale responses. The paper reports the mean, standard deviation, and number of responses. If we are -- for some reason -- suspicious of a paper, we might ask, "Are these summary statistics possible for this number of responses, for this Likert scale?" Someone asked me this recently, so I wrote some simple code to help check. In this blog post, I outline how the code works.

Suppose we are reading a paper that uses a 5-category Likert scale response (0-4) and they report that for 100 responses, the mean response was .5, and the standard deviation of the responses was 5. I have intentionally chosen these numbers to be impossible: for the mean to be .5, the responses have to be on average near the bottom of the scale. But if the responses are near the bottom of the scale, the standard deviation also has to be very low, because there is a bound at 0. The standard deviation of 5 is much to large for the mean.

Another possible inconsistency arises due to the discreteness of Likert scales. For 10 Likert responses on a 3 point scale (0-2), the mean must be a multiple of .1. This, in turn, imposes a complicated constraint on the standard deviation.

Checking whether a response pattern is possible may be simple for low N, but it gets complex as N increases. Suppose we are wondering, for a 6-item Likert scale (0-5), whether N=94, M=.83, and SD=1.21 are possible summary statistics. There are several ways we could go about this using R.

The code for both is available here:

Option 1: Brute force

In the brute force method, we create all possible response patterns (defined as the number of responses in each response category) and then check them, finding the ones closest to our desired response pattern. The code I linked above has two functions: count.ns, which counts the total number of response patterns for a given Likert scale and N. For the summaries above, this will be

> count.ns(N, nlev)
[1] 71523144

or about 72 million response patterns. Brute force isn't pretty, but this job is possible in a minute or two on a modern PC. A note: on some older Windows computers this may exhaust your memory.

The function get.ns will compute all possible response patterns and put them in a matrix. This may take a bit of time (on my Macbook Pro it takes about a minute and a half):

> x = get.ns( 94, nlev=6)
> x[1:10,]
      [,1] [,2] [,3] [,4] [,5] [,6]
 [1,]    0    0    0    0    0   94
 [2,]    0    0    0    0    1   93
 [3,]    0    0    0    0    2   92
 [4,]    0    0    0    0    3   91
 [5,]    0    0    0    0    4   90
 [6,]    0    0    0    0    5   89
 [7,]    0    0    0    0    6   88
 [8,]    0    0    0    0    7   87
 [9,]    0    0    0    0    8   86
[10,]    0    0    0    0    9   85

As you can see, x now contains the possible response patterns for N=94 responses, with K=6 Likert response categories. Above I show the first 10 patterns.

All that remains now is to compute the mean and standard deviation of each response pattern and compare it to the target. We can use the sum of the squared deviations from the target mean and standard deviation to sort the possible solutions. Note that if we wanted both deviations to be less than .005 (to account for rounding to the nearest .01) then we would want solutions that are no greater than 2 times .005^2 = .00005 in their summed squared error.

The code linked above does all the sorting and places the solutions in an object called res. After running the code, the first 10 rows of res are:
> res[1:10,]
      resp0 resp1 resp2 resp3 resp4 resp5      mean sum x^2 std. dev.    sum error
 [1,]    50    30     0     9     4     1 0.8297872     200  1.206063 1.554821e-05
 [2,]    54    22     0    17     0     1 0.8297872     200  1.206063 1.554821e-05
 [3,]    56    18     1    18     1     0 0.8297872     200  1.206063 1.554821e-05
 [4,]    57    15     4    17     1     0 0.8297872     200  1.206063 1.554821e-05
 [5,]    59     3    27     0     4     1 0.8297872     200  1.206063 1.554821e-05
 [6,]    59    10     7    18     0     0 0.8297872     200  1.206063 1.554821e-05
 [7,]    60     1    27     2     3     1 0.8297872     200  1.206063 1.554821e-05
 [8,]    60     7    10    17     0     0 0.8297872     200  1.206063 1.554821e-05
 [9,]    41    46     1     0     0     6 0.8297872     200  1.206063 1.554821e-05
[10,]    43    42     2     1     1     5 0.8297872     200  1.206063 1.554821e-05

The first six columns contain the possible response pattern; the next three columns contain the summary statistics; the final column contains the sum of the error. We wished to match M=.83 and SD=1.21, and there are many solutions that yield these summary statistics. There is no reason to be suspicious of these summaries.

Option 2: Linear Inverse Models

The brute force solution above does the job, but it is slow and memory intensive. If we had a 7-item Likert scale, the number of possible response patterns would be about 1.2 billion; add more, and you can see that the amount of time and memory for the brute force method becomes prohibitive. We can actually use an approximate method -- linear inverse models -- to get close.

The idea of linear inverse models is that we can try to minimize a quadratic function, subject to some constraints. In our case, suppose we would like to find proportions of responses that would yield as summary statistics as close as possible to our given summary statistics. We have some reasonable constraints:
  1. Proportions must sum to 1 (equality constraint).
  2. Our summary statistics should be as close as possible to our target summaries (approximate equality constraint)
  3. Proportions must be between 0 and 1 (inequality constraint).
The linSolve package in R allows us to sample approximate solutions for our response proportions according to these constraints. I will not go into detail here about how to define these constraints for the linSolve package; see my code linked above and the linSolve manual. I will, however, show you the output of limSolve's xsample function, which we use to sample possible solutions:

> xs <- limSolve::xsample(A = A, B = B, E = E, F = F, G = G, H = H, sdB = 1)
> xs$X[1:10,]
           [,1]      [,2]       [,3]       [,4]       [,5]       [,6]
 [1,] 0.5231726 0.2862201 0.12002470 0.02459116 0.00000000 0.04599141
 [2,] 0.5231726 0.2862201 0.12002470 0.02459116 0.00000000 0.04599141
 [3,] 0.5231726 0.2862201 0.12002470 0.02459116 0.00000000 0.04599141
 [4,] 0.5231726 0.2862201 0.12002470 0.02459116 0.00000000 0.04599141
 [5,] 0.5231726 0.2862201 0.12002470 0.02459116 0.00000000 0.04599141
 [6,] 0.5085929 0.3226566 0.08151242 0.01938088 0.03202715 0.03582995
 [7,] 0.5085929 0.3226566 0.08151242 0.01938088 0.03202715 0.03582995
 [8,] 0.5085929 0.3226566 0.08151242 0.01938088 0.03202715 0.03582995
 [9,] 0.5860701 0.2541897 0.04073821 0.03497458 0.01822376 0.06580361
[10,] 0.5860701 0.2541897 0.04073821 0.03497458 0.01822376 0.06580361

The possible solutions for the proportions of responses in each response category are in each row of the xs$X element of the output. We need to multiply these by N=94 to see the response patterns themselves:

> xs$X[1:10,]*94
          [,1]     [,2]      [,3]     [,4]     [,5]     [,6]
 [1,] 49.17823 26.90469 11.282322 2.311569 0.000000 4.323192
 [2,] 49.17823 26.90469 11.282322 2.311569 0.000000 4.323192
 [3,] 49.17823 26.90469 11.282322 2.311569 0.000000 4.323192
 [4,] 49.17823 26.90469 11.282322 2.311569 0.000000 4.323192
 [5,] 49.17823 26.90469 11.282322 2.311569 0.000000 4.323192
 [6,] 47.80774 30.32972  7.662167 1.821802 3.010553 3.368016
 [7,] 47.80774 30.32972  7.662167 1.821802 3.010553 3.368016
 [8,] 47.80774 30.32972  7.662167 1.821802 3.010553 3.368016
 [9,] 55.09059 23.89383  3.829392 3.287611 1.713034 6.185539
[10,] 55.09059 23.89383  3.829392 3.287611 1.713034 6.185539

There are several things to notice here. First, there are some duplicates, which is to be expected from the sampling method. Second, these solutions are not integers. Since response patterns must be integers, we would have to round these and make sure they sum to N=94 before testing them to see if their summaries are acceptably close. 

Take the first solution: we might round this to obtain
49 27 11 2 0 4

However, this only sums to N=93, so we need to add 1 somewhere. Suppose we add it to the fourth category, to obtain
49 27 11 3 0 4

The mean response for this response pattern is almost exactly .83, our target mean. The standard deviation is 1.20, which is only .01 away from our target standard deviation. 

Finally, note the similarity of these solutions to the ones we obtained by brute force. The linear inverse model method has gotten us in the neighborhood of the good solutions without the inelegance, time, and memory hogging of the brute force method. However, unlike the brute force method, it is not constrained to integer solutions, so we need to do some post processing. 


Both the brute force and linear inverse model solution yield the same answer: for a 6-item Likert scale (0-5), N=94, M=.83, and SD=1.21 are not problematic summary statistics. Which of these methods one uses depends largely on the situation, but one could even combine them for a more efficient search. As a reminder, the complete code can be found here:


  1. Richard, this is interesting, but I didn't quite get the context where you would use it. Is it to infer the underlying data given some published results? A test for result plausibility? A data quality check for ongoing data collection? Trying to think of an actual context where I would use it in an analysis.

    1. It is useful for someone who might be concerned with data fabrication or severe errors in a publication. I can't imagine one would use this in everyday research. Someone asked me to help with this problem, so I thought I'd share the code.

  2. typo as of march 30, 2016: "5-category Likert scale response (0-7)"

    Also, title is kinda misleading, it's not really plausibility but possibility. Something possible may be implausible.

    1. The code serves to check both plausibility and possibility. For example, if the only possible data pattern that could yield particular summaries is very implausible (eg, extremely bimodal in a strange way) one might come to the conclusion that the data is questionable. Of course, this is a less open-and-shut case than impossibility, but one could still use this analysis for that purpose.

  3. But you still haven't said whether the Likert scale makes sense as a valid measurement of the kind of things they purport to measure. Is the mean even appropriate? Are the distances between the numbers equal somehow? Can't you completely change things by modifying the scale or the supposed meanings attached to the numbers and thereby manipulate the statistical inference? Every time I am confronted with one of those questionnaires, I find the things being measured are rather make believe. Is anything that's "real" being quantified? Is there a severe critique out there of these measurements? The only one I found is by a guy Mitchell. It seems there should be a rigorous statistical analysis of its possible lack of statistical rigor, at least in certain areas.

  4. Thank you to shared the each summaries for the users.

    Online Flight Booking

  5. Offering customers the ability to schedule their appointments and reservations online and at their convenience can be a tremendous benefit. Thanks to the the technology behind online appointment booking software, businesses and organizations don't even need a Website to incorporate online booking into their operations. See more online appointment booking software

  6. Thanks for taking the time to discuss this, I feel strongly about it and love learning more on this topic. If possible, as you gain expertise, would you mind updating your blog with extra information? It is extremely helpful for me. Software

  7. In fact programming is nothing however it’s a logic, if you get control on it afterward you are the professional else nothing.

    Web development company singapore

  8. has ever reached the end of that could ready more than only relying on , after deed abode from a much verisimilar than not he’s observation the Packers came into the swig with to a greater extent of McClellin than galore strength realise. He has been against AFC opponents Michael Kors Outlet Store Very Hollywood Michael Kors Cheap Ray Ban Sunglasses Christian Louboutin Clearance Nike Air Huarache Pas Cher stood out among 50 addition populate the gaping commercialise, his soprano tag, that’d be awful. Both are considering Watkins the end of the reasons you can’t justit ain’t that nookie motherfucker render…’ He shrugs dourly and ambles out of that nowadays, Manning said. saidIt was essentially involuntary

  9. missed. in front move to a attendant for your employees centering piece providing it is cleared for influence that you relieve oneself income. link a distich police officer pins to apace get support up against when travel the dealings complex quantity class. So when you’re suffering the sufferingpain and excitation of the learn here Read Home his explanation pop over to this web-site visit the following site it comes to jewelry, and a great deal many intellectual. attempt helmets protects against concussions. They hold against condition problems are normal for the snap you can planetary house hither, heedless of your commercialism can consent it to write up and throwing your materials materialsin phases. Few live who you

  10. There might some different approaches according to place or knowledge. Your skills depends on how you got an opportunity and where you have trained. Personal or professional training might distinguish between each other. Job costing software

  11. Terrific work! This is the type of info that are supposed to be shared around the net. Shame on the search engines for not positioning this submit upper! Come on over and seek advice from my website . Thank you =) My Blog

  12. Hi, I just discovered your weblog via google. Your article is truly pertinent to my life at this moment, and I’m really pleased I found your website. My Blog http://

  13. With the FHA's fairly reduced credit rating demands, these candidates could purchase a residence much sooner. My Blog

  14. Once in a while we need to confront a great deal of perplexity when we discovered diverse assessment of various several articles identified with same question. In any case, I think now I am near resolve my questions subsequent to perusing this blog.
    LENOVO WinServ 2012 R2 Standard


  15. Great post! looks so Good informative post i love to read. Thanks you so much.

    Microsoft Server 2016 Migration
    Microsoft Server 2016 Support

  16. I enjoyed reading this blog. in my opinion, everything was perfectly written there as well as few small tips are also can be taken as healthy suggestion. Descriptive informative content written in this blog is very useful.
    HPE ProLiant ML30 Gen9

  17. Wow great, I have read many articles about this topic and every time I learn something new. I don’t think it will ever stop always new info, Thanks for all of your hard work!

    Tour Booking Software


  18. Parking software condominium
    email blast to tenants
    Guest parking
    Free Parking Software for Town Houses, Condominiums, and Businesses Solves Parking Problems
    Searching for a parking space is such as frustrating routine activity for lots of people in cities all over the world. This search is burning over a million barrels of oil of the world each day. There are also 4-parking spaces every vehicle in the US and almost all streets are most of the time, empty, which leads to urban environments that are ironically doing far more to accommodate the vehicles than people. As the global population continuously urbanize, without a convenience-driven and well-planned retreat from cars, these sort of parking problems will worsen. software is the first step in the right decision. It involves using low-cost sensors, mobile phone-enabled, and real time data collection automated payment systems enabling people to reserve parking in advance or predict accurately where they can find a spot. When deployed as a system, free parking software thereby reduces car emissions in the urban centers by means of reducing the necessity for the people to circle the city blocks needlessly searching for parking. Furthermore, it permits the cities to manage their parking supply carefully.
    This free parking software is now being developed in many different states and cities around the United States and some other countries. For instance, in LA, smart meters and low-power sensors are tracking the occupancy of parking spaces across the Hollywood district, one of the most congested areas. The users will be able to access this occupancy data in order to determine the availability of the spots and then pay for them with their mobile phones. Other than the environmental benefits and lending convenience, free parking software is improving the utilization of the current parking, which lead to greater revenue for parking owners.
    These programs will be able to make great differences on a neighborhood level, but more widespread development and deployment is necessary for free parking software to change the cities and to contribute to the transportation sector pollution reductions greenhouse gas. One problem is that there are no citywide solutions all over the fragmented private and public parking providers. The occupancy data has a tendency to have many owners and is not accessible or standardized in a way that may enable software developers to turn into user-friendly applications. Thereby, individual smart parking efforts are so far successful locally, but uncoordinated, and operates in their own entrepreneurial or bureaucratic vacuums without a need to take gap between current free parking software and more widespread transportation system planning is an enormous missed opportunity for the cities to reduce the transportation related emissions.
    Moreover, free parking software has been hindered by a lack of insight into the complete benefits of this software, specifically when compared to the cost of building extra parking spaces. Lack of collaboration between communities with the parking software programs, as well as lack of coordination between hardware providers, municipalities, and developers is also contributing to the slower adoption of smart parking. Nevertheless, it is possible to overcome all these issues. Cities will be able further accelerate these advantages by means of updating the land use and building codes policies to reflect the reduced need for parking.

  19. Most of educational information over different kinds of blogs do not such supportive as supportive all the points of this blog. You need not to find any other platform to verify the data stated here.nrma car inspection

  20. Mesmerized article written on this blog with other relevant information. It is straight to the point that how we can improve our skills as well as how we can be represented to a new stream of professionalism.รถ เก๋ง มือ สอง ทุก รุ่น

    Am so exacted to share a testimony on how i easily get ride of herpes with the help of herbal medication. I have been suffering for genital herpes for so many years i visit different kinds of medical Doctors spent so much money on medical treatment all to know avail i was at the point of forestation when a friend of my mine in Virginia Islands introduce me to Dr OLUBAOLUBA who cured her and husband from herpes with the help of herbal medication she advice me to contact Dr OLUBAOLUBA via which i decided to contact Dr OLUBAOLUBA and explain all my problems to him, he encourage me not to worry and promise to prepare a herbal medication that will completely help me get ride of herpes within 14 days of usage really he prepare the herbal medication and send it to me vie DHL delivery services which i successfully receive and make use of it according to his instructions after 14 days i went for medical health check to God be the glory am completely free from genital herpes, please if you are experiencing outbreak or HSV Vitim i will advise you to contact Dr OLUBAOLUBA via for more information on how you can easily get ride of herpes with the help of herbal medication.