Unpacking the behavioural auditor’s toolbox

Manja Knevelbaard & Jan Otten explore how the Q-method can help develop a better understanding of organisational feeling and culture.

We released the first publications on behavioural auditing more than a decade ago and cautiously undertook the first implementations of the theory. The approach has since grown into a mature and very successful audit product. A behavioural audit is carefully designed to obtain insight into organisational members’ behaviour, to report on the results, and by doing so influence organisational socio-sychological climate and culture. A behavioural audit is an inductive audit form to investigate complex organisational issues at the level of mental models, motivations, and beliefs concerning managerial control. Examples of research questions are: “How effective is our change program?”, “Why do many of the projects fail?”, and “Are we a learning organisation?”

The following starting points have been brought together in the methodology of a behavioural audit:

  1. The Action Theory of Argyris & Schön;
  2. The Learning History, developed by Roth & Kleiner;
  3. Qualitative research with an emphasis on the Grounded Theory of Glaser & Strauss.

The latter point in particular will occasionally cause hesitation among both clients and auditees. However, the principles of quantitative research − in which mathematical figures and statistics play amajor role − are deeply rooted in many ideas about research. This often causes surprise regarding the size of the ‘sample’ and the generalisation ability of the results of a behavioural audit.

The sample of qualitative research must be designed in such a way that research results are represented in a qualitative sense. ‘Representativeness’ has a different definition within qualitative research than within quantitative research. In quantitative research, the term ‘representativeness’ is defined numerically and based on (large) numbers of respondents. Qualitative research, in contrast, is small-scale because representativeness in this context is not based on numbers and numbers of respondents. The results are qualitatively representative if the sample is composed in such a way that all relevant variations in opinions, views, feelings, and motivations are given the opportunity to manifest themselves within the sample and the research. We speak of representativeness in qualitative research, therefore, in terms of how well those factors manifest in the sample.

Samples of 12, 15 or 20 respondents are common in qualitative research. Those small samples are usually
surprising. One of the most frequently asked questions about qualitative research is: ‘how can you draw valid conclusions based on a study with 12, 15 or 20 respondents? You always need at least 100 people, right?’

However, with subjects about which little is known, the demand is usually for exploratory research in which qualitative methods are often appropriate. This research is not about adding facts and figures but
unravelling a ‘why’ or ‘how’ question. Qualitative research is often aimed at very specific target groups, where the variance in behaviour is smaller than for ‘all employees’. As a rule, the focus is on underlying motives and choice arguments. In a behavioural audit, we look for motives and motivations. We speak of ‘mental models’, responsible for how people think and act. Therefore, we are mainly looking for the underlying causes of certain situations or events. In short: an answer to the ‘how’ and ‘why’ questions!

At the level of ‘how’ and ‘why’, you will find less difference in variance than at the behavioural level (‘where’, ‘what’, and ‘how much’). It is remarkable how behaviour is often driven by a limited set of motives and values!

The results of qualitative research cannot be generalised statistically, but theoretically. This means that the results may be generalised to environments whose context is the same as the context of the research group. For example, a behavioural audit delivers a so-called ‘narrative’ as one of the end products. An effective (and efficient!) possibility to generalise the results of the audit, when in doubt, is to discuss the narrative (possibly (further) anonymised) with members of other organisational units of which we do not know whether the context is the same as that of the research group.

Sometimes the behavioural auditor is confronted with organisational members who are less open to the
principles of qualitative research. Situations are also conceivable in which it is important not only to gain insight into the variation of motivations and mental models, but also into their quantitative distribution among the organisational members. When it is clear which views are held by which groups of people, the client can draw up a policy aimed at specific groups within the organisation. In those situations, the so-called Q-method comes in handy.

The Q-method

Many auditors have never heard of the Q-method. That in itself is not so strange. However, even though the method has been around for almost a century, it is not or hardly discussed in the regular teaching of methods and techniques. It is a pity, because the Q-methodology can be a valuable addition to the toolbox of any behavioural auditor.

With the Q-method, subjective research areas can be portrayed in a reliable and detailed way. The underlying idea of the methodology is that subjective assessments can best be examined by weighing them up against each other. This retrieves information about a respondent’s relative perception.

An opinion usually cannot be captured in a simple representation on a Likert scale in a questionnaire. Yet subjective information (such as opinions about politics, points of view, beliefs, etc.) is often retrieved and reported in studies in this way. We also see this reflected in audit practice, where (a limited number of) audit departments make an inventory of so-called ‘soft controls’ using a questionnaire. This, of course, should not cause any problems. For topics such as which political party is voted for, the market share of a candy manufacturer or the lease car of the year, it is useful if the answers are shown in percentages or numbers. However, it is a different story when it comes to complex subjective topics, such as people’s perception of psychological safety in the organisation or the attitude towards a new management philosophy to be implemented.

With such topics, it is necessary to delve deeper into an issue than the level of a simple ‘agree’ or disagree’ to get a good picture. The Q-method offers an excellent opportunity to investigate the nuance and complexity of subjective subjects. The Q-method was originally developed and introduced by the
psychologist and physicist William Stephenson (1902-1989) and is mainly used to reliably retrieve subjective information from individuals. The underlying idea of the Q-method is that subjective statements are relative and can therefore be assessed in relation to each other.

The Q-method consists of a Q-set, the Q-sort, and the Q-analysis. The Q-set is a set of statements that are ordered by the respondent based on his or her preference. This creates a representation of the participant’s subjective experience, their personal Q-sort. For the respondent, there is also a game element; it is fun to slide the cards and arrange them. The Q-sorts of different respondents are then analysed with the Q-analysis. Through this analysis, correlations between individual variables are not sought separately, but the complete Q-sorts of the respondents are compared with each other. Subsequently, factor analysis is used to find out which groups can be distinguished. The Q-analysis provides, on the one hand, a representation of the subjective experience of the participants: the ‘intrapersonal’ is based on the order in which one respondent made the statements. On the other hand, the prevalence of which opinions exist within a selected group is assessed as ‘interpersonal’, based on the
grouping of corresponding sortings of several respondents. These features make the Q method a valuable tool for examining subjective topics at an aggregate level while still preserving the level of detail.

During applying the Q-method, respondents are asked to sort a set of statements based on a so-called Q- sort table (see figure 2). In a ‘stand-alone’ Q method, the statements often come from multiple sources, such as newspaper articles, literature, social media, etc. When we use the Q method as an addition to a behavioural audit, the statements are directly derived from the audit results. It concerns respondents’ views, opinions and other expressions as they emerged during the interviews and the data analysis.

The Q-method as an addition to Behavioural Audit

The Q-method can provide insight into the subjective experience of (groups of) organisation members. In fact, the step is made from inductive and, therefore, theory-forming research to deductive, theory-testing research.

The analysis of data obtained from interviews and possibly behavioural observations takes place in a behavioural audit according to the principles of qualitative data analysis. An essential step in the analysis process is developing themes. The researcher translates these themes into propositions and forms the basis for the Q-set. The analysis usually yields around 20-30 themes. These themes can have both a positive and a negative direction, making them suitable for the format of a thesis.

One can consider adding statements on this point that arise from scientific literature, newspaper articles or social media, and the like, if it can provide additional insights and is relevant to the research question. It is, of course, important that the number of positions corresponds to the number of squares of the normal distribution, as shown in figure 3.

The next step is to have the respondents place the statements in the boxes of the normal distribution. The first is to sort the statements about which one has a strong opinion. These positions are placed in the
outermost boxes of the distribution. Then the positions are placed in the middle part of the distribution. This step of the Q-method can be performed entirely online, via specially developed software or under the supervision of the researchers. Although the latter method is, of course, much more labourintensive
and therefore more expensive, the advantage is that additional information can be obtained, for example, in the form of an explanation by the respondents of the choices that have been made.

According to Q-methodologists, the number of respondents should be between 40-60. Alternative sample sizes, such as a mathematically substantiated sample size, are also conceivable. In those cases, the number of respondents will usually be considerably larger. The assessment must then be made to weigh the
research’s purpose and importance against the costs and certainty margin desired by the client.

After all respondents have submitted their Q-sorts, a factor analysis is used to determine which statements are most relevant to which groups of respondents. It is then up to the researchers to find out the specific characteristics of these groups and distinguish them meaningfully.

Finally, the results of the Q-investigation are reported and discussed with the client. The insight that the research provides can be a reason for the organisation to implement a targeted policy for specific groups and thus allow the organisation to develop further in the desired direction.

Manja Knevelbaard & Jan Otten

Jan studied organisational psychology and business administration. During and after his education, he conducted research for institutes such as Radboud University Nijmegen and the Institute of Applied Sociology (ITS). Most of these studies related to work motivation and aligned with the socio-technical paradigm. In 1994, after 10 years at KPMG, he started ACS. His main activity is conducting and supervising audits on behaviour and culture. He also provides coaching, training and education in this area. After working for several years in the energy and IT sector, the semi-government and financial industries, Manja joined ACS Partners in 2022. She researches behaviour and culture (behavioural audits) and supervises Agile Audits with multidisciplinary teams. Manja graduated in Controlling and Change, completing her thesis on the interaction between auditor and auditee and what this means for the adoption of recommendations. She graduated cum laude.

This article was published in 2022 in the Ethical Boardroom (www.EthicalBoardroom.com).

 

Geef een reactie

Je e-mailadres wordt niet gepubliceerd. Vereiste velden zijn gemarkeerd met *