Published in The Clarity Journal 80 – 2019.
Every reader of this journal will be convinced that plain language revisions improve reading success. But in this article, I will argue that this improved success might not help the people you wanted to help in the first place; people who do not read as easily as us, academics. And we might even increase the gap between lower and higher literate people.
This increased gap is referred to as a Matthew-effect Merton (1968). Plain language interventions might cause a Matthew-effect while the plain language practitioners actually aimed for the opposite. This idea is based on my research on improved readability of the Dutch court summons in small claims. I will clarify the Matthew- effect in PL-revisions giving you a broader context of reading research, explaining some of the results of my research, and I will explore some ways to avoid the Matthew-effect.
Matthew’s parable of the talents
In the parable of the talents, a master entrusts his capital to his three servants. One gets 5 talents, the second gets two and the third one gets one talent. On his return the master asks the servants what they have done with the money. The first two doubled their money by investing, and they are being praised and rewarded. But the third, who was afraid to lose his one talent and fearing punishment, buried the silver. He digs up the money and returns it to his master. The master is furious with his bad and lazy servant and he gives his share to the other two. It is this story Matthew has in mind when he says that For to everyone who has will more be given, and he will have abundance; but from him who has not, even what he has will be taken away (Matthew 25:29)
What does increased reading success mean?
You are a technical writer, communicative lawyer or researcher and you have worked hard on your text by revising it according to plain language instructions. You want to know if it actually works. You can simply test how many readers comprehend document 1 and how many comprehend the revised document 2 and guess what? The document 2 shows better results. Yes, reading success increased, but do we know what happened and why?
In this example, we actually don’t know if readers who performed poorly on document 1 actually show improved comprehension in document 2. It is likely that the average comprehension is increased by the readers with high reading proficiency performing even better. Unless you have tested all factors in the reader that might be responsible for reading success, there is no way of telling if lower literate benefit or not from the improvements.
Three factors to evaluate reading success
Reading success is the result of interaction of three factors (Kintsch, 1998, Kirsch, 2001, 2005, Kirsch, Jungeblut & Mosenthal, 1998):
- the reading proficiency or literacy level of the readers;
- the function of the document;
- the difficulty of the task that is set for the reader.
When testing the effectivity of revisions, every technical writer needs to take into account these three factors. That is an expensive and complicated job, especially because the lower literate, poor and indebted don’t participate much in adult reading research. In the past years I looked for reading research results where literacy of the test person was a factor, but not much was found. Exception is Lentz, Nell & Pander Maat (2017).
Taking literacy into account
The major factor in reading success, that is way more important than document features, is literacy (or reading proficiency). If we do not take literacy levels into account, we have no idea who actually benefits from our revisions and it is quite likely the group that already performs rather good and pushes up the average.
In the legal field where I work and conduct research, it is the comprehension of the low literate readers that drives me to improve the access to justice, which seems to be guarded by impressive documents. These readers usually are not only weak in reading, they are more likely to have a lower economic status and therefor are more vulnerable when dealing with complicated financial products or when involved in a legal (debt collecting) procedure. To fully understand the Matthew-effect you need to know a bit more about these specific readers.
The reading success of indebted readers
Dutch court documents are notorious for their incomprehensibility. This is a big issue because these documents can have a huge impact on the receivers, especially for the over represented group of poor and indebted readers. For example, after a conviction, debtors have to deal with a seized lone, or the risk of losing their home.
But what do we know about the receivers of court documents in debt collecting cases? They are lower literate (Jungmann, Madern & Geuns, 2016). Lower literacy is linked to poverty, depending on welfare (Christoffels & Baaij, 2016), Buckingham e.a. (2013), Hernandez (2011) and the lower literate show more risks in health (Berkman e.a., 2011). In the past years I conducted a research on the comprehensibility of the court summons in debt collection cases. When joining the judicial officer on his route delivering these court summons, I was visiting neighborhoods, families and loners mostly all living in poverty and I can’t recall having seen books in any of these houses, but one.
So most of the receivers of court documents in their debt collecting case belong to the group that is most vulnerable in our society.
Testing court summons
I tested four documents in a 2×2 design on over 200 readers. The content in all four documents was the same.
The first, classic summons is normally being used throughout the legal professions involved in serving writs.
In the second version, the structure was manipulated by inserted headings and an altered order of both sentences and topics.
In the third version, only words and sentences had been manipulated, the structure however remained identical as in the classic summons.
In the fourth version, both manipulations are combined.
The results indicate that when both style and structure are revised (version 4), readers with lower literacy and a lower educational level are no longer disadvantaged and show equal reading success. But the highest average score was not on this combined version, it was on the other two versions with single manipulations (versions 2 and 3). The best average was actually showing a Matthew-effect, where the combined version showed the opposite.
The Matthew-effect
Yes, it is likely that measures aiming to help the deprived are actually the most beneficiary for the ones that aren’t. For instance: educators set up a program to help poor readers in school and who benefits the most? The good readers, thus increasing the gap between poor and good readers. Or, a program set up to help low income households with financial measures, ends up being used mostly by households not in the most need of that help (Deleeck, Huybrechs & Cantillon, 1983; Pfost, Hattie, Dörfler & Artelt, 2014, Rigney, 2010).
This is how the parable of the talents is interpreted by famous sociologist Merton (1968) when he described the chance of academics being published and cited. This phenomenon has been described many times in many fields. But what does that mean for us, communication experts and lawyers? Does the Matthew-effect actually occur in our field? Alas, yes.
Davis e.a. (1996) describes how an easy to read intervention can increase the gap between poor and good readers, while the intervention wants to close that gap and not widen it. Ben Shahar & Schneider (2011) linked the unequal distribution of reading proficiency to how well people could make informed decisions with disclosing documents.
My own research showed the best results on the court summons with either an enhanced structure or style, but only because the higher literate performed so much better. And there is an abundance of reading research where we simply don’t know, because reading proficiency was not tested. How can we deal with this?
Solutions to avoid the Matthew-effect
We can try, measure and see what works and what doesn’t in order to create a reversed Matthew-effect, a Martin-effect, named after the Patron Saint of the city of Utrecht (Sikkema, Lentz & Pander Maat, 2018). It is interesting to know that Martin is not just the Patron Saint of the city of Utrecht, where we live and work, but also of the poor. My own project revealed a reversed Matthew-effect in reading success of the court summons as long as both style and structure revisions were combined.
Two other review studies (Garcia Retamero & Cokely, 2017; Houts, Doak, Doak & Coscalzo, 2006) showed good results with visuals, better document design, plain language and legal design thinking. This is a new perspective for the legal (communication) professionals and their toolkit to address, analyze and communicate about law. There are also worldwide initiatives to improve access to justice with non- verbal interventions, such as Hiil Innovating justice (Netherlands) and Stanford Law Legal design lab (United-States). But there is a screaming shortage of academic, independent evaluations of what works for whom.
We also might focus on legal decisions as a result of shared decision making: with visuals, document design and plain language we can support comprehension by the social network of the reader who doesn’t understand and indirectly improve his decision making. But we must acknowledge that it’s the quality of the decision that is most important, not the quality of the document.
Finally, in our communication for the end users of our work, we have an obligation to explore other instruments to support fit decisions for lower literate readers: video or animation or even personal contact. Especially complicated financial, legal and medical decision making is too important to be solely based on written information, however plain that might be.
References
Ben-Shahar, O. & Schneider, C. E. (2011). The failure of mandated disclosure. University of Pennsylvania Law Review, 647-749. DOI: 10.2139/ssrn.1567284
Berger-Walliser, G., Barton, T. D., & Haapio, H. (2017). From visualization to legal design: A collaborative and creative process. American Business Law Journal, 54(2), 347-392.
Berkman, N. D., Sheridan, S. L., Donahue, K. E., Halpern, D. J., & Crotty, K. (2011). Low health literacy and health outcomes: an updated systematic review. Annals of internal medicine, 155(2), 97-107.
Deleeck, H., & Huybrechts, J. B. Cantillon (1983). Het Mattheüseffect. De ongelijke verdeling van de sociale overheidsuitgaven in België. Antwerpen/Deurne: Kluwer
Garcia-Retamero, R., & Cokely, E. T. (2017). Designing visual aids that promote risk literacy: a systematic review of health research and evidence-based design heuristics. Human factors, 59(4), 582-627.
Houts, P. S., Doak, C. C., Doak, L. G., & Loscalzo, M. J. (2006). The role of pictures in improving health communication: a review of research on attention, comprehension, recall, and adherence. Patient education and counseling, 61(2), 173-190.
Kintsch, W. (1998). Comprehension: A paradigm for cognition. Cambridge: Cambridge University Press.
Kirsch, I. S., Jungeblut, A., & Mosenthal, P. B. (1998). The measurement of adult literacy. Adult
literacy in OECD countries: Technical report on the first international adult literacy survey.
Washington, DC: US Department of Education, National Center for Education Statistics.
Kirsch, I. S. (2001). The international adult literacy survey (IALS): Understanding what was measured. ETS Research Report Series, 2001(2), i-61.
Kirsch, I. S. (2005). Prose literacy, document literacy and quantitative literacy: understanding what was measured in IALS and ALL. International Adult Literacy Survey. Measuring Adult Literacy and Life Skills: New Frameworks for Assessment, 88-136.
Lentz, L.R., Nell, L, Pander Maat, H.P. (2017). Begrijpelijkheid van pensioencommunicatie: effecten van wetgeving, geletterdheid en revisies. Tijdschrift voor Taalbeheersing, 39(3), 191- 208. DOI: 105.5117/TVT2017.2.Lent
Mechanic, D. (2002). Disadvantage, inequality, and social policy. Health Affairs, 21(2), 48-59. DOI: 10.1377/hlthaff.21.2.48
Merton, R. K. (1968). The Matthew effect in science. Science, 159(3810), 56-63.
Pfost, M., Hattie, J., Dörfler, T., & Artelt, C. (2014). Individual differences in reading development: A review of 25 years of empirical research on Matthew effects in reading. Review of Educational Research, 84(2), 203-244. DOI: 10.3102/0034654313509492
Rigney, (2010). The Matthew Effect: How Advantage Begets Further Advantage. New York: Columbia University Press Sikkema
Sikkema, T., Lentz, L., & Pander Maat, H. (2018). Naar een betere dagvaarding. Tijdschrift voor Taalbeheersing, 40(3), 327-356. DOI: 10.5117/TVT2018.3.003.SIKK
