"RIA Window" - a mystery thriller!
Does e-learning improve knowledge retention to 25-60% compared to 8-10% for face-to-face training? We investigate!
Have you ever read these words before? – “The Research Institute of America reports that learning retention rates improve from 8 to 10 percent for face-to-face training to 25 to 60 percent for e-learning.”
Statements like this currently feature on learning solutions and technology industry websites around the world. In this week’s issue of Caveat Emptor we explore the mystery surrounding this hyperbolic advertising claim.
You might be tempted to accept that statement at face value when you see the results of this Google search. Surely these ‘facts’ about learning retention must be correct, given the large number of virtually identical listings revealed here? However there are a number of things about this that aroused our curiosity.
The infographic below was posted by By Danielle Arad on November 2nd, 2016 under the heading ‘Virtual Learning – Everything You Need to Know’. If you click the image, you can see this in its original location.
More lists of lists!
As you can see from Arad’s footnote above, the source cited for this is a list prepared by Christopher Pappas for elearningindustry.com called ‘Top 10 e-Learning Statistics for 2014 You Need To Know.’ The statement below ranks at number 8 in that list:
There are a couple of points worth noting here.
Firstly, Arad saw fit to switch the words ‘Virtual Training’ for the word ‘e-Learning’ used by Pappas when she reproduced the claim in her own post in 2016. It is still unclear why that was.
Secondly, Pappas refers to a ‘recent study conducted by the Research Institute of America’ but does not provide any further information about this. The link highlighted in blue leads to an unrelated article by Matthew Guyan entitled: ‘5 Ways To Reduce Cognitive Load In eLearning’. There is no mention of the ‘Research Institute of America’ in that article. So that proved a dead-end.
Those that read Issue 1 of Caveat Emptor may recall that the name Christopher Pappas cropped up as part of our investigation into ill-founded claims about the level of productivity generated by investments in e-Learning. In that instance our enquiries led us to the current version of a very similar list: ‘Top 20 eLearning Statistics For 2019 You Need To Know’.
As we reported last week, the 2019 list appeared to have been used as a source for many virtually identical but ill-founded marketing claims about productivity. This led us to wonder if a similar proliferation of claims about retention arose as a result of the earlier list compiled in 2014?
Reassured that we were seeking a ‘recent study conducted by the Research Institute of America’ our researchers set out to track the organisation down. This quest took far longer than we ever expected!
Seeking the Research Institute of America!
After much searching and some false trails we established that the Research Institute of America Inc. was a corporate-oriented, economics consultancy with a tax emphasis, based in Fifth Avenue, New York. In the 1980’s it was publishing ‘State and Local Taxes’ (comparison charts), ‘What you should know about your social security now’ (RIA Employee Handbook); The RIA complete analysis of the Tax Equity and Fiscal Responsibility Act of ’82; and other exciting titles.
By 1989 the RIA had become a US legal and tax publisher of some scale. In that year RIA was acquired by Thomson Corporation to augment it’s growing array of financial and tax information services businesses. Research Institute of America (RIA) ultimately became a division of Thomson Tax & Accounting.
The RIA OnPoint System
In the late 1990’s the RIA announced the release of their OnPoint System, a new CD-ROM based program designed to assist those with an interest in tax information. A number of versions of OnPoint were produced. These were specialist products marketed through full page advertisements in magazines for accountancy and tax professionals. Here is an example from the ABA Journal issue of November 1995.
Users needed training to use the RIA OnPoint System so Student Versions of the CD-ROM were produced. These were abbreviated versions of the complete RIA OnPoint System computerized tax research database, designed to teach the user how to conduct tax research utilizing the database. For example, the RIA Onpoint System 4 Student Version Multimedia CD-ROM was issued on July 21, 2000.
WR Hambrecht & Co
As you can see from the information outlined above, the Research Institute of America was not an independent research institution that published academic peer-reviewed studies on learning effectiveness. On the contrary, they were a commercially motivated business forming part of Thomson Corporation. Moreover the organisation had a beneficial interest in selling computer based training and more latterly e-learning, for tax professionals. Their specialism was tax – not educational research.
Today the RIA brand lingers on at Thomson Reuters, Tax & Accounting but the Research Institute of America organisation itself disappeared decades ago. This is why it proved so hard to investigate this story via the Internet!
After a lot of work our researchers finally tracked down a report published by WR Hambrecht & Co in March 2000 entitled ‘Corporate e-Learning – exploring a new frontier‘. The following paragraph on page 6 finally allowed us to place a timeline on the claims associated with the RIA:
Given the date of publication for this report (March 2000) the source information for the highlighted text was probably gathered in the late 1990’s. Footnote 5 below explains where that information actually came from:
The ‘Endnotes’ are annoyingly vague. However this document was not a peer reviewed academic research study. It was simply a white paper written by a group of investment analysts who arguably had an axe to grind in promoting the benefits of e-Learning in the year 2000!
WR Hambrecht & Co made a market in the securities of DigitalThink, Inc. (DTHK), having underwritten of a public offering of securities of DigitalThink, Inc. within the previous three years. WR Hambrecht & Co also held private equity investments in Ninth House Network and Pensare, Inc. All three e-Learning vendors featured prominently in their report.
The forgetting curve?
The WR Hambrecht & Co ‘Endnotes’ above suggest the RIA stated the following information at some point in the 1990’s:
- That 33 minutes after a lecture is completed, students usually retain only 58% of the material covered in the class
- That by the second day, 33% is retained
- That three weeks after the course is completed, only 15% of knowledge is retained.
Many will recognise this as a bastardisation of the ‘Forgetting Curve’, which Hermann Ebbinghaus proposed after he ran a limited, incomplete study on himself from 1880 to 1885. He taught himself nonsense syllables and then tested himself after the initial exposure, recorded how many he remembered, reviewed them again and then repeated the process. The RIA may have done something similar or were simply citing results from another researcher. However it is not scientifically justifiable to draw broader conclusions about rates of forgetting from these kinds of results. As Will Thalheimer eloquently explains in his excellent white paper, everyone forgets things at widely differing rates.
It is unclear why the RIA were taking an interest in this particular subject. It is possible that the RIA was publicly promoting the benefits of self-paced learning as embodied by their Onpoint System 4 Student Version Multimedia CD-ROM in 2000 or shortly prior to that. That would certainly explain why a financial information business was advancing educational effectiveness claims. Given the role of WR Hambrecht & Co within the investment sector it is not unlikely that the two organisations were known to each other.
WR Hambrecht & Co do not quote the RIA as their source for the claim that e-Learning results in superior levels of retention. Confusingly, they reference a book by Jack E. Bowsher, called ‘Revolutionizing Workforce Performance: A Systems Approach to Mastery,’ published in 1998, a publication by D. Peoples called Presentations Plus published in 1992 and finally ‘Training Magazine, 1998′.
It remains unclear what mix of information from these varied 1990’s sources led to the authors to make their statement on page 6. However we have no doubt those sources could be tracked down if necessary.
One or more e-Learning vendors later extracted the wording from Page 6 of the WR Hambrecht & Co paper and simply credited the entire finding to the impressively (but misleadingly) titled Research Institute of America. The legend was born!
The Smoking Gun
There was little point progressing our enquiries back beyond the year 2000. The impression given to prospective buyers when e-learning vendors cite this statistic today is clearly misleading:
- The the study is not ‘recent’ – it is at least 20 years old and predates e-Learning as we know it today.
- The claim was the work of investment analysts gathered from multiple sources from the 1990’s rather than a direct finding from any single research study.
- The organisation cited (the RIA) is not an academic organisation publishing peer-reviewed research but a vendor of technology based training and financial information.
- Most importantly, this hyperbolic claim is unsupported by any of the well-documented peer-reviewed academic studies carried out since 2000!
We therefore encourage vendors to withdraw this claim immediately and not to repeat it.
Our findings from this investigation parallel our findings from last week. In both instances an apparently reputable source is being incorrectly cited for information that prospective buyers cannot rely on. This brings the provenance of many similar advertising claims into question.
In next week’s issue of Caveat Emptor we look at the broader issue of bogus learning effectiveness claims, how to spot them and what to do about it when you do!
Pau
Thank you for this post! I was trying to look for the original study and you saved me a lot of time.
annechien helsdingen
Thank you, too bad that the claim is still repeated: https://www.weforum.org/agenda/2020/04/coronavirus-education-global-covid19-online-digital-learning/
AdrianSnook
Sadly, the Internet is an echo chamber and some ‘facts’ keep reverberating around the world! Many thanks for highlighting this.
Kind regards
Adrian Snook
Kyle
I was also skeptical when I saw this elearning statistic and I did some Googling around to find it mentioned in so many different places but nobody links back to the original study by the Research Institute of America.
It definitely sounded a little too good be true, which means it most likely is!
Thanks for digging into this and revealing the actual truth :=)
AdrianSnook
Thanks for taking the time and trouble to check this out Kyle. It is tempting to simply accept these hyperbolic claims at face value and then repeat them. Verificationis much much harder. Glad we could help out!