Frequently Asked Questions

What is the "no significant difference" phenomenon?

What does it mean for a research study to show "no significant difference"?

What do the "no significant difference" findings mean for distance education?

Some articles show a "significant difference". What is the best way to interpret these articles relative to the "no significant difference" phenomenon?

Do the "significant difference" articles tend to show greater achievement in face-to-face or in mediated instruction?

Has anyone conducted "meta-analyses" (studies which summarize the results of several prior research studies into a single estimate of their combined result) of the findings in these articles, and where might I find such resources?

How is this website different from Thomas Russell's book, "The No Significant Difference Phenomenon"?

What information can I expect to find on this website?

How do I submit an article for inclusion in this website?

How do I purchase a copy of Thomas Russell's book, "The No Significant Difference Phenomenon?"



What is the "no significant difference" phenomenon?

In the early 1900's, as correspondence courses came into vogue, there was a question that weighed on the minds of educators: could students learn as well at a distance as they could face to face? As with most controversial issues, there were proponents on both sides: traditionalists held face to face as the gold standard, while innovators held that distance courses could deliver equivalent, if not improved, learning outcomes. Both sides were eager to gather evidence to substantiate their claims - and thus began the movement in media comparison studies (MCS) in education. In these studies, researchers looked to compare student outcomes for two courses that were delivered through two different methods, thereby identifying the "superior" method for teaching effectiveness.

The "No Significant Difference" phenomenon refers to a body of literature consisting of a particular type of MCS - those comparing student outcomes between face to face and distance delivery courses. This body of literature was originally compiled by Thomas Russell in his book, "The No Significant Difference Phenomenon: A Comparative Research Annotated Bibliography on Technology for Distance Education" (IDECC) to answer the following question: Does taking a course via distance education lower a student's chances for success as compared to the same student taking the same course in a face-to-face format?

Mr. Russell collected research studies addressing this question from as far back as 1928. The studies included in his collection involve a wide array of distance delivery modes including correspondence (printed materials sent out to students), radio, television, video, and online. Mr. Russell found that an overwhelming number of studies showed that when the course materials and teaching methodology were held constant, there were no significant differences (NSD) between student outcomes in a distance delivery course as compared to a face to face course. In other words, student outcomes in distance delivery courses were neither worse nor better than those in face to face courses. Mr. Russell referred to this collection of results as the "No Significant Difference Phenomenon", thus coining the now-common identifier phrase for this body of literature.

For a short introduction to research study design and the significance of "no significant difference" findings, please refer to the question, "What does it mean for a research study to show 'no significant difference'?", below.



What does it mean for a research study to show "no significant difference"?

Though it may seem tempting to discard findings that show "no significant difference", it is important to remember that "no significant difference" does not mean "unimportant". No significant difference (NSD) findings are as important as significant difference (SD) findings. However, instead of showing us that the two items being compared are different, and possibly how they are different, they show us that the two items being compared are NOT different.

In any experimental research study, researchers begin with a question. From this question, researchers derive a null hypothesis - the prediction that a change in the independent variable (the element of the study that is altered in the experimental condition) will have no effect on the dependent variable (the element of the study that is observed for changes upon application of the experimental condition). In media comparison studies (MCS), the independent variable is the mode of delivery, and the dependent variable is student outcome. If the MCS question is, "What effect, if any, does the mode of delivery of education have on student outcomes?", the corresponding null hypothesis would be, "The mode of delivery of education has no effect on student outcomes."

Researchers design experiments to test the prediction of their null hypothesis. Experimental results either support, or do not support, the null hypothesis. In MCS, researchers "experiment" by changing the independent variable - mode of education delivery - and observing the effects on the dependent variable - student outcomes. The great majority of results of MCS research demonstrate No Significant Difference (NSD) in student outcomes across different modes of education delivery. Therefore, these results support the null hypothesis that "The mode of delivery of education has no effect on student outcomes."

In any research study, there are many variables that come into play. To isolate the effects of a single variable, researchers try to control the other variables in the experiment. In MCS, some variables other than delivery mode that affect student outcomes might include curriculum materials, instructional method, and student learning preferences. Thus, some important considerations for MCS include the following:

  • Were the two courses being compared truly comparable? That is, were the two courses taught using the same course materials, teaching methods, faculty? The effects resulting from changes in any of these variables between two "comparable" courses could very well outweigh effects of delivery mode.

  • Did the faculty who taught the distance courses have facility with the media used to deliver the course? A faculty's lack of facility with the chosen distance technology, whether it be video, television, or internet, can seriously affect the quality of a distance-delivered course, thereby affecting study results.

  • Were the two student populations in the study truly comparable? If students in one delivery mode tended to be generally better prepared, more motivated, or more interested in the subject matter than those experiencing the course in the alternate delivery mode, study results would be affected to favor those students.

One of the common criticisms of MCS is that they fail to control some, if not all, of these external variables. Although this is troubling, it is also a general problem across educational research - we are, after all, working with real students in the real world, not controlled experimental conditions in a laboratory. However, we must remember to use caution when generalizing across MCS results. While the NSD finding may be prevalent enough across a large enough pool of MCS research results to be considered a fair conclusion despite the lack of controlled variables, other findings limited to certain studies may not be as widely applicable. Therefore, a good practice in performing meta-analyses (studies which summarize the results of several prior research studies into a single estimate of their combined result) of the MCS research might be to first sort for collections of MCS that control for similar sets of variables, and then analyze the findings from those collections.



What do the "no significant difference" findings mean for distance education?

Quoting Richard Clark from the introduction to the fifth edition of Thomas Russell's book,

"The NSD media finding in studies where adequate learning occurs can be interpreted to mean that compared treatments are equal in their impact on learning."

In other words, if the amount of learning produced by different media is similar, then each of those media are equally valuable for learning. As long as the message remains the same, it doesn't matter what media are used to deliver that message - the effect for learning will also remain the same.

The "no significant difference" literature in media in education can be further interpreted in two ways. First, the NSD findings demonstrate that delivering education at a distance does no harm. That is, students who opt for distance delivery are not immediately put into a compromised position simply because they are not receiving their education in a "face to face" format. Second, the NSD findings indicate that simply converting a face to face course into a technology-mediated distance delivery course does not help improve student outcomes. To achieve gains in student outcomes, we must do more than just deliver the course through a different medium. Quoting Mr. Russell from the introduction to his book,

"These studies tell me that there is nothing inherent in the technologies that elicits improvements in learning. Having said that, let me reassure you that difference in outcomes can be made more positive by adapting the content to the technology. That is, in going through the process of redesigning a course to adapt the content to the technology, it can be improved."

This idea is reflected in the history of the No Significant Difference literature. Over the last 50 years, the question for media comparison studies (MCS) has evolved from, "Can students learn at a distance?" to "What is the effect of distance delivery on student outcomes?" Over the years, especially since the internet revolution, the conviction that distance delivery is necessarily inferior to face to face instruction has faded a bit. As we accept that it is not the technology itself, but the application of technology, that has the potential to affect learning, it is our hope that future research will strive to identify the instructional methods that best utilize technology attributes to improve student outcomes.



Some articles show a "significant difference". What is the best way to interpret these articles relative to the "no significant difference" phenomenon?

Both the No Significant Difference (NSD) website and Mr. Russell's book are meant to provide a historical perspective on media comparison study (MCS) research. By amassing MCS research studies from the early 1900's onward, these resources provide us with a picture of how MCS research evolved over time.

In the early days of correspondence courses, the original question addressed in MCS research was "Does delivering courses at a distance hurt student outcomes?" Later, upon introduction of radio broadcast courses, then televised courses, then video education, then online education, the question became "Does delivering courses through technology hurt student outcomes?" Thus, the null hypothesis (see question 2 for a definition) that early MCS research studies sought to examine was "Distance delivery (or, as appropriate, technology-mediated delivery) of courses does not hurt student outcomes." In these studies, the best possible results for distance education were those that supported the null hypothesis; that is, they demonstrated that distance education "does no harm." And, from the results collected in Mr. Russell's book and website, it appeared that different modes of delivery for the same materials actually did not hurt, so the great majority of these early studies show "no significant difference" (NSD) between student outcomes in face-to-face courses versus those delivered at a distance.

In the 1980's, with the accrual of several generations of NSD findings, MCS research began to shift focus. Researchers began asking, "Can we improve learning by using technology tools in education?" This led to studies that examined ways in which technology might actually help improve student outcomes. In these studies, the null hypothesis became "Use of technology tools in education does not improve student outcomes."

As the question changed, so did the results. MCS research studies showing "significant differences" (SD) in student outcomes started appearing in the literature, and most showed improvement with technology; that is, they tended not to support the null hypothesis. In many of these studies, courses were redesigned to take advantage of the unique aspects afforded by technology - asynchronous discussions, archives, links to resources. Mr. Russell suggests that by this very effort, tech-mediated distance courses were improved. Quoting Mr. Russell from the introduction to his book,

"These studies tell me that there is nothing inherent in the technologies that elicits improvements in learning. Having said that, let me reassure you that difference in outcomes can be made more positive by adapting the content to the technology. That is, in going through the process of redesigning a course to adapt the content to the technology, it can be improved."



Do the "significant difference" articles tend to show greater achievement in face-to-face or in mediated instruction?

In the great majority of studies compiled for this website, the "significant difference" articles show greater achievement in technology-mediated instruction. However, it is also important to note that in most of these cases, courses were adapted to the technology being utilized for mediated delivery. It is likely that this very adaptation created a course that allowed students to achieve higher outcomes, rather than the technology itself resulting in the higher outcome. Quoting Mr. Russell from the introduction to his book,

"These (NSD) studies tell me that there is nothing inherent in the technologies that elicits improvements in learning. Having said that, let me reassure you that difference in outcomes can be made more positive by adapting the content to the technology. That is, in going through the process of redesigning a course to adapt the content to the technology, it can be improved."



Has anyone conducted "meta-analyses" (studies which summarize the results of several prior research studies into a single estimate of their combined result) of the findings in these articles, and where might I find such resources?

In June 2009, the U.S. Department of Education released the report: “Evaluation of Evidence=Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies”. After reviewing more than 1,000 empirical studies of online learning, the report states: “The meta-analysis found that, on average, students in online learning conditions performed better than those receiving face-to-face instruction.” The online newspaper Inside Higher Ed wrote an article summarizing the report.



How is this website different from Thomas Russell's book, "The No Significant Difference Phenomenon"?

Mr. Thomas L. Russell's book, "The No Significant Difference Phenomenon: A Comparative Research Annotated Bibliography on Technology for Distance Education" (2001, IDECC, fifth edition), is a fully indexed, comprehensive research bibliography of 355 research reports, summaries and papers that document no significant difference in student outcomes based on the mode of education delivery (face to face or at a distance). The book also includes a foreword by Dr. Richard E. Clark, one of world's most cited researchers in the area of media research design. Previous editions of the book were provided electronically; the fifth edition is the first to be made available in print from IDECC (The International Distance Education Certification Center).

The No Significant Difference (NSD) website has been designed to serve as a companion piece to the book. The primary purpose of the NSD website is to expand on the offerings from the book by providing access to appropriate studies published or discovered after the release of the book. In addition to studies that document no significant difference (NSD), the website includes studies which do document significant differences in student outcomes based on the mode of education delivery. The significant difference (SD) entries on the website are further classified into three categories:

  • better results through technology - improvement in outcomes when curriculum is delivered at a distance;

  • better results in the classroom - improvement in outcomes when curriculum is delivered face to face; or

  • mixed results - some variables indicate improvement when curriculum is delivered at a distance, while others indicate improvement when curriculum is delivered face-to-face.

The website is designed to function as an ever-growing repository of comparative media studies in education research. Though the current website is not meant to be a comprehensive collection of such studies, we would like to move towards that goal. Thus, both NSD and SD studies are constantly being solicited for inclusion in the website. New entries can be nominated through a form on the website.



What information can I expect to find on this website?

The primary purpose of the No Significant Difference (NSD) website is to expand on the offerings from Tom Russell 's book, "The No Significant Difference Phenomenon," by providing access to appropriate studies published or discovered after the release of his book. In addition to studies that document no significant difference (NSD), the website also includes studies which document significant differences (SD) in student outcomes based on the mode of education delivery (distance or face-to-face).

The NSD website was originally designed to serve as a companion piece to Mr. Russell's book. As such, the website contained:

  • a short biography of Mr. Russell,

  • a link to purchase Mr. Russell's book,

  • a selection of entries from Mr. Russell's book, and

  • new entries of pertinent studies published or discovered after the release of the book.

The website was redesigned in 2005 to simplify navigation and provide additional features. In addition to the components of the original website listed above, the redesigned website:

  • Includes a top navigation bar. The redesigned website adds a persistent top navigation bar that makes it easier for users to both explore various parts of the NSD website and to return to any part of the site at any time.

  • Presents supplemental materials pertinent to the site. The redesigned website adds an "About" page and a "Frequently Asked Questions" (FAQ) page to help new users familiarize themselves with the No Significant Difference Phenomenon, Mr. Russell's book, and the website.

  • Combines NSD and SD entries in a single database. In the previous version of the website, No Significant Difference entries were kept in a separate database from the Significant Difference entries; thus searches could be conducted on either NSD entries or SD entries, but not both at once. In the current site, NSD and SD entries are clearly labeled, but contained within a single database. When combined with the current site's new search functions, the single database offers users the best of both worlds. Now, users may:

    • perform a single search without specifying a type of entry (SD or NSD), thereby identifying both NSD and SD entries that match a set of search parameters; or,

    • specify a single type of entry (SD or NSD), thereby narrowing their search results to either only SD or only NSD entries.

  • Partitions SD entries for greater clarity . In the original website, entries were classified either as a No Significant Difference (NSD) entry or a Significant Difference (SD) entry. To improve the informative quality of the entries, the Significant Difference (SD) studies on the redesigned website have been further classified into one of three categories:

    • better results through technology - improvement in outcomes when curriculum is delivered at a distance;

    • better results in the classroom - improvement in outcomes when curriculum is delivered face to face; or

    • mixed results - some variables indicate improvement when curriculum is delivered at a distance, while others indicate improvement when curriculum is delivered face-to-face.

  • Adds a search engine. The original website allowed users to quickly identify NSD studies by year through the left navigation bar. The redesigned website retains this feature, but extracts both NSD and SD articles for the selected year. The redesigned website also adds a search engine, so entries can now be searched through a simple Keyword Search or a more detailed Advanced Search. Among other features, the Advanced Search allows users to identify studies by the type of finding (NSD, or one of the SD categories outlined above).

  • Provides a form to nominate new entries. The redesigned website is intended to function as an ever-growing repository of comparative media studies in education research. Though the current website is not meant to be a comprehensive collection of such studies, we would like to move towards that goal. Thus, both NSD and SD studies are constantly being solicited for inclusion in the website. New entries can be nominated through a form on the website. Mr. Russell makes all decisions about inclusion of studies in the website; entries that are approved will be added to the website database and can subsequently be located through the left navigation bar or the search functions.



How do I submit an entry for inclusion in this website?

Both NSD and SD studies are constantly being solicited for inclusion in the website. New entries can be nominated through a form on the website. Mr. Russell makes all decisions about inclusion of studies in the website; entries that are approved will be added to the website database and can subsequently be located through the left navigation bar or the search functions.



How do I purchase a copy of Thomas Russell's book, "The No Significant Difference Phenomenon?"

A copy of the fifth edition of Mr. Russell's book may be purchased through the publisher - IDECC, the International Distance Education Certification Center. The book may be ordered online.