“A scientist must be like a child. If he sees a thing, he must say that he sees it, whether it was what he thought he was going to see or not. See first, think later, then test. But always see first. Otherwise you will only see what you were expecting.” Douglas Adams, So Long, and Thanks for All the Fish
The Scientific Method, A Short Primer:
Children will often apply scientific method: they look without bias and then report with unflinching truthfulness! As we shall soon see, understanding the minimum requirements of scientific method is of great help for all of us⏤at any age⏤when identifying which theories may have scientific support. Many people consider a theory “good” if it is backed by science. How do we identify if a theory is actually backed by science? The answer is in knowing the basic requirements of scientific method. To adhere to scientific method, a theory must fulfill three criteria. First, its claims must be testable; second, its explanations must be difficult to vary; and finally, its test results must be truthfully reported.
To identify if a theory meets the basic requirements of scientific method, we first organize a theory’s parts into the form “Claim because Explanation.” For example, if the claim is “the ice will break when you walk on it,” and if the explanation is “the ice is too thin,” then the “Claim because Explanation” form is, “The ice will break when you walk on it because the ice is too thin.” In order to evaluate whether a theory meets the basic requirements of scientific method, the theory must be put into this “Claim because Explanation” form.
Let us look closer at the “Claim” part of a theory. A scientific claim must be constructed to meet the following conditions: the claim must be measurable(1) and the claim must be disprovable(2). In other words, it must be testable. The claim in our example, “The ice will break when you walk on it,” is both measurable and disprovable. The ice breaking or not breaking is measurable, so the claim meets the measurability requirement, and if the ice does not break when walked upon, the claim would be disproved, which meets the requirement that it is possible to disprove the claim. Since the claim exhibits both measurability and disprovability, this claim meets the first requirement of scientific method, testability.
Before we look at the “Explanation” part of a theory, let us investigate an example of a claim that sounds scientific at first, but turns out not to meet the testability requirement of scientific method. Is the claim, “A person cannot live forever,” a scientific claim? To answer, we check whether or not it is measurable. We certainly can measure (observe) if the person is alive or dead. Is the claim disprovable? To disprove the claim, we would need the possibility that someone has lived forever. Since that is not possible, the claim is not disprovable; it does not meet the testability requirement. We note that it is highly reasonable to say, “A person cannot live forever.” We are just pointing out that the claim is not constructed in such a way as to meet the testability requirement, and thus is not useful when using the scientific method.
Now that we understand how to check the “Claim” part of a theory, we can move on to the “Explanation” part. To meet the basic requirements of scientific method, an explanation must be difficult to vary. This is not as tricky as one may think. A scientific explanation is one that cannot be easily varied or changed as a result of the collected measurements and observations.(3) To check if an explanation is easy to vary, use the following repeatable sequence: consider possible test results that would disprove the claim, then check whether a slight modification to the explanation can disrupt this disproval. If it is easy to disrupt the disproval with a slight modification, the explanation is highly variable. If it is difficult to disrupt the disproval, then the explanation is difficult to vary. This low variability of the explanation is a requirement when using scientific method. In our example, “The ice will break when you walk on it because the ice is too thin,” if the ice does break during the test, then there is no need to vary the explanation. The first and second requirements of scientific method are met. If the ice does not break, disproving the claim, it will be difficult salvage the theory by varying the explanation that the ice was too thin. In the disproved case, the ice did not break; the basic requirements of scientific method are still met. Nevertheless, the claim and explanation are simply wrong, and the theory should be abandoned.
To recap, we now know that a scientific theory’s parts should be organized into the “Claim because Explanation” format. We also now know that the “Claim” must be testable (measurable and disprovable), and the “Explanation” must be difficult to vary. The third basic requirement of scientific method is that test results be reported truthfully. Here we must address the fact that scientists are also human. Being human means being prone to human foibles. Scientific method requires truthful reporting of the facts, yet humans are sometimes not truthful. Scientific reporting must also resist taking sides or having bias. It has been said, “Truth doesn’t have a side.”(4) Humans, including scientists who are reporting results, sometimes make misleading claims, lie by omission, and just outright lie. When this happens, it becomes difficult or impossible to verify results. If the reporting of results is not truthful, a basic requirement of scientific method is not met. Simply put, the scientific method requires the veracity of results. This is why the scientists rely on a public exchange of information, so others can independently confirm or dispute the publicly reported results.
Knowing the basic requirements for scientific method will be of great help when evaluating and sorting the many claims and explanations that are offered. In summary, scientific method requires the theory’s claims to have high testability, explanations to have low variability, and results to have honest veracity. Now when someone exclaims with excess enthusiasm that their theory is scientifically “proven,” we need not be disadvantaged in our response.
(1) Francis Bacon was born on January 22, 1561 in London, England. Bacon took up Aristotelian ideas, arguing for an empirical, inductive approach. biography.com
(2) Karl Popper (July 28, 1902-September 17, 1994) defines empirical falsification as follows: a theory in the empirical sciences can never be proven, but it can be falsified, meaning that it can and should be scrutinized by decisive experiments. wikipedia.com
(3) David Deutsch (2009), “A new way to explain explanation”
(4) Bennet Omalu (2017), “Truth Doesn’t Have a Side: My Alarming Discovery about the Danger of Contact Sports”
We are careful to define the terms we use in our data analyses, in order to be correctly understood. The term ‘Rider’ or ‘Motorcyclist’ are not used, because they are ambiguous.
We use the following terms:
Motorcycles1: Mopeds, two-wheeled motorcycles2, three-wheeled motorcycles, off-road motorcycles, scooters, mini bikes, and pocket bikes are all motorcycles.
Our definition for “Motorcycles” follows the NHTSA definition for motorcycles.
Click here for the page of the FARS Coding Manual that explains the different Body Types for Motorcycles. Note that Vehicle Body Type (80) Motorcycle means Two-Wheeled-Street Motorcycles2.
Please see this table of the FARS Analysis Manual that explains how the different body type elements are grouped for motorcycles. Here is where NHTSA defines motorcycles as Vehicle Body Types (80-89). There is a subtlety here. On the table, motorcycles (plural with an s) is defined as body types 80-89. Then body type 80 is defined as motorcycle (no s).
1NMI definition of terms accept the realities of fatality data collection, especially as performed by investigative (law enforcement) officers at crash scenes. Fatality data collection has a long history and vehicles have evolved greatly since the first fatality reports. We carefully track Vehicle Body Type in our fatality data. We acknowledge that training and licensing may require terms and definitions that may reasonably vary from what is identified as Vehicle Body Type coded from law enforcement crash reports.
2Two-Wheeled Street Motorcycles are what has historical been called “motorcycles.” Crashed Two-Wheeled Street Motorcycles in the NHTSA Body Type Code are coded, 80 Motorcycle. Sidecars? What about motorcycles equipped with sidecars (sidecar rigs)? Motorcycles equipped with a side car are included in the definition of (two-wheeled) motorcycles (FARS Body Type 80) when discussing data. We acknowledge that training curricula and licensing may have other useful and necessary definitions for motorcycles. Motorcycles with sidecars are purposefully included in NHTSA’s and our two-wheeled motorcycle definition and trikes/spyders are purposefully included in three-wheeled motorcycle definition due to data collection realities. Historically, motorcycle crashes began with both types of crashes, motorcycles and motorcycles with sidecars attached. Crashes began to occur and data tracked. Since motorcycle and motorcycle with sidecars fatalities began simultaneously, they are forever coupled in the data. We strongly acknowledge motorcycle sidecar driver training and licensing as separate from motorcycle driver training and licensing. The dynamics and techniques for controlling a sidecar rig are different than those of controlling a motorcycle without a sidecar. Regarding licensure, states currently license/endorse sidecar rigs differently, some states use Class C, some use Class M, and some have a special class of license endorsement, possibly for pragmatic reasons.
Motorcyclists are persons riding upon motorcycles. A motorcycle is a type of motor vehicle. Motorcyclists include both passengers and operators (drivers) of motorcycles.
Occupants are persons riding within motor vehicles. Occupant counts do not include motorcyclist counts. Occupants include both passengers and drivers (operators) of vehicles. An Occupant-Vehicle is a motor vehicle.
Person-Miles-Traveled equals Motorcyclist-Miles-Traveled plus Occupant-Person-Miles-Traveled.
Vehicle-Miles-Traveled equals Motorcycle-Miles-Traveled plus Occupant-Vehicle-Miles-Traveled.
Two people on a motorcycle traveling 10 miles equals 20 motorcyclists-miles-traveled.
Five people in a minivan travelling 10 miles equals 50 occupant-vehicle-miles-traveled.
Persons carried within or on a motor vehicle in transport: Driver, Passenger, Occupant.
We use the definitions of “Person Type” provided from NHTSA and the FARS coding manual.
Driver of vehicle in transport: the person carried within or upon a vehicle, who is operating or controlling the vehicle.
Passenger of vehicle in transport: a person carried within or upon a vehicle, who is not operating the vehicle.
Unknown Occupant of vehicle in transport: is used when it cannot be determined if the person was the driver or passenger, but it is known that the person was an occupant of a motor vehicle in-transport.
Total Occupants of vehicles in transport: This is the sum all drivers and passengers. This sum includes “Unknown Occupants,” because even if an unknown occupant is in a crash, the unknown occupant was either a driver or passenger and thus would be included in the sum of drivers and passengers.
Vehicle Body Types: Vehicle Body Types Motorcycles and Vehicle Body Types Passenger Vehicles.
Table of the FARS Analysis Manual that explains how the different body types elements are classified.
FARS Coding Manual that explains the different Vehicle Body Types for Motorcycles.
Crash: First we determine the Vehicle Body Types, for example “Motorcycles.” With the definition established, it can be stated that, “A motorcycle crash is a crash that involves one or more motorcycles.”
Danger: the chance of bodily harm to a person or persons.
Fatality: motorist or non-motorist involved in a crash with a motor vehicle traveling on a traffic way customarily open to the public, and must have resulted in the death of a motorist or a non-motorist within 30 days of the crash. Ref: Fatality Analysis Reporting System data base.
HDF – Helmeted Driver Fatalities
Killed: fatality of a person through unnatural causes, such as a motor vehicle crash.
LDF – Licensed Driver Fatalities
LMDF – Licensed Motorcycle Driver Fatalities
LPVDF – Licensed Passenger Vehicle Driver Fatalities
Morbidity: a measure of gruesome or grisly injury to a person, whether or not fatal.
Rate: a calculation of the relative frequency of an action, compared to a base such as population or exposure time.
Society: the general population of people in a given geographical or political area.
TDF – Total Driver Fatalities
TMDF – Total Motorcycle Driver Fatalities
TPVDF – Total Passenger Vehicle Driver Fatalities
Transfer Momentum: Momentum is a vector quantity that can be transferred from one body to another.
Transform Energy: Energy is a scaler quantity that can be transformed into different forms, such as Kinetic, Potential, Chemical, Mechanical, Electrical and other types of energies.
We use crash, slide, and collision this way: Slides, collisions, and combination of slides and collisions are all crashes. If the slide is very short in time and distance, we use the term collision. If the collision happens for a long time and over some distance, we use the term slide.
VMT – Vehicle Miles Traveled
VMT data should not be used for historical comparisons because the statistical methodology for both VMT and vehicle registrations was changed over the years. We are confident in using the VMT data starting with 2012. We believe that VMT is a superior exposure measure to vehicle registrations, so we downplay current registration data.
For more regarding the warnings for not using registrations from historical comparisons, please see this link:
NHTSA’s WARNING to not use registrations for historical comparisons
Caveat: Just because a ratio is mathematically correct does not make it scientific or meaningful. In addition to the ratios of the well defined fatality counts listed above (for example, the ratio LMDF/MDF = percent licensed), we use population (Pop) with count “All Fatalities” and Vehicle Miles Traveled (VMT) with count “Driver Fatalities.” These ratios are scientifically meaningful.
We are able to access raw data from NHTSA’s Fatal Accident Reporting System (FARS). We consider this fatality data very reliable. Although injuries may go unreported, fatalities occurring on public roadways are reliably reported. Although we typically depict fatality rates based on “fatalities per year”, there are other ways to provide additional insight into motorcycle fatalities. For example, to better compare the danger of motorcycling to other vehicles, we can ratio All Fatalities/Motorcycle (AFMC) and All Fatalities/Passenger Vehicle (AFPV), comparing the annual relative dangers of the two classes of vehicles.
Some additional meaningful fatality rates are:
Pop: Fatalities per general Population for a given area. Pop can be used to measure the danger of specific vehicles to society (Societal Danger). We have high confidence in the accuracy of the population count reported by the U.S. Census Bureau because the population count is very exhaustive, and used throughout the government and society for many purposes.
We use AFMC/Pop extensively to MODEL Societal Danger. This numerator/denominator combination is meaningful. All fatalities in the crash were members of the population. Governments should use the AFMC/Pop to track, and make adjustments to their safety programs, so as to minimize this rate. Additionally, AFMC/Pop rates gives us a way to make meaningful comparisons in time for a given geographical location as well as comparing different geographical locations during the same time period. For example, “Compare the 1990s fatality rate to the 2000s in California” and, “Compare the fatality rate of California to New York for 2005.”
VMT: Fatalities per Vehicle Miles Traveled. VMT can be used to measure the exposure to danger of a specific category of driver (Driver Danger). VMT is very important data and much improvement has been made on techniques to determine accurate numbers. This is because certain taxes will be based on VMT in the near future.
VMT is an excellent measure of exposure to danger. Whatever the danger, the more miles traveled, the more time of exposure to the danger for the driver.
We use TDF/VMT to model Driver Danger.
We use the Highway Statistics books published annually by the US DOT Federal Highway Administration to obtain VMT.
For example, here is a link to the 2013 VM-1 table.
“Counting Cars” and checking VMT: Practicing counting passenger vehicles can help you gain insight to the vast difference between Motorcycle VMT and Passenger Vehicle VMT. While observing traffic, count the number of passenger vehicles you see. You can use particular stretches of road, or simple try to count for five minutes from any observation point. It is exciting to learn how to do this!
Vehicle Registrations (Reg or VR): Fatalities per total vehicles registered (VR). This has difficulties in accuracy like VMT. VR is an exposure measure, the more registrations, the more miles will be driven. VR can be used to support VMT claims. Vehicle Registrations (VR or Reg) have been used in the past to measure fatality rates, such as Fatalities per total vehicles registered. VR is theoretically a measure of exposure, assuming that more registered vehicles indicates more miles driven. VR can provide a useful measure of danger over time for a specific type of vehicle. For instance, it is reasonable to compare the fatality rate of motorcycles from one year to another, since the use of motorcycles can be assumed to not change dramatically from year to year. Likewise, it is reasonable to compare the fatality rate of passenger cars in the same area from one year to another. The limitation of using VR for rates is attempting to compare different types of vehicles that have obvious differences in use. For example it is unreasonable to compare the fatality rate of classic cars to taxicabs, or compare the fatality rate of passenger cars to the fatality rate of motorcycles. NMI cautions that any comparison of danger between different types of vehicles based solely on registration numbers is likely to be inaccurate and misleading.
Fatalities per new vehicle units sold (NVU). Industry exerts effort for accuracy with these numbers. NVU can be used to support VR claims.
NVU and VR can be used to determine the popularity of a specific type of vehicle. Generally, NVU is provided by a manufacturers’ organization, the Motorcycle Industry Council; while VR is provided to NHTSA by state licensing departments.
Avoiding Confirmation Bias
“We are not taking a side on an issue. We share information that is factual. The truth does not have sides.” NMI
Confirmation bias is the seeking out of information that confirms the statement or belief, and discounts information that conflicts with the statement or belief. We are watchful to prevent confirmation bias from eroding our analysis of the data. Note that “bad” or “easily varied” theories are strongly susceptible to confirmation bias. This is why our explanations are not easily varied.
We have found the following terms useful for monitoring confirmation bias:
The quality of being factual.
Truthy or Truthiness:
The quality of preferring facts one wishes to be true, rather than facts known to be true.
“Just because it sounds right [has good truthiness] does not mean it is right [truth].”
We carefully use the 4-Box scientific method to check for “Truthiness” of a statement. This 4-Box method is the simplest test for checking whether a statement belongs in the “Old Paradigm – Motorcycles can be safe and can be enjoyable” or the “New Paradigm – Motorcycles are dangerous and can be enjoyable.” (Examples will be posted here in the future.)
Raw Data Comment
National Motorcycle Training Institute always posts the raw data, with reference to where the data was obtained. This is so anyone else may go to the source and verify or our analysis, or provide conflicting analysis. We encourage you to apply the scientific method to our work. For fatality counts, we use National Highway Transportation Safety Administration (NHTSA) data from the Fatal Accident Reporting System (FARS) database. We have a high degree of confidence in NHTSA’s fatality counts. We are careful to monitor the published changes in the method of counting through the years of the data base’s life. Starting in 1991 the data we look at in this database became well defined. We caution readers to understand the changes made to the data base regarding FARS data prior to 1991 before splicing any data from FARS from years prior to 1991.
Assessing the Danger of Motorcycling to Society
We use the rate and changes in rate of Total Motorcycle All Fatalities divided by Population, TMAF/Pop, to assess Societal Dangers.
We compare the fatality rates and changes in rates of Motorcycle Related Fatalities with Passenger Vehicle Related Fatalities to measure relative danger.
Assessing State Motorcycle Licensing Programs
We use the rate and changes in rate of Licensed Motorcycle Driver Fatalities per Vehicle Miles Traveled LMDF/VMT, to measure the danger for licensed drivers. We use the percent of Licensed Motorcycle Driver Fatalities/Total Motorcycle Driver Fatalities, LMDF/TMDF to measure the effectiveness of licensing programs. If effective, licensed drivers should be under represented and the unlicensed drivers should be over represented in the fatality rates.
Assessing Effectiveness of Motorcycle Safety Programs
We compare the results of all analyses above when assessing the effectiveness of motorcycle safety programs.