Human Factors Testing and Clinical Research

• 

Tim Reeves, CHFP
Founder and Managing Director
Human Factors MD

Abstract: Human factors simulated-use studies provide a valuable tool for evaluating the usability and safety of medical devices, including combination products such as inhalers, injection devices, and transdermal patches. Through human factors studies, researchers evaluate how users and devices perform in low-frequency high-risk situations that may not occur in controlled clinical trials. This article provides an overview of the role of human factors testing in combination product development as well as how human factors studies complement traditional clinical trials in establishing safety and efficiency in drugs and devices.

Introduction

Just as clinical studies are necessary to demonstrate that a drug in a combination product is safe and effective therapeutically, human factors studies are necessary to demonstrate that the device in a combination product can also be used safely and effectively. Clinical studies and human factors studies are both necessary and complementary.

Combination products are becoming more and more ubiquitous, primarily because of the need to offset the cost of healthcare and to require patients to take more responsibility for the delivery of their healthcare, which includes their medications. While combination products may seem simple, they often cause problems for users.

The U.S. Food and Drug Administration (FDA) has published a long list of known issues with combination products in its guidance documents (Table 1). For example, people may not know how to store inhalers, or they may have difficulty opening the packages or priming meter-dose inhalers. People may not use the best inhalation technique, or they may not clean the inhalers.

People may have difficulties with injection devices, such as selecting the wrong dose if there are multiple doses, not ensuring that the drug is injectable, and not checking the appearance of the product. They may inject the product at the wrong site or not remove the cap. They may not know how to activate the injection device, or they may remove it from the injection site prematurely before all of the medication has been delivered.

Known issues with patches include not entirely removing the adhesive, placing one patch on top of another patch, or cutting patches in half to modify the dose. There have been many instances reported to the FDA of children being overdosed because a disposable patch was not disposed of and the child ingested the patch.

Human Factors and Medication Error Prevention

Human factors is a methodology and a body of knowledge that tries to eliminate or reduce the impact of errors related to the use of products such as medical devices. Regulatory agencies such as the FDA are proponents of human factors studies. Human factors can be considered a marriage of psychology and engineering. Engineering covers the design of the product, which in this case is a medical device. Psychology covers understanding the people who may use the device, including their capabilities, limitations, and predispositions. 

Human Factors Studies

The FDA and other regulatory bodies see human factors testing as a way to mitigate use issues with medical devices and to reduce medication errors. Since the early 2000s, it has become increasingly more difficult to obtain approval of medical devices without human factors testing to demonstrate that people can use a device safely and effectively as intended.

The two types of human factors studies, formative and validation, are similar in terms of their approach (see Table 2). After recruiting people who are representative of product users, researchers introduce them to the product in a way that is similar to the way that they might learn about the product. For a commercial product where people will be trained when the product is released, users would receive training in the study. If users will learn about a product by reviewing the information that comes with it, then the study will be run that way.

Tests are conducted in realistic use environments and evaluate realistic use scenarios. Participants use the device while researchers observe them. Human factors studies are largely about observing users and then questioning them about their experience. Researchers might ask questions about what users were doing, why they did things, and how they interpreted various aspects of the device and its use. 

Formative studies are conducted early in the development of a product and are intended to inform development. Results are used iteratively to modify the product and test it until developers are confident that the device will work appropriately and that people will know how to use it.

While the FDA expects device manufacturers to conduct formative studies, it emphasizes validation studies, which are conducted at the end of the development process on the final finished combination product. The device used in a validation study must incorporate the final commercial labeling and packaging designs. Validation studies are post-market simulations of whether people will be able to use the product appropriately after it reaches the market. They are intended to demonstrate that the design supports the safe and effective use of the device when used by actual users in realistic use environments.

Validation studies are almost always simulated-use; noone is administered an actual drug product. As per FDA guidelines, a validation study must include a minimum of 15 people per user group. Typically, these studies have between 15 and 120 people. They are small studies in comparison to major clinical trials.

All of the ways in which people could use the product inappropriately that might lead to harm or ineffective therapy must be identified, and the manufacturer must demonstrate that it has reduced the risk of these issues happening as much as possible.

Researchers collect two types of data – objective performance data and subjective assessment data. They observe people actually interacting with the device (performance) and collect their subjective impressions of what this was like (subjective assessment). Subjective assessments are important. People may have difficulties with things that cannot be identified through observation. For example, a participant may have been tempted to perform some action that would have resulted in a safety error; however, the participant changed his or her mind at the last minute. None of these issues may have been evident to an observer. This is called a “close call.” Another user may have gone ahead and made the safety error without self-correcting.

Interviewing people is necessary to uncover these situations. Researchers also collect subjective assessment data for things that are difficult to observe in a simulated-use study. For example, it is difficult in simulated-use studies to test whether participants know that a product must be refrigerated prior to use. Instead of creating a more elaborate simulation to test whether users understand this requirement, researchers simply ask users where the product should be stored prior to use, and researchers assess the participant’s understanding based on their response.

Human Factors Studies and Clinical Studies

Human factors studies start as a preliminary analysis during the Pre-investigational New Drug application activities. Preliminary analysis involves understanding the users and the use environment as well as doing some initial development of the device. During Phase 1 and Phase 2 clinical trials, formative testing is started, followed by validation testing. Before Phase 3, the FDA expects a finished product that has undergone human factors testing that has shown that users have minimal safety-related issues with the use of the device. The timing of human factors studies in relation to clinical studies is illustrated in Figure 1.

It is possible that knowledge gained from the clinical trial will lead to modifications of the device, its labeling, or its packaging. In this case, another human factors study may be needed for the New Drug Application submission. 

Table 3 outlines differences between human factors validation tests and major clinical studies related to timing, the device, study participants, and sample size. Validation tests are conducted prior to Phase 3 clinical trials. Use of the device is simulated in a human factors study, whereas study participants receive the actual medication in clinical studies. Both validation tests and clinical studies use the final commercial product.

The participants in a clinical trial are patients, whereas the participants in human factors studies may include everyone who is using the device. Participants could be patients; however, they could also be healthcare professionals such as pharmacists who will be dispensing the product to patients. If the product has multiple doses, researchers must ensure that the pharmacists can differentiate the doses in order to dispense the right dose. In this situation, the validation study would involve a mock pharmacy where pharmacists are asked to fill prescriptions in different doses.

Validation studies have been done with physicians to ensure that they understand what they are prescribing or administering. Participants could also be caregivers who will be administering the drug to the patient. While clinical trials could have thousands of participants, human factors studies typically have 45 participants to 120, with 15 participants in each user group.

The main difference between human factors studies and major clinical studies is that clinical trials want to minimize variations that could alter the integrity of the data that are being collected, whereas human factor studies test the variation among users (Table 4). In clinical trials, the data focus on the performance of the drug, so participants are trained in the appropriate administration of the drug. The use environment is tightly controlled and variations in device use are minimized. Human factor studies, on the other hand, study the variation in device use among users. Use errors are the dependent measure in a human factors study, which are used to ensure that when the product is released, people will be able to use it safely and effectively, given variations in users, their experience, their training, and the use environments.

Table 5 outlines differences between human factors validation tests and major clinical studies related to data collection, statistics, and endpoints. Everything in the human factors studies focuses on use-related issues, whereas clinical trials seek to control issues in order to prevent interference with data collection. Data collection in human factors studies focuses on performance of critical tasks: successes, failures, operational difficulties, and close calls. In clinical trials, data collection focuses on the performance of the drug. Human factors studies focus on how people interact with the device and whether they understand it well enough to use it safely.

Unlike clinical trials, there are no statistics in human factors studies, which provide descriptive information and counts of the number of people who made each type of use error. The endpoints are quite different. Human factors studies are qualitative and have no acceptance criteria; instead, these studies focus on how people perform when using a device, and they focus on the root cause of any issues that arise during use. In clinical trials, the endpoint is evaluation of whether the drug is safe and effective.

Conclusion

Clinical studies and human factors studies are complimentary. Regulators require both types of studies. The best drug will not be beneficial if users cannot use it appropriately.

TABLE 1

Known Issues with Combination Products*

• Known use issues with inhalers:
  • Improper storage prior to use
  • Unable to open package
  • Unable to assemble or assembled incorrectly
  • Not priming at all or priming incorrectly
  • Improper inhalation technique
  • Improper seal of mouth on mouthpiece
  • Not waiting long enough between doses
  • Failing to clean or maintain
  • Storing under wrong conditions
  • Failing to properly dispose of device
• Known use issues with injection devices:
  • Selecting incorrect product
  • Failure to check the drug appearance or drug expiration
  • Not priming at all or priming incorrectly
  • Identifying incorrect injection site
  • Not removing injector cap
  • Holding injector upside down
  • Unable to activate the injector
  • Premature removal (wet injection)
  • Unable to determine if dose is given
• Known use issues with patches:
  • Not removing all or some of the protective layer
  • Taping patches on rather than using patch adhesive
  • Placing new patches on top of old patches
  • Applying multiple patches at once
  • Failing to remove previous patch
  • Misunderstanding dosing including replacement schedule
  • Cutting patches to “adjust” dosing
  • Failing to safely discard used patches

Source: FDA Draft Guidance – Human Factors Studies and Related Clinical Considerations in Combination Product Design and Development, February 2016

TABLE 2

Types of Human Factors Studies

• Formative:
  • Prototype(s) of combination product
  • Iterative in nature: inform design changes
  • Inform the content of the human factors validation study
  • Generally completed early in Investigational New Drug application development
• Validation:
  • Final finished combination product
  • Demonstrate that design supports safe and effective use with representative users in expected use environment(s)
  • Ideally, human factors validation testing should precede any major clinical study (Phase 3)

TABLE 3

Human Factors Validation Tests Versus Major Clinical Studies: Part 1

• Timing:
  • Human factors validation test:
    • Before major clinical studies (before Phase 3)
  • Major clinical study:
    • Follows human factors work (including validation study)
• Device:
  • Human factors validation test:
    • Prototypes (formative) or final commercial product (validation)
  • Major clinical study:
    • Final commercial product
• Study participants:
  • Human factors validation test:
    • Representative users (patients, caregivers, healthcare professionals, and adults or children)
    • Healthcare professionals could include those prescribing, dispensing, and administering
  • Major clinical study:
    • Patients
• Sample size:
  • Human factors validation test:
    • 15 in each user group, about 45 total
  • Major clinical study:
    • Thousands

TABLE 4

Human Factors Validation Tests Versus Major Clinical Studies: Part 2

• Training:
  • Human factors study:
    • Should mimic commercial practice
    • If training is a risk mitigation, participants should be trained
    • If not, or if training is not reliable, participants should be untrained
  • Major clinical study:
    • Patients trained on proper use of device
    • Minimize variations in how device is used on clinical data
• Use environment:
  • Human factors study:
    • Should mimic actual use environments incorporating situations factors that may adversely affect use 
  • Major clinical study:
    • May be controlled
    • Minimize variations in how device is used on clinical data
• Use scenarios:
  • Human factors study:
    • Common and low frequency (but safety-related) use scenarios
  • Major clinical study:
    • Common
    • Controlled
    • Minimize variations in how device is used on clinical data
• Use:
  • Human factors study:
    • Simulated
  • Major clinical study:
    • Actual
• Use errors:
  • Human factors study:
    • Dependent measure
    • Monitor for evidence of failures, operational difficulties, and close calls
  • Major clinical study:
    • Independent variable
    • Controlled
    • Want to minimize use errors to ensure data integrity

TABLE 5

Human Factors Validation Tests Versus Major Clinical Studies: Part 3

• Data collection:
  • Human factors study:
    • Performance on critical tasks: Successes, failures, operational difficulties, and close calls
    • Subjective assessments of task difficulty
  • Major clinical study:
    • Drug effectiveness
• Statistics:
  • Human factors study:
    • Descriptive only
    • Typically, just sums
  • Major clinical study:
    • Inferential
    • Mean estimation and confidence limits
• Endpoints:
  • Human factors study:
    • Residual risks have been mitigated as much as possible
    • Product benefits outweigh risks
  • Major clinical study:
    • Drug is efficacious and safe