Evaluating the Protocol: Can the Clinical Research Site Really Do this Study?

Monika Gaa, MBA, CCRP
Senior Feasibility Coordinator, Nationwide Children’s Hospital

Christine Baker, BA, CCRP
Senior Regulatory Coordinator, Nationwide Children’s Hospital

Abstract: Protocol evaluation is critical to determining the feasibility of a study. This article describes the key elements for evaluating protocols for financial feasibility, recruitment, and practical application. Strategies that have worked well at Nationwide Children’s Hospital are highlighted.


Clinical Research Services at Nationwide Children’s Hospital is a streamlined coordination of services from idea to manuscript, and it is available to clinical and research investigators who need assistance with their clinical studies. Services include feasibility assessment, regulatory coordination, and coordination of the clinical services required for clinical study management.

Assessing study feasibility is crucial because roughly 80% of studies are delayed due to inadequate staffing and/or an inadequate number of subjects who meet the inclusion/exclusion criteria. Spending more time on feasibility upfront increases the chance that a study will be successful.

The Feasibility Process

The feasibility process at Nationwide Children’s Hospital is mandatory for all clinical research studies that are paid on a per-subject and per-visit basis and that have procedures that are outside of standard of care. The process became mandatory about eight years ago to ensure consistent pricing, appropriate budgeting and overall compliance.

Table 1 provides an overview of the feasibility process at Nationwide Children’s Hospital, which is mandatory for industry-sponsored and investigator-initiated studies. Each study is reviewed for scientific adequacy; “Does the study further advance the science behind the disease? Is this study scientifically significant and beneficial to the subjects?” Competing studies and patient populations are also reviewed because this determines the number of eligible subjects. Retention and time commitment are also vital components of feasibility because frequency of visits and study duration greatly impact participation.

Subject availability is a key consideration. The clinical research site must have a sufficient subject base to meet the enrollment goals. The site may see two hundred patients a year with the study indication; however, only a portion of those patients will meet the protocol’s inclusion and exclusion criteria. Of those, only some patients would consider participating in the study. Principal investigators who want to conduct a study often overestimate the number of patients who are truly eligible to participate.

Other factors determining the feasibility of a study include recruitment, which is impacted by not only visit frequency but also whether the disease indication is common or rare as well as whether there is use of a placebo. All of these factors need to be considered because they affect the budget, staffing, logistics, and resources for a study.

Study staffing is another key consideration in determining whether a study is feasible at the research site. Eager principal investigators do not always have the time to conduct the studies that interest them. They must be able to meet the protocol requirements for study visits, attend monitor visits, sign off on labs and reports timely, and many other time-consuming tasks to complete regulatory requirements and ensure compliance.

This mandatory process ensures that everything is set up in the hospital’s electronic tracking system and that no research fees would ever be charged to the patient’s insurance. It also ensures that study budgets cover all of the costs of conducting the study. Feasibility meetings include the principal investigator, the feasibility and regulatory coordinator, and the study coordinator if assigned, who all come together to make informed decisions about whether to participate in each study. The meetings include a thorough review of the protocol and the purpose of the study, the visit schedule, and the study logistics and staffing.

Calculating Study Costs

Consistency is the key in calculating study costs (Table 2). Everyone must be charged the same amount for study procedures. Nationwide Children’s Hospital has a centralized laboratory review process and uses codes with pre-assigned costs that are applied to every study.

Study preparation and start-up costs are very important. The same activities and work is necessary to start up a study regardless of whether the clinical research site ends up enrolling one subject or ten subjects. Having a standard, non-refundable, non-negotiable fee schedule for start-up costs has been crucial for Nationwide Children’s Hospital. This is based on the average amount of time spent on study start-up, and the fees vary depending on whether the study is interventional, academic, or observational. Start-up fees include PI and coordinator preparation time as well as regulatory and pharmacy fees.

Nationwide Children’s Hospital has a few institutional policies and fees that both the Research Institute and the Legal Department insist upon and are considered ”deal breakers” if they are not able to be negotiated. Withholding is not allowed because the costs have already been incurred and need to be reimbursed in full; a standard indirect rate must be applied to study budgets.

Nursing and study coordinator time can have a major impact on the budget. At Nationwide Children’s Hospital, only nurses are assigned to interventional studies in order to ensure subject safety. In order to understand the required data points, clinical research sites should obtain a copy of the case report forms (CRFs) from the sponsor. A study with twenty data points, for example, will take far less time than a study with eighty data points. Some studies need a nurse coordinator and a non-nurse coordinator due to the extent of the data and subject travel requirements. Estimating the amount of time necessary for the data points is one of the most difficult parts of calculating costs. At times, staff time has been grossly underestimated, which has a major impact on the budget.

Industry studies and investigator-initiated studies are very different in terms of budget preparation. With industry studies, the sponsor will generally provide a budget template. Clinical research professionals must ensure that the sponsor’s budget template incorporates the appropriate indirect rates and that the fees match the institution’s charges for study procedures such as laboratory tests, radiology, and pharmacy. The budget template should be modeled after the schedule of events, and Clinical research professionals should compare the budget template and the schedule of events to ensure that all protocol requirements are reflected in the budget. A checklist can be helpful to ensure that all required costs are covered in the budget, including reimbursement for often-missed fees such as phone calls, stipends, and ancillary services.

Appropriate amounts for stipends and travel needs to be determined with consideration of what the institutional review board (IRB) will approve. It is much easier to include subject stipends and travel in the budget upfront than to later renegotiate with the sponsor or realize that the study is running at a deficit, especially for investigator-initiated studies.

Investigator-initiated studies require that the clinical research professional work from the protocol to develop the budget. Key considerations in developing the budget are the number of subjects, procedures, laboratory tests, and ancillary services. It is necessary to obtain all of these costs in order to prepare a complete budget. If grant funding is used for an investigator-initiated study, a finite amount of money has generally already been allocated, and the investigator needs to ensure that the protocol requirements and associated fees are conducted within the total allowable budget.

Nationwide Children’s Hospital involves statisticians to determine the possible number of subjects to ensure that the proposal will provide valuable data with the number of subjects that can be included as per the study budget. In some cases, it is necessary to adjust some of the procedures or laboratory tests in order to include sufficient subjects in the study.

Pitfalls in Creating a Study Budget

Table 3 highlights some pitfalls in creating a study budget. Principal investigator and study coordinator time has a tremendous impact on the study budget. This is often underestimated. Serious adverse event (SAE) reporting, for example, is very time consuming and is sometimes left out of the budget. Clinical research sites should include either an hourly fee for SAE reporting or a flat fee per report. The overall study load, the protocol requirements, and the visit demands must be considered in terms of the time required for the principal investigator and the study coordinator. This includes considering things such as follow-up phone calls and SAE reporting.

Standard of care versus research procedures is a compliance issue as well as a budget issue. Billing procedures to insurance must only be done according to the normal standard of care. For example, if the standard of care is to see patients in the study population at three, six, and nine months, and the protocol requires the principal investigator to see subjects at two, four, and eight months, this is not the normal standard of care and cannot be billed to insurance.

Screen failure is another consideration because some studies have very high screen failure rates. During screening visits, it is common to complete most of the requirements only to find out at the end that the patient is a screen failure. When sponsor budgets pay for half or a third of screening visits that result in a screen failure, clinical research sites lose money because the screening visit is the same amount of work whether the patient enrolls or is a screen failure. Clinical research sites should be paid in full for screen failures. Start-up fees should cover the start-up costs.

Estimating the number of eligible subjects based on the inclusion/exclusion criteria is difficult. Clinical research professionals should run queries to determine the number of patients who are eligible for the study and then review the inclusion/exclusion criteria against those patients. Another person should double check the review.

Hidden Costs in Protocols

The hidden costs must also be considered:
• Study design
• Safety events
• Local versus out-of-town subjects
• Monitoring time
• Recruitment.

Study coordinators or other members of the research team with experience managing studies can provide crucial expertise related to hidden costs. They provide a different perspective beyond determining feasibility by analyzing costs.

In the study design, the timing of procedures, number of visits, number of blood draws, use of home health agencies, and the inclusion/exclusion criteria can all involve hidden costs. For example, if the study requires multiple blood draws with immediate processing of samples and vital signs to be taken shortly after the blood draw, this requires more staff. A floor nurse or a second study coordinator must be available to take the vital signs while the first study coordinator processes the sample (or the floor nurse or a second study coordinator could process the sample while the first study coordinator takes the vital signs). The location of the laboratory, within the unit or across the facility, also affects the study budget.

If a study has infusions for forty-eight weeks and subjects must stay in the clinic for four to six hours, this takes clinic space and affects the amount of space that is available for patients. It also influences the number of study visits that the study coordinator can realistically handle. Another nurse or another member of the research team may be required to handle study tasks such as processing and shipping samples and data entry, while the study coordinator handles the clinical tasks.

The number of blood draws in a study can greatly affect the budget. As a pediatric institution, Nationwide Children’s Hospital always considers the age of the subjects and how they will respond to blood draws. Some children do not mind blood draws while others are very scared of them. Young children may need to be distracted during blood draws, which requires involving another person and additional costs. Other considerations are whether the blood draws can be done in the clinic space, whether the subjects need to go to the central laboratory, or whether the IV team needed.

Home health agencies can be very helpful to subjects; however, using them in studies is problematic for principal investigators and study staff. The individual nurse at an individual agency can make or break the study. The principal investigator is responsible for the home health agency nurses who work on a study. If the home health agency nurse does not show up because the weather is bad nor does the paperwork incorrectly, this will result in deviations that will most likely need to be submitted to the IRB. Any safety events must be reported immediately, which takes time.

Inclusion/exclusion criteria are an important consideration, including proving that the subject has the specific indication being studied. The diagnosis may be subjective or require a genetic test or an exam by a specialist. These things can be costly. It may be necessary to obtain documentation for the local physician or a specialist and obtain written authorization. This often requires follow up, which could be time-consuming.

Vague inclusion/exclusion criteria are especially problematic. In one study at Nationwide Children’s Hospital, for example, subjects were not supposed to take a certain drug within three months of the study, as measured by a test during screening. The test, however, only measured the drug’s use for about a month, and the study population could not urinate on demand. Therefore, the test could not be run.

Some protocols require that ECGs are read prior to study drug being administered. However, these protocols often have ECGs that are centrally read that are not immediately reviewed. In order to not delay the administration of the study drug, the ECGs are read locally in addition to the central read, which will create an additional charge. Depending on the number of subjects and ECGs per year, this could be quite expensive. The local reading fees should be included in the budget.

Processing SAEs is time-consuming, especially when medical records from an outside institution must be obtained. In studies where SAEs are anticipated or some subjects live far away, clinical research professionals can obtain an authorization for outside medical records in advance. This can save a great deal of time, which is important because SAEs need to be reported to the IRB promptly.

Requesting payment in the budget for processing SAEs is helpful, especially since clinical research professionals need to complete and submit paperwork on events that may not meet the criteria for being unanticipated. The amount of time required to prepare Suspected Unexpected Serious Adverse Reaction and Investigational New Drug safety reports is very variable. If the study drug has not been tested before or if it is a marketed drug that is approved for multiple indications or is being tested in multiple indications, there will be extra safety reports.

In one study at Nationwide Children’s Hospital of a new drug, there are a number of SUSAR reports from all study sites that may or may not technically meet the criteria for being unanticipated. All of these reports require the principal investigator’s signature and the signature of every sub-investigator. Routing the paperwork and ensuring that all of the necessary signatures are obtained takes a tremendous amount of study coordinator time.

With out-of-town subjects, travel planning must be considered. Clinical research professionals need to know whether the sponsor will handle this or whether it is their responsibility. It can be very time consuming to book hotels and flights and arrange for transportation and meals.

Coordination of ancillary services is a consideration. Services must be coordinated as per the protocol. For example, say that a study involves surgery, cardiology, and physical therapy. The physical therapy must be done before surgery, and the subject must see the cardiologist after surgery. The surgeon is only available one day each week. If the physical therapist is not available one week, surgery cannot be done, and the procedure may not occur within the specified window. Consideration must also be given to how to handle out-of-town subjects who miss a study procedure and must come back for an extra visit. This could involve considerable cost.

Monitoring considerations include the use of multiple monitors over the course of the study and on-site versus remote monitoring. Nationwide Children’s Hospital tries to incorporate a monitor change fee up front. In one four-year study, the same monitor worked on the study for the first three years. In the fourth year, however, the clinical research site had seven different monitors. Each monitor verified all of the documentation from start to finish, which took a tremendous amount of the study coordinator’s time. Each new monitor wanted everything done differently than had been done by the previous monitor. Dealing with queries and changes took a tremendous amount of time.

Remote monitoring can take a tremendous amount of the study coordinator’s time if, for example, the clinical research site is required to make copies of subject records and fax them to monitors. If bound files need to be taken apart, copied or scanned, and then put back together correctly, this takes time.

Recruitment considerations include verifying the inclusion/exclusion criteria, the recruitment methods, and the amount of time that will be required for recruitment. Recruitment methods may include having the study coordinator sit in the clinic and see who comes in (e.g., for a flu study), sending out letters to recruit healthy controls, obtaining physician referrals, self-referrals, and advertising. Funds for advertising should be included in the budget.

Potential Problems

When reviewing a protocol, it is important to look for potential problems related to study feasibility (Table 4). Clinical research professionals should read all sections of the protocol, even sections that they might not normally read.

The informed consent section, for example, is generally fairly generic; however, issues have been identified during the feasibility process related to informed consent. In one case, a registry study that used questionnaires to collect data required that a medically qualified person do the informed consent. The site normally would have assigned a non-nurse coordinator to handle informed consent for this type of study.

One high-risk study that required both parents to provide consent required a study physician to be present for informed consent and assent, and it would not allow nurse practitioners, even though they were listed as sub-investigators, to handle this. The study was for a new drug and multiple re-consents were necessary, resulting in a significant underestimation of the amount of principal investigator time that was involved in the study just for the informed consent process.

Footnotes and investigator discretion can cause problems. Tables sometimes have footnotes with information that is not anywhere else in the protocol. It is important to read all the footnotes. In one study, the investigators were having parents give their child antihistamines at home. A footnote, however, required the antihistamines be given within fifty minutes of dosing of the study drug. Thus, the study had many protocol violations. Anything that says “investigator discretion” is open to a great deal of interpretation, and is likely to generate many follow-up issues.

Reference ranges may be different for central laboratories and local laboratories. As a pediatric institution, local laboratory reference ranges are not necessarily the same as the central laboratory’s reference range. In one study, the principal investigator deemed some results as non-clinically significant because for the population at Nationwide Children’s Hospital, they were not significant. The protocol, however, required retesting if the results were out of the central laboratory’s reference range a specified number of times. This was not done because the principal investigator determined that the results were non-clinically significant, which created issues.

The Feasibility Process at Nationwide Children’s Hospital

  • Scientific adequacy:
    1. Benefits
    2. Competing studies
  •  Subject availability:
    1. Population
    2. Retention
    3. Time commitment
  • Staff availability:
    1. Experience levels of principal investigator and coordinator

Calculating Study Costs

  • Procedure/laboratory costs:
    1. Consistency
  • Study preparation/start-up costs:
    • May differ depending on study type:
      1. Interventional
      2. Academic
      3. Observational
  • Institutional policies:
    1. Indirect rate
    2. No withholding
  • Nursing/study coordinator time:
    • Interventional studies:
      1. Assign only nurses
    • Visit and data requirements
  • Extent of required study data:
    1.  Obtain copies of CRFs

Study Budget Pitfalls

  • Principal investigator and study coordinator:
  1. Overall study load
  2. Protocol requirements
  3. Visit demands
  • Standard of care vs. research procedures
  • Screen failures
  • Start-up fees
  • Estimating eligible subjects

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.