![\"a](\"https:\/\/www.socra.org\/blog\/wp-content\/uploads\/2019\/08\/iStock-1035963640-1024x460.jpg\")
<\/figure>\n\n\n\nAbstract<\/em><\/strong>: Detailed\nprotocol analysis and feasibility objectively translated into a protocol grade\nor quantitative score is an effective way to manage overall workload\ndistribution, personnel resource allocation, and financial management before\nthe clinical trial is implemented. This article provides an overview of\nprotocol areas for evaluation, scoring, and current effort tracking models used\nin clinical research. One model, the Protocol Acuity Rating Scale (PARS), is\ndescribed in detail with examples of its application to different types of\nclinical research studies<\/em><\/p>\n\n\n\n\n\n\n\n Due to the increasing complexity\nof clinical research protocols and the need to justify staffing, it is critical\nfor clinical research sites to have methods to evaluate, quantify, and document\nthe amount of effort that will be required to effectively execute a clinical\ntrial. It is also necessary to estimate the financial support needed to justify\nand to properly carry out the work. Determining equitable workload allocation\namong staff at a site is crucial. Large sites need accurate projections of total\nstaff (research coordinators, clinical trial managers, data managers, etc.) and\na balance of work between staff members. Smaller sites need to provide\njustification to hire additional personnel or provide decision points for\naccepting or rejecting a research project.<\/p>\n\n\n\n Protocol scoring, according\nto the author, is a detailed review of a clinical research protocol and\nactivities needed to carry out that protocol, including moving the subject\nthrough the protocol during project execution. All of the activities are\ncategorized and graded so that they can be assigned a point value. The points for\nthe project are then totaled, and this determines the project\u2019s overall\ndifficulty \u201cgrade\u201d or score. The most important benefit of protocol scoring is\nthat it can then be translated into work effort projections or used to\nappropriately assign staff to various projects. <\/p>\n\n\n\n The balance between effective\nutilization of personnel (full-time equivalents or FTEs) to support the work and\nbeing overstaffed and having idle staff is very delicate. If the workload is overwhelming\nor a project is beyond the scope of the personnel, the participants, the project,\nand the data may all be at risk and the quality of the unit will diminish. If staff\nmembers are not fully utilized or there are unequal distributions in workload,\nfinancial resources may be deemed to be mismanaged. High error rates,\ndeviations, or overspending on underperforming staff never goes unnoticed. <\/p>\n\n\n\n When evaluating deliverables\nand scoring a protocol, it is important to keep in mind the abilities of the\npersonnel and the role each person will play in the research study. Questions\nto be answered include:<\/p>\n\n\n\n The Houston Methodist\nResearch Institute conducts many first-in-human and early phase studies. The author\nneeded to evaluate one of the research sections to determine the feasibility\nfor the section to take on a high-profile, federally-funded, first-in-human trial\nthat required effective management and aggressive recruitment to see the\ninvestigational product into the pivotal phase. <\/p>\n\n\n\n The evaluation consisted of a\nreview of the number of FTEs, the number of treatment studies open within the\npast five years, and annual accrual for each study as reported in the continuing\nreviews. Over the review period, the research section had a total of 6 to 8 FTEs\nand 48 treatment trials, most of which were open 3 years or longer.<\/p>\n\n\n\n Average enrollment was low,\nranging from 1.3 to 2.6 subjects per study per year for the research section.\nThis indicated that something was amiss. Perhaps FTEs were overworked or only\nenrolling subjects to some trials. Other possible problems were that some\ninvestigators were very prolific and others were not. A review of the studies\nby specialty did not identify any prolific investigators and also showed low average\nenrollment per study per year. The specialty clinic has a targeted orphan population;\nhowever, a review of this population did not improve the results. <\/p>\n\n\n\n Thus, the review showed that\nthe research section had issues with feasibility, and by the demonstration of\nmany trials going annually with zero enrollments, failed to perform feasibility\nprior to commitment and initiation. The research section had allocated FTEs to\nexecuting these trials; however, it was unable to meet the minimum burden of\nenrollment, demonstrating an ineffective allocation of personnel. <\/p>\n\n\n\n There are several available, well-published\nprotocol scoring and workload tracking tools: <\/p>\n\n\n\n Table 1 provides an overview\nof each model, most of which were developed in the field of oncology. <\/p>\n\n\n\n The NCI Trial Complexity and\nElements Scoring Model was one of the earliest models for scoring protocols. It\nrates 10 elements, such as the length of the study and the complexity of the\ninformed consent process. Each element is rated Level 0 (standard), Level 1\n(moderate), or Level 2 (high). The model also provides an overall score for the\nprotocol. <\/p>\n\n\n\n The University of Michigan\nResearch Effort Tracking Application is a detailed effort tracking clinical\ntrial management system. It is a Web-based service that tracks effort allocated\nto all clinical research activity and can be used to compare projections with\nactual personnel expenditures as well as to project feasibility or the FTEs\nneeded to carry out future studies. This is a very effective tool.<\/p>\n\n\n\n Two more recent tools are the\nOntario Protocol Assessment Level and the Wichita Community Clinical Oncology\nProgram Protocol Acuity Tool. The widely used Ontario Protocol Assessment Level\nuses a pyramid rating scale ranked from levels 1 through 8. The base of the\npyramid is Phase 1 highly interventional studies. Each increment represents\nincreasing complexity. The score can be increased based on the number of\nsubjects and contacts per subject. The Wichita Community Clinical Oncology\nProgram Protocol Acuity Tool ranks protocols on six workload-related\ndeterminants and scores protocols according to their estimated workload using a\nrange of one to four. These are good tools that have a great deal of utility.<\/p>\n\n\n\n When the author was working\nin pediatric clinical research in a previous position, she found that the oncology-based\ntools did not facilitate comprehensive analysis of non-oncology research. Additionally,\nalthough many of the scoring models provided ways to analyze and result in\nquantitative values, this was rarely translated back to the actual amount of\nestimated personnel effort or cost. The author and her team wanted to evaluate\ndifferent portions of studies. Since the clinical research site conducted more\nindustry-sponsored trials than most other academic clinical research sites, she\nwanted to be able to analyze the heavy data requirements of these studies.<\/p>\n\n\n\n The Protocol Acuity Rating\nScale (PARS) was developed based on post\ninitiation operational aspects of clinical trials and all participant\nmanagement at the pediatric clinical research site. Table 2 highlights the\ncriteria used to review protocols in PARS, which is expressed in a grid. Criteria\nincluded the phase and type of the study, the participant setting (inpatient or\noutpatient), and data reporting requirements (paper case report forms, whether\nit is a consortium study, and electronic data capture). Oversight and\nmonitoring is another key criterion used in PARS. The amount of time required\nfor preparation for oversight and monitoring by an external sponsor can\nsometimes be significantly more than expected. The encounter procedure and\nfrequency as well as the duration of the study are also part of PARS. <\/p>\n\n\n\n Because rate of enrollment\nand total number of subjects placed on study can increase or decrease total\nclinical trial workload exponentially, the rate of accrual is deemed the \u201cX\u201d\nfactor in any study. All previous criteria are evaluated and totaled, then\nmultiplied by the rate of accrual score. If the rate of accrual is moderate or\nslow, study staff members can pace themselves, or the hours per week of effort\nmay be lower. If, however, the rate of accrual is very fast, the study needs a\nhigher FTE allocation. <\/p>\n\n\n\n It\u2019s difficult for one method\nof protocol scoring to effectively evaluate everything required to conduct a\nclinical trial. Thus, PARS does not evaluate the regulatory effort required to\ndraft, gain approval for, and process amendments. Although initial approval may\nbe projected based upon protocol evaluation, changes throughout the study\nlifecycle can be unpredictable. The number of amendments can be highly\nvariable. The author and her team reviewed more than 50 treatment protocols and\nfound a mean of four amendments; however, the range was two to six amendments over\nan average of three years active study period. Additionally, the complexity of\nthe required regulatory effort does not necessarily correlate with the trial\nphase or operational protocol complexity, nor does it relate to the scope and\ndepth of the amendments. Cooperative grant or limited funding studies, for\nexample, commonly arrive with amendments containing \u201ctiny\u201d changes that require\n10-fold additional hours of FTE work.<\/p>\n\n\n\n PARS also does not evaluate:<\/p>\n\n\n\n Other metrics could be used\nto evaluate these activities.<\/p>\n\n\n\n The PARS grid has the\nreference or categories (low, which is 1 point, medium, which is 2 points, and\nhigh, which is 3 points) in the first column. Across the top, the items are: phase,\ntype of study, participant setting, data requirements, monitoring oversight,\nencounter procedure, lab\/samples, encounter frequency, study duration, and rate\nof accrual. This is similar in some ways to the NCI Trial Complexity and\nElements Scoring Model.<\/p>\n\n\n\n In Example #1, PARS is used\non a non-interventional, cooperative group protocol with the objective of establishing\nand maintaining a disease history and outcomes database to evaluate this disease\u2019s\nimpact on health-related quality of life. The exploratory objective is to learn\nabout genetic modifiers of clinical phenotypes of the disease (an optional sub-study).\n<\/p>\n\n\n\n There is a baseline exam\nassessment and then subjects come in every six months during the two-year\nstudy. The study includes questionnaires, basic exams, measurement of vital\nsigns, medical history, and the same disease-specific clinical evaluation that the\nsubjects would be getting in standard care. All exam results and questionnaires\nhad to be entered into the central database. The monitoring burden is very low,\nwith monitoring to be done about once a year.<\/p>\n\n\n\n The rate of accrual can\ndramatically change the protocol score for this study, as shown by this example\nof two different study populations. If the study population is a low-prevalence,\nrare disease such as Duchenne muscular dystrophy, cystic fibrosis, or Huntington\u2019s\ndisease, overall patient recruitment and rate of recruitment will be much lower.\nData requirements are medium, based on the number of hours per week expected to\nbe required to acquire the data, translate them, review the medical notes, and\nenter questionnaire and other data. Laboratory samples for the sub-study will\nrequire drawing the samples, and then prepping, packing, and shipping them. A\ntwo-year study requires some scheduling and follow up. <\/p>\n\n\n\n The rate of accrual will be\nlow, even in a specialty clinic, and is projected to be about one or two subjects\nper month. The total study score for this study with a low-prevalence, rare\ndisease study population is 14.<\/p>\n\n\n\n In a diabetes, atrial\nfibrillation, or obesity clinic, however, everything is different. The number\nof subjects are recruited into a study at a faster rate, resulting in more\noverall encounters and a higher data burden over the work week, demand more of\nan FTE. The total study score with a high-prevalence study population is 42. Carrying\nout the exact same study requires more personnel effort in a high-prevalence\nstudy population than in a low-prevalence, rare disease study population.<\/p>\n\n\n\n In Example #2, PARS is used\non a Phase 3 industry-sponsored pivotal multicenter trial. Data from the study\nwill be used in the New Drug Application. It is a three-arm randomized\ndouble-blind placebo-controlled study of the efficacy and safety of two doses\nof the study drug as adjunctive therapy in patients with genetically-induced\ncatastrophic syndrome. The primary objective is to compare the clinical\nresponse to the different drug doses on the primary endpoint: reduction in the catastrophic\neffects of genetically-induced catastrophic syndrome. The study population is\nan orphan population, usually very ill at baseline and complicated medically.\nThis increases the potential for adverse events and serious adverse events. This\npopulation has no viable treatment alternatives, and this is the first possible\ndisease-modifying investigational product that has been identified. <\/p>\n\n\n\n The study includes\npharmacokinetics to compare the efficacy of different trough levels versus\nplacebo, as well as quality of life and developmental assessments. The\nthree-part study consists of baseline\/screening, blinded core, and an extension\nopen label phase lasting until approval of the drug. The entire study period is\nexpected to be about three to four years.<\/p>\n\n\n\n Many studies are like this.\nSponsors, whether they are industry or academic, are trying to gather as much\ndata as they can from studies, especially in orphan drug development programs\nwhere available patients to enroll in research studies are fewer.<\/p>\n\n\n\n This is an interventional,\npivotal Phase 3 study. It is outpatient; however, it requires seeing subjects\nfrequently, sometimes with only days between visits. Many laboratory samples are\nrequired, and the encounter frequency is high. Like many other studies, this\none continues until marketing approval. The rate of accrual will be very high,\nsince this is an orphan population with no standard alternative disease-modifying\ntreatment. The total score for this study is 69. <\/p>\n\n\n\n When developing a protocol\nscoring tool or using one of the available tools, each clinical research site\nmust compare the scores and the workload to determine the best way to translate\nthe scores for that particular site. This can be done by modifying or changing\nthe areas of protocol operation evaluation or by scaling point values to be\nappropriate to their staffing structure or job responsibilities. <\/p>\n\n\n\n Houston Methodist Research\nInstitute translates the protocol score into FTEs, reviewing the total number\nof points and how that translates into carrying capacity in FTEs (Table 3). Carrying\ncapacity, however, varies by knowledge and experience, which is not limited to only\nclinical research experience. It includes overall education level, prior\nmedical training, professional maturity, social awareness, and the ability to\nhandle stressful high-pressure work. The author has worked with junior\nemployees who are new to research but are very socially and professionally mature\nand able to handle pressure. She has also worked with research nurses who have\nextensive disease and treatment area experience who can easily extrapolate\ntheir medical training and expertise to working with complicated research\npopulations, allowing them to carry immense workloads. <\/p>\n\n\n\n Concurrent, overall workload\nis the other key factor in translating the protocol score into FTEs. This\nvaries by the size of the overall research department and the segregation of\nwork tasks. A one-man show will be very different than a multi-disciplinary\ndepartment where work tasks are highly segregated. In a single-coordinator\nmodel, the research coordinator handles budgeting, regulatory work, interface\nwith the U.S. Food and Drug Administration, and so forth, as well as\ncoordinating the studies. This research coordinator cannot do as much operationally\nwith subject movement as a research coordinator in a compartmentalized\ndepartment, where she\/he only coordinates the studies and the department has a\nregulatory coordinator, a finance manager, a clinical trial manager, and so\nforth. That changes the relevancy of those point values.<\/p>\n\n\n\n Generally speaking, in a\ncompartmentalized department which has research coordinators (responsible for\ncoordination and data management), a regulatory coordinator, a finance manager,\nand a clinical trial manager, the FTE \u201dcarrying capacity\u201d is as follows:<\/p>\n\n\n\n The expectations for a given\ntarget capacity can be adjusted to suit the complexity of the studies seen in\nthe department or for the particular interventions that are being tested. For\nexample, if the clinical research site is the applicant and needs to interface directly\nwith the U.S. Food and Drug Administration, the burdens of reporting and monitoring\nwill be higher. Other types of studies that increase the protocol score include\ninvasive, high-risk surgery protocols and translational, first-in-man studies where\na high incidence of toxicity is expected.<\/p>\n\n\n\n Utilization\nof a protocol scoring model or tool can help clinical research sites objectively\nevaluate the requirements of carrying out a protocol. A protocol scoring model\nor tool also removes bias, which the author has found to be extremely effective\nin determining protocol acceptance, and workload leveling. <\/p>\n\n\n\n It\nis critical to evaluate protocol feasibility before implementation of the\nclinical trial; a protocol scoring model or tool facilitates this. Pre-implementation\nassessment aids in preventing the acceptance of protocols for which the\nprocedures or subject recruitment is unattainable for the study site. Moreover,\nit assists in gauging the available capacity or \u201cbandwidth\u201d available in the\nworkgroup to accept taking the additional workload. It prevents the problem of\nopening many clinical trials where no subjects are being enrolled and patients\nare missing out on participating in research because the feasibility was never\nworked out. <\/p>\n\n\n\n The protocol score can be extrapolated into estimating how many FTE employees are required to support the trial and the overall group workload. This can be used as an objective measure to justify personnel projections and costs per year, per project, per FTE, and anticipated growth needs. . This is not a one-time assessment. If there are amendments causing protocol modifications\/extensionor staff changes or reassignments, protocol scoring should be done again. Protocol scoring does not take long to do and can easily be repeated.<\/p>\n\n\n\n Review of\nAvailable Protocol Scoring Tools<\/strong><\/p>\n\n\n\n Activities\nEvaluated in the Protocol Acuity Rating Scale<\/strong><\/p>\n\n\n\n How Protocol Scores Relate to FTEs at Houston\nMethodist Research Institute <\/strong><\/p>\n\n\n\n Christina Talley, MS, RAC, CCRP, CCRC Houston Methodist Research Institute, Office of Strategic Research Initiatives Abstract: Detailed protocol analysis and feasibility objectively translated into a protocol grade or quantitative score is an effective way to manage overall workload distribution, personnel resource allocation, and financial management before the clinical trial is implemented. This article provides an … <\/p>\nProtocol Scoring <\/strong><\/h2>\n\n\n\n
How Analysis Benefits a Clinical Research Site: An Example\n<\/strong><\/h2>\n\n\n\n
Review of Protocol<\/strong> Scoring Tools<\/strong><\/h2>\n\n\n\n
Development of the <\/strong>Protocol\nAcuity Rating Scale <\/strong>Tool<\/strong><\/h2>\n\n\n\n
PARS Example #1 <\/strong><\/h2>\n\n\n\n
PARS Example #2 <\/strong><\/h2>\n\n\n\n
Protocol Scores and FTE <\/strong><\/h2>\n\n\n\n
Conclusion<\/strong><\/h2>\n\n\n\n
\n\n\n\nTABLE 1<\/strong><\/h2>\n\n\n\n
TABLE 2 <\/strong><\/h2>\n\n\n\n
TABLE 3<\/strong><\/h2>\n\n\n\n