Transcribed from the recorded session and lightly edited for readability. The speaker's views are his own and do not represent ARC-PA; always confirm specifics against the current ARC-PA Standards and Compliance Manual.
0:00 — Introduction: why C1.01 is a turning point
when we talk about C1.01, I think that in terms of being consequential, it is very significant in that you are now going to make your own decision about what data to analyze and what conclusions to draw.
And I think that one of the things that's unknown is that when do you know you're correct? And I'm hoping that by the time we finish, you'll have at least some insights about how you go about the process. Okay. So this webinar really is about the strategic process of approaching C1.01. So just typical kind of perspectives.
Obviously I don't speak for ARC-PA. My perspectives are based upon I'm now writing some of these now. So I'm getting some insights. And again this is new. Many programs are not still are not writing sixth edition SSRs yet. There's still a lot of fifth. But going forward it's going to be everybody writing these. And so I think that the insights I wanted to gain really was because I wanted to give, I guess the participants out there some insights as you start doing this, having been knee deep in the fifth edition, standard SSR.
You know, my company does a lot of work with that. I think this will be less work in the preparation and writing. It does not mean less work in terms of the collection aggregation of the data. Okay. But the writing and I think some of you can probably identify that some of the fifth edition SSRs went on for 200 or 300 pages.
If you go back to back, like anything else, you know, I have concerns about ambiguities. So I think that we'll find out in the next year. Exactly. You know, kind of like what the perspectives are from the ARC-PA viewpoint, because I think everybody has a different lens and many have the readers will have their perspectives. Right. So I think that if you follow a process, which I'm going to kind of go through, I think that you're less likely to get into trouble with this and more likely to be able to defend whatever answer that you are giving for each question.
So, as we know, there's five questions. This webinar does not include C1.02. I am toying with the idea of expanding this webinar into C1.02. But for now it's just these. So each one of these questions, think about it this way. They stand on their own. There is overlapping data obviously, but every time you go to answer these questions, you have to ask yourself, what is the specific aspects about this question that's different than the others?
Okay. And we'll be going through each of those in terms of, you know, kind of like what's the data? What do you start with? So that's kind of from my perspective.
3:24 — Critical analysis: trends, comparison, contextualization
So the language that ARC-PA has used for a number of years is obviously critical analysis. And there's some there's some new kind of I think emphasis on some I would say expanded perspectives about critical analysis. You know, one is trend analysis over time. And that is does that mean that a downward trend analysis basically if it's above your benchmark, is not a problem?
Okay. That's a question comparison about how data relate to each other. That's going to be something you'll be doing a lot of in this particular process. And then contextualization where you have to kind of think about influences or contributing factors. The best way I would describe that is that if you have an area of weakness, say for example, in your PANCE and you go back retrospectively or look in the past.
Was there any, I guess, impact of courses or course performances or anything that might have a contextualization relationship with that, where basically things are related? Okay.
4:43 — Drawing conclusions and choosing the right data
So in drawing conclusions, basically you have to identify whether the program meets its expectation or needs improvement based upon the data analysis and the what we call an ANI in the fifth edition area needing improvement is the same thing. I mean, basically the action plan is the same, but the pitfalls is the one question is did you write, did you choose the right data?
Okay. The combination of data. Was there any additional data sets beyond the predetermined templates? I think you're going to find that as you get into this, you're going to say to yourself, I think that would be helpful. I think that would be helpful and allows, you know, some additional charts to be added. Right. And the program has to have a consistent definition of what qualifies as an ANI, I will tell you that the materials that ARC-PA has does not define what an ANI is.
You have to define what an ANI is from your perspective.
5:54 — Defining an area needing improvement (ANI)
So in determining basically you got triangulation of data, three or more data sets, you have, you have declining trends and you have precipitous drops. And when I mean by precipitous drops is that you got a critical data set that all of a sudden falls off a cliff. Okay. Or what if you have declining trends but it's still above your benchmark?
Okay. Does that mean that you ignore that? And I think that there's like an opening up of interpretation here, that in the fifth edition, if there wasn't triangulation of three data sets, then basically you didn't play ball, right. You know, it didn't qualify. I think that now there's a little bit of an opening up of interpretation of this from the viewpoint of ARC-PA.
So essential language again benchmarks not is not is not new. But in your C1.01 template the Excel spreadsheet, you have to put down your benchmarks for every data set. And if you don't have those, the first thing that I would suggest to do is to work through those to make sure you have them, because as you self populate that template, you're going to want to know what your benchmark is to see where you are trending toward.
Right. Data summaries basically is labeled charts. We know about that data sets, which data sets are used for the questions. And then the analysis process. So that involves identifying data below benchmark or trends over time. Triangulation of data sets. So we all know this to be the case.
So how do you know how does the program document its ongoing self-assessment process. One is you arrive at a conclusion and it's kind of like you go through the process of analyzing the data. You synthesize the data, you apply some of these constructs to it. And then you say to yourself, does is the program effective or is the program not effective?
Okay. The supporting evidence basically will help you go in one direction or the other. I will say truthfully that if everything is an ANI, you're probably not going in the right direction. But I would say also, it's possible that you will go through all five questions and have no answer. That's very possible. I would say don't have too many ANIs, because it might be that you're going into a rabbit hole with information.
The other thing that the sixth edition standard now talks about is the comparison concept. We are looking at two or more subjects, data sets like side by side. And that might be for example, the, you know, the course grade. And of course that teaches specific content. You look at PACKRAT results for a specific organ system or task area.
You look at, you know, the PANCE. You could look at these comparison data sets side by side to see if there's any relationship. And within trends you look at long term direction and the templates allow you to put in I think, 5 or 6 years of data. So think of these templates as being your vehicle for the future, that you can start putting that data into those templates and you start to watch, you know is the is there is there a data moving upward?
A growth pattern, in other words, is improving over time or is it a downward decline? And that's what I mean by this. Is that what if you are going down very quickly even though you're above benchmark, does that mean that's an area that could mean improvement? The answer is potentially yes. Or basically there's no change whatsoever.
And then of course we know what triangulation is in triangulation. Basically, it helps to basically mitigate bias by converging information. And I think that if you think about how that's useful to you, is that you do not want to rely on a single data point and that and that's still the same. Truth holds, you know, holds truth. But.
It kind of reinforces the notion that as you go through the analysis of this, if you have one data point among the three or 4 or 5 you choose that's below benchmark or appears to be a problem. You want to pause to see whether or not you want to declare an ANI. So.
10:56 — A systematic, defensible approach
So let's talk about the systemic approach to this. First of all, you review the required data sets for each of the five questions. So I would start there and say is it above or below. Is there any issues there. And make sure that you look at it from a macro viewpoint in a micro viewpoint. So obviously if it's course evals or whatever it might be, you look at the whole thing and if there's like 1 or 2 below benchmark courses among 20 over three years, it's probably not a problem, right?
But you still want to annotate that. And number two is benchmark is in effect. So you make you have to make sure that you have that in each of the templates. And then evidence triangulation is where you start to investigate additional data points. So you start you look at the initial data point and then use other data points to see if there's any thematic evidence or any additional data sets that are reinforcement.
The root cause diagnosis is basically diagnosing potential causes using both quantitative and qualitative signals. And so the one the one difference I see is the increased reliance or increased requirement for qualitative data. If you're not collecting qualitative data, you the areas that are required without having that it's incomplete. Okay. And there are some qualitative data components like for example unsuccessful PANCE taker which is difficult to get.
But when you look at all the other data that you're going to be gathering, you're going to have to gather that qualitative data, because that's going to be part of the, I guess, the overall puzzle that you're going to be solving. And then you determine compliance. So again, no data sets below benchmark. Pretty easy right? You pretty much can say I think it's probably in compliance.
The question is probably effective right. And but that's where basically you have to be a little cautious as I go forward. So critical assessment. If there's the little benchmark data you got to decide whether a truly constitutes a program level area needing improvement. There are basically what I would say, you know, operational needs of improvement or basically small, I guess, you know, coarse level things that need to be done.
But again, you've got to say to yourself, is this like, does this mean that your admissions process is not effective? Does this mean that your program is ineffective? You got to really think critically about that decision, and then you have an evidence based strategy that follows the steps. That's really a compilation of the steps that I just talked about.
So this is kind of like a starting point.
14:05 — Evaluating each question through its own lens
And then when I want to say the lens for independent evaluation and contextualization, remember to look at every question differently. And remember that you're looking at a specific element. When you look at when you look at basically our faculty effective outside the classroom, that's a different question than basically, is the program effective? It doesn't mean that they couldn't be related.
It doesn't mean they couldn't. They basically data points within the five could be related. But you have to say to yourself, is it likely that it's related? Right. So you look at every question differently. Remember that you're evaluating specific elements without always overlapping the others. And so you're looking through your own lens. And then the term is contextualizing where the faculty as a whole can discuss that and say what is it?
What does this data mean to you? Because you know the data better than anybody else does.
15:08 — Question 1: Faculty effectiveness outside of teaching
So let's get started. The first question in our program faculty effective in operating the program outside of teaching. So the first thing that you've got to understand is that there are defined rules of teaching that for the most part, we're not evaluated consistently in the in the fifth standards there. There's three standards. The principal faculty, the program director and the medical director there must be evaluated, but not through teaching.
It's through. Think of it as operational aspects of the program or other aspects of operating the program besides just being in the classroom. So think about like I'm not in the classroom. What am I doing? So.
One of the things that started putting together is looking at basically putting together surveys. So surveys can be developed basically for faculty based on standard A 2.05, for example. And the question you got to ask yourself is whether that's a question for students, a question for faculty, or a question for your program director to evaluate you on it.
There's no rules for that, but it may be that students may not know some of the things that you do outside the classroom. And again, this is a limited list. You got to remember that there's many things you do outside the classroom, including things like institutional based for example, service or even those in tenure track scholarship. Okay.
But this is just a few things. Like for example, I'm selecting for admissions providing. And actually this is not applicable evaluating student performance among other things. And you can see like academic counseling of students remediation, which I think is a direct kind of kind of. You know, skill that you want to evaluate. And the advocacy of the medical director and the curricular oversight of the medical director.
The question will be is that can students evaluate that or should that be faculty? More than likely it's faculty. But again, that depends on the program.
I just took this from data like from the template. And I made a few comments here just for your purpose. So the suggestion is to start basically in the area basically considered to be a data set. Outside of teaching. So in this one admissions is outside of teaching. But then as you look down here, attrition, PANCE, course grades, course evaluations, instructor evals.
There's really limited number of data sets that are what I would say directly connected to this question as much as others. Okay. So let's look at this again. So data sets for number one admissions data. Absolutely right. You and you. And you have to use the evaluation of faculty outside the classroom you know attrition data. What does that mean in terms of your role said the classroom does PANCE data.
Is that ultimately basically a reflection not as directly course grades, not as directly? Okay. So this is one where it's not quite as easy as it might look to discover. Like what is the way that you can determine that this is compliant because you have to, at the end of the day, have enough evidence. And this is the one that I find programs scrambling to find the information or develop the surveys that are going to help you with that as well.
Okay. So again, look for questions that address faculty effectiveness that may be triangulated. Right. So a faculty effectiveness remediation. And again remediation is outside of teaching. You can look at this one critically as well. Sources can include program evaluation of faculty functions as well. So I have seen some creative things like for example program directors evaluating faculty and annual reviews.
So there's no one way of doing it. That's just kind of a reiteration of what I just talked about. And then some helpful strategies going forward. So as we get into questions two through five, I want to say to you, is that data beyond the templates could include PANCE benchmarking against national average. It could include PACKRAT benchmarking against national average, your composite score, EOC composite score, and even you may or may not want to get down into the nitty gritty of the EOR content areas or the EOC content areas, but again, it's an option for you.
20:18 — Question 2: Admissions
Okay, so I find this at the very least to be effective as a as a data point. So let's look at question number two. And that is admissions. So I find it kind of interesting that admissions data itself is not mandated. But the data is the most critical. And this is not easy because I think a lot of you have discovered that admissions is not very effective.
At least admissions variables is not very effective in predicting student success, right? So if your admissions is ineffective, attrition would theoretically be high. But pitfalls and correlation exist. So again sometimes they don't always, you know, go hand in hand. Then you look at things such as PANCE data. I would look at other variables first before determining that's an area to measure effectiveness I think course grades, early trends, course loads, low grades, especially in the early part of the program might be an indication of that.
Now again, admissions process is really are you selecting the right students who will complete your program. And that's the question you have to ask yourself is what are the signals telling you that. Supplementary points basically courses again an increase. Of course grades below C could indicate an appropriate data points. instructor evals summative exam. Again I would look at some of the examiner patterns.
Probably last exit survey is another potentially fruitful area because there may be questions directly related to admissions there, or things that can help you do to see some additional information. The satisfaction. The satisfaction with faculty could be in a connection, but again, faculty are the ones that are running the admissions process. And then I think that remediation, an extraordinary number of remediation could be an indicator.
So again start with the required data and then start moving basically in proximity with data-wise. And decide whether you're seeing any signals or any data that's leading one direction or the other.
So and then if you've chosen three appropriate data sets and none of these below benchmark, then you can legitimately declare this being compliant. So an evaluation. Cautionary note you know, be cautious about looking at some of the results or PANCE results as the sole reason. And I know there's been literature that ARC-PA has talked about is that it's not just about passing the PANCE.
Be careful about the PANCE being the main data point that you're always looking at. And also remember, folks, that it's not just about knowledge components, it's about other components in your competencies that you have to think about that are measurement of whether students are successful or not. Obviously, they've got to pass the PANCE.
23:48 — Question 3: Didactic curriculum
So let's look at it. Question number three is didactic curriculum effective in preparing. So now you're going to kind of compartmentalize. You've got to think about this. And it's not the clinical phase. Even though didactic phase prepares you for the clinical phase. Don't go to the clinical phase data I mean stick to the phase data first. Right.
Look at variables that are directly attributable. Right. And there could be things such as, you know, preceptor preparedness surveys that might be effective for example. But again, it's not something you're going to look for first. It's not the main underlying theme. So obviously as far as the I mean, the admissions data ineffectiveness of admissions process could be related to poor performance.
So basically, if you truly do have a ineffective admissions process, then you're probably going to have an effective didactic phase. They probably are going to go hand in hand. Okay. Attrition data. And again I want to say that think about the data that you look at, not just the students, but think about basically how the faculty administer the program and how that might impact the students.
Like, for example, of course, grades. It's easy to say that the students aren't studying hard enough, but it may be that there's an inappropriate number of large number of low grades because there may be, maybe, you know, maybe the level of expectations too high. Maybe the students are not prepared. There's a lot there's you got to think on both sides of the coin and look through the lens of the students and the faculty as well.
So again, courses are required. Remediation is also look for large numbers of remediation in specific courses and then exit surveys and other things like sub scores. So again if you think about course evals very deeply, think about as you look at that data. Do you find trends? Do you find specific courses below benchmark for two years for example.
That's fertile ground for looking at related data points that might lead you forward. Okay. Especially if the data point is connected to an organ system or task area. Right. So you want to look basically at other data points that could be related to that. So you're always looking organically for, you know, how might that be impacting a large number of remediation within specific courses could indicate that something is amiss potentially in that course, especially if there's not as many remediation in other courses okay.
So some caveats for evaluation. Number three, avoid looking at questions that is overarching or overlapping with question number five okay. Because it's different a major challenge is when certain datasets data sets are below benchmark. For example first-time taker pass rate. But does that trump all others? And I've come to this conclusion myself. And if you have a low percentage of pass rate, say for a couple of years in a row, and you look at your data and your data all says that your academic year is your academic curriculum is effective, does a low pass rate automatically make the curriculum ineffective?
You got to ask yourself that question, because I don't know that you can say that for sure. Okay. And then because I think that performance is a continuum, you know, didactic outcomes spill over into the clinical year. So I think it's unlikely that you're going to have major issues with clinical year without major issues with the year.
So you look at some of the low performing areas, and oftentimes they spill over and eventually they spill over into first-time taker pass rate. So I know that I'm kind of like going through a lot of different avenues here. But again, I just don't get don't get tunnel vision about that.
28:19 — Question 4: Clinical curriculum
Now we get into clinical curriculum and you're asking yourself basically just about the clinical curriculum first of all. And so again, don't get tunnel vision with first-time taker pass rates. Right. And I talked about the continuum performance already. So I think we already talked about that. I see that as a warning sign. So if there's some low performance and didactic year as you get to, you know, you get to that point, then you're going to basically be seeing that as well.
I think I went backwards I'm sorry folks. Okay. So question number four, you're looking at admissions data unlikely to be a direct connection. PANCE data should not be the first data point to look at. I would look at alignment between your composite scores and PANCE for triangulation course grades many times affects failed grades, right. The number of failed EORs in certain disciplines could be a hint that there could be an issue that could be downstream, it could be related to maybe students weren't prepared.
And again, it's a student's responsibility to look at those templates. But again, you got to think about it as a continuum. And then the course evaluations. Now you're looking at clinical sites and preceptors. And the way that the data is put in is you're looking at the aggregate, yearly averages for each of the disciplines, and then you're looking basically at preceptors that were below benchmark.
And what happened with that. Okay. So we'll get more into that.
So just a few things about number four student evaluation of preceptors. Because the data is aggregated, it's difficult to identify specific weaknesses. I mean it's unlikely just because we know the nature of that. It's unlikely that there's going to be so many low performing preceptors that they're going to come up below your benchmark. Okay. If there's a large number of preceptors that are recorded that also required remediation and if you can connect the students to it somehow in terms of performance, then maybe.
Yes, but that's you're getting deeper into a rabbit hole. Okay. The summative exam can be a direct indicator of the clinical years. So essentially the clinical year is preparing the student for the OSCE, preparing the student for the EOC. So it's kind of a kind of an indicator. Okay. The exit survey may be used if you have alignment of questions of clinical year.
One question or one caveat for you is that whenever possible, make your exit survey connected to your competencies, because then you can start looking at competencies within your learning outcomes in the clinical year, because you have to have those competency based glossary definition terms from ARC-PA and then the faculty effectiveness. Could that be related? I mean, it's possible because maybe the students are not getting effective counseling, remediation, etc..
And then I think remediation really is more about failed during this year, less likely to be clinical remediation or professionalism remediation as well. But and then last thing I'll say is the sub-scores on the EOR may be helpful if you see a systemic weakness in one area. So when I say about not going into a rabbit hole, there's all these sub scores and all the EORs.
I wouldn't do that first. But if I saw a trend like, you know, on the PANCE on the PACKRAT, that there's specific content areas and then you see a composite scores on the EORs that are low, then. Yes. Then I would start looking at maybe there's some content areas that then can be triangulated. And then you could look basically at your didactic for example.
And these are some indicators I just talked about. Just in summary, aggregated data, you know, provides some, you know, specific curricular weaknesses. Exit survey, align clinical competency questions, remediation, monitor the EOR failed clinical years because that may give you a indication of weaknesses and rotation sites. The summative also. So think about the summative. Now you're aligning with your competencies.
So if there's weaknesses in specific competencies that can give you a hint that the clinical curriculum is not effective. And then faculty effectiveness as well. I think that this could be aligned with remediation and non-teaching activities. And I already talked about this already as far as sub scores. So this question basically is really about how the clinical curriculum prepares students for practice.
Okay. So think about the triangulation of data points starting with required. And then basically determine whether they truly indicate noncompliance. You can tell that I'm being cautious about telling you to declare ANIs until you have done your due diligence with this analysis. Right. Because effectiveness extends beyond cognitive components. And so, as I said, clinical reasoning, diagnostic skills, tech technical abilities.
All these things are part of clinical curriculum. And getting tunnel vision with just the knowledge based can get you in trouble.
34:34 — Question 5: Overall program effectiveness
So question number five. And now question number five really is about you know, I've heard it called the 10,000-foot level the overarching effectiveness of the program and preparing for practice. So now you're looking at more kind of like I'm going to say like you're taking a step back and you're looking at the program as a whole now, and you're saying how effective is the program?
And again, there was also literature about it's not just about the PANCE pass failure, right. There has to be evidence that the program is preparing students for all clinical skills throughout the program. So again, as you look at the data points, PANCE is required as is summative required. You can crosswalk these okay. So if you crosswalk EOC with PANCE, if you look at basically the summative components.
Right. And then from there start looking at what bunch of course grades. What about your instructor evals? What about course evals? If there are widespread issues throughout and you're starting to see basically that you've got high attrition, you've got lots of sees, you've got lots of remediation, some of the performance is low. And then your first-time taker pass rate is way below benchmark.
Then yes, you had a question whether you have an effective program. Right. So it's kind of like you know, how many areas are telling you that okay. So I mentioned about exit survey may be a good source about competencies. Remediation may be applicable especially if you think about if remediation is truly ineffective and students are dropping out of your program like flies, then that's an issue.
Again, we can't control remediation if they want to become a fashion designer. I mean, those kinds of things. Other data points like preceptor, evaluation of students, you know, one and two scores may be contributory to this. And so just kind of in summary, these are the things about this particular. And what I wanted you to go to next is the elements of the narrative.
36:52 — Writing the narrative (under 1,500 words)
I obviously can't go through the complete narrative with you, but I want to give you some hits about this. So first of all, I can tell you that I've written several. And by using this process, you can definitely get it under 1500 words. So when you craft it together, remember that an effective narrative presents all necessary analysis. So how do you go about doing that?
One is to prioritize analysis over raw data. I would I would be very cautious about including all kinds of data points in your narrative. If it's already in your templates, okay. You may have some of it that makes a point, right? But if you're if you're taking up space with lots of data points, you're going in the wrong direction.
Presenting the overview of your analysis. And so you can start off by basically talking about basically the overview of your process. And this is an example of question number two. In this case there were five data points to evaluate the effectiveness of admissions process. So you can use kind of like somewhat of a boilerplate approach to this in how you go about doing this a, you know, trends patterns benchmark deviations to determine there's a concern.
Right. And then basically systematically evaluate that and then choose the variables based on their conceptual alignment with admissions process. So you can see how this is kind of written. And then basically from there the next section is your data. So your data points. Then you would write basically a synopsis of the data point. What's important, what's really important about the data that you compiled.
And remember I think what you can play with basically as you get through this is you can start looking at word counts, and if you go over, then you can go back and compress it. But these are the data points chosen. The high points are included like you know below. Benchmark for two consecutive years PANCE data below benchmark you know in 2023.
But above you know the following two years you know so you try to use, I would say, descriptive analytic approach to these kinds of things. And then you conclude with comparisons, trends, triangulations. So now you come to kind of your summation statement right where you triangular the data to determine patterns of low benchmark bindings. Right.
And then you try to define areas needing improvement as three or more aligned data sets, demonstrating below benchmark or a sustained decline in program outcomes. Notice that there's an or there. That's because that there's a little more flexibility now about declaring that it's not just about three data points below benchmark. There could be other things. And I know that there's been some comments from members of the staff, for example, that's like, if there's something that's really bad going on, you better pay attention to it, right?
So if something's going on and there's decline in multiple areas, you got to say there's potentially an ANI here. So I know that I want to make sure there's this time for questions, but I want to just kind of like summarize with this is that you use a systematic evaluation process. You collect and analyze these data points in accordance with C1.01.
So this is kind of like the final determination here. And so it requires programs to engage in systemic valuation of program effectiveness using multiple data points okay. Each selected variable was examined in direct relationship to guiding analytical questions regarding effectiveness in this case the admissions process. Okay. The ongoing action plan refers to the fact that everything is always going trends, right?
So if there's something that was below a benchmark and that was above you continue to evaluate trends, then the final determination really is available. Evidence support the conclusion that admissions process is effectively selecting students. So after all is said and done, you have to say to yourself that the data points in this direction. But if you find enough evidence throughout the process that there needs to be further investigation in one or more areas, then you probably would default to an area needing improvement.
41:56 — Final thoughts and annual evaluation
So final thoughts. This brief webinar really was about salient points regarding the approach to writing C1.01 I wanted to really kind of like give you the following final, you know, perspectives. Think about each question as I statement. So you're either going to support or rebut the compliance. Right. So like that's the way I think of it is that you're going to investigate, analyze and say I think you're compliant.
I think it's not okay. Make sure you always approach each question through a different perspective or lens. Right. Because it's a different question. So you have to you can't just be a cookie cutter approach to this. And so my last statement is I think it's a standards evolution; the rigidity in the fifth edition standards seems to have eased.
Now again, it's early folks, but I'm optimistic that there's that this is a step forward. And think about the five C1.01 questions and two C1.02 questions as a guide for you in the future as to how you annually evaluate your program, because you have to do it annually in order to for this system to work well for you.