There are several reasons for developing an online assessment tool for evaluation of students during their advanced practice year and sharing among the Colleges. (Table 1) The foremost reasons are that there is a limited quantity of preceptors in the State. Many take students from all of the Colleges. An assessment tool that would be shared by all seemed appropriate. There is difficulty in being objective when assessing the learning by a student. Often students are given the benefit of the doubt, and grades are higher than what the student earned. This grade "inflation" is not unique in experiential education, but the process lends itself to the opportunity in ways that do not occur in didactic courses. There are ways to assess the knowledge, skills and attitudes of students and provide them with a grade that is commensurate with their performance. Think of this assessment tool is an opportunity for the preceptor to reflect with students on their learning, rather than an evaluation.

The group of faculty met and reviewed all of the evaluation tools in place at each school. Several years prior the American Association of Colleges of Pharmacy had developed a set of outcomes that they felt every pharmacy student should achieve in order to practice pharmacy. These were called the CAPE (Center for the Advancement of Pharmaceutical Education) outcomes. The group decided to develop an assessment tool that incorporated the CAPE outcomes. The result is an assessment tool that evaluates students learning based on the acquisition of knowledge, building a skill set, and developing attitudes and values that are required of a pharmacist.

The group developed a set of 13 competency statements with related skills. Each of the skill statements has been further defined to provide a definition for the student's ability. These skill definitions are divided into Excellent, Competent and Deficient. (See Figure 1 for definitions) To be a master means that the student knows and goes beyond what is required of them to practice.

Table 1: Reasons for Implementing a Shared Online Evaluation tool:   

Adjunct or clinical affiliate faculty will weight the value of the 13 competency areas for an overall student score. The background part of the online program will then do the math, and calculate a grade based on the information that was input into the evaluation tool, and a total score will be provided.

Competency #1	Design, select, implement and/or manage drug distribution systems for various practice settings.
Competency #2	Disease State Knowledge
Competency #3	Drug therapy evaluation and development.
Competency #4	Monitoring for Endpoints.
Competency #5	Patient Case Presentations
Competency #6	Patient Interviews.
Competency #7	Patient Education/Counseling.
Competency #8	Drug Information
Competency #9	Formal Oral Presentations
Competency #10	Formal Written Presentations
Competency #11	Professional team interaction
Competency #12	Professionalism/Motivation
Competency #13	Demonstrates sensitivity and tolerance for culturally sensitive issues and manages them appropriately.

For complete listing of competencies and related skills, click here .

Figure 1 Sample Competency with Definitions

•How is grading decided?

The preceptor designates how much each area of evaluation is worth in the grading scheme for his/her rotation. The SUCCESS system was put into place to create a more objective, and consistent format for advanced practice experience rotation grading. This does not mean, however, that all rotations are the same, or even that rotations with the same name will be graded identically. We want to be sure that preceptors maintain control over the portions of their rotation that they feel should be given the most weight and ultimately how the grade for the rotation is decided. A sample of a the screen that preceptors will use to assign weighting to the rotation is shown below:

Each competency contains many sub-competencies, which the preceptor will score as it relates to each individual student. The score levels are "Excellent, Competent, Deficient, and No Opportunity." When the preceptor assigns a Excellent, Competent or Deficient to a sub-competency, the grade is weighted internally as a part of the Competency area. Those various sub-competency grades are added together to provide a Competency grade, which is then weighted in accordance with the guidelines set by the preceptor. The result is the student's grade for the rotation, which is recorded in the program for the administrators at each school to see. Once the grade is reported to the school, the student would be able to find out their grade via the normal grade review mechanisms.

Due to the multiple categories of competencies and sub-competencies, there is expectations that grading will logically fall along the familiar A-F scale.

•What about rotations that are not clinically based?

For the time being, we feel that most rotations of this type can still use the competencies with most of the weight for the courses being assigned to competencies 2,8,9,10,11,12,13. Since many non-patient care rotations are primarily project based, these should allow for assessment of performance in those areas.

•Will our grades be lower than students’ grades from past years?

There is a good chance that the average grade for rotations will decrease. This is expected and planned. Rotation grades in previous years have had a very high average, which was secondary to grade inflation and a lack of consistency from rotation site to rotation site. We hope the SUCCESS program will correct for this. At first this may be difficult for the students.

•How will this affect my chances of getting a residency, anywhere in the country, after graduation?

Concern regarding the impact this might have on transcripts and residency/position competitiveness is valid. This new evaluation system was modeled from the ASHP residency evaluation system and is very similar. Since the ASHP residency system is nationwide, it is understandable that this program will be widely viewed as acceptable by most of the nations residency sites. The Colleges using this system will make available to students a letter describing this change in grading systems, which they can include in their application materials. Since this change to a competency based evaluation system mirrors the ASHP residency evaluation system, most residency directors should understand this change.

•How will I know what competencies and skills I will have to master and how much weight each will carry?

It will be the responsibility of the student to initiate communication with the preceptor as to the competencies that will be emphasized at each specific rotation site. This communication should take place early in the rotation to provide clarity of expectations for that rotation. This information will be valuable for the student to focus and organize goals for each rotation site.

•Are all competencies the same, or are some special?

Some skills are classified as "critical skills," and students must show competency in these. Despite achieving competency in all other skills for that main competency area, a student can potentially fail a competency area if they do not achieve competency in the critical skill. Such skills are ones that would produce harm to a patient or to the practice site were the student to perform at the level defined as "Deficient." These critical skills are designated on the complete list of competencies and skills--click here.

•How do preceptors submit evaluations?

Preceptors submit evaluations through an online process in which they evaluate competencies. The calculation of the individual student grade transpires after all the competencies are rated. The preceptor does not submit a direct final grade, as such, but the grade is calculated from the student's performance of the competencies.

•Will the preceptor be aware of the grade he/she is submitting?

The preceptor will be aware of his/her ratings of the student's competencies and of the weight of each competency, but will be unaware of the final outcome of the scoring. Only the student will know the final grade once it has been reported.

•Will set rotations (i.e. Adult Medicine) be equally graded from site to site?

Not necessarily. Each preceptor will weight the competencies for their rotation at that particular site. It is assumable that these may be different, since each preceptor and each site is unique. However there should be similarities within a rotation "type"

•How will assignments and presentations work into the grading system?

Currently, we do not have a mechanism in place to enter raw scores for assignments into the system. The reason for not including a place for these grades is that it makes it too easy to bypass the grading system and thereby invalidate the whole premise of a standardized competency grading system. Preceptors will now think of their assignments as mechanisms to determine competency in the areas that are tested and then translate those assessments into the 13 competencies, as they exist.

•Will I be able to discuss grading with my preceptor?

A preceptor will be able to talk to their students about specific competency areas that need improvement or are performed very well. They will not be able to discuss final grading because this will be reported automatically after their assessment of the competencies. The preceptor will not immediately be aware of your final grade.

•Grade Change’s and Grade appeals: How will we monitor post rotation Grade Change’s and how will the student appeal the grade?

The SUCCESS program will not modify any college policies in place regarding grade finality or appeals.

•Will we evaluate preceptors, as was done in the past?

Yes, students will continue to evaluate preceptors at each rotation site in the same fashion as was done in the past.

•How can I be assured that I will not fail?

This has not changed from previous years. By showing up, working hard, using your time efficiently, and performing tasks that are asked of you; you will likely be successful.

•How can I get an "A?"

As in every other academic endeavor an "A" reflects excellence in academic achievement. The SUCCESS system is in place to allow for this achievement to be rewarded while limiting subjectivity in the process. Communication with the preceptor in order to be informed of heavily weighted skills and expectations is a must in order for the student to be successful.

---