Software Engineering



Course Projects Grading Scheme

Quality and quantity of student project deliverables will be evaluated based on two aspects:

  1. Quality: Knowledge and mastery of course materials, demonstrated through proper use of class concepts, clear communication of ideas and solutions, and professional preparation of project deliverables
  2. Quantity: Estimated project size, measured by software metrics (use case points and cyclomatic complexity) calculated by the student team and verified by the grader

Contents:

  1. Project Quality Evaluation
  2. Project Quantity Evaluation
  3. Combined Grade Assignment
  4. Public Review

1.  Project Quality Evaluation

Aspects of quality evaluation include:

Additional details about quality evaluation can be found on these webpages for Report #1 grading  and for Report #2 grading.

For example, assume that there are 10 project groups with six members each, except for Groups 4 and 6 that have only five members.  Groups 1 and 6 said that “all members contributed equally,” and other groups reported different percentage contribution for different team members. Table I shows the grade distribution after the reports were graded for quality and the points were assigned based on the reported contribution breakdown. The second row (highlighted in yellow) shows the total number of points assigned for the given deliverable as a whole. Each team can earn a maximum of 100 points on different aspects of the project deliverable. Earned points for individuals may be different from their reported contribution breakdown because their deliverable may after quality grading be assigned less than 100 points.

Table I:  Earned points after grading a project-deliverable quality and distributing the points based on the reported contribution breakdown.
 Group 1 Group 2 Group 3 Group 4 Group 5 Group 6 Group 7 Group 8 Group 9 Group 10
Total points 92 60 90 90 83 64 73 94 75.5 72
Student 1 15.3 6 22 19 20 12.8 19 13 13 7
Student 2 15.3 23 33 21 10 12.8 0 10 11.8 17
Student 3 15.3 2.4 3.4 23 3 12.8 14 15 12.5 3
Student 4 15.3 21 1.5 18 23 12.8 14 35 12.8 15
Student 5 15.3 2 0.1 9 11 12.8 5 7 12.4 15
Student 6 15.3 5.6 30 22 21 14 13 15
Average 15.3 10 15 18 14.83 12.8 12.17 15.67 12.58 12
Std.dev. 0 9.5 15.1 5.4 8 0 8.1 9.9 0.46 5.6
Ratio s.d./ avg 0 0.96 1.01 0.3 0.54 0 0.67 0.63 0.04 0.47

To assign the grades in a meaningful and fair way, we need to consider additionally each project size. It is possible to have a neat report for a small and simple project, or to have a poorly written report for an ambitious project. For this reason, we also consider the quantity (size) of each project.

2.  Project Quantity Evaluation

2.1  Use case points

For Report #1, estimate the size of your project in terms of use case points only for the use cases elaborated in this report.
Revise the size estimate from Report #1 in Report #2 to include only the use cases for which interaction (sequence) diagrams are provided and which will be demostrated in Demo #1.

Use case points are described in Section 4.2.1 of the lecture notes and in the slides (also check the Wikipedia page).
At the end of Report #1, show the detailed work for estimating the size of your project in use case points, based on your detailed use cases as well as the domain model.
Students must show step-by-step the process of calculating their project size. Every element estimate must be justified. For example, the number of points assigned to each actor or to a scenario of each use case must be explained.

The “Complex internal processing” technical complexity factor (TCF) should be guessed based on your Mathematical Model (if any is included as part of Domain Analysis in Report #1).
Other TCFs should be calculated only if such features will be implemented in your system. For example, “Performance objectives,” “Reusable design,” “Portability,” etc., should be assigned a non-zero Perceived Complexity only if you can demonstrate that you are actually working on supporting such features in your system.

We assume that all students will work under the same “environmental factors.” Therefore, do not calculate Environmental Complexity Factor (ECF) for your project; i.e., calculate only Unadjusted Use Case Points (UUCP) and Technical Complexity Factor (TCF).
To calculate your Use Case Points, use ECF = 1.

Continuing with the example from Table I, assume that student groups reported their project size in Use Case Points as shown in Table II.  The last row of Table II (highlighted) shows each project size normalized to the largest project, which is the project from Group 1 that has UCP = 116.

Table II:  Use case points calculated by student groups as part of their project report.
 Group 1 Group 2 Group 3 Group 4 Group 5 Group 6 Group 7 Group 8 Group 9 Group 10
UUCP 127 74 93 68 97 76 69 104 85 92
TCF 0.91 0.72 0.97 0.89 1.0 0.88 0.91 0.75 0.8 1.07
UCP = UUCP × TCF  116 53 90 61 97 67 63 78 68 98
Normalized UCP 1 0.46 0.78 0.53 0.84 0.58 0.54 0.67 0.59 0.84

The self-estimated size may be revised by the grader if found to inadequately represent the design described in the report. Use cases that are found to be a slightly retouched carbon copy of existing use cases will be discounted at a 50% rate. (Recall that redundancy/duplication is discouraged!)
If the project size estimate is not provided in a timely manner, the grader will assign it the size 10% lower than any of the reported sizes for other projects in the class.

2.2  Cyclomatic complexity

At the end of Report #2, revise the earlier project size estimate from Report #1, this time to include only the use cases for which interaction (sequence) diagrams are provided and which will be demostrated in Demo #1. In addition, this time your size estimate should use cyclomatic complexity metric applied to your sequence diagrams. The cyclomatic complexity estimate should be used instead of the “Complex internal processing” technical complexity factor (TCF) that was guessed in Report #1 for computing the use case points (as described in the previous section above).  Show the detailed cyclomatic complexity calculation for your project.
Cyclomatic complexity is described in Section 4.2.1 of the lecture notes and in the slides (also check the Wikipedia page).

3.  Combined Grade Assignment

The quality score (range: 0 – 100) will be multiplied by the estimated project size, which will be normalized to the range [0, 1]. The resulting combined score for each project represents its quality score adjusted by its size.

Multiplying the earned points of each student (Table I) with the normalized size of their project (Table II, last row) will yield the final individual grades as shown in Table III.

Table III:  Individual grades adjusted for the project size
 Group 1 Group 2 Group 3 Group 4 Group 5 Group 6 Group 7 Group 8 Group 9 Group 10
Quality points 92 60 90 90 83 64 73 94 75.5 72
Normalized UCP  1 0.46 0.78 0.53 0.84 0.58 0.54 0.67 0.59 0.84
Student 1 15.3 2.8 17.2 10.1 16.8 7.4 10.3 8.7 7.7 5.9
Student 2 15.3 10.6 25.7 11.1 8.4 7.4 0 6.7 7 14.3
Student 3 15.3 1.1 2.7 12.2 2.5 7.4 7.6 10.1 7.4 2.5
Student 4 15.3 9.7 1.2 9.5 19.3 7.4 7.6 23.5 7.6 12.6
Student 5 15.3 0.9 0.1 4.8 9.2 7.4 2.7 4.7 7.3 12.6
Student 6 15.3 2.6 23.4 18.5 11.3 9.4 7.7 12.6

For easier understanding, we could normalize the final individual grades to the range  0 – 100,  based on the highest score (Student 2in Group 3), as shown in Table IV.

Table IV:  Final individual grades normalized to the range:  0 – 100
 Group 1 Group 2 Group 3 Group 4 Group 5 Group 6 Group 7 Group 8 Group 9 Group 10
Quality points 92 60 90 90 83 64 73 94 75.5 72
Normalized UCP  1 0.46 0.78 0.53 0.84 0.58 0.54 0.67 0.59 0.84
Student 1 59.4 10.7 66.7 39.1 65.3 28.8 39.9 33.8 29.8 22.8
Student 2 59.4 41.1 100 43.2 32.6 28.8 0 6.7 7 14.3
Student 3 59.4 1.1 2.7 12.2 2.5 28.8 7.6 10.1 7.4 2.5
Student 4 59.4 9.7 1.2 9.5 19.3 28.8 7.6 23.5 7.6 12.6
Student 5 59.4 0.9 0.1 4.8 9.2 28.8 2.7 4.7 7.3 12.6
Student 6 59.4 2.6 23.4 18.5 11.3 9.4 7.7 12.6

A remaining issue is how to encourage teamwork and at the same time avoid punishing hard-working members of teams with slackers:

My solution is that, in this situation, the grader should review again the deliverables from the groups that reported “equal contribution” but received relatively low scores, and decide if assigning few extra points is appropriate.
On the other hand, the poorly collaborating teams should be advised to make effort and collaborate better.

4.  Public Review

Because project size estimate plays a critical role in grade assignment, in an effort to reduce errors and ensure uniform and fair approach, we will solicit peer reviews of each project’s size estimation.  Software review is a well-known and well-respected practice in the industry, see Wikipedia article.
After each full project report due date, the reports will be made available in a public folder on the Sakai course site.  All students in the class are invited (optional, not required) to review the use-case-point calculation by other teams and notify the instructor and graders about potential errors or ambiguities.
Students are also welcome to make corrections to their own use case calculations if they discover better approaches in the reports by other teams.  In this case, the corrected estimate should include the explanation of what exactly was corrected and in which report this practice was observed.  Corrected calculation without such explanations will not be accepted.
Deadline for comments or corrected estimates:  Five (5) days after the full report due date.


BACK TO:


Ivan Marsic
Sat Oct 27 13:14:05 EDT 2018