The Citizenship Project at CACPS
Citizenship Project: A Little Theory
Citizenship Project: Initial Results
Citizenship Project: 2nd Year Results
The Citizenship Project at CACPS
Codman Academy Charter Public School, the high school I helped found in 2001, regards the development of character as equal in importance to the development of academic skill. This goal is easier stated than met, of course, especially in an inner-city high school where students often display disruptive behavior for a large number of (often understandable) reasons. The School's faculty are old hands at classroom management, but they also want students to take responsibility for their own behavior, academic as well as personal. Beginning in 2004, they devised a technology-supported project called Citizenship to provide an environment fostering both personal and community growth. In addition to weekly reflection in single-sex advising groups, and the requirement to state intentions for character development at the year's beginning, the Citizenship project also states explicitly the behavior expected -- and not expected -- of community members. This article describes how the Citizenship project also uses technology to track student behavior in detail, giving both faculty and students the information needed to foster civil behavior in and out of class.
Deltas and Kudos
The Citizenship project began in the fall of 2004 with a list of infractions compiled by the faculty after discussion of the actions most disruptive to learning and teaching at the School. Each infraction, called a 'delta,' carries a (negative) point value that indicates the infraction's level of disruption. The faculty also identified a base set of positive behaviors, called 'kudos,' each with a (positive) point value. The following table lists these behaviors for the 2007-8 school year. (Earlier years had somewhat different lists -- the choice of deltas and kudos is a key configuration decision.)
Improper Technology Use
Minor Disrespectful Speech/Behavior
Cutting Class/Unexcused Absence
Rough Housing/Play Fighting
Serious Disrespectful Speech/Behavior
Inappropriate Sexual Behavior
Saturday Class Tardy/Unexcused Absence
Together, these deltas and kudos would be compiled to generate a numerical score for each student, who would begin each semester with 500 points. Citizenship scores would be published on the School's intranet where they would be available to everyone in the community -- students, family members, administrators, faculty, tutors. The scores would also be reported as averages for each student advising group, encouraging competition among groups and mutual support within them. Students and advising groups with high Citizenship scores would be publicly rewarded at the School's weekly Community Circle. Those with low scores would perform various extra community services (e.g., lunch cleanup). The example at right shows the listing of student Citizenship status by adv ising group. (The CACPS advising groups are called 'crews,', and are named for traditionally black colleges.)
Citizenship Technology Support
The Citizenship point system thus designed could be accomplished entirely on paper, as similar systems have been implemented in the past. Collecting, compiling and communicating enough data to be useful, however, would be a daunting task, even for a small school like Codman Academy. Each Codman faculty or staff member, however, has a personally-assigned laptop, used to support record-keeping as well as writing and research. Teachers, staff and family members enjoy instant access to timely web reports of attendance, course assignments and grades. It seemed natural, then, to build a parallel, technology-based approach to behavior tracking and reporting.
The Citizenship project's technology support1 consists of:
- a MySQL database that holds the information and makes it available over the internet,
- a client application for each staff laptop that communicates securely with the database, allowing teachers and staff to add, edit and delete behavioral information any time and place they have internet access,
- a client application for each student laptop that reports the student's personal behavioral record in detail, and
- a reporting engine, part of the School's intranet website, that reports each student's current Citizenship score, listing by name, by score, and by advising group.
Using this technology, teachers can quickly record incidents of constructive and destructive behavior by their students and even by entire classes or advising groups. Every entry can also (and often does) include a descriptive comment. Staff members also are encouraged to use the system to record disruptive or exemplary behavior wherever it occurs on school grounds (and on off-campus activities).
Administrative staff and faculty can review the up-to-date list of these events for any student at any time. Students are able to examine their own record of behavior whenever they wish. At right is a portion of a real record; kudos are listed in green, deltas in red. (Student and staff names have been blurred to protect privacy.)
These records prove invaluable in meetings with students and their families, since they present a cumulative view of a student's behavior in all classes. 'Deltas' (negative behaviors) and 'kudos' (positive behaviors) can be examined, revealing patterns. Does the student show disruptive behavior only in a particular teacher's class? Are staff always reminding the student about gum-chewing? Does the student make frequent offers to help other students? Much more information is available to help encourage civil behavior by every student, and to identify problems for intervention.
Although the current behavior score is listed for each student on the School's intranet site, teachers and students can also see a graph of behavior score versus time since the beginning of the semester. (Students can only see their own graphs.) Here's an example:
The graph offers a perspective shorn of detail, a useful aid in seeing how things are going for the student. This view helps locate hopeful progress to be encouraged, as well as disturbing patterns to be addressed.
Faculty members can also review a list of every behavior-management incident they themselves have recorded, across all the students in all their classes. Am I doing a good job of rewarding positive behavior? Is there a behavior I'm having to correct much too often, a behavior I should be working on explicitly with students? Am I attentive to the behavior of all students, not just the disruptive ones? How does my monitoring of behavior compare with my fellow teacher's? Again, the information is there to help teachers analyze and improve their class-management skills.
The Citizenship client also enables an administrator to email complete behavior reports weekly to the student and to designated parents, guardians, or supporters such as pastors or big brothers/sisters. This feature allows other key adults to support the improvement of student behavior and the recognition of problem situations; it also ensures that the student cannot claim to be ignorant of a doubtful record of behavior.
The Codman Academy Citizenship project has become an integral part of school life, and is strongly supported by faculty, staff, and families. Students have had no difficulty adapting to a new way of setting goals and recording progress. They talk about kudos and deltas as often as they do about grades, and are generally as aware of their behavioral standing as they are of their academic standing. No doubt there will be further modifications and improvements to both the design of the project and the technology that supports it. But the Codman Academy Citizenship project is definitely here to stay.
The Citizenship Project: A Little Theory
The Citizenship project at Codman Academy Charter Public School aims to teach students the behaviors expected in a civil learning community, and to quench behaviors that are disruptive, dangerous or illegal. (See The Citizenship Project at CACPS in this section.) A key element of the program is a system that tracks events of both positive and negative behavior, accumulates a status score for students and advising groups, and makes this information readily available to students, faculty and families. My article The Citizenship Project: Initial Results suggests that such a system can indeed improve student behavior. And, at this writing (November, 2007), the CACPS faculty has continued and extended the tracking system into its third year. Still, it's worth asking the question: Is there any theoretical basis for implementing such a system?
In his book Smart Schools: Better Thinking and Learning for Every Child, David Perkins of the Harvard Graduate School of Education states what he calls a "Theory One" of learning:
"People learn much of what they have a reasonable opportunity and motivation to learn. "
He continues by listing four minimal conditions for creating "a reasonable opportunity and motivation to learn:"
- Clear information. Descriptions and examples of the goals, knowledge needed, and the performances expected.
- Thoughtful practice. Opportunity for learners to engage actively and reflectively whatever is to be learned -- adding numbers, solving word problems, writing essays.
- Informative feedback. Clear, thorough counsel to learners about their performance, helping them to proceed more effectively.
- Strong intrinsic or extrinsic motivation. Activities that are amply rewarded, either because they are very interesting and engaging in themselves or because they feed into other achievements that concern the learner.
Although Professor Perkins has much more to say on the subject, I myself have never needed more than Theory One to analyze a learning environment and evaluate its likelihood of success. (For example, large college lecture courses often rate very poorly on the last three conditions.) How does the CACPS Citizenship program, and its behavior-tracking system, stack up when viewed from the perspective of Theory One?
In adult society at large, we assume that what constitutes "appropriate" and "inappropriate" behavior is generally understood; it need not be stated explicitly. High-school students, however, are not yet adults; they are learning the rules of conduct. Codman Academy students, in addition, are urban young people of color from varied cultural backgrounds. (Five home languages are represented among Codman students, for example.) Many students deal daily with difficult family or living circumstances, and all are more accustomed to the vernacular of city streets than they are to the speech and conduct expected in the school and indeed in the adult environments the students aspire to -- business, law, medicine, education. In sum, Codman Academy students face two learning challenges: learning to behave as adults, and learning to meet when needed the standards of an unfamiliar educated, professional culture. The Citizenship program acknowledges this challenge by stating explicitly the behavior expected, and not expected, of members of a civil, adult, professional society: the "goals and the performances expected." In addition to addressing the first condition of Theory One, this approach has other advantages:
- Consistency. The Codman faculty realized that they were not always treating student conduct in the same way -- from one teacher to another, and from one day to another. By agreeing on lists of positive and negative behaviors, they sought to be clear and consistent about goals and expected performance, both for themselves and for students.
- Focus. Stating too many learning goals at once can overwhelm learners and encourage teachers to pick and choose among them. Both faculty and students have found it very helpful to identify just a few behaviors -- the most common disruptive behaviors, the most desired productive behaviors. More, in this case, is not better. (See the list for 2007 in Tracking Student Behavior: The Citizenship Program at CACPS).
Students at Codman Academy soon understand that they are always practicing appropriate behavior. Teachers monitor and note their behavior in classes; staff, tutors, adjuncts and teachers monitor and note their behavior everywhere on school grounds and anywhere the students participate in school activities. (Freshmen are astonished to find that kudos and deltas are still being recorded at the school's 3-day camp in the fall.) Students are aware of earning points -- or losing them -- throughout the day. It's also customary for Codman teachers to pause for the last few minutes of class to have the students reflect on their behavior as well as their academic progress. Consistent expectations and the event-tracking and scoring system make active, reflective practice of behavior just as widespread and educationally powerful as active, reflective practice of academic skills.
The Citizenship program's tracking and reporting system itself supplies informative feedback to students in several forms. First, instances of infractions or productive behavior comprise a cumulative record or log of the student's behavior during the term. Students can view their own record at any time, and it is also emailed to parents or guardians once a week. Often the entries in the behavior logs include comments by the teacher to clarify the reason for the entry -- when and how the student was disruptive or exemplary.
More importantly, however, the tracking records make 'clear, thorough counsel' much easier and more productive for others to give. Teachers and advisers have the data needed to observe patterns of behavior that need attention. Parents have much more information to reference when it's time to talk with their child about behavior as well as academics. Awards for good citizenship can cite specifics when students shine as community members. And disciplinary hearings with students and parents can proceed on the basis of facts rather than hearsay.
The ultimate goal of the Citizenship program is a community in which good behavior is recognized as its own reward. Getting to this state with adolescents, however, requires some attention to extrinsic motivators. Some students are competitive about their status score or the average score of their advising group. They want to be the best. In fact, as I've noted elsewhere, the status score opens a route to success for many students who struggle academically. Other motivators include recognition at the school's weekly Community Circle, with privileges (e.g., off-campus lunch) for high achievers and community chores for those with low scores. Satisfactory behavior as well as satisfactory academic performance is required for participation in athletics, a regulation that may be the most effective of all the motivators. And of course there is the regard of significant adults and peers, which now can be based as much on evident citizenship as on evident scholarship.
It seems to me that the elements of the Codman Academy Citizenship program, and particularly its technology-supported system for tracking student behavior, align well with Professor Perkins' requirements. Can the design be improved? I'm sure it can. But the present program''s theoretical basis is strong enough to justify optimism. At least I think so.
The Citizenship Project: Initial Results
Codman Academy's Citizenship project and its technology support (described in The Citizenship Project at CACPS) had its first full year of implementation during the 2005-2006 school year. This article describes key results from program, based on data gathered through the point-system technology using the client application Citizenship Status which I wrote.
Codman Academy Charter Public School is a small public charter high school in Dorchester, one of Boston's poorest neighborhoods. Selected by lottery, its 110 students in grades 9-12 encounter an environment quite different from the schools they have previously attended. School hours are 9-5 Monday through Friday, and a half-day on Saturday for enrichment courses. Students, wearing uniforms, make use of educational resources throughout the city, taking Friday each week for activities off-campus. (For example, 9th- and 10th-graders spend two Fridays a month at Boston University's Huntington Theatre, where they study writing, stagecraft, and the season's plays; the other two Fridays offer varying off-campus science and math days.) All of the students are from minority backgrounds; about 80% are eligible for free or reduced lunch (although the school provides a free lunch to all students anyway); 16% have identified special education needs; five non-English languages are spoken in students' homes. An energetic and innovative faculty and staff set high standards for both academic and behavioral performance using an expeditionary, project-centered approach to learning and teaching. To learn more about the school, its educational philosophy and programs, see http://codmanacademy.org .
The Citizenship project quickly established itself as an integral part of life at the school. In 2005-2006, students began each semester with 500 points, and accumulated behavioral records from teachers and staff members as the weeks passed. Both positive and negative points (then called 'Status' points) could be assigned according to the student's constructive or disruptive behaviors. Students could not accrue less than zero or more than 1000 status points. Point totals were reported continuously to the school community on its intranet, where they were listed alphabetically by name, by range (0-399, 400-599, 600-1000), or by advising group average. Once a week, students with status scores in the lowest range were given extra tasks and other disincentives, while students in the highest level received public recognition and extra privileges.
So, how (and how well) did the Citizenship program work?
Overview of Results
Implementation was easy:
Since the Citizenship point system arose from the faculty's own thinking about how to manage student behavior, only a few tweaks on the faculty software for posting events were needed to get the system running smoothly. Students and families were already used to looking up grades and attendance data on the School's Powerschool server, so the addition of behavioral data was easily accepted. Almost from the start, students discussed Citizenship scores and grades (their own and others') with about the same frequency.
New student stars appeared:
The Citizenship program created a new opportunity for demonstrating excellence, and a number of students quickly established themselves as stars by gaining and maintaining scores of 1000. One mother wrote the Head of School to praise the system, saying that her child had always struggled academically but could now 'be the best' at something important.
Status records proved helpful in student and family consultations:
Teachers and administrators have found it very helpful to have the equivalent of every teacher's gradebook available in PowerSchool when talking over a student's academic performance. The Citizenship point system provided a corresponding complete record of the student's behavior which has proved equally helpful. It's easy to identify areas for improvement as well as areas to be celebrated. It's also difficult for students to wriggle or argue or dismiss, as they so often do when confronted on their behavior.
Two types of student distinguished themselves:
Every semester, the originally flat distribution of Citizenship status scores (everyone at 500) quickly developed two peaks at either end of the scale. Some students worked hard to gain points through constructive behavior; a somewhat smaller group distinguished themselves with frequent and continuing disruptive actions. Although students could be found with scores across the whole range, clustering students into three ranges produced a 'bimodal' distribution in all three semesters of 2005-2006 (see the figure for an example from the second semester).
Girls and boys showed different behavior profiles:
Perhaps not surprisingly, the CACPS girls behaved better than the CACPS boys -- in general. Individually, both girls and boys starred or sank, and every point range had both genders represented. However, the pattern for girls was consistently skewed to the high end of the scale, while that for boys was skewed to the low end of the scale (see the figure for an example from the second semester).
Special Education students showed special behavior:
How did the 18 students with IEPs (Individualized Educational Plans) distributed themselves? The results for the third and final term are shown below:
Notice that none of the Special Education students remained in the middle range as the term progressed. Also, the girls clearly took advantage of the opportunity to demonstrate constructive behavior, while the boys were evenly divided.
Student behavior was generally consistent from semester to semester:
Although individual students had very different final status scores from one semester to the next (high to low, low to high), in general student behavior relative to peers remained remarkably consistent. The correlation of status score from semester 2 to semester 3 was +0.69, judged a medium to high correlation for the behavioral sciences. (Note: Calculating correlations rests on an assumption of a normal distribution, in this case, of status points. We know that's not the case! Nevertheless, counting individual movement from one status range to another supports a significant degree of consistency. Sixty percent of the students stayed in the same status range and an additional thirty percent moved up or down just one range.) Here also, girls and boys showed different levels of consistency. Again using semester 2 and 3 data, the consistency for girls was +0.72, while that for boys was +0.59.
Still, certain behaviors improved markedly and the overalll number of infractions decreased:
As discussed below in A Closer Look, student behavior for certain infractions improved sharply from semester to semester. For these infractions, we can assert that student behavior did improve with the use of the Citizenship program. In addition, the total number of infraction entries declined from term to term. The pattern of behavior -- the number of students in each status range -- did not change much. Individual students did show remarkable improvement (for example, 4% of the students jumped from the lowest to the highest status range), but most students showed no improvement in comportment relative to their peers.
Student behavior and grades showed a correlation:
Students in grades 11 and 12 (for whom the School has a cumulative GPA) showed a correlation of about +0.60 between GPA and final status score in both semesters 2 and 3 of 2005-2006. Remembering again that the correlation calculation is only approximate for these data, it's useful to take a look at the data itself. (See the example below from semester 3, which includes a least-squares fit linear trend line.) Although GPA doesn't seem to be a very good predictor for average students, high GPA (> 3.0) correlates well with high status score, and low GPA (< 2.0) with low status score.
We've been looking at summary data so far -- final Citizenship status scores for a term, cumulative GPA. The data available, however, permits much more detailed investigation. Each student's event trail is individually available. So is each teacher's trail. What infractions are most frequent? Do teachers use the system the same way? Do students behave the same for different teachers? The data bears a closer look.
A Closer Look
Looking more closely at the data, we can answer some more detailed questions: Which infractions are most likely? Are kudos (positive points) given as often as deltas (negative points)? Do all students receive about the same attention, or are the majority of entries related to just a few students? Are all teachers equal users of the system? Are teachers generally even-handed in their assignment of kudos and deltas, or are there major differences among faculty members? Answers to these questions might help the School improve the Citizenship point system and the student behavior it seeks to track.
Kudos vs. Deltas
In the first term use of the Citizenship point system, the Codman Academy faculty became concerned that deltas exceeded kudos, and resolved to pay more attention to good behavior. The system software was also altered so that entire classes could be recognized for good (or bad) behavior with one action. The table below shows results from the second term of 2005-2006. The first column in the table shows the number of entries made for each type; the second column shows the total number of points added or subtracted by these entries.
Recognizing that it's perfectly possible for students to engage more often in one or the other kind of behavior, these figures suggest a relatively even-handed approach by teachers.
The distribution of behavior record entries made in the second term is shown below for the full-time classroom teachers at CACPS. (Teachers of working sessions and Saturday courses are not included.)
Point system use by teacher
Teachers clearly did not use the point system with the same frequency; the number of entries made during the term ranged from 18 to 513. The relative balance of kudos (positive point entries) to deltas (negative point entries) also varied widely from teacher to teacher, from about 1/3 kudos to 2/3 kudos. A very similar distribution results when all classroom teachers are included. Would the effectiveness of the system have been different if teachers used it more consistently?
Frequency of Infractions
The Codman Academy faculty created a list of infractions they deemed most disruptive to learning and teaching at the school. The chart below shows the number of Citizenship point system entries in which each infraction appeared. (The same infraction sometimes occurred multiple times in one entry, which the teacher ordinarily makes at the end of class or later from notes. An entry showing 3 counts of gum-chewing would only count 1 in this measure, then.)
A few observations:
- Students were pretty well-behaved, on the whole. None of the serious infractions (at the top of the chart) had many entries; most had zero. Students concentrated their energies instead mostly on the standards (gum-chewing, tardiness) or the annoying (dress code violations, minor disrespect).
- The most troubling infraction was 'Disruptive or Disengaged.' Students with this behavior can't possibly be learning when they exhibit it, and often they make it impossible for other students to do so, too. The other frequent infractions have far less impact on teaching and learning.
Infraction Entries Over Time
As noted above, 60% of the students remained in the same status range from trimester 2 to trimester 3, paralleling a high correlation between student scores in these two terms. This result could be taken to mean that the Citizenship point system as configured did not improve student behavior. It's essential, however, to see how the number of infractions changed over time, from trimester 1 to trimester 3. After all, it's possible for the total number of infraction entries (and thus infractions) to decrease, while the pattern (some students sinking, some rising, some saying put) remains the same. In fact, there was a small decrease in total infraction entries, as shown here:
|Total Infraction Entries||5808||5632||4315|
Such a decrease could have other causes than improving behavior, of course. What happens if we look at individual infractions? Did the number of entries change, and if so, how? The chart below shows how the total number of infraction entries changed by term for the most frequent iinfractions:
Consider Minor Disrespect (cut by 75%), Tardy or Absent (cut by 30%), and Serious Disrespect (cut by 50%). These are very encouraging results. On the other hand, the most troubling behavior, 'Disruptive or Disengaged,' actually increased. A mixed picture, but not without bright spots.
How well did the Citizenship point system work during the school year 2005-2006? On the positive side , it did a good job of providing data useful to teachers and students in assessing behavior. It also seemed to quench certain disruptive behaviors, although more data would offer needed confirmation. The point system gives students, some of whom do not excel academically, a new way to demonstrate excellence. And finally, it sets forth explicitly, and emphasizes consistently, the behavior not wanted at the school, and in fact not wanted in adult life.
On the negative side, the results display a troubling pattern where some students sink, through misbehavior, and do not manage (or do not care) to climb out of the hole they have dug. Also, recording the behavior required extensive, diligent work by teachers; nevertheless, the faculty decided at year's end to continue the point system. Finally, for behaviors like gum-chewing that are seemingly unaffected, the point system demonstrated that the incentive structure was not reinforcing enough to limit minor misbehavior, especially when it has immediate, pleasant consequences.
The Citizenship Project: 2nd Year Results
The school year 2006-2007 saw the second full implementation of Codman Academy's Citizenship program and its system for tracking student behavior. (See The Citizenship Program at CACPS.) What observations seem to hold true now that there are two years of data available? How do the two years compare? Are there any new patterns that look significant?
Changes in the 2nd-Year Citizenship Project
In its second year, the program was no longer a novelty or experiment, but a normal and expected part of school life. Even the ninth graders had been introduced to the program during a summer 'boot camp.' The program could therefore be considered to be mature in its implementation.
Although the aims and structure of the program remained the same, there were a few changes in detail. Most importantly, the faculty decided to re-define several infractions to focus attention on the most important and to permit more clarity in expectations and accuracy in reporting.
- 'Disruptive or disengaged' were separated into two infractions, 'Disruptive' and 'Disengaged,' to better distinguish events which only affected one student's learning from events which could affect many students' learning.
- 'Food' and 'Gum' were combined, in recognition that they were essentially the same kind of behavior.'
- 'Lying/Dishonesty' was reclassified as a form of 'Serious Disrespectful Speech/Behavior.'
- 'Complaining' was reclassified as a form of 'Disrespectful Speech/Behavior.'
- Rare mis-behaviors ('Harassment' and 'Vandalism') combined minor and serious degrees into one infraction.
- Minor misbehaviors ('Smoking Tobacco' and 'Littering' were eliminated entirely from the list of infractions.
Overview: Key Year 1 Results in Year 2
The following table lists some key results from the first year of Citizenship; the column at right indicates whether or not the same result was found also in the second year.
|Year 1 Finding||Year 2 Result|
|Implementation was easy.||Same|
|INew student stars appeared: students took advantage of a new way to demonstrate excellence. .||Same|
|IStatus records proved helpful in student and family conferences.||Same|
|ITwo types of student distinguished themselves: the distribution of status scores developed peaks at either end of the scale.||Two peaks appeared only in term 1; terms 2 and 3 showed a peak only at high status, with a flat distribution below.|
|Girls and boys showed different behavior profiles: girls predominated at high status, boys at low.||Similar results, although not as pronounced.|
|Students with IEPs showed different behavior: final status scores either high or low, none in between.||Not checked.|
|The status pattern was generally consistent from term to term; overall behavior didn't improve much.||Not true: behavior in 2nd and 3rd terms improved with a skew to the high end.|
|Student behavior and grades showed a correlation.||Not checked: status distribution is so abnormal that it's difficult to draw conclusions.|
Further comparisons are made difficult because of the changes in the Citizenship system between the two years. In particular, it's not useful to compare the frequency of infractions, since the list changed considerably. (The pattern showing the majority of reported infractions to be minor ones, with essentially no major ones, did persist, however.)
I think the most important of the new results is the emergence of a status score pattern with a peak only at the high end, after an initial term with the previously-encountered high-low dual peak pattern. The graph below shows the term-end status pattern for all three terms of 2006-2007. Students in terms 2 and 3 were able to adjust their behavior enough to achieve a relatively flat distribution in the lower status divisions.
Zero-status and Thousand-status Students
The CACPS faculty have been quite concerned about the students whose behavioral scores steadily decreased until they pegged at zero. For students who spent a significant fraction of the term in that position, the tracking system became ineffectual because continued infractions made no difference to the score (although the infractions did register in the students' records). The faculty concluded that the penalties for remaining at zero score for a period of time should be increased, and they instituted a new rule that detention would automatically be assigned to such students.
It's likely that some of the results graphed above reflect this change, although low-scoring students could still avoid detention by maintaining a small status score (as some did). The table below shows the striking change that resulted in the number of term-end zero scores. Notice also the increase in the number of students achieving a final status score of 1000. Both of these results encourage me to think that tracking and reporting student behavior can make real differences in the quality of student life.
|Score||Term 1||Term 2||Term 3|
Included here are additional data graphs that I don't think require further discussion.