Center Assessments
From Center for Friction Stir Processing Guide
Contents |
CENTER ASSESSMENTS
Monitoring performance is a critical part of center performance. In particular, it is important to collect feedback from constituents to improve the management of the center activities and to report progress to all funding entities. Four tools are of primary importance for center assessment: the NSF Annual report, the best practices rubric, the Annual Member's Report, and the external evaluator’s report.
NSF Annual Report
The NSF Annual Report is submitted with a required content to the NSF I/UCRC Program Office using Fastlane (http://www.nsf.gov/eng/iip/iucrc/iucrcannualreport.jsp ). The report is compiled by the CFSP Center Director with inputs from each of the CFSP Site Directors. Annual reports are due 90 days before the anniversary of the center’s inception. At the CFSP, one (1) NSF Annual Report is prepared for the entire center with each site uploading to Fastlane a copy of the report under their separate NSF award requirements. Each site submits the NSF Annual Report on the original award date of the CFSP regardless of the actual award date when new sites are admitted (Appendix O).
The format for the report is somewhat flexible, but the data must include:
- Participants
- What people have worked on your project?
- What other organizations have been involved as partners?
- Have you had other collaborators or contacts?
- Activities and Findings (Keep this short)
- Describe the major research and education activities of the project.
- Describe the major findings resulting from these activities.
- Describe the opportunities for training and development provided by the project.
- Describe outreach activities your project has undertaken.
- Publications and Products
- What have you published as a result of this work?
- Journal Publications
- Books or other non-periodical one-time publications
- What web site or other internet site have you created?
- What other specific products (databases, physical collections, educational aids, software, instruments, or the like) have you developed?
- What have you published as a result of this work?
- Contributions
- The principal discipline(s) of the project?
- Other disciplines of science or engineering?
- The development of human resources?
- The physical, institutional, or information resources that form the infrastructure for research and education?
- Other aspects of public welfare beyond science and engineering such as commercial technology, the economy, cost-efficient environmental protection, or solutions to social problems.
- Special Requirements
- A brief summary of the work to be performed during the next year of support if changed from the original proposal.
Since the CFSP is a multi-university center, each Site Director collects information about local activities and sends it to the Center Director. The Center Director is responsible for compiling the information from each site. The CFSP NSF Annual Report is organized as follows. Note that data provided in the report is separated by partner university sites where possible.
- Major Accomplishments for this period
- Research Goals
- Short Description of Communication with Center Members
- Project Selection Process used by the Center
- Quantitative Information
- Number and diversity of students
- Number and diversity of faculty and senior personnel
- Industrial members
- Degrees granted to students involved in center activities
- Amounts and sources of income, patents, licenses, and papers created
- General Center Identification Number
- Year of initial funding
- Center Director
- Partner University Site Directors
- Operating Budget and total funding
- Capital and in-kind support
- Industry Membership Descriptors for the current award
- Directors Descriptors
- Center Director
- Site Directors
- Center Outcomes
- Students receiving degrees and type of degree earned
- Students hired by industry by type of degree
- Publications with research center
- Publications with IAB Members
- Number of presentations
- Intellectual property events
- Invention disclosures
- Patent applications
- Software copyrights
- Patents granted/derived
- Licensing agreements
- Royalties realized
A relatively recent addition to the I/UCRC toolkit is a spreadsheet created by North Carolina State University (NCSU). The spreadsheet provides an alternate data collection mechanism for compiling the information needed for the NSF Annual Report. Use of this spreadsheet is not mandated, but it provides numerous pre-defined summations and calculations to reduce the burden on collection of the required data from multi-university centers. However, the most effective use of this tool requires significant advanced planning. If each Site Director can fill out the relevant Section and forward the form to the next director, the Center Director’s job is noticeably easier. If all Site Directors fill out their portion and individually send them to the Center Director, the same level of effort is required to provide a complete summary of the data. The size of the worksheet also makes it difficult to view and print.
At the present time, no mechanism exists for automatically compiling the NSF report from the site university reports or for combining the individual spreadsheets into a single report. Such a management tool would remove the need for tedious and error-prone cut and paste operations. Examples of the type of summary data available from the NCSU spreadsheet are shown in Appendix Q.
Best Practices Rubric
The Independent I/UCRC evaluator has surveyed the Center and Site Directors and compiled and published a list of “best practices” from the responses. The CFSP has taken those best practices and created a visual tool for displaying center progress toward implementing those practices. A sample is presented below. A simple color-coding scheme quickly identifies best practices which are in place and working effectively (green), have not been implemented or are not fully implemented (yellow), or have not yet been addressed (red). This rubric is incorporated into the semi-annual State of the Center document prepared by the Center Director.
Annual Member’s Report
As noted in Section 4, two major reports are prepared during the year. One is the NSF Annual Report described above and the other is the Annual Member’s Report. The NSF report is prepared for the Spring IAB meeting and includes the External Evaluator’s report prepared at the end of the previous year. Thus, the 2008 NSF report details center activities for 2007 and includes the evaluator’s assessment of the center through 2007. The Annual Member's Report is not required by NSF but has proven to be an effective tool for communicating center progress to all members, particularly to industry partners. The Annual Member’s Report is prepared for the Fall IAB meeting, distributed to the attendees, and posted on the web site for all members to access. The report is several hundred pages and is bound into a book for ease of use and archival purposes.
A standard format for the report is used which contains the following:
- Program Overview (Section 5.3.1)
- FSW Technology Development Roadmap (Section 5.3.2)
- Project Master Schedule (Section 5.3.3)
- Project Executive Summaries (Section 5.3.4)
- Current Project Annual Reports (Section 5.3.5)
Program Overview
The program overview summarizes the objectives of the CFSP, lists the current year’s projects, and includes Sections 5–12 from the NSF Annual Report. This includes center demographics, a description of industry members and center personnel, budget and funding information, center outcomes, and intellectual property events.
Technology Development Roadmap
The Technology Development Roadmap Section contains three planning tools: the graphical representation of the Technology Development Roadmap, technology development needs chart, and LIFE form summaries.
The Technology Development Roadmap:
The Technology Development Roadmap is used to discuss the direction the center is taking. The roadmap was developed at the creation of the center and provided a multi-year plan of attack for addressing the issues known at the time in friction stir processing and friction stir welding. The document must, of necessity, be dynamic and reviewed on a regular basis. The discussion of the Technology Roadmap is part of the State of the Center presentation and the revised version becomes part of the Annual Member’s Report.
A sample of the roadmap graphic is given in Section 2.1.2. Note that it defines a matrix which grids basic research, applied research, manufacturing technology, and industrial implementation on one axis against a timeline for developing specifications and standards, design guidelines, an educated workforce, and low-cost equipment and tooling.
Initial Technology Developments Needs List:
The initial technology developments needs list was developed at the inaugural IAB meeting for the center during the planning grant phase. This is a list of development areas deemed critical to accelerating the integration of friction stir welding and friction stir processing into industrial environments. At the same time, the university or universities which would take the lead in each area were selected. Each year, progress in each area is assessed and assigned a color code to visually display the level of achievement.
The areas identified as “green” are considered to be adequately addressed under the current and proposed projects. Those areas identified as “yellow” are being addressed – but additional efforts are desired. Those identified as “red” are not being adequately addressed under the current program planning. In addition to revising the assessment of the areas each year, the document must be updated as new members join the center. This is accomplished at the fall IAB meetings.
LIFE Form Summary Lists:
The LIFE form summary lists each project and the level of interest indicated by center members at the last IAB meeting. As noted in Section 4, the CFSP generates a summary of LIFE form ratings for each project, provides the opportunity for each project lead to respond to the LIFE form comments during the IAB meeting, and reserves time during the meeting for each sponsor to meet with the project leads to discuss the LIFE forms and the responses.
Industry partners have also asked for a summary of the small group discussions to be presented to the entire group. This will be incorporated into the next IAB agenda. The Annual Member's Report contained the LIFE form summaries. A complete set of LIFE form ratings/comments and project lead responses are given to the center members at the IAB meeting.
The extent to which the current and proposed programs meet the needs of the IAB Membership is reflected in the interest level of the LIFE FORM analysis from the Spring IAB meeting. VI = Very Interested; I = Interested; IWC = Interested with change; NI = Not Interested; A = Abstain. A Section of the Annual Member’s Report a summary is presented below.
Project Master Schedule
The Project Master Schedule conveys several pieces of information about projects in a concise graphical format. Not only are project initiation dates and duration mapped to the timeline, but the status of each project is also indicated by color-coding. A sample timeline is presented below. Note that completed projects are easily identified as green, current projects are in gray, proposed projects are designated as pink, and canceled projects are coded as yellow. A timeline is prepared for each university site in the center.
Note that TIE projects with non-center members are also indicated on the timeline. The timeline also indicates projects funded entirely by NSF such as REU, RET and Supplemental Awards (Section 3.6).
The timeline is reviewed at each IAB meeting and updated as needed. The typical project duration is three years, but as indicated in the example, some projects are completed in a shorter time, and some are extended. However, the center members maintain a three year horizon when planning projects and timelines.
Project Executive Summaries
The format and contents of individual Project Executive Summaries were described in detail in Section 4, and an example template is shown in Appendix L. The Annual Member’s Report contains a compilation of all Executive Summaries of current and proposed CFSP projects. The Executive Summary is a brief description of the objective of the project, the approach taken, and the progress made in the past year. LIFE form evaluations (Section 4.2.2) are completed as part of the Executive Summary presentations.
Current Project Annual Reports
The annual report for a project follows the same format as the quarterly report described in Section 4, but, as the name implies, it covers progress made on the project for the entire year. It is an extensive technical discussion of the work completed in the previous year and planned for the next year if the project is continuing. The template for the annual report (Appendix M) is somewhat flexible but should, at a minimum, include: - an introduction to the project - the objective of the research - a description of the methodology/approach used - experimental data - analysis of the data, including graphical representation of results - a summary and conclusion - description of future work
Annual reports are generated for every on-going project and every project completed during the past year.
External Evaluator’s Report
The NSF requires a formal evaluation of the center to be conducted by an independent evaluator. The duties of the evaluator are defined by the NSF I/UCRC Program Office: There must be an independent evaluator who cannot be from the department within the institution receiving funds for the I/UCRC award. The Center evaluator is responsible for: (From: National Science Foundation Evaluation Project (1997) Evaluator’s handbook: National Science Foundation Industry-University Cooperative Research Centers Program (Rev. ed.). Raleigh, NC: North Carolina State University.)
- Preparing an annual review of Center activities with respect to industrial collaboration during the previous year (which is appended to the Center’s annual report to NSF);
- Conducting a survey (using an instrument that will be provided by NSF to all Centers) of all Center participants to probe the participant satisfaction with Center activities;
- Compiling a set of quantitative indicators determined by NSF to analyze the management and operation of the Center;
- Participating in the IAB and any other relevant meetings;
- Performing exit interviews to determine why members chose to withdraw from the Center; and
- Feeding information on the quality of the industry/university partnership to NSF and back to the Center for continuous improvement.
The external evaluator attends the IAB meetings and prepares a report after the fall meeting. The fall meeting concludes center business for that year. The evaluator also surveys industry members and includes their feedback in the evaluation. A blank example survey is included in Appendix P. The evaluator’s report is included with the NSF Annual Report described above, is discussed at the Spring IAB meeting, and is available on the web site as a part of the documentation for the spring meeting.
The required contents of the report can be found at http://www.ncsu.edu/iucrc/EvalReport.htm. In summary, the report must include an overview of the center, its goals and objectives, changes at the institution or in the industrial environment which impact the center, center organization, the research program, center accomplishments, analysis of center progress, and a timeline for events that have occurred at the center.
