FINAL REPORT

ALBRIGHT-WIRTH EMPLOYEE DEVELOPMENT FUND PROGRAM EVALUATION

For

The National Park Service and The National Park Foundation

March 12, 2004

Emmalou Norland Cindy Somers

e-Norland Group

1425 P Street NW #714 Washington DC 20005

1-202-797-2023 (Phone & FAX) emma@e-norland.com

A. INTRODUCTION

The Albright-Wirth Employee Development Fund Program In 1987, the National Park Service (NPS) and the National Park Foundation (NPF) established an endowment fund in the name of former NPS Director Horace M. Albright to provide grants each year to qualifying employees for their professional development. The first year, 28 employees received grants for a total of $69,289. In 1995, as a result of a generous donation by Laurence Rockefeller in the name of Conrad L. Wirth, the fund became known as the Albright-Wirth Employee Development Fund (A-W). Throughout the years it has had many successes but as all programs do, it has also experienced a number of challenges. In 2003, the National Park Service National Capital Training Center requested an evaluation be conducted. The following three issues were highlighted as precipitating factors for the evaluation. Decreasing Applications

Most recently, the numbers of applicants appear to be declining. For example, applications numbered163 in 2001, 135 in 2002, and 99 in 2003 and although numbers were up in FY 2004 (174) the trend has still been downward. One wonders about interest, relevance, park support, marketing, or other issues that may be affecting participation. Incomplete Projects

Another puzzling phenomenon is the inability of many grantees to complete projects in a timely fashion. Not only does this produce administrative problems for grant management but the grantees most likely run into problems on-the-job when agreed-upon timelines must be changed. Changing Internal Processes A third consideration for this program goes beyond the program itself and is better framed as a strategic alignment and management question. With major changes occurring within the NPS in the training arena, how does the leadership see this program aligning within the overall areas of training and leadership development? What implications will this have on the management of the program and the application process?

Purpose of the Evaluation

The purpose of the evaluation was formative in nature, providing the organization information on how the program can be improved. The specific type of formative evaluation conducted was an Implementation Evaluation which provided information about whether the program is being implemented as it was designed and if not, why not, and how changes can be made.

3

Objectives of the Evaluation

1. Determine applicants’ perceptions of the program (including issues outlined above) 2. Determine selection panel members’ perceptions of the program (including issues outlined

above) 3. Determine NPS and NPF leadership associated with the program perceptions of the program

(including issues outlined above) 4. Using perceptions and historical data, determine any gaps existing between intended program

implementation and actual program implementation (including issues outlined above) 5. Provide recommendations for program improvement based upon results of Objectives 1-4

Methodology

Because the request was for a fast track evaluation, there was no stakeholder group assembled to guide the evaluation as is the usual practice for evaluations conducted by the e-Norland Group. The evaluation process was guided by Victoria Clarke-Wood, NPS, and Marjorie Thomas, NPF. The standards by which the evaluation was guaranteed, as always, were the Standards for Educational Evaluation (1975): Utility, Feasibility, Propriety, and Accuracy. [See www.eval.org for more information on these standards]. Instrumentation Qualitative Telephone and Face-to-Face Interview Schedules were developed based upon issues identified in the background description above. There were different interview schedules for each of the following groups: applicants receiving grants, applicants not receiving grants, selection panel members, leadership, and applicants receiving grants but not finishing projects. Questions were reviewed by Victoria Clarke-Wood and Marjorie Thomas for content and face validity. Suggested changes were made and all interview schedules were deemed appropriate for use in the evaluation. Data Collection and Analysis

Telephone Interviews were conducted during February 2004 by Cindy Somers of the e-Norland Group with the following persons: grantees from the last 3 years who had and had not finished projects (n=12), applicants who did not receive grants (n=4), and selection panel members from the last 3 years (n=5). Names were selected and provided by Victoria Clarke-Wood. John Tyler, Deputy Chief of Training, NPS and Albert Werking, former Deputy Chief of Training, NPS, were also interviewed via telephone. Face-to-face Interviews were conducted with the following persons: Julia Washburn, Vice President, Grants and Programs, NPF, Marjorie Thomas, Program Manager, NPF, Sandy Taylor, Training Manager, Supervision, Management, and Leadership Team, National Capital Training Center, NPS and Victoria Clarke-Wood, Grants Manager, Supervision, Management, and Leadership Team, National Capital Training Center, NPS.

4

Interviews were not recorded. Questionnaires were sent ahead of time and respondents were asked to briefly write their responses in preparation for the interviews and then return their written responses after the interviews. During the interviews, notes were taken and data were then analyzed using the respondents’ written responses and interviewer notes. Additional data were obtained from historical documents from the National Park Service and the National Park Foundation including program announcements and letters regarding the program.

Using This Document This remainder of this document contains key findings and supporting data, conclusions, and recommendations. It is organized as follows:

CENTERED HEADINGS are general categories of conclusions which were derived from initial data analysis.

1. Numbered Items are conclusions drawn from findings and supporting data.

• Bullets are key findings usually with supporting data (direct quotes from

documents and interviews appearing in italics). NOTE: Some comments were paraphrased to protect confidentiality or other reasons. These are not italicized.

The recommendations, which appear at the end, are suggestions and ideas for decision-makers in both organizations to reflect upon. Recommendations stem from single and grouped findings and conclusions.

B. THE NPS-NPF PARTNERSHIP (Roles, Participation, Communication)

1. Roles and participation of the partners have changed since 1987.

• Original responsibilities of the partners are outlined in the 1987 Program Announcement:

“The NPF solicits outside contributions in coordination with the Director, manages the interest-bearing account, makes annual grants of HAEDF monies to the service” “The NPF works with the Director, the Division of Employee Development and an NPS project management team of six NPS employees recruited from a cross-section of job classifications and grade levels.”

5

“The NPS project management team (currently, the selection panel) determines categories of developmental experiences to be available, develops program criteria and guidelines, reviews employee applications and recommends recipients to the Director, recommends program changes, evaluates program effectiveness, completes all program reporting requirements” “The (NPS) Division of Employee Development serves as the Service’s representative responsible for overall program management and administration including providing a project manager, providing program administrative support, publicizing the developmental opportunity to all employees, conducting operational liaison with the NPF”

• Partner responsibilities, as defined in the 1995 Program Announcement: “In partnership

with the National Park Foundation, quality control systems are provided by the National Park Service. This process, facilitated by the Employee Development Division in Washington DC includes the development of operational criteria; publication of beneficiaries; evaluation of projects, and the monitoring of conformance to legal mandates.”

• Currently NPF manages the endowment, invests the funds, and annually transfers total

amount used for grants and NPS implements and manages the program. NPS Leadership 2. Public communication about the partnership has decreased and division of

responsibilities has changed.

• “The National Park Service and the National Park Foundation are announcing the establishment of an endowment fund in the name of former Director Horace M. Albright to provide grants each year to qualifying employees...All employee grants are awarded by the foundation.” and “As a lasting tribute to one of the Service’s most influential directors, Horace Albright; the National Park Service (NPS) and the National Park Foundation (NPF) have established an endowment fund in his name...” There is additional information regarding the role of NPF. - 1987 Program Announcement

• “Through effective cause-related marketing promotions, the National Park Foundation is able to channel critical outside resources into the Nation Park System for specifically-targeted projects and programs. The National Park Foundation is a 100% private, non-profit organization dedicated to helping meet the needs of the National Park System. The National Park Foundation manages the Albright-Wirth trust fund. The interest accrued from the trust fund is used to support the projects that are selected each year. Contributions to the Fund are appreciated. Interested employees should send their tax deductible donations to: [address of NPF] - 1995 Program Announcement

• ”Special appreciation is extended to the National Park Foundation for their continuing support of this program. For information on the National Park Foundation, visit their website www.nationalparks.org.” - 2004 Program Announcement

6

• During evaluation interviews, only one panel member and one grantee acknowledged (without being asked) that NPF plays any role in the A-W program.

• The 2004 recipient notification letter signed by the NPS Director contained no mention of

the NPF partnership.

C. THE ENDOWMENT (Growth, Restrictions on Use)

1. The endowment has not grown to the extent intended.

• “The endowment fund will eventually supply an endowment of approximately $450,000 per year for development grants to employees.” - 1987 Program Announcement

• Original letters from contributors indicated varying amounts to be raised. - Rockefeller

letter • In 1988, 28 employees received grants, for a total of $69,289. - NPS Documentation

• Currently, the funds available each are based on a formula:“6% of the 3 year average

value of the endowment = ~ $120,000-140,000 year” - NPF Leadership

• Data from NPS documents indicate the last four annual A-W grant totals have averaged a little less than $134,000 suggesting a growth much less than the original goal of $450,000 annually. FY2001, 80 employees for total $140,599 FY2002, 50 employees for total $139,693 FY 2003, 49 employees for total $142,241 FY2004, 36 employees for total $112,298

2. An original restriction placed on the funds when the Wirth portion was added is

currently not acknowledged.

• The letter, conveying initial funding for the “Connie Wirth portion of the Albright-Wirth Fund”, designated that that particular portion of the fund is “devoted to Ecology and Nature Study...as Connie requested.”

3. Beyond the above restriction, there are no other restrictions on the fund which

can be determined from original documentation. Evaluation participants suggested new uses for the money.

7

• Evaluation participants suggested separate pots of money to do different types of projects and to focus on teams and parks as well as individuals.

“I think anybody should be able to apply, but I believe it’s already difficult enough for individuals to get it, so if they do something like that [go beyond individual awards] it should be a different pot of money and one shouldn’t take away from the other.” - Grantee “If they want to do this [award the grants to different configurations] then maybe they could rotate how they target the different audiences...” - Grantee “I think the reality of it is that at the [national level] we are being encouraged to link with partners either inside or outside the agency. So it behooves them to seriously consider partnerships, or teams, etc. as an option. I think it’s a great idea.” - Grantee “I think a group grant would be nice. Communication classes would do our whole office good. In those classes there are 20 people so what difference does it make if they are from the same park or different parks?” - Grantee “If there is something a team can do, then that’s appropriate. That’s moving into the modern world – and that’s something the Park Service needs to do.” - Grantee A group of employees might apply to develop and offer workshops on particular topics to other NPS employees. - Grantee “I think it would be great if a group of people want to get together and do something together. I’ve always thought of professional development as an individual thing, but if a group of folks can develop an appropriate proposal, why not?” - Grantee “Perhaps it would be appropriate if a supervisor could apply on behalf of a work team.” - Grantee “[How about a] large park project that cannot find any funding sources through the fee collection project. Half of the professional development I’ve had in my career has been working on projects where I’ve learned new skills while developing a product for the park.” - Grantee A-W’s schedule of ‘once-a-year awards’ causes people to plan a year in advance and doesn’t allow employees to take advantage of opportunities that come up with shorter notice. “Perhaps one solution to this would be to accept applications and make awards throughout the year.” - Grantee “I love the idea of teams, or groups getting awards. This is an area where seasonal and term employees could be involved. I would think also that (teams/groups) might do a better job of reporting and finishing projects on time as a multitude of skills and personalities could be drawn upon to do the right thing.” - Panel Member

8

“We saw a lot of people who wanted to complete their Masters or Ph.D.s, but not all of them did a good job of showing how this would benefit the Park Service. You kind of wish there was a second fund for college education. The other side was the basics – basic computing skills, literacy, G.E.D. etc. So, it was kind of hard to fairly judge those “higher” level requests because of the competition with those folks who were just after the basics and just wanted to grow.” - Panel Member “I think a greater focus on the group or team project approach will probably hurt the small parks – unless they went together with other parks. Maybe they should have a second grant or separate pool with a different amount of money for groups. I think there should be encouragement for groups to apply – but not compete against those who are applying individuals.” - Panel Member “More and more parks are being re-formulated into networks of parks and are supposed to work together. So, team projects seem to fit with this approach - as long as it’s not for basic training – the parks should commit to doing that.” - Panel Member “I like the team idea.” An example might be two people from different parks or even teams made up of folks from different disciplines, get trained as a team and then provide training to others as a team – the individuals couldn’t do this by themselves. Plus, Parks are beginning to work more closely together and this is good. Since regional offices are being de-emphasized, park-to-park networking is becoming more and more important. Allowing or encouraging teams to apply would reflect these changes. - Panel Member “We should use the program more strategically for the Service. There’s information available about what gaps are out there – both from the Foundation and from surveys done by the Park Service – maybe this program should be targeted toward filling those gaps. If this was done, then the marketing could be better developed and targeted appropriately.” When asked how this type of restructuring would fit with the original mission and purpose of the grant, she responded: “It’s a fine line to walk – to accomplish the spirit of the original grant and fulfill specific needs for the Service – but it’s worth consideration. Maybe we could set aside a part of the funding or for a certain amount of time to focus on specific Park Service needs.” They have gotten a lot of requests over the years for employee development, but they can’t raise a lot of funds for this through the private sector. But, they do have this pot of money. So, maybe this could be more focused on meeting needs of the training community in the Park Service. Or it could be used to fund the NPS employee development needs related to the Foundation’s four program areas. - Leadership “In addition to individuals, maybe we can have calls for specific training – perhaps related to the Foundation’s program areas – that are then offered to groups of NPS employees to attend – field staff in particular. The grant would cover the costs associated with developing and providing the workshops and for employees to attend the workshops. A Park could send a group (3-4) to attend or maybe even provide the training at the park if there are enough people to make it worth it. The people who are out there with the visitors are not as aware of this as they should be.” - Leadership

9

“I think it’s still meeting the original mission, but I think it can be adjusted and revisited and changed to adapt the current situation. This goes back to tying it into the Foundation’s program areas and being able to reach more people through an emphasis on group projects - especially if we can better show how something like this will benefit their park and/or a large number of employees at the same time.” - Leadership “These are very tough times for the parks with minimum staff and no travel budgets, need to look at other approaches to the grant program – review what are other agencies are doing to promote grant programs. Look at running the application cycle year round. If we have an open-continuous announcement, and then we would have certain times that we look at the applications or even that could be on-going. That is potentially scary, because it would require (potentially) major changes in program management. Right now it would be very difficult, if not impossible, to dedicate additional personal – a process would have to be figured out for it to work.” - Leadership Provide more money for travel. Congress has restricted travel, so even if there’s money available for training, the participant has to be able to cover travel costs – or find a way to do it where travel isn’t necessary. This is especially a concern at small parks – or in remote locations. - Leadership “I don’t think it’s good to deviate from the original spirit of the endowment, but I think we can use it more strategically.” - Leadership “We should probably have categories for teams – and maybe even partnership teams. If we’re looking at the future in government, then we should encourage this and market this to applicants” – maybe have a category only for partnership efforts. - Leadership Team or multiple person projects should be options, but too much focus on this approach could lead to the program taking on more of an organizational focus rather than the individual focus it currently has. - Leadership They should set aside a portion of the Albright-Wirth funds to develop training programs for groups of people. The funding for this would be separate from the ‘traditional’ individual grants – maybe put a $100,000 cap on the individual part and use the rest for group projects.” - Leadership “Perhaps they could set aside pots of money for three or so specific categories and then market to the appropriate people.” - Leadership “If the program has categories, maybe they should relate to where we want to be in the future – such as distance learning – technology related projects, etc. Maybe the Albright-Wirth program can be used as a way to test out some of our ideas on how to provide training. Maybe we should look at providing more basic skills training – and remember that sometimes the folks who need this kind of training are not out there looking for them.” – Leadership

10

D. PURPOSE AND MATCHING CRITERIA (Description, Changes, Interpretation)

1. The original focus of the fund was to benefit National Park Service personnel.

When the program was actually implemented, the program added that a benefit to the NPS as an organization should be considered as well.

• The fund was described as being “a special fund in his [Horace M. Albright] honor to

be used for the benefit of the National Park Service personnel. Such a fund is most appropriate because of Horace’s tireless efforts over nearly 70 years on behalf of employees of the National Park Service...” - Original letter from Director Mott (NPS) to John Bryant (President, NPF) establishing the fund

• When actually implemented, the program added that a benefit to the NPS as an

organization should be considered as well. - 1987 Program Announcement 2. Applicants do not know the history of the A-W Employee Development Fund.

Two people claimed to know who Albright and Wirth were but nothing related to the grant. Another grantee suggested, “I think Albright and Wirth are a couple of professionals who think Park Service people need professional development.” One person tried to find out the history on the Internet and “could not find anything”. No recipients of the grant who were interviewed knew the history of the grant.

3. Although the description of the balance between individual and organizational

benefits has been fairly consistent throughout the life of the program, interpretations and implementation have varied.

• The purpose, as specified in the 1987 Program Announcement, was for professional

development that resulted in benefits for the individual and subsequently for NPS.

“A key criterion...will be the benefit to the development of the individual and subsequently the Service’s ability to fulfill its mission.” – 1987 Program Announcement

• The purpose, as taken from the first sentence in the 2004 Program Announcement,

“The Horace M. Albright-Conrad L. Wirth Grant Program provides financial assistance to National Park Service employees for their professional development.” – 2004 Program Announcement

11

And further on in the announcement...“The rating criteria are based on the applicants stated interest in their professional development, as it relates to the quality and continuity of National Park Service programs, including the management and interpretation of agency resources.”

• During selection, panel members have varied in the way they implicitly place value on

individual and organizational benefits.

“I think the distinction is lost in what is appropriate for Albright-Wirth to pay for and what should be funded by the parks. There’s lots of wording about New and Innovative, vs. Fundamental. It’s important for the parks to pay for the basics and for the Albright-Wirth program to focus on those things that are beyond the basics.” The participant gave an example about a park that had purchased a new HVAC and was looking for funding to train three employees on how to properly operate it. The panel felt the Park was clearly responsible for paying for this training and should have figured it in when deciding to purchase the new heating unit. However, the participant fluctuated on this a little regarding lower grade employees. - Panel Member One panel member feels professional licensing fees might be a bit much and also has some concerns about the money being spent to fund work projects rather than enriching the person participating – but realizes that sometimes these are closely connected.

When asked about a 1995 directive to “Fund those proposals that show the greatest promise for ‘multiplication of results’,” one panel member responded: “I like this one a lot. It just makes sense to have money spent in a way that would affect the most people. I remember a few proposals while on the panel that just seemed so right and it was around this premise that I supported them so strongly.” - Panel Member “Training should be designed to meet individual goals and should allow someone to dream and hope and have a chance for their dream to come true. That’s what’s really going on. People might be isolated frustrated about how to make things better and at least the people who are applying to this program are trying to find a way to do that. And that should be encouraged. And if not with money, at least with other types of support and encouragement. It can mean a lot to people.” - Panel Member There are many programs available to fund professional development (outside of the required competencies and programs), and most if not all the other programs have are “very organizationally oriented, whereas the A-W program allows individuals to focus on their personal goals.” The program “must maintain its individual focus.” Projects should somehow be related to an employee’s career goals but not necessarily to their current assignment. The bottom line should be “is the individual, because of this opportunity, going to be a better employee for NPS.” - Leadership “People are attracted to it because the program is specifically intended for individual use as opposed to meeting departmental goals. As it is right now, they can apply for just about anything they need for enrichment – it’s more personal.” - Leadership

12

“We should use the program more strategically for the Service. There’s information available about what gaps are out there – both from the Foundation and from surveys done by the Park Service – maybe this program should be targeted toward filling those gaps. If this was done, then the marketing could be better developed and targeted appropriately.” When asked how this type of restructuring would fit with the original mission and purpose of the grant, she responded: “It’s a fine line to walk – to accomplish the spirit of the original grant and fulfill specific needs for the Service – but it’s worth consideration. Maybe we could set aside a part of the funding or for a certain amount of time to focus on specific Park Service needs.” They have gotten a lot of requests over the years for employee development, but they can’t raise a lot of funds for this through the private sector. But, they do have this pot of money. So, maybe this could be more focused on meeting needs of the training community in the Park Service. Or it could be used to fund the NPS employee development needs related to the Foundation’s four program areas. - Leadership “In addition to individuals, maybe we can have calls for specific training – perhaps related to the Foundation’s program areas – that are then offered to groups of NPS employees to attend – field staff in particular. The grant would cover the costs associated with developing and providing the workshops and for employees to attend the workshops. A Park could send a group (3-4) to attend or maybe even provide the training at the park if there are enough people to make it worth it. The people who are out there with the visitors are not as aware of this as they should be.” - Leadership

4. During grant selection, the extent to which an emphasis is placed on funding unique professional development opportunities rather than projects reflecting customary NPS training is unclear.

• Originally the program was meant to provide “opportunities beyond the reach of regular

NPS training funds and, in no way, replaces normally appropriated training or any other program funds.” - 1987 Program Announcement. This statement still appears in the 2004 Program Announcement but is not used as a screening or formal selection criterion.

• Participant quotes:

“People apply to this program because of the individuality. Some people think more of the professional development and training programs should be like this so people can choose their own method of training. A lot of the funding provided by the Park Service is available for certain people to do certain things and if you fall outside of those lines, you can’t access the money. People are attracted to it because the program is specifically intended for individual use as opposed to meeting departmental goals. As it is right now, they can apply for just about anything they need for enrichment – it’s more personal.” - Leadership

13

A-W is set apart from other NPS programs in that it “allows for an individual approach that’s still applicable to the job. That out-of-the-box idea. There are many ways to develop knowledge and skills related to any career or job, and the A-W program allows for a more creative approach than the other standardized courses offered by the government.” - Panel Member “I think we need to get back to the original purpose of the grant. Too many people are looking for money to go to school. Too much emphasis is on getting a degree as opposed to more skills-related training, innovative ideas, doing something rather than getting a certificate. I think the AWG has moved too far away from looking for inventions, creating things with/from the earth, and the development of mechanical apparatus. I think the AWG should be looking more at the actual preservation and conservation of ‘things’ the NPS is entrusted to preserve and conserve e.g., artifacts, wildlife, and curatorial treasures.” - Leadership

5. The extent to which the grant money reaches as many employees as possible is

unclear but the process for accomplishing this original goal has been inconsistent.

• The original intention of the grant program was “to reach many employees at all levels of the organization rather than a select few.” “...at any level, in any job series and at any NPS park/office/Center...” The 1987 Program Announcement. This intention has been described similarly but not necessarily applied consistently during the selection process.

• There is no limit on the number of years an individual can apply for or receive a grant. • Half of the grantees interviewed in this evaluation had received more than one grant. • Panel members who were interviewed indicated that prior recipients were treated

differently. (See data in panel section below.) • Panel members who were interviewed indicated that GS level was a consideration during

selection. (See data in panel section below.)

• Eligibility for A-W applicants has changed.

The original 1987 program announcement indicated that employees who were eligible to apply for a grant were those “with five or more years of NPS experience, including temporary seasonal appointments.” Temporary seasonal employees are currently not eligible to apply but may evaluation participants felt they should be. “My personal opinion is that someone with five or more years of seasonal work should be allowed. Maybe you pick a certain number or percentage of seasonal employees. Many folks have 15 years or more experience trying to get a permanent position.” - Panel Member

14

“Here is an area I would like you to look into – having a category for seasonal employees. Just the name makes them sound impermanent but in fact there is a large number of employees who are seasonal for many, many years and something like one of these grants could make a direct step towards a permanent job. I would like to champion the seasonal cause, as getting a permanent job in the Park Service is very difficult. Some of our best and most creative employees are seasonal some by choice and some not. They are often treated like second-class citizens and this is an area of recognition that could go a long ways to building morale. Also are term employees eligible? If not why? We have many of them in our Park and I would like to see their acknowledgement also.” - Panel Member “With the focus on outsourcing – this is a reality, we work them like permanent employees, yet they aren’t eligible – maybe they should be considered if the development clearly relates to the job their doing with the Park Service.” - Leadership

6. Employee renewal and the development of fresh perspectives, an original intent of A-W, is not currently identified as a selection criterion.

• The original 1987 Program Announcement describes “[In the spirit of Horace Albright]

grants enable employees to renew their personal commitment, gain new insights, and learn new skills for a fresh look at their NPS careers.”

• The renewal aspect is not currently a formal, explicit criterion for project selection nor

was it identified as an implicitly-used criterion in panel interviews.

One of the program leaders expressed strong feelings about this aspect of the program stating: “This is exactly what it should be and I have resisted efforts to change this focus.” One year, science and scientific research was of great concern in the Service and the suggestion was made that the A-W program should focus exclusively on science that year and only fund science-related projects. This individual “actively opposed this idea,” emphasizing that that was not the purpose of this particularly grant program.

7. Descriptions of project selection criteria appearing in the program

announcements have varied throughout the years.

• The original selection criteria, as appeared in the 1987 Program Announcement, were “designed to help insure the nomination/selection of employees who have (1) demonstrated a genuine interest in their personal development, (2) identified a specific operational need to be aided by the experience, (3) shown a tangible value to the Service for project results, (4) committed themselves along with their supervisors/managers to specific application(s) of experience outcomes for park operations, and who (5) are judged most likely to reinvest their new experience into continued service with the NPS.”

15

The 1987 Program Announcement continues with a more specific list of the original criteria used by that screening committee:

1. Activity Purpose/Objectives: What are the proposed results of this activity? 2. Identified Need: What specific employee/organizational need(s) will be helped by this

activity? 3. Benefits: What is the extent and perceived value of activity results? 4. Products: What is the perceived value of any tangible products resulting from the activity? 5. Plans/Methods: To what extent do activity plans and methods help insure activity results? 6. Sharing: To what extent can activity products/results be practically shared with others? 7. Budget: To what extent is the size and nature of the costs appropriate for HAEDF support? 8. Evaluation: What response did the activity proposal receive fro the reviewing Service-

wide program manager(s)?

A key selection factor is that of “mutual benefit.” Grant awards from the National Park Foundation will be based on the anticipated benefits to NPS operations as well as to the individual – both will have to benefit.”

• Selection criteria are mentioned and/or described in the 2004 program announcement.

“Be sure to address the following criteria in your application: What the goal or objective of the project is. How the project will be accomplished or developed. Why the project is creditable.”

8. Current criteria in the program announcement and formal, explicit criteria used

during project selection do not match. • When asked what the selection criteria were (during an interview with members of

NPS/NPF leadership in February, 2004), the following were constructed from the rating sheet the selection panel used during the 2004 grantee selection:

1. Activity/Project Description: Tells a complete “win-win” story which has a logical

sequence of events. 2. Budget: Reasonable budget in which the expenses are justified. 3. Time Frame: The applicant is likely to achieve/accomplish the proposed

project/activity within the 1-year time limit. 4. Personal Commitment: The level/amount of the applicant’s prior or on-going

personal commitment of time or resources. 5. Benefit to the Individual: Measurable, clearly-defined positive outcomes for the

participant. 6. Benefit to the Park Service: Measurable, clearly-defined positive outcomes for the

Park Service. 7. Supervisor/Management Team Support: The level of support from the applicant’s

management team and/or a partner/organization having some relationship to the project.

16

9. The selection panel has used implicit, self-defined criteria in addition to or in place of current, explicit criteria given to them. (See data in the Selection Panel and Process.)

10. NPS employees applying for the grant do not know the criteria to be used in

selection. • When asked how to increase the number of applicants, one grantee suggested: “provide

an explanation of how the proposals will be scored – what are the specific criteria.” • Those applicants interviewed who did not receive a grant were asked if they knew what

the selection criteria were. Their responses were: “No, I don’t think I do know the criteria. If they are written somewhere, “I must have overlooked them because I am not aware of any criteria.”

“No, I don’t know what the criteria are.”

“I don’t have a clue what they are. There are some sort of very vague statements in the application. If they’re not anything other than that, that’s a problem. They need to be better than that.” And another person said that s(he) didn’t remember anything about them and could remember if they were listed anywhere or not.

• When asked about the effect of the supervisor on the review process, non-recipients said:

“Who the supervisor is may have an effect on whether a proposal is funded. It depends on who’s on the panel.” Another person doesn’t think supervisor support has any bearing on the outcome of the application and review process IF the supervisor DOES support the individual. However, “If your supervisor does not support you then there is no sense in applying.” “If there are problems with a supervisor, an applicant should have a way to work around this and still apply...especially since it’s not coming out of the supervisor’s budget.”

• When asked what the criteria should be, these same individuals suggested:

Give it to a person “who is willing to invest some time themselves into the project...those who are willing to make a sacrifice before asking the park to make a sacrifice.” Some of the projects that were funded “seemed to provide very little benefit for the costs incurred. While it is nice to provide employees with training opportunities, I think there needs to be more benefit to the NPS.”

17

“The project also has to clearly benefit the park as well as the individual.” The criteria “probably need to be directly related to what the grant is supposed to be for. So that it’s not about OTHER people’s development – not about funding [park projects]. Specifically about professional development.” “The criteria should be based on whether or not the project is something the person is directly responsible for or doing in their present job. How beneficial it will be for the Federal Government and how it’s going to benefit the person. It should be connected to the person’s current job – if it’s not, then it should be put on a second pile.” (S)he’s not aware of what criteria were used to judge the application, but (s)he thinks one of them was the contribution the applicant would make to the project. This was fine because “If they’re willing to give me funds, then I’m willing to sacrifice as well.”

11. Panel members and leadership varied in the extent to which they felt the criteria by which the applications were judged were explicit, clear, and fair.

• Evaluation participants said the following: “I don’t remember all the criteria – most of it was very clear, especially from the employee development side – in terms of how the employee met those criteria were sometimes a little less clear.” - Panel Member “They gave us very broad guidelines to approach the ranking, but then left it up to us to create specific criteria for the ranking - to the point where there was no real valuing approach. We had to create our own rating sheet. We were given an introduction and told there were 5 categories in which people could apply and we they gave us general comments about how the project should have value to the employee and to park service. But very little was really said about how we should or shouldn’t rank people.” “I wanted to see clearer definitions of what should and shouldn’t be considered – more of a rubric of what criteria to apply and how to apply them.” - Panel Member “A rating guide was given with clear explanations.” - Panel Member “Yes, I believe the criterion was appropriate. I found it all easily useable and understandable. There were a lot of applications to look through so the given guidance was appreciated. I did not spend a lot of time questioning the process. On a gut level it seemed fine to me.” - Panel Member “I felt there needed to be more structure because there are always panel members that are more dominant than others.” Since so much was left up to the panel, then the more dominant members had a lot of effect on what happened. “In the end, I feel there are factors that are inappropriate that went into the selection.” - Panel Member “To me, the criteria, at the time, were easy to apply.” - Panel Member

18

“The criteria should be revised by a small panel. I think it needs more. I’m not sure what that it is; something that is more conforming to the competencies” which is the direction all the training courses are moving in. “It needs more structure: needs to be more scientific,” especially regarding the competencies so that “if we were audited, we could give them more measurable criteria – the current ones are more soft – we need ones that are harder.” - Leadership

E. GENERAL PROGRAM MANAGEMENT

(Planning, Marketing, Solicitation, Application Assistance, Budget Management)

1. The sequence of events for the operation of the A-W program is not tied to a

specific time line.

• The timing of implementation of program components varies from year to year. • The amount of time between applying for a grant and hearing about the results of the

selection process is not shared with applicants. • Applicants do not receive communication between application deadline and notification

about selection (or non-selection). (They don’t know what comes next.) • There is not a set of established benchmarks (dates) to use to judge whether the program

is being implemented in a timely way.

2. Marketing of the A-W program is limited and perceived as insufficient.

• Word-of-mouth was the most frequently mentioned method for learning about A-W.

“My supervisor had received a grant and told me about it.” - Grantee A mentor told this person: “I honestly don’t know how I would have known about it otherwise.” - Grantee “I’ve known people who have applied since my early years.” I know about it through “Word of mouth.” - Grantee “Probably word of mouth from my supervisor.” - Grantee “The education specialist told me.” - Grantee “I heard about it from a professional peer.” - Grantee

19

• Other locations and methods for how grantees learned about (or could learn) about A-W included Inside NPS, e-mails, and the NPS website.

“I think I learned about it from Inside NPS.” - Grantee “I would love to see it more prominently advertised on the communication networks... Inside NPS – in big, broad letters, and not just one day when the announcement comes out, but across the entire course of the application period.” - Grantee This person recalls that there “used to be a flyer sent out, but it may be all e-mail now. And that’s an issue; they need to remember those who don’t have e-mail.” - Grantee “Through a posting – either by e-mail or other notification.” - Grantee “There should be a note on the Morning Report.” - Grantee (S)he assumes it is on Inside NPS, and thinks it should be on there. “From a remote duty worksite, it’s really the savvy people who are going to know about or search this out.” “I think the Park needs to get more involved. I think that’s the key – supervisors and superintendents need to get the word out and encourage people to apply.” - Panel Member “Put it on the front page of Inside NPS – It’s coming, why is it important, let them know it’s effective. Let managers know it is beneficial and not threatening – not taking them away for lots of time and not costing them lots of money.” “Most employees don’t know how to get to Learning Place and they don’t care about Learning Place. Need to get it right in front of their faces – they won’t dig for it. Use the Arrowhead – they’ve got to keep using the printed press.” - Panel Member “I actually hear very little about the program. I think there should be a better job done in this area. Reminders could be sent to employees in mass on the E-mail system. My only concern is that there are a number of employees who are not strongly computer oriented and will miss this message.” - Panel Member “I think the training community really tries – but mostly through a barrage of emails. They either post flyers or send out a list of classes. The flyers don’t stand out – don’t grab your attention – you have to be looking for something to notice them. It’s frustrating more than anything.” - Panel Member (S)he doesn’t know much about the marketing of the program. “I know it’s placed on the NPS website. I still think it’s ‘insider knowledge’.” Within the Training Department, they could do more to advertise the program and encourage participation at least to those who participate in other training programs. They should also send it directly to superintendents. They might consider holding an A-W workshop for supervisors to help them understand the program and how having them participate would benefit them, their people, and their park. (S)he thinks they could use grant funds to do something like this. Thinks funds can be used from the endowment to promote the program. - Leadership

20

• But most evaluation participants felt the marketing needed improvements:

“I don’t remember how I learned about the program.” - Grantee One individual said that other than the supervisor, (s)he had not seen or heard anything else and doesn’t remember anything ever being sent directly. - Grantee After actively seeking out ways to enhance knowledge and skills, this individual “finally learned about A-W from [someone in training].” - Grantee “The only reason I heard about this [A-W] was because I called people asking for information. The way this is advertised is very poor.” - Grantee “It is hard to hear about it. When I applied...I had to look for it – I went searching for information.” - Grantee “In the big-wide-world of NPS communication tools, I don’t see it.” - Grantee This individual believes marketing is dropping: “Originally, I heard about it several years ago. I haven’t heard much since, and I only know one other person who has applied.” - Grantee “It’s hidden fairly well.” - Grantee One individual tells the story of asking the supervisor for an example of another A-W application; the supervisor laughed and indicated this person was the first one ever to apply. - Grantee It takes effort to get information out to staff, especially at parks with remote assignments. Even though (s)he thinks the announcement now comes to everyone through ccmail – it’s important to remember that not everybody gets on email. Even some people who have access won’t use it. Also, people in remote work sites don’t always connect with people in-the-know very often. Supervisors need to do a better job encouraging people to apply and getting hard copies out as well as computer-based information. “People who are savvy know to apply. Then they get one and apply again and so forth. People who are more disconnected or less savvy (perhaps at lower grade levels) are less likely to know about it or feel qualified to apply.” Panel Member “Some employees are intimidated – it’s not a one-page form with yes or no questions. It seems like too much because they don’t understand the value – don’t see the return in investment to the employee or the supervisor. So, they need to find a way to communicate the value – market to the supervisor – if you don’t, it won’t get to the employees.” Panel Member “Haven’t heard about it much at all – maybe gotten an email – which, you know is not bad, but I haven’t seen much.” - Panel Member

21

“I think many are not far-sighted enough to apply on time. This is where maybe the month before applications are due reminders are put out in a way that reaches as many as possible.” Panel Member “They have to find ways to get word to folks out in the field, especially if they are remote or perhaps even at lower grade. They need a little special attention.” Her concern “goes back to not everyone is on the computer. They are not getting the information. It’s a problem to solely rely on the computer at this point.” Panel Member “This information comes from the top down. Here, I don’t even know how many of the maintenance staff are on our email. The message might come into the Park Superintendent, but I don’t think it gets out there to all of the people. Somehow they have to get the word to all the staff. Broader commitment is needed in the park or on the part of program staff to get supervisors excited about the program. By and large, the applicants are in professional series, higher-graded.” - Panel Member “Often, if you want people to see things, you need a poster or someone to mention it at a briefing.” Panel Member Those who are aware of it know they can use for their advantage. “But it is kind of like a secret – I don’t if it’s well known across the Service. Some people have seen the information on the website, and those who have used it are familiar with it and the regions are aware of it – and so, when someone comes looking for information and if they ask the right people, they are able to get information about the program.” (S)he’s not really sure the wider field knows about it, though. - Leadership Poor marketing of this program “is mostly a marketing/communications framework issue within NPS. Currently, the channel/process used to distribute announcements related to the program (which, at this point consists of the announcement of available funding, application deadline, etc.) is as follows: the program administrator sends the announcement to the Regional Offices who then are responsible for sending it to the 387 parks. Someone at each of the parks is then responsible for seeing that it gets distributed to each of that park’s employees. There are 20,000 employees. You can imagine that there are many opportunities for the process to get interrupted and for information to not get passed on.” - Leadership “Marketing of the program leaves much to be desired. Our websites/bulletin board updates have a slow response time. Is often ‘too little, too late’. By the time we receive feedback that certain folks aren’t getting the information, then it’s too late for us to get information to them in time for them to respond with the highest quality application. Feels like we’re always putting out fires. It can be challenging to find the time/resources to remain pro-active and spread the word quickly and comprehensively, and there’s been very little support for the My Learning Manager system.” - Leadership

22

“Two years we tried doing the process through electronic mail – didn’t work because some don’t have access and others aren’t comfortable with it. I think it needs some more uniformity – something that can be identified year after year when you see it.” The program needs to have a consistent announcement date – it varies from year to year. The length of time it’s open has varied from year to year as well. Some of this has to do with limited help and with changing landscapes of administration over the years.” - Leadership “We may need to find ways to develop learning that is more focused on distance learning opportunities - market the idea that you don’t have to go away for training, but that it is something you can do in the workplace; provide information to regional training coordinators to assist with marketing the program - we assume people know what’s happening in the training community, but it’s not really on the average employee’s radar screen; and prepare marketing information for supervisors to understand employee development as a responsibility - we need to market professional development and training to supervisors as a way to improve their employees. We must convince them that it is valuable and worth the sacrifices they may have to make in the short run while the employee is doing the project or attending the professional development activity.” - Leadership Marketing of this program is “marginal, at best, due to a variety of factors.” One problem is that there is a certain, small set of channels through which it is usually marketed, and these channels have often been disrupted over the years resulting in uneven and uncertain distribution of information. A marketing plan needs to be developed that includes a variety of media and doesn’t rely so heavily on the usual, formal channels. Marketing of the program should be “constant and ongoing, not just an announcement when it’s time to apply.” Information should be communicated about who has received grants, how those grants are progressing, and the outcomes of the grants. Other ideas for this include sending postcards to all 20,000 employees (wouldn’t cost very much); add messages about the program into payroll statements (no cost); access existing informal publications and newsletters that go to all or segments of the employee base; etc. - Leadership “There’s really a lack of awareness about the program. Superintendents don’t even know about it. The reason behind [the Albright-Wirth program] is admirable, but it’s a little scattered” It’s up to the employee to determine how they want to spend the money and the audience it’s directed toward is very broad. This might result in a diffuse message being sent out – any one individual might not see where they fit into the program depending on what examples are provided in the marketing. Some might think this isn’t for them or “I’ll never get this, so why try? The current, very broad nature of the grant may make it harder to market. “I think the program could have more impact if it were more focused and then the marketing would be focused – better targeted toward the appropriate groups and better at showing how the grant can benefit them.” - Leadership

23

“I’m not sure how well it’s being marketed in the service. I’m not sure if there’s ever been any training for grantees on how to apply or for supervisors” to make sure they know about it and how to access it. – Leadership TEL – Electronic training – maybe they could do commercials during the training sessions – they already put up pictures and music – to advertise A-W and other professional development and training opportunities. “These people are sitting there looking at their screens – show them stuff – at least enough to get them interested or prompt them to look further into it.” And “how do we say to the organization this has value? – by giving people who have done it some exposure and highlight their accomplishments. We also might want to look at offering a different application process – having a coach or input from previous grantees. And maybe a different way to submit applications – video, verbal, … How can the process be more inclusive of people with different, or limited skills?” - Leadership Suggestions from one Leadership participant included the following: • A “traveling trunk” show that could be used around the NPS at various roundtables,

conferences, special emphasis program that highlight the accomplishments/successes of the AWG recipients.

• A permanent repository (a training center exhibit, visitor center exhibit, etc.) where AWG products produced by the funding could be exhibited/marketed.

• TEL (Technology Enhanced Learning) satellite broadcasts to enhance marketing - develop a marketing video to incorporate into these courses. These are captive audiences and probably one that is already interested in receiving training.

• Open application process versus an annual process - Accept and review applications more than once a year. This would allow us to deal with some of the problems associated with applicants needing to plan so far in advance (e.g., college or conference registration) or things that only come along once or only a few times. How would this effect administration of the grant? This would require a different way to review the applications. It might even require different criteria and a different way of notification.

• Providing more financial support programs to choose from - divide Albright from Wirth: perhaps with different emphasis in the funding – Operations and administration from one and natural/cultural resources from the other. The openness of the current grant is nice, but maybe that has made it unmanageable.

• Direct marketing (flyers/brochures to employees at their home or park address.) “We’re all struggling with My Learning Manager right now. In the short term, there’s the learning curve issue. Once it’s up and running, that’s the only place people will be able to go to get training. That will probably help with the marketing – of Albright-Wirth and all the development programs. It will really focus people. But during the interim, it’s going to be painful.” It will be at least a year before it’s all-together. - Leadership

• Several participants suggested promoting the accomplishments of the recipients more widely.

24

“People who have been funded should be called on to market the program in their cluster or region. - Grantee “Provide a list of the types of projects that have been funded – let people know just exactly what you can do with this program...” - Grantee

“How do we say to the organization this has value? – by giving people who have done it some exposure and highlight their accomplishments.” - Leadership “Information should be communicated about who has received grants, how those grants are progressing, and the outcomes of the grants.” - Leadership They should “have a ‘traveling trunk’ show that could be used around the NPS at various roundtables, conferences, special emphasis program that highlight the accomplishments/successes of the AWG recipients.” It would also be nice to have “a permanent repository (a training center exhibit, visitor center exhibit, etc.) where AWG products produced by the funding could be exhibited/marketed.” - Leadership “I think we should promote the recipients a little more.” - Leadership “I don’t know what happens with the reports, but it would be great to use information from these reports. It would be great to really spotlight the employees and their accomplishments at all levels. And also look at folks like Patty who used the Albright to jump start her career.” Highlighting the accomplishments of the participants would also be a way to solicit more funds for the endowment. “If we were marketing the program better and showing the benefit, we might be able to grow it even more.” There’s no reason employees couldn’t contribute to the endowment and a natural group to ask is those who have participated in and benefited from this grant over the years. This would make more money available for the grant each year and this would also raise awareness about the grant in the Park Service. - Leadership

• Within My Learning Manager, the Albright-Wirth program is listed under a separate

heading for training opportunities but does not appear in listings of profession-specific training offerings.

3. The timing of solicitation of applications has been inconsistent.

• Because there is no formal time line for administration of the Albright-Wirth program, program announcements have been made at various times during the first half of the year.

One grantee suggested that “It would help if the announcements were announced the same time each year. I have really been going wild looking for it...”

• Potential applicants cannot count on a specific time period for the application process

therefore cannot pre-allocate the time needed personally or with the supervisor to apply.

25

“I think it needs some more uniformity – something that can be identified year after year when you see it. The program needs to have a consistent announcement date – it varies from year to year. The length of time it’s open has varied from year to year as well. Some of this has to do with limited help and with changing landscapes of administration over the years.” - Leadership

4. The program information and current application appear to be compatible with My Learning Manager.

• Based upon a quick examination of the software in action, there appear to be no major

problems migrating the application and information to MLM..

5. There is no formal process in place to offer application assistance. Help is limited to information located in the program announcement and self-sought help from others; employees desire additional application assistance.

• Mentors and prior grant recipients could facilitate completing applications.

“I would love to see contacts for people who have been awarded the grant in the past who are willing to be contacted for questions and advice by people who are applying for the grant.” - Grantee

“I think it would be nice to have a mentor that has been successful in their bid for a similar grant to take an applicant under their wing and help. It would be nice to have a pool of people from each grant category for the new applicant to call on. These folks could provide advice and review applications prior to submission.” Grantee

• Supervisors and training officers can be, but are not necessarily, supportive and helpful in the application process.

“Upper management at the park have been excellent mentors; immediate supervisors, however, are not as supportive.” - Grantee “If a supervisor doesn’t value professional development and doesn’t have a lot of education, then they might not care and might not be supportive of professional development. This can definitely be a negative on the employee. Sometimes if someone doesn’t value professional development, they may actually stand in the way of an employee pursuing it. – Grantee “I don’t think everyone knows about the process or just forgets that it is there. I also wonder about supervisory support especially today when budgets are tighter and it is hard to send someone when there is so much work not getting done.” - Panel Member “If the supervisor doesn’t support it there’s not point in applying.” This panel reviewed the letters from supervisors the year this person participated. - Panel Member

26

“What was atrocious – the applications require that supervisors provide their approval, and some only wrote ‘I concur’ whereas others wrote a paragraph or more. In some cases – especially for lower grade levels, the letter expressed the need better than the applicant.” Based on the letters, it seemed hat some supervisors actually didn’t want their employee to do it. - Panel Member “Supervisors need to be willing to help employees, especially if someone needs assistance with writing or developing their application. I’m not actually sure how willing some supervisors are to do this or to have their employees taking part in professional development rather than doing their job.” - Panel Member “Supervisor support is really important in a lot of cases. Especially if doing something like this is new to the applicant. It almost seems like the application should be a joint application between the supervisor and the employee - help with writing the application, writing the reports, … It’s easy for the reports to get put off.” (S)he can see where supervisors can play a role in keeping employees on track. - Panel Member “Supervisors should provide assistance, especially at certain grade levels.” - Panel Member “I certainly think supervisors can play a positive role here. A good supervisor can do wonders for an employee. Maybe there should be an appointed liaison on the A-W committee.” - Panel Member

• Other ideas for assistance:

“Perhaps the people involved in the grant program could research past grants and develop a training guide that would address several issues: how to prepare a competitive grant using past grant applications as examples; ways in which grants have been successfully used to enhance a person’s career; ways to realistically prepare a proposal and budget; creative ways to enhance both employee development and park needs. - Grantee

• When asked if they would apply again, three non-recipients said yes and one added “If they could provide better guidelines, helpful hints, standards, that would be great.” One person indicated “I don’t think I’ll bother. It’s a lot of effort, not huge, but not worth it.”

• “As a panel member, it would be nice if the panel could give some feedback to the

applicants that might be useful to them for writing their next application.” - Panel Member

6. Unused funds, due to recipients declining awards or returning extra funds, have been inconsistently managed.

• An NPS-managed account held unused A-W funds (FY1987 - FY2001) which totaled

$105,525 in July 2002 - document accompanying NPS 2003 A-W Program Summary

27

• According to NPF leadership, funds were subsequently used for an alternative purpose.

• The total amount of funds for the year are sent from the Foundation to the Park Service, and then an account is held at/through the Park Service. Each recipient receives a set of requirements and instructions each year that provides the account number and how to access it. Grantees must use the appropriate account number to get their expenses paid for. If someone asks to do alternative activities (e.g. if the original course they signed up for is canceled and they want to go to a different one), the program manager will approve this only if it conforms to the original intent of the applicant’s proposal. It is unclear what happens if the request doesn’t match.

7. Unused funds from incomplete or inappropriately budgeted projects are not

consistently tracked.

• There is no formal system used to regularly track project completion and whether the grant funds have been spent.

• Currently, expenses are approved at the park (through whatever their normal procedures

are) and money is requested through a single A-W account number and sent directly to the park. This seems to be working pretty well, but there are some concerns. From the program perspective, auditing or monitoring project expenses is a reverse process – the recipient gets approval from his/her park, the park approves it and requests it from the A-W account. The program manager then receives a (monthly?) summary of expenses, which is entered into another database to compare with grantee awards. If there is a problem, someone must then contact the park/recipient to clarify or get money back.

• When asked about their experience with unused funds, grantees said:

“[I] did not submit anything, and I never contacted the office to let them know [I] wasn’t going to use the money. “It seems they would have known I hadn’t used the money and call to ask me why.”

8. Partner communications regarding the endowment, available annual funds, and

funds management have been inconsistent.

• NPF and NPS leadership agree that communication regarding all aspects of endowment funds needs improvement. “They send a report every year on this list of projects that will be funded by the grant. That’s the only formal reporting that comes from the Park Service back to the Foundation.” - Leadership

28

They [NPF] get nothing back concerning what happens to the money they send to NPS. They send over the sum of the selected projects and had always assumed that it was all spent and that all projects were completed. “I have no idea what happens to any money that isn’t spent. Two years ago, I know there was an account with a large amount of money in it and then several months later the money was gone – where? I was quite surprised when I first met with them two years ago and learned that projects weren’t being completed.” - Leadership

F. APPLYING FOR A GRANT – FROM THE APPLICANT’S PERSPECTIVE

(Value of Professional Development, Reasons for Applying, Perceived Reasons Others Do Not Apply,

Perceived Reasons for Not Funding Certain Projects) 1. Professional development is highly valued by grant applicants.

• Grant recipients described the value of professional development.

“Critical, vital. I don’t know how you can’t do it to improve your performance on the job. This is what keeps me excited about my job.” “I think education is becoming the great equalizer in our society.” “It makes me a more savvy manager.It keeps me interested and engaged in my job. And it provides professional connections...and it’s fun.” “It’s important to see what others are doing and keep up with anything new coming in. It provides ‘personal satisfaction’ also. It helps me do my job better, makes me feel more effective.” “I feel personally that if you enhance your own abilities you can give more to the organization. It’s important to continue to grow – both personally and professionally.” “I’ve gotten no more money, no promotions...so, it’s not helping me professionally at all. The Park Service benefits, and the public benefits, I hope.” “Everybody could learn something and it does everybody good to learn something. I value education.” “Very high value.” “I feel it’s essential.” “Priceless and indispensable.”

29

“You can never have enough training.” “Makes me a better steward of the Park’s natural and cultural resources.” “It is very important to my job – you have to know how to do your job.” “I’m into lifelong learning. I enjoy learning. I enjoy the challenge.

• Applicants not receiving grants described the value of professional development.

“Very high priority because I know it is required to advance. You need to know how to do things the park way. There is a chain of command and you need to follow it.” “Training and attendance at professional meetings is important to stay current with the latest techniques.” “I think it’s very important. Part of our job is to be leaders in the field. This is what the Park Service does.” “I place it pretty high. I think it’s very important to have as much education as possible. ...I’ve learned how competitive it is, and how hard it is, and my struggle to move up the ladder in government...”

2. The number one reason for applying for an A-W grant was the perception that it was the only way to fund and accomplish the specific project.

• Lack of other funding drew persons to A-W.

“I knew the only way I could fund the training would be to get the grant. No grant, no training.” “There is no funding available in the park due to budget limitations,” “Money.” “Because there was no other way I could fund it. We are strapped for cash in the agency...” “It was Albright-Wirth or nothing.” “The park would not pay for it.” “I didn’t have money to do this on my own.” “Funding, without a doubt. There’s no other way to pay for this training.”

30

“No other way to find funding to continue the research I have been doing.” “I exhausted my sources of money.” “I applied because they don’t give me any training money.” “Training money is the first to go when money is tight.” “The Park Service does not take care of it’s own employees so we must take care of ourselves.”

• Other responses included:

“To enhance my skills and provide a service to my park.” “To better my work performance.” “Professional development and it’s available to NPS people.” “It was really for professional development.” “To meet new people.” “I felt I had an excellent opportunity to be funded.” “I wanted to gain more knowledge and improve my personal development.”

3. The number one perceived reason why people do not apply to the A-W program is poor communication about the program.

• Applicants believe NPS employees just don’t know about the A-W program.

“They don’t know about it...they don’t have folks at the parks who have done this and who they can use as a resource to help them complete the process the first time..” “They possibly don’t know it exists.” “Well, I didn’t know about it, so maybe other people don’t know about it.” “Communication issues – the probably don’t know about it. It’s important to remember that [some NPS employees] still have no or little access to computers. Perhaps people don’t know or understand the value of the program.” “People don’t know about it.”

31

“Because people don’t know about it – it is not widely advertised. I never heard about it and I’ve been in the Park Service [for a long time]. You need more press on it – a video for people who can’t read.” “Don’t know about it – very simple – I hope that the NPS new employees’ orientation training are providing this information.”

• Other perceived reasons for not applying varied.

“Lack of self-motivation.” “Fear of failure – unsure of likelihood of being selected and so don’t apply.” “Fatalistic attitude...’I’ll never be funded’.” “Some might be pessimistic about their chances of getting a scholarship.” “It is somewhat time consuming to write the essays and some don’t have the energy to do the work.” “I feel people are too LAZY to do the paperwork. They DON’T want to LEARN anything new. They DON’T want to have to participate. They’re afraid they may FAIL. “Desks are piled high. We’ve got a lot on our plates.”

4. Non-recipients beliefs varied about why their projects were not funded.

“I was not detailed enough.” The proposal was very generic and s(he) didn’t realize the competition would be so tough. “I don’t know. Looking over the list of people who received funding, it appears to me that my proposal had more benefit to the NPS than many of the others.” “I don’t have any idea. But, it strikes me that money is going to things that aren’t actually professional development.” S(he) thinks the park service is using the money to meet their needs rather than the needs of the applicants. The A-W program is “starting to be treated as a pot of money.” “I believe there were quite a few folks that had some great things they wanted to do as well. And there’s only so much money. I don’t know what the selection criteria were. I submitted and the best man or woman won.”

32

G. SELECTION PANEL AND PROCESS (Role, Selection, Review Process, Criteria Use,

Benefits to Panel Members) 1. The role of the selection panel has changed.

• The original selection panel, as described in the 1987 program announcement, had more responsibilities than do the current selection panels:

“The NPS project management team [currently the selection panel] determines categories of developmental experiences to be available, develops program criteria and guidelines, reviews employee applications and recommends recipients to the Director, recommends program changes, evaluates program effectiveness, completes all program reporting requirements”

• The current duties are described in the 2004 program announcement:

“A diverse, rotational panel of experienced National Park Service employees reviews the applications.”

2. The current process and criteria for selection of panel members are unclear and

so the match to the original intention is uncertain.

• Originally, the “NPS project management team of six NPS employees [was] recruited from a cross-section of job classifications and grade levels.” And “The six employees serve[ed] on alternating two-year terms.” – The original 1987 program announcement

• The original purpose of the nature of the selection panel and its specific participation was

described in this quote from the 1987 program announcement:

“For the credibility of the HAEDF, [A-W] the screening and recommendations of applications by the Director to the Foundation is made to the Director by this six-person peer group.”

• As noted in (1) above, the current program announcement describes the panel and role in

the following way:

“A diverse, rotational panel of experienced National Park Service employees reviews the applications.”

• Applicants shared their perceptions of panel member selection criteria.

“The people should themselves be invested in professional development – not just people who are willing to come to Washington for a week. They should have demonstrated they are interested in professional development for themselves and for their staff.”

33

“I think the panel should have had to apply for the grant in the past to be considered as a member of the panel. They should know what we as applicants are going through.” “They should come from a range of professions from throughout the park service.” “They should be from programs as well as parks.” “I would assume these people are in employee development. That they have to have firsthand knowledge of what’s actually going on with employee development – what programs are available, what’s popular, and so forth.” “I would think these people would have firsthand knowledge about a variety of subjects – ethics, fairness, being able to make good decisions based on what’s in front of them and not based on who they know and who that person is or who they’re tied to. I want to make sure that those people on the committee are upright, level-headed folks.”

• Panel members are unsure how they are selected.

“I asked to do it. I had written a note 10 years prior but had never gotten a call. But I don’t know what the process is for selecting panel members.” (S)he assumes/hopes they were going to get a mix of all different disciplines and ethnic backgrounds and for people from all over the country. The individual hopes (s)he was chosen based on her experience in administration but doesn’t feel panel members need experience reading applications. (S)he wrote a year ago expressing interest in doing it again and didn’t receive a response, but assumed (s)he couldn’t because (s)he has already served. “I don’t know what the criteria or processes are, but I would like to know.” - Panel Member “I don’t know for sure what the qualifications are, but [the panel] seemed to be multi-disciplinary. Different parks, WASO, and a couple of different regions were represented. It seemed to have a good balance men and women and ethnicities. I thought that it was a good panel.” - Panel Member “I can’t say [how the panel members are selected]. I know I was recommended by somebody. They do ask you to recommend people for the next year. I believe I was selected to be the Natural Resource person on the panel. My understanding is that they have trouble getting people to do it. I’m not positive, but I’ve gotten that feeling.” - Panel Member Didn’t know how panel members were selected, but believes (s)he was selected because of insight with the employee development community, years of service in Park Service, previous jobs, active interest in education, and support of these types of activities. Believes panel members “should be leaders in their field and have a passion for what they’re doing.” - Panel Member

34

“I am not aware of specific qualifications. I think I was selected on a recommendation. I do think there should be specific qualifications but they are difficult to define. I believe having someone on the panel who actually works in the field is a good idea.” - Panel Member

• Leadership is unsure how panel members are selected. While they did not express any concerns about the adequacy of previous panels, they did have ideas about what the qualifications should include.

“They need to be open-minded and have a commitment to understanding what learning is all about – that learning is indispensable to leadership.” They need to value training and development themselves. Display good leadership and learning. They must have been actively involved in their own professional development. “Vicki always looks for diversity among the panel members – they must have an understanding of training and development, but different understandings.” - Leadership “The panel should be representative of the NPS workforce in all senses, ethnicity, job type and level, gender, etc..” While it is impossible for any one panel to represent all types and ranges of diversity, over the years, the panels as a conglomerate should be representative of the NPS workforce as a whole. “There is nothing wrong with someone who makes minimum wage sitting on a selection panel such as this. If they work for NPS, they should be able to be on this panel.” Past grant recipients should be represented on the panel as well. - Leadership

• The A-W Program Manager described the process for selecting panel members as follows:

“There are no specific qualifications. I look for people who are open-minded, experienced, committed/focused on the mission, receptive to new ideas, well-rounded, effective communicators/listeners, willing to accept diversity at all levels, and interested in volunteering their time. I look at people that have what I consider a wealth of experience in the organization – either a lot in one area or dispersed into a range of work responsibilities. I look for people who are open-minded and receptive. I look for different geographical locations, different age groups, different ethnicities, different ability levels, different disciplines, etc. I interview everybody. Some people volunteer – very few. Some are recommended by recipients, managers, … I’ve done research before – called the EEO office and requested names, called parks, etc. I have to have people who are able to focus on reading a lot of data in a short amount of time and have at least reasonable interpersonal skills to be able to participate in the discussions. Only occasionally has a panel member infuriated other panel members. The person’s character matters more than anything – they have to be an ethical person.”

3. The panels seem to be diverse in many aspects.

• Panel members from various years described their panel in the following ways:

35

“The committee I served on lived up to my expectations - especially regarding the ethnicity component. It had a nice representation, except perhaps from law enforcement.” “The panel seemed to have a very good balance; disciplines, regions, sex, ethnicity.” “Looking at who was on the panel – it wasn’t just ethnic diversity, but they tried to get a group that crossed major divisions and disciplines in the Park Service. As a group we had a pretty good knowledge base of different functions of the Park Service. Regarding ethnicity, we were reasonably ethnic diverse. I’m not sure about different grade levels, but I don’t think there was anyone very low. I think were probably all GS-9 or higher.” “The panel I set on was highly diversified and this really showed. It was valuable and needed to be there. The panel was very well represented – someone with a Ph.D. and someone at a small park with a high school diploma; ethnicity; age; sex; all of those things. And all of those played into acting as a team to bring this thing to completion.” “This is an interesting question because some of my preconceived notions did not mean much. There was one woman who was Chinese Cuban, two black males, a white female, a black female, and one white male. I found that age, not ethnic background or direct Park exposure was the common denominator. Values and ethics were somewhat universal.”

4. The actual application review process varies from year to year.

• Panel members from various years described the process in the following ways:

I didn’t have any defined expectations. I did expect the process to be a little more formal than it actually was. I don’t think it was complete. When you get down to the nitty-gritty, there needs to be a more formal rating system.” “We were certainly given plenty of freedom to make decisions as a group with minimal micro managing. The desired end results were made clear and how we got there was up to us.” “All I remember is that the process was thorough and excellent. Vicki was extremely professional. I was very, very impressed. I remember that they brought someone in from the Park Foundation to provide some history of the program.” “To me, the criteria, at the time, were easy to apply. I wasn’t expecting to put things into Excel – and this was a little difficult. I’m not sure how this fit in exactly, but it was kind of confusing and I’m not clear that it was even necessary. But, I guess it worked out in the end.” The panel member also expressed a little bit of concern about the collection of notes from the panel members. “I was making notes throughout the process and didn’t realize these were going to be turned in. My notes were sloppy and contained some personal thoughts and I’m not sure what happened with them afterwards.”

36

“I think there were some points that could be improved there. My recollection was that Vicki briefed us and then she didn’t want to interfere or bias the process and so she wasn’t there part of the time. But, I could have used a little more input and guidance. The rating system, for me, needed to be clarified and better explained. It might be good for them to provide a few examples of how to apply the criteria – maybe the top-rated proposal in each category from the past few years.”

“They sent pretty brief introductory material (a copy of the application) and gave us some introductory material on how to rate and then spent a few hours explaining what they wanted us to do. For their intention – to leave it open-ended and leave it up to us to decide who to proceed – then it was fine. I just wanted more formality.” “Our duties were made very clear – to review the applicants that made it through the initial cut.”

“I received some information before arriving and on the first morning we spent the first half of the day going over expectations and possible ways to do our task. A rating guide was given with clear explanations. I actually appreciated the freedom given to come up with our own way of organizing the evaluations as long as consensus was achieved by a given time. I think there should be more information provided ahead of time, thus giving panel members time to process the task ahead of them.” “I don’t perfectly recall what the criteria was.” Whatever they were, “it wasn’t really an easy process – but that could just be my personality because I tend to weigh things a lot and when you’re dealing with that number of applications and a limited time, it gets difficult. The first 10-15 applications get a lot of attention and the next 20-30 get a lot less. It’s hard to be as fair with the last as you were with the first. For me it was very time consuming.” It would be nice if they could find a way to “narrow the number of applications the panel has to deal with.” “We were provided examples of good applications and were able to sit with WASO staff for a couple of hours when we first started” but this wasn’t sufficient for this panel member. “They gave us very broad guidelines to approach the ranking, but then left it up to us to create specific criteria for the ranking - to the point where there was no real valuing approach. We had to create our own rating sheet. We were given an introduction and told there were 5 categories in which people could apply and we they gave us general comments about how the project should have value to the employee and to park service. But very little was really said about how we should or shouldn’t rank people.” (S)he was really bothered by the lack of structure to the rating process. (S)he wanted to see clearer definitions of what should and shouldn’t be considered – more of a rubric of what criteria to apply and how to apply them. “I think it’s left up to the committee to decide how they’re going to handle the review process. We mostly read each one as a group - some we read independently. They left it up to us and some folks wanted to go out and see things and so we shut down early and took things home so they could go see DC.”

37

“We were given a long list of names and applications and it was up to us to apply to a summary rating number and brief comment.” They did not read and rate individually – did it as a group discussion and came up with a single number. They eventually agreed to come up with a mean. “The discussion was valuable because people in different fields could ‘enlighten’ others about different aspects of proposals they didn’t understand. We ended up grouping them into classes – 1 through 5. We made recommendations, but we were also told a ‘committee’ above them made the final decisions.” It made them wonder what the next steps were. They discussed that this panel is not really “for real” - that some people would be selected regardless of their say. “I never got a list of who was ultimately funded, so I don’t know how our recommendations stacked up with the final list.” “I don’t remember all the criteria – most of it was very clear, especially from the employee development side – in terms of how the employee met those criteria were sometimes a little less clear. The idea was that we would each individually rate – bring ratings to the table – throw out highest and lowest ratings and then agree. We respected each other, we respectfully listened, and it was an open discussion.” “The process needs to be made more formal, what can and cannot go into evaluating the proposals and our summary and documentation. Specifically, individual ratings should be there and there should be more than a few words to explain why we did or did not select a proposal. Perhaps we should have worksheets or check lists – like those used to rate job applicants – and comments on these sheets could be provided back to the applicants. It would have comments like ‘The committee thought this proposal was too expensive for the benefit gained, or you didn’t explain this aspect of the proposal sufficiently’, and so forth. All we had was a list of names to squeeze all this (the rating and the comments) on to. It just wasn’t enough.”

• Applicants suggested that it would be good “if it was much more transparent. Let people know what they didn’t match where they fell on the list, why they didn’t get funded.”

• A member of the leadership felt the process “seems very involved. It’s too lengthy – it

seems like it could be a one-day process in D.C. “Maybe we could mail applications out to panel members for review prior to coming to D.C. There are a number of ways the process could be shortened.

5. Panel members enjoy their participation.

• Regardless of the particular year of service, panel members noted their enjoyment with participation:

“I loved the training – it was the best.” “Serving on the selection board was a very positive experience for me and gave me an understanding of the process I did not have before.” “I wish I could do it again.”

38

“My expectations were pretty much met. I have a much better understanding of how it is managed and how it is agreed upon. I really came to understand that it is one peer judging another – one employee to another.” “I enjoyed the experience. I liked working with the people – they were interesting and very nice. It was excellent.” I expected to work hard, be objective, be fair, and have some influence in the process. My expectations were exceeded in all ways. Victoria was a very gracious host who made my feel right at home instantly. In a nutshell I found the process fair, as objective as one would expect, very rewarding, and certainly time consuming.” “For me it was a good experience. It was frustrating to see the incredible spectrum – trying to come to consensus, I think we did a good job of it, but weeding it down was difficult. Sometimes had to set some aside and come back to them.” “It was rewarding. I really did enjoy it.” “I felt it was a great honor to serve on the panel. It was like I received an award myself. I gained an understanding to the selection process and came out feeling even more positive about the National Park Service.” “I have nothing but good to say about it. It was very gratifying and grounding to see people from such diverse backgrounds have such similar values. I came away feeling the world was a better place.” “It was intense, but it was a good intensity. I would do it again in a heartbeat.”

6. Panel members are using implicit selection criteria in the process and those implicit criteria vary from year to year.

• Panel members interviewed during the evaluation mentioned various criteria they used:

“I participated on the panel because I wanted to represent administration and lower-grade positions - wanted to make sure they were getting a fair shake. I felt, and think my group agreed, that it was more important to give someone at a lower grade a real chance to get training to move ahead rather than give someone already at a higher grade level a plum.” The first option “offers hope to someone and the latter is much more of an extra or bonus.” “We created several criteria of own. There were things we decided as a panel that we valued. For example, there were people who were clearly ‘tapped into the system’. If someone was trying to get their third Albright-Wirth grant, they got rated lower. We favored lower graded employees. We figured they get lower pay and so this grant represented a greater opportunity for them. We also were less favorable toward projects that should be funded under other park sources – training stuff - unless this was a lower grade person looking for the basics.”

39

“I felt there needed to be more structure because there are always panel members that are more dominant than others.” Since so much was left up to the panel, then the more dominant members had a lot of effect on what happened. “In the end, I feel there are factors that are inappropriate that went into the selection.” For example, (s)he was concerned about the role of the “sob story” on the process. “Every time someone said ‘I’m a single mother with children at home,’ they wanted to give the money. This wasn’t fair because not everybody says this and so it wasn’t based on the worthiness of what was being proposed but on personal factors.” In addition, the panel tended to know a lot of the applicants and this influenced the process. One person on the panel started explaining all these details about an applicant and even started crying. This person ended up getting rated really highly, but (s)he was very concerned about the conflict of interest this represented. (S)he feels they should excuse themselves from judging an application if they personally know the applicant. (S)he did share his own knowledge of others “once he learned it was okay to do so and others were going to do it, but I really felt the proposals should have been judged based on the merits on what was proposed not on the personal information provided.” Another example of a troublesome priority that was the emphasis placed on law enforcement proposals. “Someone wanted to purchase a trained canine. I felt this really should be funded by the Parks – but the panel had this feeling that anything for law enforcement was a good thing.”

When asked what criteria they used, (s)he was only able to remember the following: One – whether they had had a grant before – (if they had, they were eligible, but not rated as high. (S)he doesn’t remember if they were told what to do with this or if her group made this decision independently.) Two – if the grant was within money range – but this qualification had been reviewed prior to being given to the panel. “As I remember it, there weren’t many things that were given to us.” “I felt a need to speak to issues around actually working in a Park and about fairness. I also spoke about rewarding the lower level employees who often are ignored by the system. I wanted to interject reality of real Park situations and sympathy for lower level employees. This was received very well and our group discussed criteria, our biases, and pretty much were in agreement. When asked what criteria were applied, (s)he listed the following: Benefits to government; Description of project; Willing to accept less money. “I remember there were some proposal that screamed “Me, me, me” and these did not receive high ratings.” “By and large, the applicants are in professional series, higher-graded. There were Park Superintendents applying. Some times I found it offensive when someone at that level, with that pay, would apply for $3000. They have the money to pay for these things.” The number of categories “could probably be weeded down” – eliminate the conference and the details. Their group seemed to think “the benefit for the cost was not worth it compared to the benefits that could come from the other four categories.”

40

“Your location matters also. In DC, there’s all kinds of things available. Not so in a remote park. I think that was a factor in our selection process as well. Where were the people and what was available to them.” “We tended to be biased against people who requested foreign travel somewhat because of the funds needed to get there. We didn’t feel the benefit was worth the cost of doing it.” “Our group definitely focused on funding proposals that represented a true cross-section of the Service workforce. This is critical –serving on the panel I was certainly aware of the different disciplines applying and found myself supporting sometimes-inferior applications to achieve this goal. I think it is very important to demonstrate a willingness to cover the whole spectrum of employees.”

• NPS management indicated the letters from the supervisors were provided to the selection panel for review. However, panel members, grant recipients, and other leadership were mixed in their perceptions of the role these letters played in the review process. “Our panel did read the letters of support from the supervisors and they played a role in the selection process. The supervisor’s support or lack of it was very evident.” - Panel Member “When I served on the selection panel, the letter from the supervisor wasn’t part of the judgment criteria and had no effect on the selection process at all. It might not have even been with the application. That’s the way it should be.” “If the supervisor doesn’t support it there’s not point in applying.” Panel members reviewed the letters from supervisors the year (s)he participated. “There is no standard format for the letter of endorsement from the supervisor, but the review panels do receive and review the letters from the supervisors. About 2/3 of the time, it’s clear that the employee wrote the endorsement. But, I think it plays a rather limited or restricted role in the review process. It has the same function as the endorsement letter for training provided by the Park Service. It’s meant to be an inclusionary activity – to bring supervisors into the process and balance the situation.” - Leadership “Even though the supervisor’s endorsement statement became mandatory several years ago, I think most supervisors would be reluctant to present challenges/obstacles for their staff to apply. With very little funding available for training, advanced degree programs, developmental opportunities, or seeder funding for new equipment, inventions or research, management realizes they will lose their better employees to park units who support training and development, so they encourage staff to apply for grants with their blessings.” - Leadership “I don’t remember if the letter from the supervisor plays a role in the review process.” - Leadership

41

(S)he doesn’t know if the letters from the supervisors are used in the review process. “I imagine that if a supervisor doesn’t agree with the employee’s participation, they wouldn’t be able to apply.” - Leadership

H. GRANT MANAGEMENT (Notification, Reporting, Project Assistance, Tracking)

1. The process and timing of notification to grant recipients and non-recipients

(including providing feedback) is of concern to applicants. • The timeline for notification to recipients which was mentioned in the 2004 Program

Announcement is “prior to October 1, 2003” but the letters actually went out October 10.

• Grantees said:

“The lapse time between when we submitted the grant and when we were notified of recipients was long. It would have helped to know about the funding and had had funds available for use sooner.”

Another grantee needed to know about his/her award early enough to register for the desired class. That person was told (by A-W program staff) announcements would come in August but in reality, notification letters were not sent out until the end of September. This lateness caused extra costs and “I had to scramble to get registered.”

• Non-recipients said:

“I received no feedback. I didn’t know that I was not funded until I called [A-W staff] and asked about it. I think I called in early November.

This person received “No feedback what-so-ever, except ‘no’.”

This person called [A-W ]staff to find out why the grant proposal wasn’t funded and indicated “she was excellent – she pointed out my short comings and things I could do to improve my chances.”

“All I received was a letter that I did not get the scholarship. A little hand-written note – like someone was really sorry that I didn’t get it – and that was nice.” There wasn’t any information about why I didn’t get it or how to improve. That would have been nice and helpful if that was possible to do.”

2. Financial reporting requirements are perceived as being unrealistic.

“Actually sticking to the ‘within two business day requirement’ is not at all realistic.”

42

Another grantee never met the financial reporting requirement but “it was never brought up as an issue. Given that, he/she’s not sure why it’s stated as a requirement.

3. This is the first year quarterly reports have been required.

“After they became aware of the completion problem, a committee of 5 people revised the application a little bit to put a little more accountability on the grantees. They thought that if they were required to report in more often, they would be more likely to stay on track.” - Leadership “This year is a little different because of the quarterly reports. There will be more contact with recipients, and it is hoped that this will lead to higher project completion rates and more final reports being turned in.” - Leadership

• Several participants felt quarterly reports might be too much given the amount of money

people receive. 4. Final report requirements have varied and there is no record kept of those who

do not submit one.

• According to program leadership, the ‘response rate’ for completed final reports is believed to have been about 50-60%. It is believed the project completion rate is closer to 80%. There have been no consequences for not turning in a final report. And there has not been a procedure for contacting recipients or their supervisors to inquire about why the reports haven’t been done. And there’s only been very minimal inquiry from others (at NPS or the Foundation) about the final reports – what happens, what doesn’t happen.

• Perceptions of the reporting system varied with several grantees reporting that they were not asked to write reports.

“I didn’t have to do any report with my grant.” As far as appropriateness of this requirement goes, (s)he feels it depends on what the grantee does. For example, if they are going to present at a conference, enough information is included in the application, and they shouldn’t have to do this again. It’s “no different than getting training through the government. If I went to a 2-day training, then I went. That’s that.” Another example, such as a project involving learning Spanish in another country, might require and call for more documentation. - Grantee “I was somewhat confused about [this question on the evaluation questionnaire]. I prepared a report for the first grant; it was late but I did send one to both the grant program and the National Park Foundation. However for [the next grant I received] the form did not mention that a report was due and no mention was made of one during the correspondence related to the grant. I think the reporting requirements are fine. However, they should be consistent and notification should be sent upon receipt of the report.” - Grantee

43

“If these are requirements, they need to be better communicated to grantees.” - Grantee

“It does seem that they sent an e-mail a few months after the project time was up and asked about if and how I had spent the money. But they didn’t follow up on it or really check on it. I could have spent it anyway I wanted.” And “They weren’t interested in whether it was a success, was it worth your time, etc. – just about the money.” - Grantee

This individual “didn’t have to deal with these [final report] but these seem reasonable.” - Grantee

“Didn’t have to submit anything for the previous grant award.” - Grantee

“These are very minor requirements. Easily done.” - Grantee

“Directions were pretty clear.” - Grantee

“Not all types of projects need the same level of reporting.” - Grantee Nothing happens with the reports – “they get filed and that’s it.” This year, if they get something electronically, it will be forwarded to NPF. - Leadership Last year she got a few copies of final reports but that was part of planning the evaluation. They do not receive a year-end wrap-up or summary. The summary they receive is a description (list) of projects they will be funding with that year’s money. - Leadership “I think the instructions ask them to forward their report, and we occasionally get a copy of one. I don’t know what happens with the reports, but it would be great to use information from these reports.” - Leadership “I don’t know anything about the final reports or really about the details of this. Vicki only comes to me with these issues if there is a significant problem.” I assume there’s some sort of summary report of each year’s final reports – but I’ve never seen it.” - Leadership “Management information on this project is not sufficient. I’m not sure how many projects have or have not submitted final reports.” - Leadership “This is like a third job. So whatever can be done to simplify this would be great.” This panel member suggested the development of a check-off form or a report by phone or something like that. Some way to make this simple, straight-forward, and consume as little time as possible – for both ends, managers and grantees. - Panel Member “I would like to think they are an aid but making the reports simple would help that. They at least remind the recipient that they have a commitment to follow through on.” - Panel Member

44

“I think there are factors having to do with recipient’s backgrounds. Some people are not highly skilled in the professional arena and struggle with reports and maybe even time lines.” - Panel Member

5. Project assistance is provided by contacting NPS, but grantees had differing experiences with receiving requested assistance.

• Some grantees didn’t recall any problems getting assistance:

“Other than there were times that things seemed a little slow coming out of their [A-W grant] office.” “They have been encouraging and supportive. They have bent over backwards to help people.” “I had no problems at all. Everything made sense to me. It may be because I had an HR background and recognized the benefits to the government of obtaining a grant. Others may not be as aware of this.”

• Several grantees had problems with funds either being late or not received in time: After running into problems getting the rest of his/her grant money, one grantee wrote a letter to A-W staff but “[I] eventually stopped trying to solve the problem and [assumed I] just lost the funds.” Another grantee described the communications between offices at the national and local levels regarding money: “It got nasty for a while.”

• Other grantees described a variety of concerns with grant management:

One person indicated that the person associated with the A-W grant was very hard to contact. “It was almost impossible to get a hold of [the A-W person]. If I called real early in the morning, I could reach [the A-W person].” This particular grantee also had trouble receiving return phone calls and e-mails. “I feel as if I myself as the grantee should be the person responsible for the grant accomplishment [but] I think we should have a person we could reach by e-mail and ask questions concerning the grant. If the ‘e-mail aide’ is unable to help you, they should contact someone else and have them contact you with the answer, or you can ask them. One grantee recalled getting a letter saying s(he) was a recipient but that was it. There was no information on how to move forward, how to access funds, etc. Once told about the award, there was no follow-up.

45

“The central office needs to do a much better job of communicating with grant recipients – more often and more meaningful. Poor administration of the program also contributes to slow or poor performance of participants.” There should be more regulation and more communication between program administrators and grantees. Currently, the emphasis in the program is on getting out the announcement and managing the application process. After one year’s grantees are selected and notified, it seems the program moves on to planning next year’s program rather than managing this year’s grantees. – Leadership

6. Projects are not regularly tracked beyond financial reports. “Tracking – I don’t know how they’re tracking – but some kind of tracking process needs to be in place so that people can’t be put on the back burner.” This person emphasized that s(he) was never contacted by program administrators about progress or lack thereof. “There should have been some sort of contact regarding ‘hey, there’s this money...’.” “I had no communication at all. I was notified that I received the grant and the next communication was asking me to participate in this survey.” “The only place I can really see room for change is in the administration of the grants themselves. If they could put a few more resources toward monitoring and tracking the progress of projects, which would be good.” At least one member of the leadership felt “Communication between program administrators and grantees is inadequate.”

I. OUTPUTS (Grants, Projects, Participation)

1. Limits on how much grant money an individual can receive have changed.

• In 2003, it was decided to limit the size of individual grants to no more than $5000.00. The reasoning behind this decision was described differently by NPF and NPS leadership.

“Last year was first year there was a limit. The limit was set by the Park Foundation. It’s my understanding was this because of economic situations” – low return on investment last year resulted in fewer funds available for the grant. “There hasn’t been discussion as to whether this will continue – but the Foundation has the only and final say on whether there is a limit.” - NPS Leadership “We suggested a limit in order to give some guidance to grantees. The number came from no particular place – we thought we’d be able to fund more projects.” They were looking at a reduction in funds (partially due to needing money to pay for the evaluation). “So, it was initiated partially for budgetary reasons, but I think it should stay at this level.” “I’m not aware of any process for determining the limit.” - NPF Leadership

46

“The limit was set in response to concerns about getting large requests ($20,000 or more) from applicants because when large grants are awarded, it limits the total number of people who can get grants. There was also a trend of not awarding large grants (not many awards were ever made for grants over $10,000) and they might as well let applicants know there basically wasn’t’ any point in requesting a large amount because they were very unlikely to get it. The $5000 amount came out of a discussion between Marjorie, Julia, Vicki, Sam and one other NPS person. They decided to try it for a year and see how it worked and what kind of feedback they got. The group agreed to try it as an interim step knowing that an evaluation was coming and that the amount might be changed again in the future.” - NPF Leadership

• The following table details the ten highest awards for the past four years:

2004 2003 2002 2001 5,000 9,933 8,410 10,000 5,000 6,750 7,700 5,000 5,000 6,548 7,162 5,000 5,000 5,922 5,500 3,000 5,000 5,475 5,394 3,000 5,000 5,235 5,175 3,000 5,000 5,000 4,598 3,000 5,000 4,995 4,454 3,000 4,880 4,600 4,310 2,920 4,650 4,006 3,948 2,600 Total Total Total Total

$49,530 $58,464 $56,651 $40,520 • Overall, the evaluation participants felt the $5000 limit is appropriate:

“I think [the $5000 limit] is fine. I don’t recall a lot of proposals that were of a very high amount.” However, (s)he doesn’t recall what or if there was a dollar limit the year she served on the panel. - Panel Member “I think they might even go lower than this – but at some point it becomes a lot of paperwork for not much money.” - Panel Member “I think it’s very appropriate. In years past, there were people who received huge awards and dominated the funding. By putting a cap on it, it allows more people to get funding. I also think it’s appropriate to ask what the applicant is contributing as well – matching funds, their own funds…” Not only does the limit allow more people to get funding, “it also made it simpler” for the panel because they didn’t have to weigh the impact of giving one or more large grants. - Panel Member

47

“This puts it on a fair basis for everyone. It can be limiting - it doesn’t seem like enough for some things - but overall, it works.” - Panel Member “I can go with this – maybe a window could be open to larger grants for teams/groups if they seemed warranted.” - Panel Member

“It was kind of frustrating because there were a lot of things that sounded like good ideas but you got the feeling you couldn’t fund a lot of them, so it became about how well someone could write.” This panel member doesn’t think there was a dollar limit per person at that time and quickly realized that a number of similar proposals would not be funded. - Panel Member

“I was surprised by the declining number – I think people have noticed that the amount of money from the Foundation has gotten smaller and the limit ($5000 limit and need for matching funds) and this is a problem – they can’t get the match from the park. The criteria have tightened over the past few years.” - Panel Member “I support there being a set limit – then people know before hand exactly what to expect. For the program as it is now, $5000 is an okay amount. It helps ensure the money is distributed to a larger pool of people.” - Leadership “There’s a pretty wide variety of things you can do with that amount of money. I think it’s okay – but it is arbitrary. It may be a little bit low. But I don’t have a sense of the trends of what people have been asking for.” - Leadership “I think it’s appropriate.” - Leadership “I don’t think there should be a limit. Some things are simply expensive. There is a trade off of how many projects can be funded each year, but this must be weighed by the panel each year, and they must decide the best way to distribute the money.” - Leadership “Having a maximum amount limit on the grant is fine, but I’m not sure how to go about determining what that amount should be.” - Leadership

2. The role of project categories is unclear.

• Neither dollar amounts nor numbers of funded projects are tied to the six categories. • NPS employees place a certain amount of importance on categories when applying.

“With the A-W program, it is structured by the six categories, but as long as you’re within those, you can pretty much do anything you want.” This person feels the six categories offer a broad range of possible activities, “which is a good thing.”

48

• As they currently exist, program leaders think the categories no longer have a role in the overall program design.

“Why do we have to have the categories? Can’t they just apply? I don’t think we should have the categories.” Leadership “Why do we need them?” This participant had “no recollection of how they’re used.” “Maybe it helps explain what the money can be used for” as part of the grant announcement, but I don’t see any reason to keep them. - Leadership

“Those categories have absolutely no relevancy at all.” - Leadership

“Originally, the categories were used to help people understand how the funds can be used, to market the program, to ‘measure’ or get the numbers on how many people participated in each area – and originally the money was spread across the areas. Now, people check a box for which category they think they fit in – and it plays absolutely no role in the review process.” - Leadership “What is the impact of these different types of activities? I would like to know what people are applying for, are there trends in these categories? Where the greatest impact is for the recipients and the Park Service?” - Leadership “If the program has categories, maybe they should relate to where we want to be in the future – such as distance learning – technology related projects, and so forth.” - Leadership

• The project categories do serve as examples or ideas for projects.

This panel member felt the categories help people understand how the money can be spent but wasn’t sure if their panel utilized the categories in any formal way. (S)he thinks they tried to classify the awardees according to this but doesn’t really recall. It was not used to decide where to put money or who to fund. “It wasn’t part of the criteria.” - Panel Member “The number of categories could probably be weeded down – eliminate the conference and the details. Our group seemed to think the benefit for the cost was not worth it compared to the benefits that could come from the other four categories.” - Panel Member “For me, it was very relevant. It helped us to interpret what the employee was trying to do. It kept the people who were going for the conferences from competing with those who were really going for basic education and professional growth.” (Note, when asked how it kept them from doing this – the participant wasn’t really able to explain this – it was “on a subconscious level for me – it helped me to put it in perspective for the rating – in my mind.” They didn’t look at them as separate or do different amounts of funding for different amounts for different categories). - Panel Member

49

“I think there were very few that could have been categorized as ‘fact-finding’. I don’t think they played a terribly big role. I think they gave them to us by categories, but they didn’t play a role in the decision making process. Education, by far, and then Details, were the largest categories.” - Panel Member

3. The publicized descriptions of what grants may be used for could be misinterpreted.

• In the 2004 program announcement, “An Albright-Wirth Grant can be used for training, education, developmental opportunities, research, marketing, cross-over assignments, international or domestic official Government travel, procurement or contracting services, professional licensing fees, tuition, registration or conference fees, equipment, production expenses, product development, or technology-related projects. Grants can be used for honorariums, stipends, intern fees, or contracting services.”

“You need to examine this on a case-by-case basis to determine whether it’s appropriate or not. The types of activities are fine at face value, but you must examine the details of the proposal to know for sure.” - Panel Member

4. The following describes trends in the 209 projects funded within the last four years.

Certification: 14 Historic firearms 4 Aviation – plane or helicopter 4 Architecture 1 Horticulture 1 Occupational Health & Safety 1 Paralegal 1 Archeological technician 1 Human Resources 1

Conference: 20 Attendance only 13 Present session – other 3

Present research 2 Present training session 1

Promote NPS tourism 1 Note: 6 were connected with the National Association for Interpretation

Detail: 9 Administration 3 Natural or Cultural Resources 3 Education/Interpretation 2 Operations – Safety 1

50

Education / Coursework: 104 Subject Matter:

Management 23 Communications 18 Computer 16 Cultural Resources 15 Natural Resources 10 Interpretation/Education 8 Maintenance 8 Operations 6 Note: 8 grants were for Spanish language immersion courses

Provider:

Unknown 23 University (Comm. College – 4) 37 Private Company 27

Government Agency (NPS – 4) 17 Provide Employee Training: 14 Safety 8 Fire – 1

EMS – 2 Search and Rescue – 5 Other 6 Program Development: 24

Audience for program: Visitors 10

Volunteers 4 Employees 4

Park 3 Employees & Visitors 1

Park & Visitors 1 NPS 1 Research: 22 Cultural Resources 13 Natural Resources 3 Education 2 Administration 2 Operations 2 Travel: 1 Write/Publish Research: 1

51

• Evaluation participants had the following comments about trends in funding: “I think people should get some kind of meaningful feedback about their chances of getting funded – whether it’s on an individual basis or trends over time (e.g., only 5% of monies tend to go to this BECAUSE…). If most grants go to education and very few to go to conference or details, (maybe only 1-3 each were picked for fact-finding or research when she was on the panel), then I think it would be helpful to have that information analyzed and offered up to applicants.” - Panel Member “I think the AWG has become a cash-cow to pay for college tuitions. I feel very strongly about the AWG being used to pay for education – it’s almost exclusively for that. I understand the need/desire for college degrees, but again, I think people are using the AWG far too much just for that reason alone. The key has to be making absolutely certain that the project clearly results in the professional development of the person participating as well as clear benefits to the Park Service. Often going to college doesn’t seem to hit the mark.” - Leadership

5. The program has awarded 764 grants to 773 employees since its inception. The total number of projects completed is unknown.

6. The percentage of NPS employees who apply for grants is very small.

• The number of applications for the last fours years: o 2001 – 163 applications o 2002 – 135 applications o 2003 – 99 applications o 2004 – 174 applications

7. Individuals who apply for an A-W grant tend to apply again.

• Within the group interviewed, 50% of those receiving grants and 50% of those not receiving grants had applied in previous years.

8. Project completion rate is unclear. Reasons for not completing projects are varied.

• According to program leadership, the ‘response rate’ for completed final reports is believed to have been about 50-60%. It is believed the project completion rate is closer to 80%. There have been no consequences for not turning in a final report. And there has not been a procedure for contacting recipients or their supervisors to inquire about why the reports haven’t been done. And there’s only been very minimal inquiry from others (at NPS or the Foundation) about the final reports – what happens, what doesn’t happen.

• According to grantees, people who value their projects will stay on track and complete

them.

52

“I chose to do things that I really wanted to do and that I knew that I would enjoy so it was easy to stay focused.” “I have been able to stick to the plan because it was of great importance to me and I made it a priority.” “If it was something extra – something that I couldn’t directly apply to work, I don’t think I’d be able to do it.” “I am on track because I am dedicated to the project and enjoying every measure of progress.”

• Planning skills are another crucial reason why projects proceed and are completed on

time.

“Perhaps, [projects are not completed] due to unrealistic planning and expectations.” “I developed a timeline for completion of the travel and research...then I planned my work around those dates of travel. I started the project before I got the grant.” “Looking back, my request was pretty ambitious. Maybe they should look at the proposals and ask if they are really realistic.” “Once it’s approved, you should have to put out a specific plan with a reasonable time frame to accomplish what you say you’re going to do.”

• Panel members and program leaders suggested a variety of factors as barriers to completing projects.

“It’s human nature to underestimate how long it’s going to take to complete a goal. Also if people are being pulled from their duties to do this work, and if there’s fewer people around already, then the person starts getting pulled in many different directions – making it difficult for them to stay on track.” - Panel Member “People are really over-obligated and it’s hard to add this.” - Panel Member “These answers are based only on gut feelings - I think the government, as a whole isn’t the best at following time lines. Being late often is accepted and the consequences are low. Using government time to complete projects may not be easy to come by especially in times of budget problems.” - Panel Member When a project’s not completed it’s “because there’s no accountability – on the recipient’s part or maybe on their supervisor’s part.” For another training program, they’re considering requiring the park/program to pay for any training that a participant doesn’t complete. - Leadership

53

Some things that contribute to projects not being completed are: “Job performance/ productivity realities (time constraints); family obligations (time constraints); and dwindling need/interest between time of application and the disbursement of funds.” There is a lot of time between application (Jan/Feb) and funding (not available until October 1). Depending on the activity, they might not be that interested anymore; they might have changed jobs; the project may no longer apply to current duties for some other reason. - Leadership “I think that if folks don’t realize it is private dollars, this may affect how people view the importance of how and whether this money is spent. There really are no consequences to not spending government money. It either just gets shifted to other projects or purposes. I think we need to make sure recipients and their supervisors are aware that they are spending private funds.” - Leadership “There are a lot of great ideas out there and lots of wonderful projects proposed. But there’s not always adequate thought about whether they really can do the work if they get the grant. Maybe there needs to be more required in the application to get people to really think about this and be explicit about how and when they will accomplish what they propose to do.” - Leadership

• It was suggested that supervisors play an important role in employees applying for a grant

and in project completion.

“The two go hand in hand. If the supervisor is not supportive, it is less likely to happen.” - Panel Member “Supervisors can reduce the day-to-day pressure on a grantee and make the project a priority. If they don’t, it will likely interfere with completion.” - Panel Member “Supervisor support is really important in a lot of cases. Especially if doing something like this is new to the applicant. It almost seems like the application should be a joint application between the supervisor and the employee - help with writing the application, writing the reports, … It’s easy for the reports to get put off. I can see where supervisors can play a role in keeping employees on track.” - Panel Member A lack of support can make it quite difficult for an employee to complete their project. Motivation is important and a supervisor who is not supportive can certainly do damage. - Panel Member “I think the supervisor will support participation if there is travel money and staff to support the work effort while the participant is at a training event.” - Leadership “Often this is due to other pressures at work or from supervisors to spend time on other priorities.” - Leadership

54

“I think there’s little middle ground – extremely supportive or not supportive at all. Sometimes their position changes once the employee gets the grant – they endorsed them in the application, but then they don’t want to give them the time they need to complete the project.” This may call for more accountability of the supervisor in the completion of the project since they endorsed it from the start. - Leadership “Even though professional development theoretically leads to a better employee, supervisors may be focused on the here and now and not look to the future potential benefits to their park. Sometimes there might be concerns about providing professional development to someone who might then leave and provide the benefits of their training to some other park.” - Leadership One participant believes the vast majority of projects require grantees to leave the park, and sometimes “they aren’t able to leave the park. Perhaps the supervisor agreed it would be a good idea, but then in reality when it comes time to do it, they can’t.” “Those that are completed are probably clearly related to the park and so, they are a priority.” - Leadership “I think it made be tied into the budgeting in the Parks – their manpower – whether the supervisor can allow someone to be away from the park. I think there are people who can not apply because they cannot leave their position for any period of time to do this.” - Leadership “Some may feel an attitude in the work place about training and learning because they feel their supervisor won’t support it. Or feel they don’t have the time. There also seems to be an attitude about training in the organization that implies you have to be around a long time before you ‘qualify’ for training. So, if someone feels they are in the ‘in’ group, they won’t get it, so why try.” - Leadership “Supervisors need to see professional development and training as directly tied to employee performance, efficiency, and productivity. They need to understand that if it (professional development) doesn’t get done, then the work doesn’t get done. To me, it’s not the employee signing up on their own – it’s a cooperative agreement. If they just sign a sheet, then their not fulfilling their role. Their role ought to be to ensure those projects are done. If not, then maybe they have to send something in as to why it isn’t done. The supervisor needs to be directly tied to everything we’re doing and we’ve kind of let them escape – but we recognize they’re busy.” - Leadership “If supervisors agree to their employee’s participation, they should help to make sure the projects get completed and their staff has the time necessary.” - Leadership

• A variety of suggestions were made for ensuring projects are completed and dealing with grantees who fail to complete their projects in a timely manner.

“They should not be eligible for future grants. This is tough – certainly one doesn’t want punishment to be part of this process, which begins as a reward.” - Panel Member

55

“They should require a written work plan – detail exactly how the project will be completed in a timely manner. Some things get in the way – job issues, class schedules can change, but these can be dealt with as they come up. They need to understand the ramifications of not using or wasting money on the whole program. Need to be held to an agreement. To get this, the application will need to clear about what needs to be written.” - Panel Member “The central office needs to do a much better job of communicating with grant recipients – more often and more meaningful. Poor administration of the program also contributes to slow or poor performance of participants.” There should be more regulation and more communication between program administrators and grantees. Currently, the emphasis in the program is on getting out the announcement and managing the application process. After one year’s grantees are selected and notified, it seems the program moves on to planning next year’s program rather than managing this year’s grantees. - Leadership “To me, it’s not the employee signing up on their own – it’s a cooperative agreement. If they just sign a sheet, then their not fulfilling their role. Their role ought to be to ensure those projects are done. If not, then maybe they have to send something in as to why it isn’t done. The supervisor needs to be directly tied to everything we’re doing and we’ve kind of let them escape – but we recognize they’re busy.” - Leadership “If each employee is entitled to a certain amount of time for these types of activities, this should be enforced.” - Leadership “Try to get the grantees to resolve the issue for themselves, but if they don’t, then they might have to go the supervisor. Maybe the supervisors should be given/ need more information about how the process works – that an employee has received a grant, what the timelines are for completion… Maybe we should let them know if there are issues or problems, program managers may contact them to check on what’s happening and ask for their assistance in resolving the issue. Let them know up front that we expect them to be aware of and supportive of their employees’ participation in the program.” Leadership “Beyond the new reporting requirements implemented this year, very little else can be done to ensure projects are completed without being delegated more authority to impose penalties, etc.” - Leadership “Move the authority to the park level, a process that says you approved it, make it work or else…but I’m not sure what the consequence should be. I’m concerned about having them pay for it because this may discourage participation - people might not apply because they’re concerned they or their employee might not complete the project. Maybe we should focus more on making it a bonus for supervisors who help their employees complete their projects. But this would have to be something meaningful.” - Leadership

56

“Perhaps they should have a formal grant agreement with the grantees. This is standard practice at the Foundation. This makes them more aware that this is private money and there are expectations that go with getting it. There is a cycle in grants management and the cycle helps grantees be successful. There should be an agreement in place between the grantee and grantor that says you’re going to get this amount of money for doing this and you’re going to do this in this amount of time. In addition to the grantee, perhaps someone at the park level should also sign off on this agreement. There should be a mid-year report and a final report – in alignment with the value of the grant. Funding can be contingent upon reports being filed. For those who aren’t filing reports or moving along – then they need to communicate with them or with their supervisor.” - Leadership “I’m not aware of any official policy of how to handle incomplete or late projects. However, I strongly feel some type of guidelines or policy should be developed and publicized as part of the application process.” These guidelines should clearly cover how long grantees have to complete a project; the process for requesting an extension; how many times and for how long they can request extensions; how many “warnings” they receive about incomplete documentation or missing items; etc. The participant didn’t have any specific recommendations on what these guidelines should be. - Leadership “You move, you lose, and the funds earmarked for those projects is immediately returned to the trust fund for investing. This may sound harsh, but I think we’ve gotta move towards tightening the reins – the horse is leading the rider. The unspoken word is that you bend over backwards, and maybe this isn’t really appropriate.” The participant would like to see a panel convened to set some limits and criteria (10-12 people) – similar to the panel that reviews the applications – but with more people from the field – regarding contingencies and other issues. - Leadership “Funding can be contingent upon reports being filed. For those who aren’t filing reports or moving along – then they need to communicate with them or with their supervisor. The park may need to be told that they stand a chance of not receiving more grants if they don’t live up to their end of the agreement.” - Leadership

9. Even though the original intention of the program was for employees to complete

their projects as part of their regular job duties, recent written documents are unclear as to this requirement and applicants’ perceptions vary as well.

• Originally, the intention of the grant, as described in the 1987 program announcement,

was that “...all employee time devoted to any approved HAEDF activity is considered hours of work” (which was limited to the standard 80 hour pay period unless pre-approved by the Project Management Team).

• Currently, as described in the 2004 program announcement, “With supervisory approval,

time devoted to projects can be incorporated into an employee’s regular work schedule, or employees may request to use annual leave or compensatory time earned. Time devoted to a project is considered an official assignment.”

57

• When asked about the requirement: Supervisors must agree that employees be allowed reasonable flexibility and work time to complete project tasks, grantees had different understandings.

One grantee indicated that this [working on the project during work hours] was not her understanding of the grant and thought her project work had to be done after hours. Another suggested that the supervisor should be involved because the project “affects your workload.” “When you get this grant, you’re supposed to be on duty and supposed to be able to do it on work time. But everybody has more to do than time to do it. It’s not that supervisors don’t want to give people the time they need, they just can’t.” “With supervisory approval, time devoted to projects can be incorporated into an employee’s regular work schedule...”

10. The role and participation of the employee’s supervisor is unclear in the written

documents and applicants’ perceptions of this vary as well.

• In the 2004 program description, the supervisor’s role is described in contrasting and inconsistent ways: (Bolding of words has been added to contrast statements.)

o The roles are described under the heading: How do supervisors endorse their

employee’s proposal? (Endorse suggests backing, support, approval)

o “An essential part of a grant application is the supervisory endorsement statement.” (Endorsement suggests backing, support, approval)

o “Employees are strongly encouraged to involve supervisors in the application process... (Involve suggests engage, engross)

o At a minimum; employees should inform supervisors of their intention to request funding...” (Inform suggests tell, notify)

o “...negotiate an agreement that ensures productivity and performance will not be compromised if funding becomes available.” (Negotiate suggests talk, discuss, confer)

o “The supervisory endorsement is a typed statement from the employee’s supervisor...which confirms: 1) the supervisor’s acknowledgement (Acknowledgement suggests concession, acceptance, and allowance) and/or support (Support suggests encouragement, help, and assistance) of the project.”

o Also, confirmation (Confirmation is a synonym for verification) of length of time employed and verification (and vice versa) of level of performance are required by the supervisor.

58

• Grant applicants described the roles their supervisors played:

“I think the supervisor has a big part in employees getting a grant by their letter.” “They are key. There has to be a letter from the supervisor, and I think their support plays a key role.” “Supervisory input and support is very important. It affects the application process (IE whether someone applies or not). It affects whether someone receives the grant, and it affects whether someone completes the project.” “None in my experience, which is limited.” “I think the letter is just a requirement that has to be done. To be honest, I wrote those letters and gave them to my supervisor to sign. The letters actually might be a waste of time – if they really want input from the supervisor, they should call them on the phone and ask them if they think the applicant would be successful in the project they are proposing.” “The supervisor has a ‘big role’. It is definitely a barrier if a supervisor isn’t prepared or interested in helping employees apply.” This individual had the supervisor read and edit the application.

J. INDIVIDUAL OUTCOMES (Reactions, Knowledge and Skills, Attitudes, Behavior)

1. Reactions from evaluation participants were mixed regarding the program

administration staff, processes, and resulting experiences.

• Data suggest that, while NPF and NPS administrators, applicants, recipients, panel members involved with the A-W program have positive reactions to individual staff members associated with A-W, they also have strong concerns about the program administration process and the long term affects on the program’s success.

(See Sections E. General Program Management, F. Applying for a Grant, G. Selection Panel and Process, H. Grant Management and I. Outputs for specific quotes and details)

• Most problems with the process cannot be linked to specific causes.

For example, a specific communication to applicants being sent late could be due to: the process being flawed, the original time line being inappropriate, the staff being over-worked, the staff not being trained, the equipment and/or technology failing, management not being engaged in oversight, or many other reasons.

59

2. Reactions from all four groups of evaluation participants were very positive regarding the A-W program and the opportunities it provides.

• Individuals not receiving a grant had the following reactions:

“First, let them know I think it is a fantastic program.” “I think it’s a great program and I’m glad that it exists.” “It’s nice to know if you work for someone, they’re willing to help support and invest in you as much as they can. I hope they never stop offering it to their employees.” “I think it is a fantastic program and I hope to be successful in the future.”

• Grantees have positive reactions as well.

“I think the program is great. I truly feel that opportunities like this are part of why I’m where I’m at today.” “Overall, I think it’s a fantastic program. I’m grateful it exists. The programs I participated in have not only enhanced my ability to manage and implement my responsibilities, but some of them have really been life changing experiences.” “I think it is the best program the NPS has. It makes up for a lot of deficiencies in the parks. Keep it funded!” “I really feel as if the A-W grant program is a very helpful, good, and informative program. If it wasn’t for A-W, I wouldn’t have gained any training.” “It was wonderful. It’s the best training I’ve ever had in 30 years.” It was directly related to the grantee’s job and (s)he still uses the skills acquired in those courses. “This was training I wouldn’t have gotten that on the job. The Park Service seems to be standardizing to certain types of training in each field and that is the only kind of training they will provide or pay for. I would never have asked to take these courses through the Park.” - Grantee

• Panel members were also have positive comments.

“I think it offers hope. They want to gain knowledge and skills. They want to receive respect. They want to grow and contribute more to their job. They realize there’s a need among their colleagues, among their customers but they realize they need a little something to go forward and meet that need. It seemed that people are trying to respond to these needs and are trying to put themselves into the solution and but realize they can’t do it alone. When they have [the help the get from the grant] they can breath life into it.” - Panel Member

60

A-W is set apart from other NPS programs in that it “allows for an individual approach that’s still applicable to the job. Again, that out-of-the-box idea. There are many ways to develop knowledge and skills related to any career or job, and the A-W program allows for a more creative approach than the other standardized courses offered by the government.” - Panel Member “There’s training available from a variety of government agencies. These are a more passive approach – you look at a list and pick something out, whereas with Albright-Wirth, you need to have a goal or purpose in mind. Your location matters also. In DC, there’s all kinds of things available - not so in a remote park.” - Panel Member

• Program leaders emphasized the importance of the focus on the individual and flexibility the program offers applicants. “People apply to this program because of the individuality. Some people think more of the professional development and training programs should be like this so people can choose their own method of training. A lot of the funding provided by the Park Service is available for certain people to do certain things and if you fall outside of those lines, you can’t access the money. People are attracted to it because the program is specifically intended for individual use as opposed to meeting departmental goals. As it is right now, they can apply for just about anything they need for enrichment – it’s more personal.” - Leadership People apply to the A-W program “to do things they wouldn’t be able to do any other way.” Some of these things are more personally oriented and some are clearly professional focused, but there are things employees can fund through this program that wouldn’t be funded in any other existing NPS professional development program. There are many programs available to fund professional development (outside of the required competencies and programs), and most if not all the other programs have are “very organizationally oriented, whereas the A-W program allows individuals to focus on their personal goals.” - Leadership “Within the Park Service there are very few other sources of independent financial assistance. Other funding is limited to specific groups – whereas Albright-Wirth is open to everybody. Also, currently the application process is simple, straightforward, and accessible. This encourages people to apply.” - Leadership “In the private sector, your company often pays for personal professional development. Park Service never did this. That is one of the reasons the Albright-Wirth program was started. People who apply to this program are looking to expand their horizons either in their current job or in some direction they want to go in the future.” - Leadership “I think it’s really important because it focuses on the individual and then moves out. It’s a personal commitment – it’s in your own hands and here’s an avenue to help you make that happen. I think it gives people hope.” - Leadership

61

One program leader felt very strongly that “the program should remain self-initiated, because of the individual focus of the program.” Applying for this grant reflects “people taking responsibility for their own development and that’s very important. Most of the professional development and training that takes place and is offered through NPS is more ‘feeding’ of the employees what they ought to and have to know. This program is different and should continue to remain that way.” - Leadership

3. Participation in the A-W program provides many varied opportunities for NPS employees to gain knowledge and skills and improve attitudes toward the project’s content focus.

• Grantees identified gains in current and new subject matter knowledge and skills. Due to

confidentiality, few direct quotes regarding specific content have been provided but general categories are listed. Learning the history of the subject provided new job-related skills. Taking an exam along with the course provided extra subject matter learning. Gaining a complete picture of a subject rather than just focusing in on one small area provided new skills which were then not only transferred to current work within the subject area but enabled the person to teach it to others. “The best way to learn something is to teach it” so this individual can continue to learn after the A-W project is done. Grantees indicated that staying current (gaining cutting edge knowledge) in a subject matter area which is directly related to current job responsibilities was a desired and accomplished outcome. Others described how learning ‘basic principles’ of a new subject or developing additional basic skills in a current area allowed them to grow. “I want to go beyond what I’m doing now.” Learning about a new subject that is complimentary to and strengthens current subject matter expertise let grantees broaden and diversify their knowledge base and skills set. Individuals who gained their expertise in a subject ‘on-the-job’ were able to obtain new perspectives on the subject and learn the theory behind the practice. “[I] don’t have formal training in XYZ and saw this as an opportunity to expand [my] knowledge of the field.”

• Grantees gained knowledge about and skills related to being better leaders and professionals and more productive NPS employees.

The goal for one A-W grantee was to “To learn with others and through other people and to develop new skills, and the ability to motivate myself along with other people.”

62

In order to gain new knowledge and skills to improve as a leader, this individual used an A-W grant to present at an academic conference. “I had a lack of comfort in doing presentations to this kind of audience, and wanted to increase my comfort. and Regular training could never cover the types of things [I] focused on with this grant.” “Just that one grant has made me a better HR person.”

• Panel members also gained knowledge and skills from their experiences on the panel. “I had a good experience. It was a real learning experience for me.” - Panel Member “I have a much better understanding of how [the grant program] is managed and how [the final selections] are agreed upon.” - Panel Member “One thing I like about the process is that the panel spends a lot of time discussing the applications. There are many benefits to bringing the group together for a face-to-face review. I don’t think it’s a good idea to do it in isolation and send in scores – a lot is missed. It’s important to remember that serving on a review team has professional development value for the reviewers, and without the direct interaction with other panel members, the process is not as valuable for the participants.” - Leadership

• Grantees valued what they learned.

“[Applying the knowledge and skills] makes [me] feel good personally. It [the grant] had personal rewards as well. If you’re getting personal rewards, you’re going to give more to your job as well.”

4. Grantees’ on-the-job behaviors were changed.

“I use knowledge and skills I gained in those classes on a daily basis.” Individuals described how their projects gave them skills that they could immediately incorporate into their jobs. “[I] saw a direct connection between [my] current job duties and the subject matter of [the grant]...the training would make [me] more efficient in [the job]. “I have written articles for publication [in subject matter magazines], park publications, given public programs in uniform to represent the park [at a community event], written on the subject for a regional publication, used the [knowledge gained] to enhance [programs]...” “[I am] putting into practice everything [I’ve] been learning.”

63

K. SECONDARY OUTCOMES (For NPS, Parks/Offices, Other NPS Employees,

Other Individuals and Organizations Outside NPS) 1. NPS realizes multiple benefits from the A-W program.

“These skills [the skills targeted by the project] are the ones most in need at this time, economy, and social condition...so NPS gains an individual with key skills. - Grantee One grantee described how the project supported the professional image of the NPS. “It’s important to illustrate [to key audiences outside the NPS] that we are all doing work that is not only important but also meets academic standards.” - Grantee Another grantee is using knowledge and skills to keep the NPS up-to-date and on the cutting edge of the subject. - Grantee One panel member’s attitude toward the NPS was improved by participating. “I gained an understanding to the selection process and came out feeling even more positive about the National Park Service.” - Grantee

2. Parks/offices realize multiple benefits from the A-W program.

“It’s especially important for us to improve our skills, because with our shrinking budget [locally], we need to be the best at our jobs as possible in order to get the most bang for the buck.” - Grantee “We are reaching a new audience, and we are also able to offer services when a ranger is not available.” - Grantee

3. Other individuals within NPS realize benefits from the A-W program.

“The more skilled I am as a [leader], the more successful I can help my staff to be in their [jobs]. The better I can train them – the trickle down effect.” - Grantee “The better I am, the better I can help my staff to be.” - Grantee “I’ve brought the information back and shared it with other people in the Park Service to keep them up to date...” - Grantee “I have...traveled to other NPS sites to provoke interest in the subject for their audiences and personnel.” - Grantee “It [the new skills] allows us to translate some of the data so managers can make decisions with it.” - Grantee

64

“I assist other parks with their [work in the same area]; I can now use the knowledge gained at the workshop to help other parks.” - Grantee “All of these [skills] have been put to use in this park and through classes I’ve done in [audience] and other parks and the regional office. - Grantee

4. Other individuals and organizations realize benefits from the A-W program.

“We are better able to work with schools...[using new knowledge and skills from A-W project].” - Grantee “Students throughout the country can access and benefit from it [product developed using new knowledge and skills from A-W project].” - Grantee One grantee described how s(he) shares the knowledge and skills learned from the A-W project in other groups s(he) belongs to outside of the job.

L. RECOMMENDATIONS

1. Establish a steering committee comprised of representatives from each partner (NPS & NPF). The committee will review each recommendation, weigh its merits and implications, and determine whether to accept or reject it. The final set of recommendations should be given to an advisory panel to develop an action plan to implement the agreed upon changes. The panel should include representatives from the National Park Foundation, the National Park Service, past Albright-Wirth applicants, grantees, and panel members. This panel should complete its duties in time for the agreed upon changes to be implemented for the FY2006 grant cycle.

2. Partner responsibilities should be redefined to capitalize on the strengths of each partner. Along with the management of the endowment, the grant-making and management process should be moved to NPF which has the personnel and resources to address the majority of the administrative concerns identified in the evaluation. NPS would oversee the marketing, application and project assistance, and have the majority of the membership of the selection panel.

3. Based upon the overwhelming number of positive outcomes for all participants interviewed, partners should consider focusing efforts on growing the endowment so more NPS employees are able to receive benefits from A-W-funded projects.

65

4. Update the categories and utilize them to allow for diversification of awards to include partnerships, teams, parks, and other configurations of applicants beyond individuals without compromising the original intention of the fund. Consider requiring a match for applicants other than individuals.

5. Specific criteria should be determined for projects in each category. After criteria have been identified, they need to be clearly defined, shared in the application and other program materials, and applied via a rubric or other measurement tool to judge grant proposals.

6. Long-term seasonal employees should be reinstated into the pool of eligible employees.

7. No recipient (individual or group) should receive an A-W grant more than once every three years.

8. As stated in the original 1987 program announcement, “...all employee time devoted to any approved HAEDF [A-W] activity is considered hours of work.” This requirement must be clearly communicated to all relevant parties, agreement with the requirement should be obtained from all appropriate parties, and the requirement should be equally enforced across all projects.

9. The role of the applicant/grantee supervisor or management team must be clarified and clearly articulated. This would include: letter accompanying application, role in project completion, approval for project work during work hours, responsibility for appropriate use of project funds, supporting professional development related to an employee’s career verses the current job and other roles, as identified.

10. The role of the supervisor letter should be clearly specified as one or more of the following: a) acknowledgement that the employee is applying b) permission for the employee to apply c) support for the employee to apply d) encouragement for the employee to apply e) provision of factual information about the employee f) provision of subjective information about the employee. If any of the information supplied by the supervisor is used as a criterion for screening or selection, this should be clearly communicated to all.

66

11. To improve and ensure consistency, a program cycle/schedule should be established. Dates for the program announcement, application deadline, review process, recipient and non-recipient notification, and reports should be determined, publicized, and strictly adhered to.

12. A comprehensive marketing plan should be developed, implemented, and continually evaluated to determine how various strategies are working. Improved marketing strategies should: ♦ increase the presence of the National Park Foundation in program materials ♦ encourage employee contributions to the fund, especially from past

participants ♦ promote grantees, their accomplishments, and program outcomes

13. The A-W program should be listed in all categories of training appearing in My Learning Manager. However, computer and e-mail access is still a major impediment for some NPS employees and should not be relied upon as a sole or major marketing, application, and communication medium.

14. A relatively small number of grants, if any at all, should be set aside to award to individuals whose projects are strictly college courses.

15. Provide a variety of opportunities for applicants to receive assistance such as: ♦ Examples of well-written proposals accessible online or by mail as requested ♦ Previous grant recipients who will serve as mentors ♦ Guidelines for proposal writing available online and in the grant

announcement

16. Provide prompt responses to inquiries and requests for assistance from grant applicants as well as grant recipients.

17. The importance of the “renewal” aspect of the professional development project should be elevated. The project should be clearly tied to individual renewal related to the employee’s NPS career, not specifically the current job.

18. The importance of the “innovative” aspect of the professional development project should be increased. Projects that are creative, resourceful, and imaginative should be emphasized.

67

19. Develop written criteria for selection panel membership. At a minimum, each panel should include one individual with expertise in the following: professional development, cultural resources, natural resources, and operations.

The selection process should be managed similarly to other major grant selection processes. An initial screening should eliminate ineligible projects and should separate projects into selection pools based upon pre-determined categories. Applications should be read and quantitatively rated before the selection team meets. Criteria should not be negotiable nor flexible and the team meeting should be primarily to work through major disagreements on ratings. After an initial pool of proposals is selected for funding, this pool should be reviewed based on the following considerations: a) Do they represent a cross section of NPS divisions/disciplines?; b) Do they represent a cross section of NPS grade levels; and c) are the projects sufficiently diverse? As necessary, the panel will need to refine the final pool to reflect these considerations.

20. Any required report should have a clear purpose for use. Grant recipients should be held to deadlines and formats.

21. Establish a policy and procedures for tracking project progress and redistribution of funds. In particular, the policy must address how to deal with the following: ♦ Funds from individuals who are awarded grants but subsequently elect not to

accept the award. ♦ Individuals who do not follow program guidelines. ♦ “Extra” funds left over from completed projects.

22. Design an overall program informational database from which summarized

program information can be derived as needed or desired.

23. The results of this evaluation should be summarized and made available to individuals who participated in the evaluation.

24. Three years after the agreed upon changes from this evaluation are implemented, an external evaluation of the program should again be undertaken. In the mean time and henceforth, internal evaluation should be incorporated as a regular part of the program activities.