Evaluation and Monitoring

Wyoming Homeless Collaborative –  WHC
Evaluation Committee Policies and Procedures
May 2015

The Project Evaluation Committee’s purpose is to coordinate and implement the rating and ranking of new and renewal projects for the HUD Continuum of Care Grant Competition.
Grant recipients are required to rate and rank all applying project applicants for the HUD Continuum of Care Grant Competition based upon  an agency’s project alignment with HUD’s priorities for the fiscal year in question and the priorities identified by the Continuum of Care, also known as the Wyoming Homeless Collaborative.

DEFINITIONS:

Chairman of the Evaluation Committee:  The Chairman of the Evaluation Committee is a member of the Executive Board.  The Chairman recommends to the Executive Board committee members to be selected to the board.  Thereafter, the Chairman facilitates Project Evaluation Committee Meetings and voting procedures, and reports back the results to the Executive Board. The Chairman has no voting rights on the Evaluation Committee. The Chairman’s role is to facilitate educational discussion, notify members and parliamentary procedures when conducting meetings.

Collaborative Applicant:  Designated by the WHC to collect the required documentation for application information within the geographic area of the Continuum of Care and apply for the Continuum of Care Grant.  The Collaborative Applicant’s duties on the Evaluation Committee are to educate and advise the Committee on the Notice of Funding Availability, HUD priorities for that given funding cycle and the funding priorities identified by the Continuum of Care during the Annual Priority Discussion which occurs at the June general membership meeting.  The Collaborative Applicant notifies applicants, in writing, of their ranking for the WHC Grant Competition 10 days prior to the date of final grant submission.  The Collaborative Applicant  submits the final Rating and Ranking Worksheet to HUD for the Competitive Grant Competition. The Collaborative Applicant has no voting rights on the Evaluation Committee.

Applicant: An agency submitting a project proposal for funding through the Continuum of Care Grant Competition.

Evaluation Committee Member:  A project applicant or prospective project applicant selected by the Executive Board of the Continuum of Care, also known as the Wyoming Homeless Collaborative, tasked with evaluating and ranking both new and renewal grant projects for submission to the HUD Continuum of Care Competitive Grant Competition. Members must be in good standing with WHC.

Eligible proposals will be prioritized for inclusion in WHC’s coordinated application by the Evaluation Committee acting as the rank and review group.

Applications not scoring high enough will not be placed on the project funding request as part of the Consolidated Application.

WHC leadership recruits Evaluation Committee members, who have served as members in the past or who have other relevant experience. The Evaluation Committee will be composed of representatives from a cross-section of groups which might include: Faith-based and non-profit providers of homeless services and housing; city representatives; county employees; mental health; substance abuse; veteran’s services; and consumers.

  • Evaluation Committee members must declare that they have no conflict of interest.
  • Members must be appointed every year and their eligibility verified.
  • Members must be able to dedicate time for application review and committee meetings.
  • Evaluation Committee members (5-7) will participate in an orientation which includes:

* Information regarding homeless activities, needs, services, definitions and other issues that are pertinent to WHC
* A background of McKinney Vento
* Review the role of the Evaluation Committee
* Review of the scoring tools, applications, and resources
*The Evaluation Committee, along with all prospective Project Applicants and the Collaborative Applicant are assigned a portion of the recently released Notice of Funding Availability (NoFA) to be presented via teleconference at assigned Project Evaluation Committee Meetings. Unless excused by the Committee Chair, any prospective Project Applicants who fail to participate will be excluded from the competition. This is to ensure that all prospective applicants are familiar with HUD’s Eligibility and Project Quality Thresholds, that year’s Project Priorities and any requests to change funding. (i.e. 5% funding cut, etc.)

Evaluation Committee members receive eligible application proposals and scoring  materials.

  • Prior to the Ranking meeting, all Evaluation Committee members review all applications, preferably over a one-week period. Members read projects, preliminarily score and note any questions/comments to follow-up with applicants.
  • If the WHC staff have any knowledge that could lead HUD to deny granting funds to a project, they will share the information with the Evaluation Committee. WHC staff will discuss this information with applicants as part of technical assistance provided to assist project development.
  • The Evaluation Committee meets to review and discuss each application together and to individually score them. Each member  scores each applicant on WHC’s Scoring worksheet and submits their completed worksheets to the Evaluation Committee Chair and the Collaborative Applicant or WHC staff.

WHC staff is present at the Evaluation and Committee meeting to record decisions of the Committee and any comments/ recommendations they have for applicants.

* The Evaluation Committee discusses the merits of each proposal, scores the applications, and turns in score sheets to  WHC staff.
* Overall raw scores are calculated by WHC Staff.
* The Evaluation Committee considers adjustments for such issues HUD incentives or requirements.
* The Evaluation Committee considers proposed changes or project budget adjustments that may be required to meet community needs.
* The Evaluation Committee determines the rank and funding levels of all projects considering all available information.
* During deliberation, WHC staff will provide technical assistance by responding to questions of  the Committee members, correcting technical inaccuracies if they arise in conversation and reminding the Committee members of their responsibilities if they step outside their purview.
* Each applicant receives copies of their proposals with technical edits made by WHC staff.  Applicants are asked to correct their applications and send them back to WHC staff before final  submission to HUD.
* Scoring results are delivered to applicants by the Collaborative Applicant or WHC Staff verbally and in writing, no less than 10 days before the grant submission deadline. A reminder of the appellate process will be included with the written document.
* Applications which do not meet the threshold requirements will not be included in the Priority Listing in the Consolidated Application, and therefore will not be forwarded to HUD for consideration.
* If more applications are submitted than the WHC has money to fund, the lowest-scoring  applications will not be included in the Priority List  and therefore will not be  forwarded to HUD for consideration.
*The Collaborative Applicant completes the Rating and Ranking Worksheet and enters into e-snaps along with the entire WHC Competitive Grant Competition.
*The Evaluation Committee debriefs following the grant submission along with the project applicants to identify ways in which the process could be improved in the next grant cycle.
*The Chair of the Evaluation Committee reports back to the Executive Committee on the outcome and grant submission at the quarterly Executive Board meeting.

Policy for Appeals of Rating/Ranking

Eligible Appeals

  • The application of any applicant agency which a) is unranked, or b) receives decreased funding (e.g.

projects receiving reallocated renewal funds) may appeal.

  • Applicants that have been found not to meet the threshold requirements are not eligible for an appeal.
  • Appeals cannot be based upon the judgment of the Evaluation Committee.

Applicants may appeal if they can: prove their score is not reflective of the application information provided; or  describe bias or unfairness in the process, which warrants the appeal.

All notices of appeal must be based on the information submitted by the application due date. No new or additional information will be considered. Omissions to the application cannot be appealed.

Per 24 CFR 578.35(b), project applicants that believe they were not allowed to participate in a fair and open process and that were rejected by the WHC may appeal the rejection directly to HUD by submitting as a Solo Application prior to the application deadline.

WHC TIMELINE

*Before the NoFA is released by HUD the Evaluation Committee should be in place.
*The Evaluation Committee Chairman calls for interested participants to the Project Evaluation Committee. The Chairman submits the recommendation to the Executive Board for selection to the Evaluation Committee. The Evaluation Committee Chairman then notifies the new Committee members and coordinates meetings.
*When HUD releases the Notice of Funding Availability (NoFA), the Collaborative Applicant will notify all potential applicant agencies via an announcement on the Wyoming Homeless Collaborative Website and the Wyoming Homeless Collaborative e­mail list serve.
*When the NOFA is released the Collaborative Applicant  will create a  timeline of events with specific dates to be sent to the general membership.
* A call for Letters of Intent is made through both the Wyoming Homeless Collaborative website and e-mail list serve.
*The Evaluation Committee will review the NoFA, develops a Letter of Intent with a corresponding Scoring Worksheet.

WCH On-Site Monitoring Policy and Procedures
May 2015

Project Monitoring Committee is tasked by the Executive Board of the Continuum of Care to ensure grant recipients are administering Federal Grants effectively, develop quality standards for projects approved by the Continuum of Care, and ensure that grant recipients are compliant with those standards.  They are asked to provide data on the compliance and project quality of renewal project applicants.

The Wyoming Homeless Collaborative (WHC) Monitoring Committee members, to include a representative from WHC, shall make an annual visit to CoC funded projects. In order to avoid conflicts of interest, any committee member conducting the visit shall not be a staff member of any agency participating in the project.

A WHC Monitoring Committee member will contact the  grant recipients to schedule an on-site monitoring visit and confirm the monitoring visit in a written confirmation letter. ( Appendix A-1) which will include a copy of the WHC Program Monitoring Checklist (Appendix A-2) and the Monitoring Visit Summary(Appendix A-3).

On-site monitoring visits will be coordinated, when possible, for programs that receive both CoC Program and Emergency Solution Grant (ESG) funds.

On-Site monitoring visits will consist of the following activities/components:

  1. Tour facility/units. Site/unit visits are conducted to evaluate the physical adequacy of the housing  based on program type.
  2. Review and discuss most recent APR and HUD monitoring results, goals and progress.
  3. Review random sample of program files: Comprehensive file review will be completed on at least one file, with other files reviewed if necessary, based on findings.  Review results of client satisfaction survey and/or conduct client interviews.
  4. Identify technical assistance needs.

A written summary of the visit will be provided to the grantee within 45 calendar days that includes a PASS/FAIL notification, findings or audit concerns.

The grantee will have 14 days to respond in writing to the visit summary.

If, based on project performance and the grantee’s written explanation, the Monitoring Committee has no further response or performance recommendations, the  Monitoring Visit Summary and the grantee’s written response(s) will be presented to the WHC Board on an informational basis.

If, based on project performance and the grantee’s written explanation, the Monitoring Committee has continued performance concerns, the Monitoring Committee will draft recommendations to address identified concerns. Timelines for corrections and follow-up visits are anticipated to be sixty (60) days or less, but will be defined on a case-by-case basis in the approved recommendations. Correspondence will be approved by the WHC Board. The process of follow-up visits, summaries, and reports to the WHC Board will follow the same the process as the initial Monitoring Visit. However, follow-up visits will specifically address grantee’s fulfillment of the recommendations of the WHC Board and Monitoring Committee. In addition to an annual Site Visit, the Monitoring Committee or a representative from WHC, may conduct desk monitoring reviews of program audits, drawdown requests, APRs and other documentation as necessary for the prioritization process and response to the NOFA.

Requirements for WHC CoC Grant Recipients

  • Grantees will provide WHC a copy of the signed grant agreement with HUD
  • Provide WHC monthly drawdown reports, due by the 15th of the following month
  • Participate in no less than one on site WHC monitoring visit
  • HMIS Participation Required
  • Provide WHC quarterly Project Report due, no later than the 15th of month following end of quarter.
  • Provide WHC a copy of APR due within 90 days of the end of project’s grant term
  • A representative from the project will serve on Evaluation & Monitoring Committee
  • A representative from the project will participate in a peer monitoring visit of another CoC grant recipient ?
  • Notify WHC of issues with Hud/ tenants/residents

WHC Monitoring Committee Information

Monitoring Staff present:

Date of Monitoring Visit:

WHC Program Grantee: Agency and Program Information
Agency:

Program Name:
Agency staff consulted:
Grant Amount:
Contract Number:

Program Type: PSH RRH TH SH

Number to be served: Number of chronic beds/units:

Program serves: Individuals Families Both Individuals and Families

WHC Program grant funds are used for:

Rental Assistance ⎕ Leasing (no match required)

Operations Admin

Supportive Services

Is the agency a faith-based organization? Yes No

Matching funds (25% required) are: Cash/Cash Equivalent In Kind N/A

Is there an active restrictive covenant on one or more of the project properties? Yes ⎕NO

FY 2016 CoC Program Competition Renewal Project Scoring Tool
Factor Possible Points This  Score
1.   Project’s Work is Consistent with HUD Priorities 30
2.   Outcomes 40
3.   Agency Capacity 20
4.   Budget 10
5.   Reallocation Bonus 5
Total 100

Note: HMIS renewal projects and renewal projects operational less than one year will be

reviewed for threshold and placed at the bottom of Tier 1.

DETAILED BREAKDOWN BY FACTOR

  1. PRO JEC T’S W O RK IS CON S I S TE N T WI TH D PRI ORI TI ES (30 P T S.
Factor Possible Points This Score
1.A. HUD Priorities

Renewal permanent housing (PSH & RRH) = 20 points Renewal transitional housing = 5 points

20
1.B. Prioritizing Chronically Homeless Households

Percentage of new clients who are chronically homeless

• Consider: program type, household type

10
  1. PROJECT PERFORMAN CE OU TCOME S (40 P T S. )

Keep in mind that outcomes will naturally be lower in a more difficult to serve population such

as chronically homeless people, homeless people with mental and/or addictive illnesses.

Factor Possible Points This Score
2.A.1.a Average Length of Stay PSH: N/A

RRH, TH: Average length of stay for participants

10
2.A.1.b Exits to Permanent Housing PSH: N/A

RRH, TH: % of participants who exited to a permanent destination

10
2.A.2 Increasing Housing Stability

PSH: % of participants who remained in the program for at least 6

months or exited to another permanent destination

RRH, TH: N/A

20
2.B. Exits to Homelessness

PSH, RRH, TH: % of persons who exited to homeless temporary destinations (emergency shelter, transitional housing, place not meant for human habitation, or hotel/motel)

5
2.C. Increasing Income

PSH, RRH: % of adults who maintained or increased any income as

of the end of the operating year or project exit

TH: % of adults who increased their earned income as of the end

of the operating year or project exit

10
2.D. Connecting to Non-Cash Mainstream Benefits

PSH, RRH, TH: % of all adult participants who access non-cash

mainstream benefits

5

GENCY CAPACI TY (20 P T

Factor Possible Points This Score
3.A. Agency Capacity

Does the agency have the expertise, staff, procedural, and administrative structure needed to meet all grant audit, administrative, and reporting requirements?

• Any outstanding HUD findings and/or financial audit findings?

• Has HUD deobligated any of the agency’s/program’s grant

funds in the past three operating years?

• Does the letter of intent submitted reflect an agency with capacity that is sufficient to carry out the HUD administrative requirements?

6
3.B. HMIS Participation & Data Quality

Is the agency/program actively participating in the HMIS?

• Consider: the percentage of the project’s clients who have

data entered into HMIS; HMIS Data Completeness Report Card

• Domestic Violence programs that do not participate in HMIS

Will be required to participate in a comparable data

Management and reporting system.

 

6
3.C. CoC Participation

Does the agency and/or project sponsor participate in the Wyoming Homeless Collaborative and CoC-related planning meetings?

3
3.D. Quality Assurance

Does the agency and/or project maintain policies, procedures, and

actions to ensure continuous quality improvement?

• Is the agency implementing evidence-based practices and

approaches, including Housing First and Coordinated Entry?

• Does the agency train its staff to ensure high quality of care?

• Does the agency assess quality of service and consumer

satisfaction through surveys, focus groups, etc.?

• Does the agency monitor program performance using data?

5
  1. BUD GET   (10 P T S. )
Factor Possible Points This Score
4.A. Project Capacity (Utilization Rates, Participant Eligibility)

Is the project at capacity in meeting the number of homeless

people it is designed to serve?

• Consider: point-in-time and annual capacity data, unit

utilization rates, participant eligibility

5
4.B. Unspent Funds and Drawdown Rate

Is the project using its grant efficiently and regularly?

• Consider: percentage of funds left unspent

and rate drawdown

5
  1. REALLO C ATIO N BO NU S (5 P T S. )
Factor Possible Points This Score
5.A. Budget

Did the Agency voluntarily reallocate a renewal project?

• How much funding was reallocated?

• What was the project type?

5

FY 2016 CoC Program Competition New Project Scoring Tool

Factor Possible Points This  Score
1.   Project’s Work is Consistent with HUD Priorities 20
2.   Project Design and Readiness 30
3.   Agency Capacity 40
4.   Budget 10
5.   Reallocation Bonus 5
Total 100

Note: This tool will be used for reallocated projects and permanent housing bonus projects.

DETAILED BREAKDOWN BY FACTOR

  1. PRO JEC T’S W O RK IS   C ON S I S TE N T WI TH HU D P RIORITIES   (2 PTS.)
Factor Possible Points This Score
1.A. HUD Priorities

Permanent Housing Bonus Funding:

Permanent supportive housing = 20 points Rapid re-housing = 15 points

Voluntarily Reallocated Funding (from Transitional Housing):  Permanent supportive housing = 20 points

Rapid re-housing = 15 points

Involuntarily Reallocated Funding (from projects not meeting threshold):  Discretion of Review & Rank Panel based on community need

20
  1.  PROJECT DES I G  AN D R EAD INESS   (3 0Consider the overall design of the project in light of its outcome objectives, and the Continuum of Care’s goals that permanent housing programs for homeless people result in stable housing and increased income (through benefits or employment).

Threshold Criteria: The project will be ready to start by HUD’s statutory deadlines.

Factor Possible Points This Score
2.A. Program Design – CHOOSE ONE *

*For PSH/RRH: Housing where participants will reside is fully described and appropriate to the program design proposed. Program design includes provision of appropriate supportive services.

• Does the program design include the use of innovative or evidence- based practices?

• Will the project be ready to start within HUD’s statutory deadlines (e.g., can demonstrate site control, has plan to identify units)?

• Is the project staffed appropriately to operate the housing/services?

• Are staff trained to meet the needs of the population to be served?

• Does the program include involvement of clientele in designing and

operating the program?

• Does the method of service delivery described include culture- specific/sensitive elements?

• Will the program be physically accessible to persons with

disabilities?

*For Coordinated Entry: Program design is in alignment with coordinated

entry system design envisioned by CoC.

• Is the system easily accessible for all persons within the CoC’s geographic area who are seeking information regarding homelessness assistance?

• Is there a strategy for advertising the program that is designed specifically to reach homeless persons with the highest barriers within the CoC’s geographic area?

• Is there a standardized assessment process?

• Does the program ensure that program participants are directed to

appropriate housing and services that fit their needs?

*For HMIS: Program design is in alignment with CoC’s data needs.

• Will the project conduct trainings and otherwise prepare projects

for the new HUD Data Standards?

• Will the project provide data in a form that can be analyzed to assist

the CoC in assessing homeless needs, allocating resources, and coordinating services?

20
2.B. Services Partnership or Capacity

There is a committed relationship with a service provider with a signed letter of commitment or MOU; if agency is providing services itself, they have shown they have the funds to do that.  

5
2.C. Program Outcomes

Program outcomes are realistic but sufficiently challenging given the

scale of the project.

Outcomes are measurable and appropriate to the population being

served.

5

Y   (4 0 PTS.)

Factor Possible Points This Score
3.A. Agency Capacity

Does the agency have the expertise, staff, procedural, and administrative structure needed to meet all grant audit, administrative, and reporting requirements?

• Does the agency have any outstanding HUD findings and/or financial

audit findings?

• Has HUD deobligated any of the agency’s grant funds in the past

three operating years?

• Does the application packet that was submitted reflect an agency with capacity that is sufficient to carry out the HUD administrative requirements?

20
3.B. HMIS Participation

Is the agency/program actively participating in the HMIS?

  • Consider: the percentage of the program’s clients who have data

entered into HMIS; HMIS Data Completeness Report Card

  • Domestic Violence programs that do not participate in HMIS

Will be required to participate in a comparable data

Management and reporting system.

5
3.C. CoC Participation

Does the agency and/or project sponsor participate in Wyoming Homeless Collaborative and CoC-related planning meetings?

5
3.D. Quality Assurance

Does the agency maintain policies, procedures, and actions to ensure

10
  1. BUD GET   (10 P T S. )
continuous quality improvement?

• Is the agency implementing evidence-based practices and

approaches, including Housing First and Coordinated Entry?

• Does the agency train its staff to ensure high quality of care?

• Does the agency assess quality of service and consumer satisfaction

through surveys, focus groups, etc.?

• Does the agency monitor program performance using data?

Factor Possible Points This Score
4.A. Budget

Is budget clearly articulated, with no unnecessary or unexplained items?

10
  1. REALLO C ATIO N BO NU S (5 P T S. )
Factor Possible Points This Score
5.A. Reallocation

Did the Agency voluntarily reallocate a renewal project?

• How much funding was reallocated?

• What was the project type?

5