Section VII – Lessons Learned and Recommendations
Similar to the Homeless Count in 2005, there have been a number of "lessons learned." A discussion of those "lessons learned" and recommendations follow below.
Lesson Learned 1: Standardizing data extracts from HMIS
The 2007 Count was the first time that South Carolina attempted to incorporate HMIS data in its understanding of the Homeless population and used for an official count. While information was requested from all of the continua; only four of the five continua contributed data from their HMIS systems for this effort. While the continua utilize their HMIS systems to provide information on the people that they serve, that data had never been pulled together at the state level in an electronic format. In doing this, the analysis team faced several hurdles including the development and implementation of confidentiality agreements, defining and receiving electronic extracts, and analyzing those extracts in a short period of time. What resulted were four differing extracts. During this period, valuable staff time was spent attempting to understand the variations in the information. Obviously the variations in the information made it challenging to standardize across the continua and collapse the information in a state level file.
While agreements are now in place between each of the Continua and ORS, those agreements should be reviewed and expanded if necessary. More importantly, it is essential that ORS begin to receive regular extracts. New and expanded agreements between the Continua and ORS and regular extracts (perhaps quarterly) from HMIS to ORS should improve the quality of extracts and understanding among the agencies. ORS should work with each of the Continuums and develop a standardized data extract. ORS should work directly with the HMIS data managers or data consultants with each continuum for data delivery.
For use in the 2009 Count, all extracts from the HMIS systems need to be delivered within one week of the count date. Another recommendation would be to stage a "dry run" prior to the count in order to identify any data issues. This dry run should happen no later than July 2008.
Lesson Learned 2: Issues on the quality of information from HMIS
During the 2007 process, several quality issues were identified while attempting to use the HMIS extracts. In many communities outside of South Carolina, the HMIS systems provide the majority – if not all – the information for the count. It is a goal in South Carolina to use the HMIS systems primarily for the next count. For that to happen, quality issues must be addressed early on. In general it is recommended that the HMIS coordinators should address issues and begin developing a plan for improving the use of HMIS for the count in 2009. This would need to be in conjunction with ORS who can share their experiences with the HMIS coordinators.
There were several specific issues regarding quality. Data on universal elements was not consistently entered by providers within the continuums. This is a local issue that must be addressed this year with special attention focused on disabilities and exit data. In spite of HUD's emphasis, it was also believed that some of the data on chronic homelessness needed improvement.
The Continua also need to convene and come to a common understanding of data definitions to ensure that the data elements mean the same across the systems. Another recommendation would be that the HMIS systems report quarterly on universal data elements to ORS. Quarterly reporting would not only help to identify data issues and standardize – the state of South Carolina would also receive valuable information on homeless services throughout the year not once every two years. A quarterly process would also help with data cleaning.
Additionally, a consensus from the continuums needs to be reached on the management and/ or use of the domestic violence data in HMIS. Traditionally there has been heighted concern regarding the security and confidentiality of domestic violence information. However, to better serve this population and secure additionally funding, it is important to have information.
If HMIS information is not usable in the count as identified either through regular extracts or through a dry run held no later than July 2008, then the Count team should ask for paper forms from all of the shelters.
Lesson Learned 3: Reviewing the use of volunteers
South Carolina utilized volunteers during both of their Homeless counts. While training and improvement of the forms were emphasized during the 2007 count, it still remains a challenge utilizing volunteers. Some data quality issues perhaps would occur with any interviewer such as the inability to read the person's handwriting. Poor data quality resulting in the creation of "other forms" and the misinterpretation of the forms and data definitions would be better addressed with experienced or professional interviewers.
Ideally it would be best to use experienced interviewers trained in collecting data. However, if volunteers are necessary for the 2009 count, more control of volunteers would aid hopefully in improving data quality. Volunteers must receive full training prior to going into the field. They should practice filling out the forms. A dry run at a soup kitchen one month prior to the count should be mandatory thereby ensuring that the forms are finalized, pre-tested, and reviewed to see who is completing them correctly and neatly. There should be a no exception policy to this mandatory training. Perhaps teams of college students or guardsmen or others marshaled en masse and with a centralizing coordinator (like a faculty member) could be recruited for the 2009 Homeless Count. Fewer people with a longer time commitment would probably result in better quality than a lot of people who go out for only a few hours.
Lesson Learned 4: The forms, the forms, the forms...
Obviously critical to the collection of the information during the 2007 Count were the actual instruments. The Count team worked judiciously on the creation of the instruments – reviewing not only the questions asked during the 2005 Count but also reviewing questions used by other states or in national surveys. Unfortunately there is no perfect balance. Forms that have lots of forced choice questions so that volunteers would not have to make judgment calls were criticized because the lists were too long and volunteers skipped them. However, without long lists of forced choice questions, volunteers had difficulty distinguishing situations that may or may not be considered homeless or a host of other definitions that were mandated through HUD. It is unclear how to balance strict data definitions and usability of the instruments. If however, the information is instead collected through the HMIS systems, forms may no longer be as critical as an issue.
Lesson Learned 5: Not quite a lesson learned, but maybe we should explore other options
While HUD mandates the Homeless Count, it leaves the states leeway in its implementation. While South Carolina has attempted an actual "count" for both 2005 and 2007; HUD does leave the door open for other methodologies such as estimation.
Until an entity at the state level receives funding and/or authorization to implement the count statewide; South Carolina would continue to be forced to utilize volunteers. Massing an effort such as the 2007 Homeless Count is not easy task. Instead, perhaps the Count team should explore the use of extrapolation. In this methodology, a sample of counties (using critical criteria such as poverty, ethnicity, and rurality) would be picked. In those counties, resources and training would be focused. The results of those counties would be used to estimate the state as well as estimating other counties. Obviously this methodology would have to be well researched. Because the count is mandated by HUD and funding is contingent on it, the continua would need to feel secure in this methodology.