« Back to Previous Page
In this edition of Research Roundup, we discuss two studies that highlight the usefulness of the internet in connecting agencies with their clients and the importance of making web sites easy to use.
Download the report in PDF
Inside This Research Roundup:
Internet-Based Fidelity Monitoring
- Multidimensional Treatment Foster Care Program
Social Service Web Site Usability
Internet-Based Fidelity Monitoring
Internet-Based Fidelity Monitoring System for the Multidimensional Treatment Foster Care Program
Over the last 20 years, research evidence has validated the effectiveness of a number of interventions for adolescent behavior disorders. Although communities have increasingly sought to adopt evidence-based practices, community implementations of these interventions have not been as effective as the clinical trials. A number of reasons exist for this difference in efficacy; one significant barrier has been that community providers have found it difficult to replicate these interventions without intensive fidelity monitoring and support from the intervention developers, which can be difficult and costly for communities geographically distant from the developers.
Dr. Edward Feil from the Oregon Research Institute and his colleagues examined a pilot test of a multimedia internet-based fidelity monitoring system (ITFMS) as a potential solution for remote monitoring of reporting and data collection by clinicians on the community level. Developers created the ITFMS for the Multidimensional Treatment Foster Care (MTFC) program, an in-home treatment for adolescents with antisocial behavior problems and the only intervention that has been empirically demonstrated to be effective with youth who have been removed from their homes due to extensive antisocial behavior.
The goal of the ITFMS was to help community implementation sites maintain fidelity to the MTFC program model without imposing the heavy financial burden of the traditional consulting process. Ideally, the ITFMS would allow the MTFC consultants to monitor fidelity and provide feedback at an affordable cost to the local site.
The ITFMS has three major components: (1) a secure MTFC website, tailored to the individual site, with data storage, reports, and observations for dynamic feedback; (2) computer-mediated forms for clinical and supervisory activities, such as daily reporting of child behavior; and (3) computer-mediated video observation of clinical meetings. The developers designed the ITFMS with a user-friendly graphic interface, state-of-the-art security protocols with multiple access levels, and automated video recording software.
The system was used and reviewed by 20 foster parents, 9 MTFC program supervisors, and 4 MTFC consultants. Foster parents had the option of entering their youth’s behavioral data for the Parent Daily Report (PDR) measure, a basic component of MTFC, through the internet-based system, through a phone-based system, or having the agency call them directly. MTFC staff could view the PDR data from all three sources in the ITFMS for youth in their caseload. The ITFMS was also used to record agency meetings for observation and feedback.
Parents and staff rated their satisfaction with the ITFMS after a week-long trial. All staff and parents rated the system as very easy to use and said they would recommend it to other staff or parents. Both parents and staff reported that the ITFMS saved them time entering or viewing the PDR data; parents further noted that they liked being able to enter PDR data on their own schedule rather than scheduling a call with agency staff during working hours. Several foster parents did not want to return to phone reporting and continued using the web-based reporting past the 7-day trial period.
For the video observations, the ITFMS developers balanced the need to record and transfer long video recordings with the need for a high enough visual and audio quality for fidelity coding and ratings. MTFC consultants rated the quality of the video segments as easy to code.
The results from this pilot test indicate that internet-based fidelity monitoring may help agencies receive a higher level of fidelity monitoring and support even at geographical distances that would otherwise be cost prohibitive. Further studies are needed to determine whether the ITFMS or similar technology will actually improve implementation fidelity and if so, whether community-implemented interventions with strong fidelity will show the same level of effectiveness as the research trials. Future evaluations will also need to examine any potential negative aspects to this system, such as a reduction in direct communication and opportunity for assistance between foster parents and agency staff.
Usability of Agency Web Sites
Are Social Service Agency Web Sites Easy to Use?
Friedmeyer-Trainor and her colleagues (2012) recently published an article that raised two questions that have long received inadequate attention: Are social service agencies’ web sites easy to use? As social service agencies gain experience redesigning their web sites, have their web sites become easier to use? The conclusions of the article are, generally, “no” and “no.”
For the purposes of this study, the authors defined a social service agency as having “a physical location in the United States, a non-profit status, and the assertion that it provided services to one or more diverse populations.” In 2000, the authors searched the Internet and found the web sites for 407 social service agencies. They randomly selected 100 of these web sites, which they assessed on a number of measures. They then assessed the web sites again in 2005 and in 2010, if the web sites still existed.
The authors rated the reading ease of the text on a web site using the Flesch Reading Ease Scale. They rated the grade level of the text using the Flesch-Kincaid Grade Level Scale.
They checked each web site to see whether it accommodated non-English readers by offering at least one paragraph written in a language other than English.
They used the Bobby tool to determine whether or not each web site was accessible to the disabled. That tool was discontinued after 2005. Several other accessibility tools are available but none is very similar to Bobby. Therefore, the authors rated the accessibility of the web sites in 2000 and 2005 but not in 2010.
The results were not encouraging. In 2000, the mean Flesch Reading Ease score indicated that the text on the web sites was too difficult for all but one-third of American adults. In 2005 and 2010, that figure fell even further, to less than 5 percent.
The Flesch Kincaid Grade Level measures indicated that the text on the web sites was, on average, written at a level above the eighth grade in 2000, at the high school level in 2005, and at the college level in 2010. The mean reading level for American adults is the eighth grade. The results suggest that the text on these web sites is written at a level that has grown far too high.
The number of web sites that had non-English text options did grow over time. However, even in 2010, only about one-fifth of the web sites offered at least one non-English paragraph.
The Bobby measures pointed to a substantial improvement in the accessibility of the web sites for the disabled. In 2000, only about one-fifth of the web sites passed the Bobby test, but in 2005, all passed. Federal legislation to help the disabled may deserve some credit for this improvement.
This article suggests that social service agencies need to devote more effort to making their web sites easy for the public to use. The authors were concerned with the web sites of social service agencies of all sorts; they did not specifically examine the web sites of child welfare agencies.
However, child welfare agencies usually serve a very wide range of people, ranging from potential foster parents, attorneys, journalists, families, social workers, and people who are simply interested in the topic of child welfare. This article should encourage child welfare agencies to ensure that their web sites adequately advance their missions.
For more information
Feil, E.G., Sprengelmeyer, P.G., Davis, B., and Chamberlain, P. (2012). Development and Testing of a Multimedia Internet-Based System for Fidelity and Monitoring of Multidimensional Treatment Foster Care. Journal of Medical Internet Research, e139.
Friedmeyer-Trainor, K., Director, R. V. & Lynch, D. (2012). Accessibility and agency website design: Stumbling backwards? A follow-up study. Journal of Technology in the Human Services, 30, 59-71.