Government Information Quarterly 26 (2009) 295–304

Contents lists available at ScienceDirect

Government Information Quarterly

j ourna l homepage: www.e lsev ie r.com/ locate /gov inf

Supporting self-evaluation in local government via Knowledge Discoveryand Data mining

Hye-Chung Kum ⁎, Dean F. Duncan, C. Joy StewartDepartment of Computer Science, Jordan Institute for Families, School of Social Work, University of North Carolina, Chapel Hill, USA

⁎ Corresponding author. School of Social Work, UniPittsboro Street, Campus Box 3550, Chapel Hill, NC 2759

E-mail addresses: kum@email.unc.edu (H.-C. Kum), d(D.F. Duncan), jstewart@unc.edu (C.J. Stewart).

0740-624X/$ – see front matter © 2009 Elsevier Inc. Aldoi:10.1016/j.giq.2008.12.009

a b s t r a c t

a r t i c l e i n f o

Available online 10 February 2009

Keywords:

The business sector has alremany businesses adoptingbased on free flow of inform

Knowledge Discovery and Data mining (KDD)Self-evaluationChild welfare information systemsAdministrative dataInformation infrastructure

ation. Free flow of information in government agencies is just as important. Forexample, in child welfare, entities that fund social services programs have increasingly demanded improvedoutcomes for clients in return for continued financial support. To this end, most child welfare agencies arepaying more attention to the outcomes of children in their care. In North Carolina, many county departmentsof social services have successfully adopted the self-evaluation model to monitor the effects of their

ady recognized the importance of information flow for good management, withnew technology in data mining and data warehousing for intelligent operation

programs on the outcomes of children. Such efforts in self-evaluation require good information flow fromstate division of social services to county departments of social services. In this paper, we propose acomprehensive KDD (Knowledge Discovery and Data mining) information system that could upgradeinformation flow in government agencies. We present the key elements of the information system anddemonstrate how such a system could be successfully implemented via a case study in North Carolina. Thenext generation infrastructure in digital government must incorporate such information system to enableeffective information flow in government agencies without compromising individual privacy.

© 2009 Elsevier Inc. All rights reserved.

1. Introduction

In recent years, government agencies have been under pressure tobe more effective and are increasingly held accountable for out-comes. To support this new mode of operation, resources are beingset aside for quality evaluations that go beyond simple counts ofservices provided. For example, in child welfare, the Adoption andSafe Families Act (ASFA) of 1997 identified a national set of outcomemeasures that can be used to gauge state and national progress inreaching the goals of child safety, permanency, and well-being. A keyfactor in such evaluations is the free flow of government information,which is used to evaluate the operations of government organiza-tions and measure real outcomes that result from its services whilestill protecting individual privacy. With advances in informationtechnology, the business sector has already realized the importanceof effective information flow for good management (Gavirneni,Kapuscinski, & Tayur, 1999) and many have implemented large-scale information systems to enable information flow. The nextgeneration infrastructure in digital government must also enableeffective information flow in government agencies without compro-mising individual privacy.

versity of North Carolina, 3259-3550, USA.fduncan@email.unc.edu

l rights reserved.

A good example of the need for effective information flow ingovernment agencies is demonstrated in the use of self-evaluation inchild welfare agencies. Self-evaluation is a form of empowermentevaluation that is collaborative and participatory. Through self-evaluation, a county social services agency, with the assistance ofexperts, can design, monitor, and improve indicators that ultimatelyimprove the outcomes that are important to their local community. Akey element in the self-evaluation efforts is the availability of timelyand accurate data that appropriately measure the outcomes ofinterest. In the case of social services agencies, comprehensive dataon families and children served by the agency is required. However,many of the local agencies lack the resources to collect and analyzethe data for such evaluations. In addition, as many of the localagencies in a state would have similar goals, much effort would beduplicated if each local agency built its own capacity for such dataanalysis. Furthermore, it would be easier to have consistency acrosssimilar outcomes in different local governments if the state providedthe technical assistance for such efforts (Usher, Wildfire, & Schneider,2001).

In North Carolina, a KDD information system was built to supportself-evaluation efforts at the local level. In collaboration with theNorth Carolina Department of Health and Human Services (NC-DHHS), the Jordan Institute for Families at the School of Social Work atthe University of North Carolina at Chapel Hill (UNC-CH) built aninformation system to publish comprehensive outcome measures atthe local level for child welfare. The project built a dynamic website

296 H.-C. Kum et al. / Government Information Quarterly 26 (2009) 295–304

that contains information on various state and federal outcomemeasures for child welfare for all 100 counties in North Carolina. Theseoutcome measures are provided at various units such as county,groups of counties, state, as well as different time periods and sub-categories like age, race, and gender. Counties can use this vastamount of information to construct their own self-evaluation systemand monitor their progress from different perspectives.

In this paper, we propose a comprehensive KDD (KnowledgeDiscovery and Data mining) information system that could upgradeinformation flow in government agencies via a case study in NorthCarolina. We first describe the different policies in child welfare thathave played a key role in its success as well as the details of howinformation is being used. Next we will demonstrate how effectiveinformation flow among state and county government agencies hasempowered both to better serve their communities via self-evalua-tion. We then follow by presenting the key elements of theinformation system and demonstrate how similar systems could besuccessfully implemented. Such a KDD information system should bean integral part of the next generation information infrastructure indigital government.

The remainder of the paper is organized as follows. Chapter 2provides an overview of the related works in KDD and self-evaluation.In Chapter 3 we introduce the language used in child welfare with afull discussion of child welfare policies and how it uses technology andinformation to better serve its community. Chapter 4 demonstratesthe use of self-evaluation in Guilford County, NC. Chapter 5 details theKDD information system built for the project highlighting theimportant technical details. And finally, Chapter 6 concludes with adiscussion of lessons learned and future works.

2. Related works

2.1. Knowledge Discovery and Data mining (KDD)

KDD is the area of computer science that tries to generate anintegrated approach to extracting valuable information from data bycombining ideas from databases, machine learning, artificial intelli-gence, knowledge-based systems, information retrieval, statistics,pattern recognition, and visualization. KDD has been defined as, “thenontrivial process of identifying valid, novel, potentially useful, andultimately understandable patterns in data” (Fayyad, Piatetsky-Shapiro, & Smyth, 1996). The goal is to discover and presentknowledge in a form that is easily comprehensible to users (Fayyadet al., 1996). An additional important aspect is the timely delivery ofthis knowledge to thosewhowill ultimately use it (Kum, Duncan, Flair,& Wang 2003). Such timely information flow is critical in improvinggovernment services.

A key characteristic particular to KDD is that it uses operationaldata. Specifically in child welfare, operational data is administrativedata collected by agencies for management and administrationpurposes. There are different objectives for utilizing KDD, such asexploratory data analysis, descriptive modeling, predictive modeling,discovering patterns and rules, and retrieval of similar patterns whengiven a pattern of interest (Hand, Mannila, & Smyth, 2001). Manyprivate businesses have successfully applied KDD to operational datafor better management, marketing, and planning. For example,Amazon recommends books based on previous choices using KDDtechnology (Kum et al., 2003).

2.2. Self-evaluation

In recent years, entities that fund social services programs, bothpublic and private, have increasingly demanded improved outcomesfor clients in return for continued financial support. Thus, socialservices agencies are required to justify expenditures in terms ofprogram results. This situation has created the need for timely,

reliable, and valid data related to client outcomes. Furthermore, amethod of evaluation is also required that is more suited to agovernmental system that increasingly emphasizes local control andautonomy, a context not well suited to the experimental paradigmthat has prevailed in traditional evaluation research (Usher, 1999).

Detachment of the evaluator from policymakers and programmanagers is a hallmark of the traditional evaluation approach.Indeed, in their overriding concern about possible contaminationfrom measurement on the treatment or program itself, Usher(1995) stated that “many [evaluators] have taken on the mantle ofscientific objectivity in a way that causes them to define their tasksin such narrow terms that their contribution to inform policydebate is minimized.” Under this approach, evaluation becomesadversarial (between evaluators, policymakers, and program staff),can become ill-informed due to the lack of communication betweenevaluators and staff, and ultimately may be of little benefit (Usher,1995).

Usher (1995) argued that the self-evaluation approach, as long at itis technically strong, offers a viable alternative to traditionalevaluation. Self-evaluation is a form of empowerment evaluationthat is collaborative and participatory. Self-evaluation is usuallycarried out by a team of diverse people, the SET (self-evaluationteam), gathered locally for their expertise. In this approach, evaluatorsmay work for an independent organization or be employed by theagency administering the program. The evaluators work alongsidepolicymakers, program managers, and staff to establish outcomes,monitor performance, and provide continuous feedback for programplanning and management. The SET defines a concrete set ofoutcomes as target goals for the local agency. The focus of self-evaluation is to change policies and practices to make progress in thedirection set forth in these performance targets in each outcome area.In other words, as changes in programs and practices are made, theseoutcome measures are monitored to see whether there are associatedchanges in outcomes. The programs and practices are adjusted orredefined as needed on an ongoing basis. Thus, it is more a processrather than an end product. New goals are added as previous goals aremet (Usher, Wildfire, & Schneider, 2001). North Carolina's countyadministered system of social services is the kind of context in whichself-evaluation may well flourish.

In most sites, the outcome-related data is supplemented byprocess-oriented data describing program operations. The process-oriented data will confirm changes that occurred in programs andpractices. Then the outcome measures can be monitored to seewhether such changes in programs and practices have associatedchanges in outcomes. Accordingly, the programs and practices can beadjusted and refined as needed. In other words, self-evaluationinvolves both sides of the evaluation equation — interventions andoutcomes (Usher, 1999).

3. Child welfare (CW)

The primary responsibility of child welfare agencies is to ensurethe safety and protection of children from abuse and neglect. On adaily basis, that entails the investigation of reports of abuse andneglect, assistance to families and children at risk of abuse andneglect, and, finally, the temporary care of those children who can nolonger remain safely in the home.

When there is a report of abuse or neglect, the child welfare agencymust investigate the allegation in a timely manner. If the report isfound to be true, or substantiated, steps are taken to ensure the safetyof the child at risk. Depending on the severity of the situation, thesocial worker can work with the family to improve the situation sothat there is no longer a threat to the child, or he or she may place thechild in foster care if the risk is deemed too great. Typically, childwelfare agencies attempt to keep children with their parents orcaretakers unless it becomes no longer plausible.

297H.-C. Kum et al. / Government Information Quarterly 26 (2009) 295–304

When a child is removed from his or her home, the state assumestemporary custody of the child until a more permanent arrangementcan be made. The agency typically works with the family to improveconditions so that the child can be reunified with his or her parents. Ifthese efforts are not successful, then the agency will seek to terminateparental rights and find an adoptive home for the child. The full periodduring which the state has custody is termed a placement authority.During one placement authority, a child can be placed in multiplefoster homes, called placements. Placement stability is about keepingthose placements to a minimum to provide a stable environment forthe child. A child can have more than one placement authority if thechild is taken from the permanent home again. This is called reentryinto care.

3.1. Policies supporting data and technology in CW

Given the critical nature of protecting the children in our society,there is a long history of efforts to monitor and evaluate the childwelfare system. In the 1990s major efforts were put into buildinginformation systems that properly collect data on the children andfamilies being served. Two particularly important pieces of legislationwere enacted in 1986 and 1993 regarding child welfare informationsystems development and funding. The 1993 regulations establishedthe Adoption and Foster Care Analysis and Reporting System(AFCARS), which required States to submit AFCARS data semi-annually to the Administration for Children and Families (ACF).

The 1997 Adoption and Safe Family Act (ASFA) included importantchanges to the reporting regulations of AFCARS. ASFA mandated ACFto use outcome measures to assess the performance of each state'schild welfare system and required states to report additional dataneeded to measure such outcomes. To further assist states, local, andtribal child welfare agencies, and the courts in improving outcomes forchildren and families through the use of information technology, ACFestablished the National Resource Center for Child Welfare Data andTechnology (NRC-CWDT) in 1999. Currently most states have astatewide information system that collects comprehensive adminis-trative data on the children and families they serve (US-DHHS, 2007;NRC-CWDT, 2007).

North Carolina has administrative data on the children andfamilies served since the early 1990s. However, to meet changes inAFCARS requirements, amajor systemupgradewas performed in 1998resulting in a refined and more comprehensive dataset. These policiesresulting in real support of funds and technical assistance were thebuilding blocks of the KDD information system. With data collectionmethods now established, more efforts have been focused on how toprocess the raw data into information for consumption. There hasbeen much work done at both the federal and state levels to establishaccurate outcome measures that can properly assess the safety,permanency, andwell-being of the children and families served by the

Table 1Round 1 CFSR measures

Type Denominator

1. Recurrence of maltreatment Among all the children with a substantiatedand/or neglect within the first 6 months of

2. Maltreatment in foster care Among all the children in foster care duringreporting period

3. Reunified within 12 months Among all the children who had a terminatreunification within the reporting period

4. Reentry into foster care within 12 months Among all the children who entered foster creporting period

5. Adoption within 24 months Among all the children who had a terminatadoption within the reporting period

6. Placement stability (2 or fewer placements) Among all the children who entered foster c12 month reporting period

child welfare system. Raw data processed into various outcomemeasures based on solid methods can provide an accurate depiction ofthe child welfare system in any region.

3.2. Federal outcome measures in CW

In 2000, ACF started the first round of the Child and Family ServicesReviews (CFSRs) to help states improve safety, permanency, and well-being outcomes for children and families who receive servicesthrough the child welfare system. Specifically, the CFSRs measureseven outcomes. The outcomes measured include (1) whetherchildren under the care of the state are protected from abuse andneglect, (2) whether children have permanency and stability in theirliving conditions, (3) whether the continuity of family relationshipsand connections is preserved for children, (4) whether families haveenhanced capacity to provide for their children's needs, and (5)whether children receive adequate services to meet their physical andmental health needs. Table 1 details the measures.

In 2007, ACF began a second round of CFSRs which uses a morecomplex set of composite measures to overcome some of thecriticisms with the first set of CFSR measures. The first two measureson child safety were basically unchanged. The only difference wasthat they were converted into positive measures by reporting theabsence of recurrence of maltreatment and the absence of maltreat-ment in foster care by subtracting the measures from 100%. Forexample, the national standard changed from ‘5.4% or less’ to ‘94.6%or more’ and ‘0.32% or less’ to ‘99.68% or more’ respectively. However,the four measures on children in foster care, measures three to six,have been changed to use composite scores derived from multipleindicators so that the final score is more comprehensive. Forexample, measure six (on placement stability) is a weighted averageof the following three indicators: (1) of those children who were infoster care for less than 1 year, what percent had less than threeplacement settings; (2) of those children who were in foster carefrom 1 year to less than 2 years, what percent had less than threeplacement settings; and (3) of those children who were in foster carefor two or more years, what percent had less than three placementsettings. The weights have been predetermined based on principalcomponent analysis of national data. The details of the methodologyused for composite measures can be found in the Federal Register(US-DHHS, 2006).

3.3. North Carolina outcome measures in CW

In response to the first round of CFSRs, many researchershighlighted problems with using point-in-time data for evaluatingperformance outcomes. Point-in-time data select the caseload from aspecified time period for review, a practice which over-representedthose cases that stay in care for long periods while under-representing

Numerator

report of abusethe reporting period

What percent of this total had another substantiated reportwithin 6 months of the first report

the 12 month What percent of this total had a substantiated report of abuseand/or neglect where the perpetrator was a foster parent orfacility staff member

ed case due to What percent of this total had been in placement authority fora year or less

are during the What percentage of this total had previously been in foster carewithin 12 months of the most recent foster care entry date thatis within the reporting period under review

ed case due to What percent of this total had been in placement authority for2 years or less

are during the What percent had 2 or fewer placements

298 H.-C. Kum et al. / Government Information Quarterly 26 (2009) 295–304

those with shorter stays. To address this problem many researchersadvocate using longitudinal data which captures the full experience ofthe children in care (Webster, Needell, & Wildfire, 2002).

In 1994, in North Carolina, child welfare experts at the JordanInstitute collaborated with NC-DHHS to design outcome measuresusing longitudinal data to measure the performance of the childwelfare system in North Carolina as part of the Families for Kidsreform initiative (Usher, Locklin, Wildfire, & Harris, 2001). Thefindings, called the Experiences Report, were tailored to each of the100 counties in North Carolina.

The report measures five outcomes – (1) pattern of initialplacements, (2) length of time in custody, (3) experiences of childrenever placed in non-family settings (such as group homes), (4)placement stability, and (5) reentry to custody – for those childrenwho entered foster care for the first time in the last 5 years for eachcounty. A year represents an entry cohort, which is a group of childrenentering care for the first time together, and the measures can becompared among the five entry cohorts to see how the child welfaresystem has changed over time in a county. Rather than looking at allchildren who are in care at a particular time, this approach has a fairrepresentation of all types of children in care. In addition, to comparethe county's performance to the state as a whole and with similarcounties, the samemeasures for the five entry cohorts were calculatedstatewide, and with three groups of counties based on county size —

large, medium, and small. This three dimensional matrix, fivemeasures by entry cohort and by county, counties of similar size,and statewide, was formatted into a one page report and provided tocounties to assist in self-evaluation efforts via the NC-DHHS website(Usher, Locklin, Wildfire, & Harris, 2001).

By 2003, the production of the reports on a regular basis had beentransferred to NC-DHHS from the research team at UNC involved withthe creation of the reports. However, the IT department at NC-DHHSlacked the resources at the time to continue producing timely reports.Specifically, the department had difficulty processing the raw data intheir DBMS to generate these measures on a regular basis. As a result,the counties and state no longer had the valuable data readilyavailable. However, having the information was critical to continuewith the self-evaluation efforts. To solve this problem, NC-DHHScontracted with the current team in 2005 to publish the ExperiencesReport on-line.

Our team has extensive experience in Knowledge Discovery andData mining (KDD) on social welfare data. Over the last 10 years, wehave been collaborating with the NC-DHHS to build informationsystems to analyze the administrative data on welfare programs inNorth Carolina such as Work First, Food and Nutrition Services, andMedicaid. The team has assisted NC-DHHSwith various ad hoc reportsto the federal government; informed the state and counties on thestatus of the welfare programs through presentations, technicalreports, and our public websites; and trained many social workerson using data to better manage welfare programs (Kum, Duncan, Flair,& Wang, 2004; Kum et al., 2003).

4. Self-evaluation in Guilford County

Guilford County Department of Social Services (DSS) is oneexample of a county agency embracing the concept of self-evaluation.Self-evaluation began in Guilford County with the Families for Kidsinitiative in 1994 and was renewed and expanded with the Family toFamily Initiative, a child welfare reform effort sponsored by the AnnieE. Casey Foundation, in 2000. Self-evaluation was one of four corestrategies that comprised the Family to Family initiative.

The key ingredients for self-evaluation are as follows (Usher,Wildfire, & Schneider, 2001):

• Outcomes for clients are clearly defined and disseminatedthroughout the agency.

• It is a collaborative process bringing together individuals withdifferent kinds of expertise to discuss the data used to measureoutcomes and agency processes.

• It is an ongoing process with regular meetings to discuss agencystatus on outcomes.

• It requires timely and accessible data that appropriately measuresoutcomes and other indicators of interest.

• It should include ongoing attention to implementation progress ofthe core strategies to improve the outcomes.

• It requires technical expertise to ensure defensibility andadaptability.

The success of self-evaluation in child welfare at monitoringperformance inspired a similar 2003 agency-wide effort known asLeading By Results (LBR). Guilford County DSS identified six major LBRresults or outcomes, each of which has several data indicators tomeasure progress. The result related to the child welfare programwithindicators is detailed below.

LBR Result 1: at-risk children and families to be safe and healthy instable environments

• Indicator 1: Wewill decrease the rate of children placed away fromtheir home from 7% in FY 2003 to 5% by the end of FY 2005.

• Indicator 2: We will decrease the rate of children entering fostercare who are initially placed in shelter or group home care from13% to 11% by the end of FY 2005.

• Indicator 3: We will reduce the rate of children re-entering carefrom 10% in FY 2003 to 8% by the end of FY 2005.

• Indicator 4: We will reduce the number of children in care withfour or more placement moves from 14% in FY 2003 to 12% by theend of FY 2005.

• Indicator 5: We will maintain the percent of children substan-tiated/in need of services that are not repeat victims of sub-stantiated maltreatment at 92% by the end of FY 2005.

• Indicator 6: We will continue to work on addressing disparitiesassociated with race/ethnicity as evidenced by a decrease in thepercentage of African American children in care from 57% by theend of FY 2005.

The central role of data in self-evaluation is apparent. UNC-CHJordan Institute for Families provided longitudinal datasets tocounties, including Guilford, as part of its ongoing collaborationwith NC-DHHS. Faculty and staff also provided valuable training tocounty personnel on understanding and appropriately analyzing thedata. In so doing, UNC-CH ensured that the key ingredients of timelyand accessible data and technical expertise were available to thecounties and the state and thus facilitated the success of self-evaluation in Guilford County DSS. With the creation of a dynamicwebsite for child welfare data, UNC-CH has enabled seamlessinformation flow between state and county agencies demonstratinginnovation and continued commitment to high quality performanceevaluation of services to children and families.

5. KDD information system

In this section, we introduce the KDD information system our teamhas developed. The primary objective in the UNC-CH project isexploratory data analysis and descriptive modeling to inform childwelfare policy and practice in North Carolina. Child welfare agencieshave large administrative databases about the experiences of childrenand families that were served by the child welfare system. This datacan be analyzed for valuable information. Significant insight on theeffects of various child welfare programs, policies, and practices can begained by using KDD technology on administrative data to accuratelydepict the child welfare system in North Carolina without compro-mising privacy.

299H.-C. Kum et al. / Government Information Quarterly 26 (2009) 295–304

Fig. 1 illustrates the key elements of the KDD system built for theUNC-CH project. The full cycle of the KDD process is shown, from dataacquisition to consumption of the information via a public website.First, the data regarding child welfare cases are entered by countysocial workers from all 100 counties in North Carolina. This data isstored as raw administrative data tables in the state computer system.Second, UNC-CH extracts these raw data tables monthly and weavesthem together to build longitudinal files, depicting the experiences ofeach child over time. These longitudinal data files can be analyzed forvarious reports, research, and monitoring of performance measures,which are more comprehensive, accurate, and easier to use thanindividual raw tables. In particular, the project presents a set ofoutcome measures that can depict a complete view of the childwelfare system in North Carolina. The measures are calculatedmonthly and stored in summary tables (called data cubes) for easyaccess. Finally, the information from the data cubes is shared with all100 counties via a public website. The dynamic website can generateuser-friendly county reports and graphs based on user input. From thewebsite, agency staff can easily obtain timely measures regardingtheir performance by monitoring changes in children's outcomes overtime as well as across counties.

Ideally, the KDD process works best when data is generated andconsumed by the same users, which in this casewould be county childwelfare agency staff. As users access and utilize their data, they will bemotivated to submit more accurate and detailed data, which in turnwould result in more reliable and comprehensive information forperformance measures and outcomes. In fact, in North Carolina, whenthe counties saw their own outcomemeasures as being inaccurate as aresult of bad data, many took steps to fix previous data as well asensure future data quality. Furthermore, those at the county agencieswho provide the data know best how to interpret the results, giventheir familiarity with agency policy and practice regarding coding andclassification of data elements (Duncan, Kum, Weigensberg, Flair, &Stewart, 2008).

5.1. Data acquisition

Over the years, our team built a strong trusting relationship withstaff at various levels at NC-DHHS. We had direct access to the statemainframes storing the raw data files and had a protocol for pullingthe data in a secure manner. We used tape and physically picked upthe tapes. Recently, wewere able to use a utility to electronically movethe files in a secure manner. The secure transfer of information raiseda small number of challenges because the NC-DHHS data is collectedand stored under MVS/TSO and the analysis at UNC is performed

Fig. 1. Key elements of the K

under a Unix based system. Data is moved first to a small IBM systemat UNC using an IBM utility. The data is then moved from that systemto the secure UNIX system for analysis. The conversion betweenEBCDIC and ASCII is handled with SAS.

Although the initial research on the Experiences Report wasconducted by our colleagues, our team took on the responsibility ofgetting the child welfare data for them from the start. So, when wetook on the project, data acquisitionwas easy. The only change neededwas to pull the raw tables for child welfare regularly on a monthlybasis. Previously, we had been pulling the data on request roughlytwice a year.

5.2. Data preparation

Many child welfare agencies have administrative data in the formof raw data tables that are designed for managing services. However,obtaining useful and accurate information from these administrativedata is extremely difficult. In addition, with limited disk space, there isthe challenge of retaining all historical data. Therefore, analysis formonitoring agency performance over time cannot be conducted onthe raw tables until they are consolidated and archived into a usableform. Converting these raw tables into a usable form is the datapreparation step in the KDD process. Data preparation is laborintensive and involves data cleaning and integration. In KDD, the setof files in usable form are typically called the data warehouse. A well-designed data warehouse is an essential part of an effectiveinformation infrastructure.

In this project, substantial effort was devoted to the datapreparation step. The goal was to build a few analysis files that canbe used to accurately assess outcomes for children. Accordingly, theteam consolidated raw tables from multiple administrative computersystems into a few longitudinal analysis files that could properly trackthe experiences of all children served in North Carolina since 1997(over 102,000 children served). The project links records amongdifferent child welfare data tables as well as records from otherwelfare programs in North Carolina, including Work First (NorthCarolina's Temporary Assistance for Needy Families program), Foodand Nutrition Services, and Medicaid. The longitudinal files allow forthe analysis of child welfare outcomes at different levels includingindividual recipient, the entire county, groups of counties by size orjudicial districts, or the entire state. Most importantly, the data islongitudinal. Therefore, county child welfare agencies may evaluatechanges over time, which is a significantly different perspectivethan was previously available using primarily cross-sectional data. Byusing KDD technology on multiple administrative data tables, the

DD information system.

300 H.-C. Kum et al. / Government Information Quarterly 26 (2009) 295–304

longitudinal data create a more complete and accurate view of thechild welfare system (Duncan et al., 2008).

We started with a thorough review of the steps used to build thelongitudinal file in the previous project. We then decided to reworkthe process. Mainly, the old process would require much effort tomaintain with regular monthly updates. In a KDD project whichrequires regular updates, automating much of the process for easymaintenance is critical (Kum et al., 2004). We converted the process tobatch mode which would only require that a staff member check thebatch processing logs for errors each month.

Initially, we adopted the basic design of the longitudinal filedepicting the experiences of children in foster care. The file had onerecord per child, with information on multiple placement authoritiesand individual placements being stored in the same table. None-theless, we had to make changes to how some of the data werecleaned as well as how the raw files were processed. For example,previously, only information on the first 21 placements was stored inthe longitudinal file. In order to keep complete information on allchildren, we redesigned how placement information was processedand stored. Surprisingly a handful of children had more than 100placements. The new process could handle information on all theplacements. We also improved the design of the longitudinal file tobetter handle missing end dates.

5.3. Descriptive modeling

In the current KDD project, the main goal is descriptive modelingto inform child welfare policy and practice in North Carolina. Thepurpose of descriptive modeling is to extract compact and easilyunderstood information from large data files (Hand, Mannila, &Smyth, 2001). Specifically, this project uses a collection of outcomemeasures to accurately describe the child welfare system in eachcounty and statewide. Taking into account the history of outcomemeasures in child welfare, a combination of longitudinal and crosssectional measures on placement in foster care as well as reports ofabuse or neglect are used.

5.3.1. Experiences reportThe five outcome measures from the Experiences Report – pattern

of initial placements, length of time in custody, experiences ofchildren ever placed in non-family settings, placement stability, andreentry to custody –were the first set of measures to be implemented.We expanded the years covered in the report from the past 5 years toall entry cohorts after 1997.

The child welfare experts who originally designed the ExperiencesReport requested additional measures to address the follow-up timeissue for placement stability and reentry to custody. The existingExperiences Report considered the maximum possible follow-up timeon both measures. This means that those in earlier entry cohorts hadmore follow up time compared to those in the later entry cohorts. Forexample, assume a child entered care in year 2003 and another childentered care in year 2005 and both were still in care in 2007. Theplacement stability measure would look at how many placements thechild was in care for 4 years for the first child, and for 2 years for thesecond child. It might seem as if the first child had more placementswhen, in fact, the second child could end up with a higher number ofplacements given two more years of data.

Due to this follow-up time issue, the most recent entry cohortstend to have lower numbers of placements and lower rates of reentry.The difficulty is that in both measures, the appropriate length offollow-up time would be more than 1 year, which means that themeasure would not be estimated until a couple of years after the entryinto care. This is especially problematic in the reentry measure, whichwould require some number of years in care and more years afterleaving care to reentry. This issue complicated efforts to providetimely data to administrators to assess current trends in the child

welfare system. Hence, the reason for using the currentmeasures eventhough the entry cohorts have different years of follow up time. In thereport, the research team decided to use the current measures andfocused on training the county and state staff to realize that thesudden drop in these measures for recent cohorts was due to lack ofsufficient follow-up time.

Communications via a static report are more restrictive comparedto an online forum where much more information can be presentedin one location and tailored to each individual user. When we startedthe project to publish the report online, our colleagues requested thatwe try to better address the follow-up time issue through the onlineforum. Upon further investigation, we found that while bothmeasures increased initially with more follow-up time, after a certainamount of time, the measures reached a plateau. For placementstability, the graph evens out at about three years out. For reentryrate, it took up to 4 to 5 years to show stable estimates. After morediscussions with our colleagues, we decided on a different approachto each measure.

For placement stability, we felt that the estimates becameacceptable even after 18 months. All entry cohorts were alreadygiven 18 months of follow-up time before full information is given onthe measures. Furthermore, counties were interested in how manyplacements a child was in during the full duration of one placementauthority, the current measure, as well as how many placements achild was in during the first year of placement authority. The medianlength of custody was typically over 1 year. The best compromisewithout changing the Experiences Report (since county staff werefamiliar with using it in its current form) was to add a more detailedsection on the placement stability measure for those who wantedmore information. We added a new measure on the number ofplacements in the first year of custody. This will give a more faircomparison over time.

For reentry, we decided not to publish the reentry rate until 3 yearshave passed since entry. At this point, the reentry rates were muchmore reasonable, if not fully stable. Although we investigated addinganother measure with a set amount of follow-up time, such as we didwith the placement stability measure, we decided against it for tworeasons. First, the reentry rate was very low — even 10 years afterentry the typical reentry rate is less than 10%. Therefore, a shorterfollow-up time would not produce much more meaningful estimates.This is what is expected since children who have found permanentplacements should not be returning to the child welfare system.Second, a minimum of 5 years of follow-up time would be required tohave sufficient time for children to have entered care, left care, andthen to have been reinvestigated and taken back into the state'scustody. Information about children who entered care five or moreyears ago had limited usability. Furthermore, as the reentry rates werelow, there was not much difference with the full reentry rate versusthe reentry rate 5 years after entering care. Thus, the decision wasmade to keep things simple for the reentry rate.

We have also expanded the Experiences Report to include otherrelevant information. In 2008, statistics on type of exits from fostercare was added. Noting that children who have longer stays in caretend to have different exit types compared to those who have shorterstays, this analysis was broken out by length of time in care. Thestatistics show that overall in North Carolina, more than 50% of thechildren who are in care for 1 year or less are reunited with theirfamily. In comparison, about 20% of the children who are in care formore than 2 years are reunited with their families, and more than halfof the children in care for more than 2 years are adopted.

5.3.2. Child and Family Services Reviews (CFSR)Given the emphasis on federal accountability of state child welfare

agencies with the Child and Family Services Reviews (CFSR), thesemeasures were included to support the state and counties inmonitoring their own performance according to the federal standards.

301H.-C. Kum et al. / Government Information Quarterly 26 (2009) 295–304

Both Round One and Round Two CFSR measures have been fullyimplemented by 2008. To enable counties to measure how their childwelfare system has changed over time, we provide the CFSR measureson rolling quarters starting in January 2000. This allows the CFSR toevaluate children in care during any given year. Rolling quarter meansthat we calculate themeasures for each year starting every quarter (i.e.from Jan 2005 to Dec 2005, from Apr 2005 to Mar 2006, from Jul 2005to Jun 2006, from Oct 2005 to Sep 2006, etc.). Being able to breakdown the state CFSR measures at the county level for the past 7 yearsexpands the breadth of detailed information about the child welfaresystem in North Carolina. We are now able to identify which countiesare strong or weak on which measures, and how North Carolina canimprove its performance in regard to child welfare.

5.3.3. Abuse and neglect dataIn addition to the Experiences Report and CFSR measures, which

are predominantly focused on experiences of children in foster care,we have developed new measures to assess maltreatment reportoutcomes. The outcome measures include: the number of childrenreported, types of substantiation findings, re-reports of maltreatment,and the probability of placement. We provide both the point-in-timeand longitudinal perspective on reports of abuse and neglect.

5.3.4. Subgroup analysisEach set of outcome measures depicts a particular aspect of the

child welfare system. Together they can provide a fairly comprehen-sive view of the system in North Carolina. Furthermore, each of thesemeasures is calculated for different units, time periods, and sub-groups. The main unit of analysis is the county, with additionalcapabilities to analyze by groups of counties by size or judicialdistricts, or the state as awhole. In addition, outcomes can be analyzedover time or by subgroups of children's characteristics including age,gender, race, and ethnicity. This allows the county child welfareagency to view their own performance from various perspectives.

5.4. Presentation and Visualization

The last step in the KDD process is presenting the information tothose who will use it. A user-friendly website was built to share theoutcome measures estimated in the previous step with child welfareagencies in all 100 counties across the state (Duncan, Kum, Flair,Stewart, & Weigensberg, 2007). The challenge is to design a websitethat is both simple to use and comprehend, yet contains a wealth ofinformation about child welfare outcomes without requiring toomuch maintenance. Keeping applications light, in regard to beinginexpensive to maintain and support, is one of the most importantaspects of a successful KDD project. A trade off in the development ofthe website all along has been to provide a tool that is easy to learn,easy to use, easy to maintain, and easy to support.

The solution is a dynamic website driven by well-designedsummary data cubes that could easily be updated each month. In adynamic website, the contents of each webpage are controlled by userinput. That is, the tables, charts, and graphs are generated on demandusing predefined template pages and server-side scripting rather thanbeing created in advance and stored as static pages on the server. Thetemplate pages control the layout of the webpage, while the contentsof the page are driven dynamically by the data cubes.

Data cubes are multi-dimensional data tables organized in such away that information can be quickly analyzed from differentdimensions. The full set of outcome measures, developed in theprevious step, for all levels, time periods, and sub-categories amountsto a vast amount of information which needs to be shared quickly viathe website. Thus, for timely and easy access, all information is storedin well-designed data cubes. Given the public accessibility of thewebsite, to ensure privacy, individual level data is not available(Duncan et al., 2008).

Using a dynamic website can be a rich form of communication.Utilizing an online forum, users can select which information is ofinterest to them as well as how they would like the informationpresented. All the information is available in a table format which canbe easily transferred to Excel for further analysis. However, graphs andcharts depict information more visually than tables. Our websitepresents the information using line graphs, pie charts, and bar charts.For easy navigation, it also provides multiple points of entry to thesame information. For example, there is a summary page, where allinformation by race can be accessed in one location.

Fig. 2 shows screen shots from the website. Most of theinformation is accessed by counties through a clickable map asshown on the top portion of the figure. When a county is selected,users are presented with different options for information about thecounty organized by groups of indicators as shown below the map.There is a navigational menu on the left side so users can easily moveamong different types of information. Finally, when the user selectsone of the many options, they are presented with graphs and tables aswell as explanations about the information being presented. Thesurvival curve on the top right is the rate of leaving custody. Using aline graph, it depicts the length of stay measured in the experiencesreport. Here we can see that in North Carolina, children in largecounties tend to stay in care longer than children in medium or smallcounties. For large counties, 50% of the children have left care by about510 days, while in small and medium counties it took about 360 days.The pie chart on the bottom left depicts placement stability forGuildford County by race for state fiscal year 0304 (July 2003 to June2004). Here we see that white children, the top chart, tend to havelower number of placements compared to the African Americanchildren, the bottom chart. The black pie piece, the children with oneplacement, is much bigger in the top chart. Finally, the line graph onthe bottom right depicts the number of reports of abuse and neglectover time in Chatham County. There was a sudden increase in thenumber of reports in SFY9899 which drops again in SFY0304. Thebottom chart displays the breakdown of the reports by age. Roughly40%–50% are reports for children ages 0–5, 30%–40% are reports forchildren ages 6–12, and 15%–20% are reports for children ages 13–17.The tiny area across the very top represents those childrenwhose dateof birth is unknown.

5.5. Training and data requests

There are training sessions held across the state for human servicessupervisors and managers on how to use data to assess outcomes andperformance. As part of these three-day sessions, project staffdemonstrate how to navigate the site and what the informationmeans. County staff learn to track outcomes for children involved inthe child welfare system and how to use the information for self-evaluation. Staff also discuss how the information on the site relates tolocal policy and practices. In addition to these training sessions,project staff conduct workshops at statewide meetings for humanservice professionals. These workshops focus on how to use theinformation available through the website.

In addition, the UNC team frequently assists policy analysts andprogram managers from NC-DHHS's Division of Social Services (DSS)on various analysis of the administrative data using the datawarehouse built in the KDD system. Some analyses are requestsfrom federal agencies, legislators, and independent researchers whileothers are internal.

5.6. Discussion

It appears that the outcomes for children who have entered fostercare in North Carolina are improving. A large number of initiatives – anumber of which overlap – have been undertaken in the state in thelast decade. The central theme underlying North Carolina's child

Fig. 2. Screenshots.

302 H.-C. Kum et al. / Government Information Quarterly 26 (2009) 295–304

303H.-C. Kum et al. / Government Information Quarterly 26 (2009) 295–304

welfare reform effort was self-evaluation within each county. Thepremise of this approach is that those closest to the families andchildren involved with the welfare system are most capable ofimproving the outcomes, given proper resources and support. Hence,those in the local community closest to the clients should haveresponsibility for developing, implementing and evaluating thereform efforts, which can be tailored to fit each unique local context.Stakeholders in the local community including community represen-tatives, program managers and staff, policymakers, and families andchildren, along with persons with specific research and evaluationskills, can form a SET (self-evaluation team) to best lead these reformefforts (Usher, Locklin, Wildfire, & Harris, 2001; Usher, Wildfire, &Schneider, 2001).

North Carolina's success with self-evaluation is directly related tothe willingness of the state to provide strong technical support tothese self-evaluation teams with good information flow from the stateagency to the county agencies (Usher, Locklin, Wildfire, & Harris,2001). As most local teams do not have the resources to maintain aninformation system, getting the proper information needed to thelocal management team in a timely manner is crucial in makinginformed decisions. The KDD system utilizes IT to provide consistentinformation on diverse areas of child welfare to all 100 counties inNorth Carolina while still adhering to strict privacy protection rules.The KDD technology is able to encapsulate the privacy issue to ensurethat private information is easily protected. The KDD informationsystem built by UNC in partnership with NC-DHHS has proven to be asuccessful initiative with numerous benefits and is widely used bycounties and NC-DHHS staff. Other agencies involved with childwelfare in North Carolina, such as the Administrative Office of theCourts and the Department of Public Instruction, are heavy users ofthe site. Advocates and individuals interested in the child welfaresystem also use the site on a regular basis. The utilization of thewebsite is a strong indication of the crucial role the KDD system playsin various efforts by numerous stakeholders to improve the childwelfare system in North Carolina. There are also a growing number ofresearchers at universities across the state frequently using the site.Moreover, a cursory review of the web logs indicates that there are anumber of individuals and organizations outside of North Carolinausing the site as well.

There are a number of advantages to the collaborative relationshipbetween NC-DHHS and the School of Social Work at UNC-CH. One ofthe benefits to DHHS is access to an array of faculty and researchers inthe analysis and assessment of outcomes in child welfare. Thisrelationship is symbiotic: university faculty have access to a vastamount of administrative data for research while the state has accessto the benefits of that research. In addition to having experts on policyand practice, the university also provides access to individuals who arefamiliar with the latest approaches in statistics, methodology,analytical techniques, and information technology. While it may beyears before the latest analytical approaches are widely adopted,university researchers are able to apply them to a number of thornyissues in the analysis of data on child welfare and other human serviceareas.

Using their methodological skills, researchers are able to processthe administrative data to develop valid and reliable indicators of whois being served by child welfare agencies, as well as descriptions oftheir experiences and outcomes. Through the use of informationtechnologies, indicators can be produced and easily updated for eachcounty or administrative subdivision (e.g., judicial district). Thecreation of these measures relies on attributes of the children, inthis case, the county in which they live. Utilization of other attributesof the children, such as age, gender, and racial classification, make itpossible to observe differences in social work practice across thesegroupings within counties.

Furthermore, the interdisciplinary team of computer science andsocial work researchers at the university has fueled development in

both areas of research. Once the computer science researchers builtthe KDD information system, the child welfare experts have been ableto access much cleaner and more comprehensive data. For example,the team has been able to link data from the EIS system (data onWorkFirst, Medicaid, and Food and Nutrition Services) and the SIS system(data on child welfare). This allows for research on more reliableoutcome measures. For the computer science researchers, access toreal data, such as those available on this project, is rare. In KDDresearch, there is no better experimental data than working on realproblems to be applied and tested directly to real data. To the best ofour knowledge, North Carolina is the only state to have independentcounty level information on the most recent Round Two federal CFSRmeasures. Such detailed analysis of the child welfare datawas possiblebecause of the sophisticated KDD information system and theinterdisciplinary team at UNC.

6. Conclusion

A number of enhancements are planned for the KDD system. Theseenhancements involve both the information that is available as well asthe way it is presented to the users. We have plans to add informationabout the fiscal aspects of the child welfare system, as well asinformation on earnings and incarceration of former foster care youth.There are also plans to revamp the website using CSS (Cascading StyleSheets) technology to improve usability and different online trainingmethods so more people can use our website readily.

The use of operational data to improve management practices inbusiness is not new. The private sector has been using KDD technologyfor effective marketing and sales since the early 1990s (Fayyad et al.,1996). Such technology transfer to the public sector can greatlyenhance federal and state government practices as it can effectivelyshare information between many diverse entities. KDD technologymakes it possible to have consistency and diversity at the same time.To effectively address the many local problems, diversity is a mustwhile accountability requires consistency of measurements. The nextgeneration information infrastructure in digital government shouldincorporate such KDD information systems to enable effectiveinformation flow in government agencies without compromisingindividual privacy. We have learned that trust, real support throughpolicies and funds, access to good technical expertise in both thecontent area and IT, and training are key factors to successfullyimplementing the KDD information infrastructure in government(Kum et al., 2004). Access to the required expertise can be obtainedthrough strong partnerships between government agencies andinterdisciplinary teams at universities. The partnerships can lead tosuccessful implementation of KDD information systems in the publicsector while providing a priceless opportunity for research.

Acknowledgments

The authors would like to thank Kimberly Flair, ElizabethWeigensberg, Lynn Usher, and other members of our team for theirhelpful comments on earlier versions of this paper as well as theirefforts on the project. This research was supported by funding fromthe North Carolina Department of Health and Human Services,Division of Social Services. All opinions expressed and any errorsfound are those of the authors.

References

Duncan, D., Kum, H. C., Flair, K., Stewart, J., & Weigensberg, E. (2007). NC Child WelfareProgram. Retrieved from Jordan Institute for Families website http://ssw.unc.edu/cw

Duncan, D., Kum, H. C., Weigensberg, E., Flair, K., & Stewart, J. (2008). Informing childwelfare policy and practice: Using Knowledge Discovery and Data miningtechnology via a dynamic website. Child Maltreatment, 13, 383−391.

Fayyad, U., Piatetsky-Shapiro, G., & Smyth, P. (1996). The KDD process for extractinguseful knowledge from volumes of data. Communications of the ACM, 36, 27−34.

304 H.-C. Kum et al. / Government Information Quarterly 26 (2009) 295–304

Gavirneni, S., Kapuscinski, R., & Tayur, S. (1999). Value of information in capacitatedsupply chains.Management Science, 45(1), 16−24 (Jan. 1999).

Hand, D., Mannila, H., & Smyth, P. (2001). Principles of Data Mining. Cambridge, MA: MIT.Kum, H. C., Duncan, D., Flair, K., & Wang, W. (2003). Social welfare program

administration and evaluation and policy analysis using Knowledge Discoveryand Data mining (KDD) on administrative data. Proc. of the 3rd National Conf. onDigital Government Res. Boston, MA, May 2003.

Kum, H. C., Duncan, D., Flair, K., & Wang, W. (2004). Successfully adopting IT for socialwelfare program management. Proc of the 4th National Conf on Digital GovernmentRes.(DGO 2004). Seattle, WA, May 2004.

National Resource Center for Child Welfare Data and Technology (NRC-CWDT). (2007).Retrieved December 17, 2007, from http://www.nrccwdt.org/index.html

United States Department of Health and Human Services (US-DHHS). (2006). The datameasures, data composites, and national standards to be used in child and familyservices reviews, ATTACHMENT B: Methodology for developing the composites. 71Fed. Reg. 32, No. 109, 969–987 (June 7, 2006).

United States Department of Health and Human Services (US-DHHS), Children'sBureau, Administration for Children, Youth and Families. (2007). Child WelfareMonitoring. Retrieved December 17, 2007, from http://www.acf.hhs.gov/programs/cb/cwmonitoring/index.htm

Usher, C. L. (1995). Improving evaluability through self evaluation. Evaluation Practice(now the American Journal of Evaluation), 16(1), 59−68.

Usher, C. L. (1999). From social experiments to reform initiatives: Implications fordesigning and conducting evaluations. Protecting Children, 15, 4−9.

Usher, C.L., Locklin, E., Wildfire, J.B., & Harris, C.C. (2001). Child welfare performanceratings: One state's approach. Administration in Social Work, 25, 35–48. DOI=10.1300/J147v25n01_03

Usher, C. L., Wildfire, J., & Schneider, S. (2001). Family to Family. Tools for rebuilding fostercare: The need for self evaluation— Using data to Guide Policy and Practice. Baltimore:Annie E. Casey Foundation.

Webster, D., Needell, B., &Wildfire, J. (2002). Data are your friends: Child welfare agencyself evaluation in Los Angeles County with the Family to Family initiative. Childrenand Youth Services Review, 24, 471−484.

Hye-Chung Kum, PhD, MSW, is a research assistant professor at the Jordan Institutefor Families in the School of Social Work at the University of North Carolina at ChapelHill. She also holds an adjunct position at the department of Computer Science. Herresearch explores use of administrative data and KDD technology for social welfarepolicy analysis, social services program evaluation, and local self-evaluation efforts.

Dean F. Duncan, PhD, is a research associate professor at the Jordan Institute forFamilies in the School of Social Work at the University of North Carolina at Chapel Hill.His research explores the outcomes for children and youth involved with the childwelfare system.

C. Joy Stewart, MSW, is a research faculty member at the Jordan Institute for Familiesin the School of Social Work at the University of North Carolina at Chapel Hill. Herresearch interests include child welfare and social services program evaluation.