CAVITY FREE AT THREE CDC EVALUATION 2013-2018

CAVITY FREE AT THREE

CDC EVALUATION 2013-2018

Health Surveys And Evaluation Branch

TABLE OF CONTENTS

EXECUTIVE SUMMARY ......................................................................................................... i

INTRODUCTION ................................................................................................................ 1

Background ......................................................................................................1

Infrastructure Development ..................................................................................1

METHODS ....................................................................................................................... 5

Data Collection Process ........................................................................................5

Evaluation Questions ..........................................................................................5

RESULTS ......................................................................................................................... 7

Training Data ....................................................................................................7

Pre/P ost D ata ...................................................................................................8

Technical Assistance Data ................................................................................... 12

Impact Data ................................................................................................... 15

CONCLUSION .................................................................................................................. 19

REFERENCES ..................................................................................................................20

CAVITY FREE AT THREE: CDC EVALUATION 2013-2018 COLORADO DEPARTMENT OF PUBLIC HEALTH AND ENVIRONMENT

EXECUTIVE SUMMARY

stablished in 2007, the Cavity Free at Three program trains medical and dental professionals to provide

preventive oral health services for young children and pregnant women to ultimately decrease dental

disease and reduce oral health disparities among high risk populations. The CF3 model includes eight

standard practices, including caries risk assessment, oral evaluation, knee to knee exam, uoride varnish

application, counseling/education with primary caregiver, caregiver goal setting, provision of or referral to

dental provider for a dental visit, and oral health services for pregnant women. The CF3 program moved

from the University of Colorado’s School of Medicine to CDPHE’s Oral Health Unit (OHU) in 2013. In 2013, the

Centers for Disease Control and Prevention (CDC) funded the development and implementation of a 5-year

comprehensive program evaluation that concluded on August 30, 2018 (Grant #1U58DP004904). The OHU

partnered with CDPHE’s Health Surveys and Evaluation Branch to develop the evaluation infrastructure and

implement the evaluation.

The evaluation was developmental in nature and included process, outcome and impact questions. Evaluation

and programmatic infrastructure was built in tandem, and the process included frequent data reporting

and feedback, a continual honing of the evaluation focus, and attention to results and continuous program

improvement. The evaluation answered the following questions: What has been the reach of the CF3 training

program? What are the facilitators and barriers to implementing CF3 in a provider’s practice? What types of

support would be the most helpful to implement after a CF3 training? What types of support and technical

assistance did CF3 give to providers? How many providers were given support and technical assistance? What is

the CF3’s geographic and population-level impact?

The evaluation team developed data collection and reporting infrastructure through Qualtrics, an online

survey platform, Freshdesk, a technical assistance tracking platform, and Tableau, a data visualization

software package. The evaluation also utilized population-level data from CDPHE’s Child Health Survey and the

CDC funded Basic Screening Survey, as well as Medicaid billing data supplied by the Colorado Department of

Health Care Policy and Finance via an interagency agreement.

Key ndings of the 5-year evaluation include:

■ 5,026 individuals have been trainings across Colorado, from 2007 to June 2018. 3,348 of these were

medical providers, dental providers, students, or other health professionals.

■ CF3 has trained providers and key partners in 73 percent of the counties in Colorado, both rural and

urban.

■ The CF3 training is positively impacting oral health practice changes of medical and dental healthcare

providers: 56% of providers who responded to the post-training survey between 2015 and 2018 (n=195)

reported they were fully implementing and 27 percent were partially implementing the CF3 model in

their practice after the training. Implementation rates signicantly increased for six of the eight oral

health services covered in the CF3 training.

■ Since 2015 when it was implemented, Freshdesk, the program’s technical assistance platform, is

mostly used as a place for individuals to communicate with CF3 staff about upcoming trainings. CF3

CAVITY FREE AT THREE: CDC EVALUATION 2013-2018 COLORADO DEPARTMENT OF PUBLIC HEALTH AND ENVIRONMENT

staff have responded to 602 technical assistance requests, most of which focused on training requests

and registration assistance or logistics. The platform has also been commonly used to request mailed

resources or request assistance with billing for services. The implementation of a webinar and billing

assistance resource document in March 2017 resulted in a signicant decrease in requests for billing

assistance.

■ The most common facilitators of CF3 implementation for both medical and dental providers are

the CF3 training and the overall need for services in the patient population. Many providers do not

experience any barriers to implementing CF3 (45% of dental providers and 26% of medical providers).

When barriers are reported, inadequate time on the part of medical providers (27%) and staff

turnover on the part of dental providers (16%) are the most common.

■ While CF3 has made some positive impacts since 2007, population-level data trends cannot be

attributed solely to CF3’s work, as many partners within the state have been implementing strategies

to improve children’s oral health. However, it is reasonable to assume that CF3 played a key role in:

■ A signicant decrease in the percent of Colorado kindergartners with untreated decay

(45% to 31%) and caries experience (27% to 18%), between 2006-2007 and 2016-2017

■ The percent of Medicaid-recipient children (age 0-2) receiving oral health services by

medical or dental providers has signicantly increased from 23 percent in 2010 to 33

percent in 2017.

■ A signicant increase in the percent of Medicaid recipient children who receive CF3

services during a Well Child Visit with a medical provider 0.5 percent in 2009 to 5 percent

in 2017.

■ Medicaid-recipient children (all ages) who had a well child visit that included a CF3 service

were 12 percent more likely to have a dental visit within 6 months.

While signicant strides have been made to address the oral health needs of young children, there is still

more work to be done. CDPHE is committed to further integrating the CF3 program with other public health

initiatives and identifying ways to make systems-based changes that support the CF3 model and further

expand the reach of CF3 into all medical and dental provider practices.

CAVITY FREE AT THREE: CDC EVALUATION 2013-2018 COLORADO DEPARTMENT OF PUBLIC HEALTH AND ENVIRONMENT

INTRODUCTION

BACKGROUND

avity Free at Three (CF3) was established in 2007 by several local health foundations. The program

trains medical and dental professionals to provide preventive oral health services for young children

and pregnant women to ultimately decrease dental disease and reduce oral health disparities among

high risk populations. The training outlines the oral health services that are standard of care for children and

pregnant women including, but not limited to: caries risk assessment, oral evaluation, knee to knee exam,

uoride varnish application, and counseling/education (anticipatory guidance) with primary caregiver. In

addition, Colorado Medicaid offers reimbursement of limited preventive oral health services in primary care

settings, but requires primary care providers to complete a “certication” program for reimbursement. CF3

is one of the two qualied certication programs and to date, about 5,000 individuals have attended CF3

training.

In September 2013, CF3 funders worked with the Colorado Department of Public Health & Environment

(CDPHE) to move the program from the University of Colorado’s School of Medicine to CDPHE’s Oral Health

Unit (OHU). This transition allowed both the OHU and CF3 to align efforts, workforce, resources and strategic

plans. In 2016, CF3 received funding from Health Resources and Services Administration (HRSA) to expand the

work with the prenatal population and pilot health systems change work to increase the number of women

who receive oral health services during pregnancy. The Centers for Disease Control and Prevention (CDC)

funded clinical preventive services and health systems change work, which included the work of the CF3

program and its corresponding evaluation from August 31, 2013 - August 30, 2018. This report is the conclusion

of the CDC-funded evaluation.

The CF3 evaluation has evolved over the years. Prior to CF3’s move to CDPHE, programmatic data such as a

handwritten sign-in sheet and training feedback survey, were collected at each training. This data was used

to help track training numbers and provide the trainers with feedback on how the training was received by

participants. The feedback was overwhelmingly positive about the training and trainers at the time when CF3

moved to CDPHE, which prompted CF3 staff to begin thinking about what was next to evaluate.

After the move to CDPHE, CF3 engaged evaluation support through the Health Surveys and Evaluation Branch

(HSEB). HSEB worked with the CF3 program to plan the evaluation and build programmatic and evaluation

infrastructure.

INFRASTRUCTURE DEVELOPMENT

During the 2013-2018 CDC funding cycle, CF3’s greatest evaluation-related accomplishment was building its

capacity and infrastructure to support evaluation and quality improvement efforts. The developed infrastructure

helped to streamline data collection processes and improve tracking of trainings, trained providers and technical

assistance, while providing real-time data for staff and key stakeholders. Table 1 provides a summary of the

infrastructure elements developed, the technology adapted, and data collected or displayed within each

element. This report highlights the technology used to develop each element of the infrastructure, the data

collected or used for each element, and program improvements made based on the evaluation.

CAVITY FREE AT THREE: CDC EVALUATION 2013-2018 COLORADO DEPARTMENT OF PUBLIC HEALTH AND ENVIRONMENT

Table 1. Evaluation infrastructure elements developed.

Infrastructure Element Technology Adapted Data Collected/Displayed

Data collection Qualtrics # and type of trained providers, current and changes to

oral health practice provided for children and pregnant

women, facilitators and barriers to implementation,

technical assistance needs

Technical assistance Freshdesk # and type of requests for technical assistance (tickets),

# of tickets addressed

Data visualization Tableau Displays monthly numbers of providers trained, type

of providers trained, oral health practice behavior

change, facilitators and barriers for dental and medical

providers, geographic impact of trainings

DATA COLLECTION - QUALTRICS

Qualtrics, an online survey platform, is the standard online survey platform used at CDPHE. Qualtrics enables CF3

to collect pre-training data from attendees as the time they complete their online registration. The online survey

platform is set up for trainees to seamlessly pass a tablet around the training to sign in and ensures all providers

who sign in at the training have completed the pretest. Qualtrics also automates the administration of post-test

surveys and reminders, as well as, generate automated tickets or technical assistance requests in Freshdesk.

Technical Assistance - Freshdesk

Freshdesk is a free online technical assistance platform that allows CF3 staff to streamline technical assistance

and training requests. CF3 receives technical assistance inquiries through Freshdesk in two ways, 1) through

the online post-test, where trainees are able to request technical assistance support to implement the CF3

model, and 2) through a Freshdesk widget that has been embedded into the CF3 website and newsletter.

The platform serves as a centralized communication hub which allows the CF3 team to address technical

assistance needs, assign tickets to appropriate staff to respond, categorize and prioritize needs, and run

reports on several metrics (e.g. number and type of inquiries received, time taken to address the inquiry).

Also, in the event of staff turnover the Freshdesk holds all historical technical assistance requests so no

communications are lost with partners or trainees.

Data Visualization - Tableau

Tableau is the data visualization platform used at CDPHE with many internal supports available (i.e., internal

server, web developers, and learning community). Hosting the CF3 dashboard on Tableau allows for access to

all of these services and the accessible interface increases the likelihood of data utilization by CF3 staff.

In August 2016, the evaluation team created a data dashboard for CF3 which included programmatic,

community, and population level data. The dashboard’s objective is to effectively disseminate and translate

the data to program staff, internal partners, and external key stakeholders. The evaluation team continues to

discuss with CF3 staff the functionality of the dashboard and make changes in accordance with feedback. Each

tab on the dashboard reects a different data source and provides CF3 with ongoing feedback about their

program. Tabs and data sources include: Community assessment (population-level data), geographic impact

(county-level training data), trainings by month, trainings by year (pre-test registration survey data), model

CAVITY FREE AT THREE: CDC EVALUATION 2013-2018 COLORADO DEPARTMENT OF PUBLIC HEALTH AND ENVIRONMENT

implementation, facilitators and barriers, and post-training practice changes (post-test survey data). The CF3

dashboard is available online here: https://www.colorado.gov/pacic/cdphe/Cavity-Free-at-Three-Dashboard

EVALUATION FOCUS & QUESTIONS

The CF3 evaluation was developmental in nature. The evaluation and programmatic infrastructure was

built simultaneously, with continued feedback, looking at results and continuous improvements. In the CF3

evaluation plan, nine questions were proposed. While all questions were pursued, not all were answered due

to the evolution of the program infrastructure. In addition to the originally proposed evaluation questions,

two new evaluation questions were included to address CF3’s overall reach, geographic reach and population-

level impact on the oral health of infants and children. The impact evaluation questions were identied as an

evaluation area of interest from local stakeholders due to CF3’s 10 year anniversary in 2017.

The table below indicates whether an evaluation question was answered and where in the report to nd

results and details. If the evaluation question was not answered, an explanation of why this question was not

answered will be presented in the cited section.

Table 2. Evaluation questions.

Evaluation question Answered For more information:

What has been the reach of CF3 training program? [New

evaluation question, not proposed in CDC evaluation

plan]

Yes RESULTS, Pre/Post Data,

p. 8

What are the facilitators and barriers to implementing

CF3 in a provider’s practice?

What types of support would be the most helpful to

implement CF3 after training?

Yes RESULTS, Pre/Post Data,

p. 8

What factors contributed to successful implementation

of CF3 in practices?

No RESULTS, Pre/Post Data,

p. 8

What types of support and technical assistance did CF3

give to providers?

How many providers were given support and technical

assistance?

Yes RESULTS, Technical Assistance Data,

p.12

When should Support Follow-up occur?

How should Support Follow-up occur?

What can CF3 do to improve the Follow-up component?

Did CF3 have a greater impact after implementing the

Follow-up component?

No RESULTS, Technical Assistance Data,

p. 12

What is CF3’s geographic and population-level impact?

[New evaluation question, not proposed in CDC

evaluation plan]

Yes RESULTS, Impact Data,

p. 15

The evaluation and data infrastructure developed for the CF3 program coincided and helped to inform

the process and outcome evaluation which was used to inform program improvements and support

sustainability efforts.

CAVITY FREE AT THREE: CDC EVALUATION 2013-2018 COLORADO DEPARTMENT OF PUBLIC HEALTH AND ENVIRONMENT

METHODS

mixed methods approach was used to measure both process, outcome and impact measures of the

CF3 program.

DATA COLLECTION PROCESS

Pre and post-test data

In 2015, a CF3 data collection process was developed to help streamline training registration, validate training

attendance, and collect of pre and post-test data. Since then, all CF3 trainees register online for a training

which includes a pretest that collects provider demographic information and current oral health services

offered in their practice. Trainees must sign into each training to conrm their attendance and inform monthly

training data. About two months after each training, trained providers who are able to bill for CF3 related

services (e.g. medical providers, dental providers, medical assistants, and dental assistants) receive a post-

test, which assess the level of CF3 model implementation, facilitators and barriers to implementation, and

identies technical assistance needs. The pre- and post-test design evaluates whether any differences in self-

reported oral health practice behaviors exist from before to after the CF3 training.

Technical assistance data

The Freshdesk platform stores all technical assistance and training requests gathered through the post-test,

CF3 website and newsletter. All requests are categorized into themes such as: training request, mailed

resources, billing assistance, hands-on coaching, electronic medical record assistance, policy development,

Medicaid/CHP+ application assistance, or other. The data is regularly pulled and shared with the program to

inform training and program improvements.

Population-level data

In addition, population-level metrics are also tracked using the Child Health Survey (CHS), Basic Screening

Survey (BSS), and Centers for Medicare and Medicaid 416 report (CMS 416), described below. CDPHE also

receives periodic CF3 related data reports through an interagency agreement with the Department of Health

Care Policy and Finance (HCPF), as HCPF has capacity to provide.

■ Child Health Survey: was developed to ll the gap in health data in Colorado that existed for children

ages 1-14 years and is conducted annually. Through a screening process, Behavioral Risk Factor

Surveillance System (BRFSS) participating households with children ages 1-14 are called a few days

later to conduct the follow-up survey on their children’s health (i.e., physical activity, nutrition, oral

health, access to health and dental care, etc.)

■ Basic Screening Survey: is an in-mouth dental screening conducted with kindergarten and third grade

students to assess caries experience, untreated decay, dental sealants and need for dental treatment.

The survey is conducted every 3-4 years.

CAVITY FREE AT THREE: CDC EVALUATION 2013-2018 COLORADO DEPARTMENT OF PUBLIC HEALTH AND ENVIRONMENT

■ Centers for Medicare and Medicaid 416 Report: State Medicaid agencies are required to report Early

and Periodic Screening, Diagnostic, and Treatment (EPSDT) performance information annually to help

ensure children under the age of 21 who are enrolled in Medicaid receive appropriate preventive,

dental, mental health and developmental, and specialty services.

CAVITY FREE AT THREE: CDC EVALUATION 2013-2018 COLORADO DEPARTMENT OF PUBLIC HEALTH AND ENVIRONMENT

RESULTS

TRAINING DATA

his section answers the following evaluation questions:

■ What has been the reach of the CF3 training program?

While CDC funding for CF3 began in August 31, 2013 the training data below are since CF3’s inception

in 2007 which demonstrates CF3’s larger reach and impact. Data are reported based on calendar year, not

scal year. From January 2008 to June 2018, 5,026 individuals have been trained at 332 trainings across

Colorado: 839 medical providers, 476 dental providers, 1,719 students, 314 “other health professionals”

(providers who cannot bill for CF3 services such as RNs and LPNs), 1,245 “other professionals” (e.g. ofce staff,

ofce assistants, partners), and 433 individuals with an unknown provider type (they did not designate their

profession on data collection tools). Figure 1 shows the respective number of CF3 trainings held and number

of individuals trained per year.

Figure 1. Number of CF3 trainings held and trainees per year.

105

173

405

591

668

867

380

632

594

428

100

200

300

400

500

600

700

800

900

2008 2009 2010 2011 2012 2013 2014 2015 2016 2017

Number of trainees Number of trainings

CAVITY FREE AT THREE: CDC EVALUATION 2013-2018 COLORADO DEPARTMENT OF PUBLIC HEALTH AND ENVIRONMENT

Figure 2 shows the number of medical providers, dental providers, students, and other/unknown trainees each

year. The other/unknown category includes “other health professionals”, “other professionals”, and “unknown”

provider type described above.

Figure 2. Number of CF3 trainees by provider type per year.

These trends show that CF3 increased the number of trainees of all types from 2008-2013. There was one year

of decline from 2013-2014 when CF3 moved to CDPHE. After that transition period, CF3 again increased the

number of trainees from 2014-2015. In 2016, CF3 focused on training more students, which is indicated by the

increase in students trained from 2016-2017. From 2015-2017, there is a slight decrease in overall number of

trainees which may be due to the implementation of a readiness assessment.

The readiness assessment was used to screen a clinic’s ability to implement CF3 services. If the clinic

demonstrated adequate readiness, the clinic was eligible for CF3 training. The CF3 trainings were more

targeted towards clinic change efforts, rather than larger trainings where any provider or professional could

attend. While the screening process decreased the quantity of trainings and number of providers trained,

the screening process has been seen as benecial since only clinics that were ready to implement the CF3

model were trained. The hope is for greater implementation and impact with providers. In the future, CF3

will be looking for ways to help clinics become ready to implement the CF3 model and support their practice

transformation efforts.

PRE AND POST TRAINING DATA

This section answers the following evaluation questions:

■ What are the facilitators and barriers to implementing CF3 in a provider’s practice?

■ What types of support would be the most helpful to implement CF3 after training?

Unaddressed evaluation question explained:

■ What factors contributed to the successful implementation of CF3 in practices?

100

150

200

250

300

350

400

2008 2009 2010 2011 2012 20 13 2014 2015 2016 2017

Number of medical providers

Number of dental providers

Number of students

Number of other/unknown

CAVITY FREE AT THREE: CDC EVALUATION 2013-2018 COLORADO DEPARTMENT OF PUBLIC HEALTH AND ENVIRONMENT

Oral Health Services

Since June 2015, 693 medical/dental providers and medical/dental assistants received the post-test. One-

hundred ninety-ve of those providers completed the post-test, which resulted in a 28 percent response rate.

At post-test, slightly more than half of providers (56%) reported they were fully implementing the CF3 model

into practice, 27 percent were partially implementing, 14 percent were not implementing currently but plan to

in the future, and 3 percent do not plan to implement CF3 into their practices (Figure 3).

Figure 3. Self-reported level of CF3 model implementation (n=195).

To examine the impact of the training, paired samples t-test and chi-square analyses were run on pre and

post-training data to assess the changes in oral health service implementation for children under age 3 and

pregnant women. Overall, the results suggest that CF3 training is positively impacting oral health practice

changes and provider behaviors. Post-test data reveals that six out of the eight standard CF3 practices are

being implemented by medical and dental providers at signicantly higher rates that they were pre-training

(Figure 4). The percent of medical and dental providers providing counseling or anticipatory guidance

increased from 65 percent to 73 percent from pre to post-training, however this change was not statistically

signicant. Also included in Figure 4, the percent of medical and dental providers reporting that the age of

the youngest child seen by the practice as under the age of one signicantly increased from 61 percent to 82

percent after CF3 training.

0 20 40 60 80 100

Percent

Do not plan to implement

Not implementing currently but

plan to in the future

Partially implementing

Fully implementing

CAVITY FREE AT THREE: CDC EVALUATION 2013-2018 COLORADO DEPARTMENT OF PUBLIC HEALTH AND ENVIRONMENT

Figure 4. Oral health practice changes by medical and dental providers pre and post CF3 training (n=195).

* indicates a signicant difference at p<0.05.

** Denominator for dental providers (general dentists, pediatric dentists, dental hygienists, dental assistants) = 57.

Facilitators and Barriers

Overall, healthcare providers consistently report more facilitators than barriers to implementing CF3 at post-

test. Medical and dental providers reported similar facilitators about the implementation of the CF3 model,

whereas barriers differed amongst the two provider groups.

The most common facilitators for both medical and dental providers, continue to be the CF3 training and

the children under age three patient population need for oral health services. The remaining most commonly

reported facilitators, shown in Figure 5, vary between provider groups, but this is likely due to differences

between the set up and support within medical and dental ofces.

82*

67 *

54*

75*

61 *

75*

0 20 40 60 80 100

Youngest child seen at practice <age 1

Oral health services for pregnant women (dental

providers)**

Provision of or referral to dental provider for a

dental visit

Caregiver goal setting

Counseling/education (anticipatory guidance)

with primary caregiver

Fluoride varnish application

Knee to knee exam

Oral evaluation

Caries risk assessment

Pre-test

Percent

Post-test

CAVITY FREE AT THREE: CDC EVALUATION 2013-2018 COLORADO DEPARTMENT OF PUBLIC HEALTH AND ENVIRONMENT

Figure 5. Most commonly reported facilitators to implementing CF3 model by provider type.

Few barriers were reported amongst the two provider groups. In fact, 45 percent of dental providers and 26

percent of medical providers reported “none” for barriers. The most common barrier for medical providers

was “inadequate time to provide oral health services” (27%), while dental providers reported “staff turnover”

(16%). Medical providers reported not having adequate time to provide oral health services, which is likely due

to the competing priorities providers experience during a well child visit.

Program Improvements

The barriers informed improvements to CF3 technical assistance such as creating an implementation guide

for organizations that are planning to be trained by CF3 and implement oral health services in their clinical

practice in the near future. Among other tools and tips, the implementation guide provides instructions on

how to develop process charts for optimizing clinic ow, which can help medical providers allocate time for

oral health services. The best practices taught in the CF3 training are the caries risk assessment, the knee-

to-knee approach for oral health screenings, the US Preventive Services Task Force (USPSTF) grade B and the

American Academy of Pediatrics (AAP) recommendations for uoride varnish application and prescription of

oral uoride supplementation by primary care providers. The CF3 training focuses on hands-on skill building to

improve clinician competency to provide these services quickly and efciently. Clear messaging that the CF3

model is based upon the USPSTF grade B and AAP recommendations helps providers understand, and thereby

practice, the standard of care.

0 20 40

Percent

60 80

100

CF3 was a practice or agency priority

Enthusiastic staff

Adequate reimbursement

Had necessary supplies

Easy to incorporate CF3 into exam

Ability to bill for services

Practice or agency enthusiasm

Patient population need for services

CF3 training

Dental (n=49)

Medical (n=129)

CAVITY FREE AT THREE: CDC EVALUATION 2013-2018 COLORADO DEPARTMENT OF PUBLIC HEALTH AND ENVIRONMENT

Unaddressed Evaluation Question

The evaluation question, “What factors contributed to the successful implementation of CF3 in practices?”

could not be fully answered due to the nature of the self-report data available. The program was unable

to obtain or connect the self-report data to clinical outcome or practice billing data which would help to

operationalize successful implementation. Anecdotally, it appears that the CF3 training, a practice’s awareness

of patient’s need for CF3 services, and ease of incorporating CF3 components into the exam procedure could

lead to successful CF3 implementation.

TECHNICAL ASSISTANCE DATA

This section answers the following evaluation questions:

■ What types of support and technical assistance did CF3 give to providers?

■ How many providers were given support and technical assistance?

This section also addresses why the following evaluation questions were not answered:

■ When should Support Follow-up occur?

■ How should Support Follow-up occur?

■ What can CF3 do to improve the Follow-up component?

■ Did CF3 have a greater impact after implementing the Follow-up component?

Since adopting the Freshdesk system in April 2015, CF3 staff have addressed 602 tickets. Figure 6 shows the

number of tickets addressed each year. Please note that the 2018 data reect tickets from January 1 - June

30. The data in this chart indicate that Freshdesk was quickly integrated into CF3 communication processes

and has continued to be a consistent form of communication over the past few years.

Figure 6. Number of tickets per year.

178

238

151

100

150

200

250

2015 2016 2017 2018

CAVITY FREE AT THREE: CDC EVALUATION 2013-2018 COLORADO DEPARTMENT OF PUBLIC HEALTH AND ENVIRONMENT

Figure 7 shows the number (and percent) of tickets submitted by each provider type. Of the types of providers

listed, “Other/Unknown” providers (n=123, 19%) submit the most tickets to Freshdesk, followed by dental

hygienists (n=114, 19%), physicians (n=72, 12%), and nurses (n=71, 12%). These data indicate that a variety of

providers reach out to CF3 from both medical and dental professions, and are likely the providers who have

more inuence on practice transformation and may reach out to request a CF3 training or seek advice on

implementing the CF3 model, for example. The large number of unknown providers is likely due to the self-

report nature of the ticket form. Individuals are not required to ll in that eld when they submit the form, so

many people do not.

Dental hygienists have submitted the most technical assistance requests, followed by physicians, nurses, mid-

level providers, and medical staff. Dental hygienists submitted the most training requests (n=46), followed by

physicians (n=26) and mid-level providers (n=22). Dental hygienists also submitted the most requests (n=18) for

mailed materials, followed by nurses (n=14). Medical staff and physicians both submitted the same number of

requests for billing assistance (n=10), followed by dental hygienists (n=6).

Figure 7. Number (and percent) of tickets submitted by provider type.

Figure 8 shows the number (and percent) of the types of tickets addressed within Freshdesk. Training

requests are the most popular reason someone reaches out to CF3 through Freshdesk (n=227, 36%), followed

by the “Other” category (n=194, 33%), mailed resources (n=84, 14%), and billing assistance (n=55, 10%). The

“Other” category consists mostly of resolving training registration issues (e.g. re-sending the registration link),

questions about training logistics (e.g. date, time, location, training certicates), communication regarding

Master Trainers (e.g. how to become a Master Trainer), sending education curriculum scan results, and sending

the CF3 implementation guide. These data indicate that Freshdesk is mostly used as a place for individuals

to communicate with CF3 staff about upcoming trainings (e.g. requesting a training, resolving training

registration issues, getting questions answered about training logistics).

15 (2%)

17 (3%)

25 (4%)

41 (7%)

54 (9%)

55 (9%)

71 (12%)

72 (12%)

114 (19%)

123 (19%)

0 20 40 60

Number

80 100 120

Dental Assistant (RDA, DA)

Dental Staff

Student

Medical Assistant

Dentist (DMD, DDS)

Medical Staff

Mid-level Provider (NP, PA, CNM)

Nurse

Physician (MD, DO)

Dental Hygienist (RDH, DH)

Other/Unknown

CAVITY FREE AT THREE: CDC EVALUATION 2013-2018 COLORADO DEPARTMENT OF PUBLIC HEALTH AND ENVIRONMENT

Figure 8. Number (and percent) of types of tickets addressed.

Program Improvements

The top three types of tickets (training requests, mailed resources, and billing assistance) have been

consistent since 2015. In response to the high number of tickets requesting billing assistance, CF3 has

developed a webinar series to educate individuals about best billing practices and developed a billing

assistance resource document for use in clinics. An independent samples t-test was run to explore the impact

of creating the billing sheets on the frequency of billing assistance requests in Freshdesk. From April 2015 to

March 2017 (when the billing assistance sheets were created), CF3 addressed 467 tickets, 46 (n=10%) of which

were billing assistance requests. From April 2017 to June 2018, CF3 addressed 135 tickets, 8 (6%) of which

were billing assistance requests. The independent samples t-test revealed that the frequency of times people

asked for billing assistance signicantly dropped after the billing sheets were created. CF3 makes program

improvements based on Freshdesk requests, and these data indicate that CF3 is addressing the need for billing

assistance through their billing sheets and webinar series.

In response to the high number of tickets requesting mailed resources, CF3 has made their resources more

available on the website (e.g. reorganized the website so users can more easily search for and download

resources), shared resources with online listservs, and printed resources in several languages for distribution

at health care centers. These resources were released on the CF3 website on January 23, 2018. Not enough

time has passed to look at the impact of this program improvement.

Unaddressed Evaluation Question

The Follow-Up Component evaluation questions were not answered because of low response rate to the post-

test survey and lack of communication through Freshdesk. Ideally, trainees would request technical assistance

through the post-test survey, a CF3 staff member would reach out to them to address the request, and the

trainee and CF3 staff member could work with the provider to resolve the technical assistance request. In

addition, information could be gathered about the effectiveness of these interactions through Freshdesk.

Realistically, a low response rate on the post-test survey meant fewer people were requesting technical

assistance than expected. CF3 staff followed up with trainees within one week after trainees completed the

post-test, and unfortunately trainees rarely engaged in email or phone correspondence with CF3 staff about

the request.

194 (32%)

3 (1%)

5 (1%)

10 (2%)

26 (4%)

54 (9%)

83 (14%)

227

(38%)

0 50 100

Number

150 200 250

Other

Medicaid/CHP+ Application

Policy Development

EMR Assistance

Hands-On Coaching

Billing Assistance

Mailed Resources

Training Request

CAVITY FREE AT THREE: CDC EVALUATION 2013-2018 COLORADO DEPARTMENT OF PUBLIC HEALTH AND ENVIRONMENT

To improve the Follow-up Component, CF3 staff tried reaching out to providers multiple times to try to initiate

conversation before “resolving” the technical assistance request in the system. Because CF3 was unable to

gather meaningful information from the Follow-up Component, they were unable to understand if they had a

greater impact after implementing the Follow-up Component.

IMPACT DATA

This section answers the following evaluation question that was added to the evaluation after the CDC

proposed evaluation questions were created:

■ What is CF3’s geographic and population-level impact?

CF3’s vast geographic impact around Colorado has allowed a variety of different providers in a variety of

different healthcare settings to be trained. CF3 has also contributed to population-level data indicating

improvement in children’s oral health. While there has been CF3 success in certain practices, the program

recognizes the difculty in changing practice behaviors and creating sustainable practices within the clinic.

Geographic Impact

As of December 2017, CF3 had trained providers in 47 of Colorado’s 64 counties (73%). Figure 9 shows the

number of trainings and trainees by county from 2007 through 2017. CF3 has trained providers and supported

practices implementing the model throughout both urban and rural Colorado. The urban counties reached

the most by CF3 were along the Front Range: Adams, Arapahoe, Boulder, Denver, Douglas, Jefferson, and El

Paso. The rural counties reached the most by CF3 were: Alamosa, Chaffee, Eagle, Gareld, Mesa, Moffat, Rio

Blanco, and Weld.

CAVITY FREE AT THREE: CDC EVALUATION 2013-2018 COLORADO DEPARTMENT OF PUBLIC HEALTH AND ENVIRONMENT

Figure 9. Number of CF3 trainees and trainings by county 2007-2017.

Population-Level Impact

Population-level data helps the CF3 staff understand oral health needs and trends across Colorado. While

CF3 has made some positive impacts since 2007, the population-level data trends cannot be attributed solely

to CF3’s work, as many partners within the state have been implementing strategies to improve children’s

oral health. It is also important to note that Medicaid expansion in 2012 was also likely a large contributor to

changes in the Medicaid data. This section highlights the population-level data trends that CF3 has contributed

to since 2007.

Child Health Survey

The CHS asks parents to rate the condition of their child’s teeth. Children ages 1-4 with poor/fair condition of

teeth has decreased from 4.3 percent in 2010-12 to 1.6 percent in 2016-17. Also, the percent of children ages

1-4 who have received a preventive dental visit within the past year has increased from 60 percent in 2010-

12 to 64.8 percent in 2014-16. Due to small sample sizes for this age group, multiple years of data must be

combined for prevalence estimates. While these trends are not statistically signicant, they are in the desired

direction.

WELD

MOFFAT

MESA

BACA

YUMA

Number of CF3 trainees

Number of trainings

PARK

LAS ANIMAS

ROUTT

GUNNISON

LINCOLN

GARFIELD

LARIMER

PUEBLO

BENT

SAGUACHE

KIOWA

LOGAN

GRAND

RIO BL ANCO

EL PA SO

EAGLE

ELBERT

MONTROSE

WASHINGTON

DELTA

LA PLATA

OTERO

KIT CARSON

JACKSON

ADAMS

CHEYENNE

PROWERS

MONTEZUMA

FREMONT

PITKIN

MORGAN

HUERFANO

CONEJOS

COSTILLA

ARCH ULETA

HINSDALE

DOLORES

CHAFFEE

SAN MIGUEL

MINERAL

CUSTER

DOUGLAS

SUMMIT

CROWLEY

OURAY

PHILLIPS

BOULDER

TELLER

ALAMOSA

RIO GRANDE

LAKE

SEDGWICK

ARAPAHOE

JEFFERSON

SAN JUAN

CLEAR

CREEK

GILPIN

BROOMFIELD

DENVER

1-5

6-10

11-15

16-25

48-56

6-32

33-75

76-220

221-758

759-1482

CAVITY FREE AT THREE: CDC EVALUATION 2013-2018 COLORADO DEPARTMENT OF PUBLIC HEALTH AND ENVIRONMENT

Basic Screening Survey

The BSS also continues to show a decrease in the percent of kindergartners with untreated decay and caries

experience (Figure 10). The decrease demonstrated below in disease burden for young children parallels

the implementation of the CF3 program. Given that CF3 was the only long-term signicant change in access

to preventive care during this time period, and the research indicating the model leads to decrease decay

(Braun, 2017), CDPHE attributes CF3 to contributing to these improvements.

Figure 10. Percent of kindergarteners with untreated decay and caries experience.*

*The 2016-17 BSS introduced a new methodology and trends should be interpreted with caution, however, the authors believe the results of the

trend analyses are valid.

Centers for Medicare and Medicaid 416 Report

The CMS 416 report is an important data source the CF3 program follows since the CF3 training is one of the

required training options for providers to become a Medicaid billing provider for oral health services. Since

2010, the CMS 416 has shown signicant increases in the percent of Medicaid-recipient children ages 0-2 who

receive oral health services from a medical provider and from dentists or other qualied medical practitioners

between 2010 and 2017 (Figure 11).

Figure 11. Percent of Medicaid-recipient children ages 0-2 receiving oral health services by provider type.

Data source: CMS 416 report. * Indicates a signicant difference at p<0.05.

100

2006-2007 2016-2017

Untreated decay

Percent

Caries experience

100

2010 2017

Medical provider

Percent

Medical or dental provider

CAVITY FREE AT THREE: CDC EVALUATION 2013-2018 COLORADO DEPARTMENT OF PUBLIC HEALTH AND ENVIRONMENT

In 2017, HCPF conducted a linear probability model and found that Medicaid-recipient children who had a well

child visit that included a CF3 service were 12 percent more likely to have a dental visit within 6 months. In

addition, the percent of well child visits that include a CF3 service has signicantly increased from 0.5 percent

in scal year 2009-10 to 5 percent in 2015-16. The analysis demonstrates the uptake of CF3 implementation

at well child visits. However, it is important to note that Medicaid claims data can include multiple visits per

child and children between the ages of birth to 21.

CAVITY FREE AT THREE: CDC EVALUATION 2013-2018 COLORADO DEPARTMENT OF PUBLIC HEALTH AND ENVIRONMENT

CONCLUSION

F3 has raised awareness of oral health needs of young children and ways to address those needs

through the training and subsequent support of over 5,000 healthcare providers, support staff,

students and key partners in rural and urban Colorado since 2007. CF3 has contributed to practice

improvements in both medical and dental ofces, and to the signicant increases seen in the number of

young Medicaid-recipient children receiving oral health services from both medical and dental providers in

Colorado. CF3 has also contributed to signicant decreases in untreated decay and caries experience among

kindergartners. While signicant strides have been made to address the oral health needs of young children

and pregnant women, there is still more work to be done. Not enough young children receive preventive oral

health interventions and many pregnant women do not receive needed dental care.

Given the positive impact CF3 has demonstrated on the oral health of Coloradans, and the remaining gaps in

access to oral health services, it is important to continue the spread of the model across the state. CDPHE

is committed to further integrating the CF3 program with other public health initiatives and identify ways to

make systems-based changes that support the CF3 model and continue to expand the reach of CF3 into all

medical and dental provider practices.

Future Steps

There are many potential next steps for the Cavity Free at Three program. CDPHE is committed to continue

addressing barriers to the sustainability and spread of the program within health systems. The partnerships

and strategies below will be further explored by CDPHE and CF3 partners to prioritize future allocation of

resources and efforts.

■ Continuing to use systems-based approaches to increase access to oral health preventive services for

infants, toddlers and pregnant patients with: public health partners serving communities with a high

burden of oral disease, Accountable Care Collaborative partners serving publicly insured Coloradans,

health systems within dental Health Professional Shortage Areas, commercial insurers and other

partners.

■ Continuing to provide community-level data, strategies, capacity and technical assistance to local

public health agencies (LHPAs) throughout the Colorado Health Assessment and Planning process.

■ Continuing collaborations with CDPHE clinical quality improvement (CQI) partners to support alignment

of CF3 efforts with best practices and increase CQI resources for oral health interventions. E.g.:

implementing AHRQ’s public health academic detailing model to increase uptake of the CF3 model by

dentists.

■ Continuing collaborations with HCPF and other insurers to increase the types of providers able to bill

for CF3 services and, therefore, increase long-term sustainability of program implementation.

■ Supporting alignment of CF3 efforts with other programs addressing perinatal health to increase

impact and evidence-informed strategies.

■ Increasing coordination of perinatal oral health activities among the Oral Health Unit’s staff, partners

and contractors throughout new HRSA and CDC grant cycles. (E.g.: using Regional Oral Health

Specialists as CF3 Master Trainers; Colorado Community Health Network (CCHN) linking FQHCs to

receive CF3 trainings; and HCPF representative being represented on the CF3 Advisory Board to

coordinate efforts to increase the reach of the program).

■ Continuing to explore other data sources such as utilization, surveillance, workforce and other data to

inform program improvements.

■ Continuing ongoing evaluations, including rapid cycle quality improvement efforts to ensure efcient

and effective program implementation.

REFERENCES

American Academy of Pediatrics Policy Statement–Organizational Principles to Guide and Dene the Child

Health Care System and/or Improve the Health of All Children: Section on Pediatric Dentistry Reprinted with

permission of the American Academy of Pediatrics (Pediatrics. 2003;111:1113-1116). Retrieved from: http://

www.aapd.org/assets/1/25/Editorial4-03.pdf

Braun, Patricia; Widmer-Racich, Katina; Sevick, Carter; Starzyk, Erin; Mauritson, Katya; Hambidge, Simon.

(2017). Effectiveness on Early Childhood Caries of an Oral Health Promotion Program for Medical Providers.

American Journal of Public Health. Retrieved from: https://ajph.aphapublications.org/doi/abs/10.2105/

AJPH.2017.303817

Calanan, Renee; Elzinga-Marshall, Gabrielle; Gary, Dahsan; Payne, Emily; Mauritson, Katya. (2018). Tooth be

Told – Colorado’s Basic Screening Survey, Children’s Oral Health Screening: 2016-17. Colorado Department of

Public Health & Environment. Retrieved from: https://www.colorado.gov/pacic/sites/default/les/PW_OH_

BSSReport.pdf

Cavity Free at Three Data Dashboard. Accessed through the CF3 website: https://www.colorado.gov/pacic/

cdphe/Cavity-Free-at-Three-Dashboard

Centers for Medicare and Medicaid 416, 2010-2017 data. Accessed through the Medicaid EPSDT website:

https://www.medicaid.gov/medicaid/benets/epsdt/index.html

Child Health Survey (CHS), 2010-16 data. Accessed through the CDPHE VISION system at: https://www.

colorado.gov/pacic/cdphe/vision-data-tool

U.S. Preventive Services Task Force, Grade B Recommendations. Accessed through the USPSTF

recommendations website: https://www.uspreventiveservicestaskforce.org/Page/Name/

uspstf-a-and-b-recommendations/