ORLANDO, Fla. – A review of the botched 2018 SunPass system upgrade by Florida’s Inspector General’s Office details the immediate consequences of the failed system launch.

On June 11, 2018 when FDOT contractor Conduent went live with the new SunPass website and billing system, known as the SunPass Centralized Customer Service System or CCSS, and failed to function as designed, the SunPass website, mobile site, customer service and airport parking operations were all affected, according to the OIG report. It also affected the billing and financial reconciliations.

Here are key takeaways from the OIG’s findings:

Widespread problems: SunPass was expecting an average of 175,000 calls on the day of the launch but fielded more than 232,250 calls.

How much the upgrade cost customers

Due to the upgrade, 571 SunPass customers experienced unintentional duplicate charges for replenishments on their passes. Those charges resulted in 1,260 refunds to those customers.

A total of 2,247 SunPass customers’ bank accounts were over-drafted due to untimely billing by the new system. The overdraft fees were refunded to customers to the tune of $189,885.

Turnover amid CCSS project managers was high

Prior to the system upgrade, there were several key managers who left the project, including the CCSS finance director, project director and executive director positions.

Conduent is currently in the process of hiring its fifth CCSS finance director, according to the OIG report.

The Inspector General found that Conduent did not meet its contractual obligation to timely deliver the required audits for the timeframes.

How to avoid a future incident

From its findings, the IG recommended 10 things for FDOT Secretary Kevin Thibault to consider for similar projects in the future.

1. Appoint one centralized project director who is employed by and accountable to a senior level manager within FDOT to provide robust project oversight.
2. Consider segregating future contracts into multiple unique contracts pertaining to specific needs and requirements.
3. Ensure new contracts require: Final approvals for all project phases be approved by a department representative, formalized in writing, and included in the contract management file; Documentation as to the justification of extensions of deadlines and timeframes; FDOT have immediate, independent, and unrestricted access to all infrastructure monitoring systems, and not solely rely on the contracted vendor to provide insight to what is occurring within the system; Adherence to public records requirements and ensure this requirement is contractually applied to all sub-contractors in the contracts; A written recovery plan is in place should the system not function as intended at Go-Live, and that this recovery plan is tested in advance of Go-Live; Consideration to conducting a “soft rollout” and/or running the “legacy” system in parallel with the new system; and, Detailed Scopes of Service when outsourcing project management functions, including specific documentation detailing responsibilities and deliverables for project management of this size and scope.
4. Ensure the finance team is included at critical decision-making junctures, including testing of financial controls prior to signoff.
5. Obtain Independent Verification and Validation (IV&V) services to provide subject matter expertise including specific project guidance and independent reviews of the entire system, testing plans, and testing results. The department, jointly with the IV&V vendor, should create and finalize all testing plan requirements and approvals.
6. Include more specificity in the Contract and Master Test Plan to address areas including, but not limited to, a thorough description of definitions, testing procedures, resolution of discrepancies and testing durations.
7. Ensure future pre-Go-Live communication strategies more strongly relay the potential impact that a new system may have on the users of the system.
8. Verify future settlements comply with applicable law in that any intellectual property received as a result of settlements is identified, valued and properly recorded.
9. Review and evaluate the SOC 1, Type II report and Payment Card Industry (PCI) Attestation of Compliance to determine if these, once provided, comply with the intent of the contract. We also recommend FDOT act according to the terms and conditions of Contract BE087, if the provided audits do not meet the requirements and intent of Contract BE087. We also recommend that failure to conduct audits within established terms have a specific penalty within the contract.
10. Continue collection procedures to verify and recoup unpaid tolls. We also recommend FDOT review and evaluate the potential revenue loss to FDOT and act according to the chargeable failures terms and conditions of Contact BE087.

[MORE COVERAGE: Transportation secretary answers questions about I-4, SunPass in exclusive interview]

Thibault responded to the OIG’s report on Nov. 18. in a letter to the OIG saying FDOT is focused on providing “a reliable SunPass system to travelers in Florida” and committed to “doing everything it can to ensure issues with the system are thoroughly and timely addresses and we remain committed to holding the contractor responsible for its performance deficiencies.”

Thibault wrote FDOT is addressing issues in a “fully transparent manner.”

View the FDOT secretary’s full response to the Inspector General’s report below:

Conduent also responded to the Chief Inspector General’s report. It addressed the failures and detailed the improvements they made to address them.

“Inconsistent availability of the website and mobile app led to saturation of phone lines inconveniencing customers. Conduent immediately invested substantial resources to resolve all of these issues," Conduent Senior Director David Schnell wrote. "Today, the SunPass system is stable and currently operating at levels far exceeding contract requirements.”

Read Conduent’s response to the OIG findings below: