Kaiku Health
Real-World Testing Results
2023 Performance Year
Download results or visit Kaiku Real-World Testing
General Information
Performance Year: 2023
Plan Report ID Number: For DRUMMOND use
Developer Name: Elekta Inc.
Product Name: Kaiku Health
Version Number: 7
CHPL Product Number: 15.04.04.1420.Kaik.07.00.1.211015
Developer RWT Page URL: https://www.elekta.com/products/oncology-informatics/kaiku-real-world-testing
Changes to Original Plan
| Summary of change | Reason | Impact |
|---|---|---|
| The baseline data collection period was extended from 1.5 months (mid-Jan to end Feb 2023) to 5.5 months (mid-Jan to end June 2023). | In the originally planned baseline data collection period, the amount of aggregated data was not sufficient to establish any meaningful baseline error rates. | Minimal impact; after this change both tracking periods were sufficiently long to establish meaningful results for all criteria and metrics. |
Summary of Testing Methods and Key Findings
Kaiku Health is a cloud-hosted health information system that is currently sold to Radiation Oncology and Medical Oncology care settings. Cloud hosting enables telemetry connectivity for all sold instances of the health information system. This provides remote event tracking capability which was be used for real world testing of the criterion which is in scope of these results. This ensures testing results accurately reflect the real-world performance of the health information system.
The tested criterion requires that patients can view, download and transmit their health information. Remote tracking therefore focused on patients’ attempts to view, download, and transmit, their health information using the health information system. Additionally, patients need to be able to view the activity log, showing how their health information was accessed for view, download and transmit. Real-world error rates were established for each of those functionalities.
Measuring real world use of the certified functionality by remote event tracking was a successful approach. It enables data collection across all instances of Kaiku Health and gives an accurate picture of how frequently which functionality is used.
The results show that view, download & transmit functionalities of patient records are used unevenly. The “View” functionality is by far the most frequently used functionality (57% of events in tracking period), followed by “View Activity Log” (29.6% of events in tracking period). “Download” accounted for 11.9% of events in the tracking period. “Transmit” functionalities are rarely used, with “unencrypted transmit” accounting for 1.4% of events in the tracking period, and “secure transmit” accounting for only 0.2% of events in the tracking period.
Observed error rates for View, Download and View Activity Log were very low. Observed error rates for Transmit functionalities were higher than expected, but on the basis of low or very low overall event numbers. Throughout the 6-month tracking period, 76 transmit (unencrypted) events were observed across all instances and users of Kaiku Health, out of which 12 reported an error (15.7% error rate). For transmit (secure), 9 events were observed out of which 6 reported an error (66.7% error rate). For reference, this is in comparison to 3110 viewing events throughout the tracking period.
Based on these observations, Elekta performed additional internal testing of both transmit functionalities on production-equivalent software. This testing showed that the software was working as intended, and health records could be successfully transmitted for both unencrypted and secure scenarios. Also the real world data shows evidence of successful transmissions for both scenarios.
In summary, the results demonstrate that Kaiku Health is used in the real world for functionalities certified under §170.315(e)(1): View, Download and Transmit to 3rd Party. The higher-than-expected error rates for the transmit scenario are likely at least partially an artifact due to low overall event numbers for the transmit scenarios.
Standard Updates
There have been no updates to certified standards for Kaiku Health in 2023.
Care Settings
Elekta markets its products in the Radiation Oncology and Medical Oncology care settings. The results reported in this document are aggregated across all instances of Kaiku Health in the United States. This includes both care settings.
Metrics and Outcomes
Criterion: §170.315(e)(1): View, Download and Transmit to 3rd Party
| Measurement/Metric | Error rate for View |
|---|---|
| Expected Outcome | Baseline error rate is expected to be low. Tracking error rate is expected to be comparable or lower than baseline error rate. |
| Outcomes | Baseline error rate: 0% errors out of 492 viewing events Tracking error rate: 0.25% errors out of 3110 viewing events Outcome is as expected. |
| Relied Upon Software | MOSAIQ 2.82 (or higher) |
| Measurement/Metric | Error rate for Download |
|---|---|
| Expected Outcome | Baseline error rate is expected to be low. Tracking error rate is expected to be comparable or lower than baseline error rate. |
| Outcomes | Baseline error rate: 0% errors out of 35 download events Tracking error rate: 0% errors out of 649 download events Outcome is as expected. |
| Relied Upon Software | MOSAIQ 2.82 (or higher) |
| Measurement/Metric | Error rate for Transmit (unencrypted email) |
|---|---|
| Expected Outcome | Baseline error rate is expected to be low. Tracking error rate is expected to be comparable or lower than baseline error rate. |
| Outcomes | Baseline error rate: 0% errors out of 6 transmit (unencrypted)
events Tracking error rate: 15.7% errors out of 76 transmit (unencrypted) events Tracking error rate is higher than expected. It must be noted that the overall sample size is relatively low. The data shows that the transmit (unencrypted) functionality is rarely used, since transmit (unencrypted) events account for only 1.4% of overall tracked events. Based on this observation, Elekta performed additional internal testing of the transmit (unencrypted) functionality on production-equivalent software. The functionality performed as intended. |
| Relied Upon Software | MOSAIQ 2.82 (or higher) |
| Measurement/Metric | Error rate for Transmit (secure) |
|---|---|
| Expected Outcome | Baseline error rate is expected to be low. Tracking error rate is expected to be comparable or lower than baseline error rate. |
| Outcomes | Baseline error rate: no transmit events recorded in baseline period,
no baseline error rate established Tracking error rate: 66.7% errors out of 9 transmit (secure) events Tracking error rate is higher than expected. It must be noted that the overall sample size is very low. The data shows that the transmit (secure) functionality is very rarely used, since transmit (secure) events account for only 0,2% of overall tracked events. Based on this observation, Elekta performed additional internal testing of the transmit (secure) functionality on production-equivalent software. The functionality performed as intended. |
| Relied Upon Software | MOSAIQ 2.82 (or higher) |
| Measurement/Metric | Error rate for View Activity Log |
|---|---|
| Expected Outcome | Baseline error rate is expected to be low. Tracking error rate is expected to be comparable or lower than baseline error rate. |
| Outcomes | Baseline error rate: 0% errors out of 306 activity log viewed events Tracking error rate: 0% errors out of 1614 activity log viewed events Outcome is as expected. |
| Relied Upon Software | MOSAIQ 2.82 (or higher) |
Key Milestones
| Milestone | Care Setting | Timeframe |
|---|---|---|
| Development and testing of remote event and error tracking capabilities completed | Radiation Oncology Medical Oncology | January 15, 2023 |
| Baseline error rate collection period | Radiation Oncology Medical Oncology | January 16 – June 30, 2023 |
| Error rate tracking period | Radiation Oncology Medical Oncology | July 1 – December 31, 2023 |
| Analysis and report creation | Radiation Oncology Medical Oncology | January 2024 |
| Submit Real World Testing report to ACB per instructions | Radiation Oncology Medical Oncology | January 22, 2024 |
Attestation
The Real World Testing Results above are complete with all required elements, including measures that address all certification criteria and care settings. All information included in these results areup to date and fully addresses the health IT developer’s Real World Testing requirements.
Authorized Representative Name: Martin Deppe
Authorized Representative Email: martin.deppe@elekta.com
Authorized Representative Phone: +358 40 661 7170
Date: 2024-01-22