The reliability metrics for our performance dashboard are automatically generated, but we are always keeping an eye on them to make sure they make sense. We noticed a large drop in our February Green Line reliability data starting on February 8, 2017 that did not reflect the actual reliability observed for the Green Line. After reviewing the data sources, we discovered some technical issues that caused this artificial drop and will be, at least temporarily, removing the affected metrics from our dashboard.
Background
There are two main software systems involved in the production of the subway reliability metrics shown on the Back on Track Dashboard: the real-time feeds, which produce the vehicle location and arrival prediction data used by app developers and on mbta.com; and the performance tracking system, which uses the real-time feeds as an input and calculates performance metrics. The other modes rely on different systems for performance measurement.
The subway performance-tracking system is built around the concept of an event (arrival or departure). An event is recorded when a train is observed at a stop in the vehicle location real-time feed. Once we have all of the events for one day, we calculate the metrics that are shown on the dashboard for each subway line and aggregate the totals. For more on these metrics, see our post Explaining Dashboard Metrics: Subway Reliability.
The real-time feeds are produced by a data fusion engine that combines vehicle tracking and arrival prediction data from different source systems to produce one consolidated feed in the GTFS-realtime format. There are multiple vehicle tracking systems for each of the MBTA modes. For more on how the MBTA tracks its vehicles see our post How the MBTA Tracks Vehicles.
We use the data fusion engine to combine the data from each of these systems into one consolidated real-time feed to make it easier for our developers to work with our data. The data fusion engine’s processes typically take a few seconds to combine data from all of the source systems.
If vehicle location data is coming into the data fusion engine faster than it can process it, the real-time feed may miss some stop events that happen during the processing time. However, we typically only miss a few events a day out of tens of thousands of stop events. A missing event would make it look like the time between trains is artificially long and negatively affect our reliability metrics. There are processes in place for the Red, Orange, and Blue lines to identify missing events and remove the incorrect wait times. Due to limitations of the source data for the Green Line, we don’t currently have a reliable way to determine if events were missed and correct for those occurrences.
The Problem
Beginning February 8, 2017, the data fusion engine starting slowing down enough to cause delays in the provision of the real-time data feeds that resulted in our system missing events on a more frequent basis. As of March 3, 2017 the issue has been resolved and the system is once again performing as desired. The root cause for the slowdown is still under investigation.
The delays were short enough during the slowdown that real-time information was still available at all times throughout the day. For customer information systems, updates every 10 or 30 seconds are sufficient to provide vehicle locations and predictions on a constant basis. The performance tracking system, however, needs near constant location updates to ensure that we capture all stop events, and so these outages significantly reduced the number of events we captured for all subway modes.
The processes in place for the Red, Orange, and Blue lines correctly identified the missing events and removed the incorrect wait times for those lines. However, due to the Green Line source data limitations, we were not able to differentiate between long wait times that were real from those that were artificially caused by missing events and correct for those occurrences. This degraded the accuracy of the metrics shown for the Green Line. For these reasons, the data between February 8 and March 2, 2017 for the Green Line will be removed from the reliability metrics on the dashboard. We are attempting to reproduce the stop event data from the raw source systems to try to recreate the metrics for these dates, and will provide an update in a separate post in the near future if that is possible.