OSafety Score model retrain
On July 31, 2024, RightShip updated its Safety Score to improve clarity, transparency, and accuracy. These enhancements were driven by continuous customer feedback and routine updates to ensure greater effectiveness.
Since its launch in February 2021, the RightShip Safety Score has become a vital tool for stakeholders across the supply chain, providing a clear snapshot of a vessel’s safety performance. It is intended to help our risk assessment customers gain an initial perspective on the operational performance of a potential vessel as part of a comprehensive due diligence process, while simultaneously encouraging shipowners to invest in improved processes and technologies that make the entire supply chain safer.
It’s important to understand that the dynamic nature of the Safety Score may lead to changes in a vessel’s score, especially during a retrain.
In our upcoming webinar we will delve into the motivations behind the retraining, the processes involved, and what these changes mean.
2021 Model Release: Calibration and Training of the Safety Score Model
Part One: Data Collection and Evaluation
Gathered Data: Collected data across six key aspects to evaluate vessels, using a dataset from 2016 to 2020.
Ranked Vessels: Ranked over 230,000 vessels in order from highest to lowest for each of the six datasets.
Applying a Normal Distribution Curve: Applied a normal distribution curve to the rankings:
Designated specific percentiles to represent each Safety Score value in the future.
Converted percentile levels into tangible data values (e.g., 10 PSC findings = median, 5 PSC findings = low, 15 PSC findings = high).
Assigned Weightings: Added a weighting value to each dataset to reflect its importance:
Incidents received a high-value weighting.
Detentions received a medium-value weighting.
Calculated Final Scores: For each vessel, calculated the combined weighted value of all six datasets to achieve a final score for each individual vessel.
Part Two: Final Ranking and Safety Score Assignment
Final Ranking: Ranked all vessels (230,000+) based on their combined weighted value from the six data sets.
Normal Distribution Application: Applied a normal distribution curve to the rankings.
Assigned vessel counts to determine how many vessels should fall into low, medium and high percentiles.
Set percentile thresholds to determine when a vessel’s Safety Score would move from Three to Four to Five.
Launched the model.
Once these parameters were established, the model began to react to new data entering the system. Each event impacting a vessel is recalculated, and as a vessel’s recalculated score crosses a percentile threshold, its Safety Score is adjusted accordingly.
Why Retraining the Safety Score Model is Essential
Over time, the distribution of vessels becomes misaligned with the parameters set by the original data set. This is expected as the underlying market conditions evolve.
As vessels experience fluctuations—whether improving or declining in various data aspects—their alignment with the original normal distribution may drift, skewing the results. This shift can occur as vessels either improve or deteriorate, while market demands fluctuate, affecting each data aspect differently. Example:
Incidents –The industry may experience a period with decreased incidents.
DOC –Industry performance improves and DOC Fleet performance rises
PSC – Inspection findings might increase as Port State Controls (PSCs) globally tighten regulations.
DETENTIONS – Detention levels might decrease as PSCs modify their rules.
Indicators for Safety Score Model Recalibration
As vessel distributions shift across each scoring area, the weighted sum scores also become skewed. This results in:
An increase or decrease in the number of vessels allocated to each specific Safety Score than originally intended. The longer the interval between recalibrations, the more vessels drift out of alignment with the designated percentiles
At this stage, it becomes essential to recalibrate the model to align with the most recent dataset. This process involves considering key questions, such as the current market conditions that vessels are facing across the six data sets and how these conditions compare to one another.
Retraining the model achieves several critical outcomes. It ensures the relevance and reliability of vessel rankings against current market data, maintains the proper distribution of vessels across Safety Scores, and provides an opportunity to improve and enhance the model for greater accuracy and effectiveness.
Analogy: Exam Grading System
To better understand the calibration process, consider a mock exam grading system. Initially, the grading is set as follows: a score of 70% earns a C, 80% earns a B, and 90% earns an A. However, only 30% of the test-takers can achieve an A.
If all test-takers score 90% or higher, the system needs recalibration. The new calibration settings would be: 90% earns a C, 95% earns a B, and 98% earns an A. This adjustment ensures that only 30% of the test-takers achieve an A, maintaining the intended distribution of grades.
Similarly, in the Safety Score Model, the normal distribution curve and percentile thresholds are applied to ensure that the distribution of Safety Scores remains consistent and meaningful, even as the data evolves.
Update on the July 2024 Retrain
In our continuous effort to enhance the accuracy and relevance of our safety assessments, we have undertaken a comprehensive retraining of our mathematical model. This retraining involved incorporating the most recent historical data from 2019 onwards.
The primary objective of this update is to ensure that our safety scores accurately reflect the latest safety performance records, regulatory changes, and operational trends.
To ensure the scores remain meaningful and comparative, it's crucial to retrain the model periodically. Retraining recalibrates the scores, maintaining a balanced and accurate reflection of current safety performance across the fleet.
Key highlights of the Safety Score model retraining
Updated Historical Data: The model now includes data from 2019 onwards, reflecting the latest safety performance and operational trends. This update allows us to offer more accurate and relevant vessel safety scores.
Refined Scores Scale: The Safety Score and its subscores, provided on a scale of 1 to 5, have been recalibrated based on the updated data. This recalibration ensures that the scores accurately represent the current safety status of each vessel and its affiliated entities, offering a clear and precise measure of their safety performance.
Enhanced Reliability: The retraining process has strengthened the reliability of our safety scores. With the inclusion of the latest data, our model offers a more comprehensive view of vessel safety, ensuring that our clients receive the most dependable assessments.
Changes due to retraining:
Most of the changes in safety scores are attributable to the retraining process. Even if there are no major changes in the methodology, the scores still change as the thresholds and boundaries are adjusted to accommodate the latest data. Vessels may see fluctuations in their Safety Score depending on where they find themselves in relation to other vessels.
Since all scores are benchmarked against each other, the performance of vessels naturally varies over time, with some improving and others declining. This variability causes shifts in the score distribution.
Other Considerations:
How a vessel relates to other vessels in the fleet is central to the RightShip Safety Score model. Hybrid rules override relative rankings of vessels to designate a result for a specific aspect that is considered important, but not addressed by the model.