|
|
In late 2020 and then in early 2024, we conducted pilots that assessed:
Highlights of the results of items 1-4 are included below under “Early Pilots Show Promising Results”.
In early 2024, we conducted an in-depth proof of concept with a large government organization that included the development of 3 profiles to capture their risk interest areas as well as assign specific impact weightings for each of the risk measures and then evaluating one of the profiles for 4 companies of interest.
With this effort we explored the different ways of sharing and discussing the risks collected in the System of Trust Body of Knowledge to drive the selection of the risk factors and risk measures for the profiles and for reviewing those selections with a broader stakeholder group within the government organization as well as how to provide an explanation of the risk factors and measures used in the assessment and shown in the assessment results.
For sharing and discussing the risks to select the profiles we found a tabular text version (using word) of the document (shown in Figure 1 below) most useful as a read-ahead and leave-behind since it had a built-in capability to invoke track changes and allow the people to take things out of scope as well as make comments about them but it was not as useful for collaborative discussions or quick review meetings.
Figure 1 – Tabular Text version of System of Trust content.
For getting a quick overview of Risk Factors and their component Risk Measures we found a spreadsheet (as shown below in Figure 2) to be very effective and was also good supporting a collaborative meeting where the items could be discussed. The spreadsheet is not as effective as a read-ahead or leave-behind and is difficult for conveying context and descriptive detail.
Figure 2 – Spreadsheet version of System of Trust content.
Another way of communicating System of Trust content is through tables, shown below in Figure 3. These have been effective for providing an at-a-glance listing of the Risk Factors being used in an assessment profile, along with their definitions. These are good for inclusion in reports but not an effective way of showing the hierarchical relationships between Risk Categories and Risk Measures.
Figure 3 – Table version of System of Trust Risk Factor content
Once a team has culled through the overall System of Trust Body of Knowledge and created their profile or profiles of interest that reflect their effort’s needs, interests, risk appetite, and available resources they will need to be able to convey the profile(s) to others in presentations, reports, and discussions.
We found that, for profiles with more than a dozen Risk Factors this can be a difficult task with any of the above mechanisms (tabular text, spreadsheet, or table) and developed a hierarchical heatmap illustration approach (shown below) that can handle up to 70-80 Risk Factors. The heatmap can be read left-to-right with the highest level of abstraction/categorization on the left and increasing levels of deeper abstraction/categorization as you move to the right.
Figure 4 – Hierarchical Heatmap with 72 Risk Factors
The above depiction, showing a large profile, can be refined to show a sub-set / sub-profile where that may be the approach a group decides to use in their assessment approach. Figure 5 below shows an example of doing this for a sub-profile of the full set of the above Risk Factors in the Figure 4 set.
Figure 5 – Hierarchical Heatmap of Profile/Sub-Profile
The above depiction can be adorned with the assessment results for each of the Risk Factors as well as showing how those assessment results are bubbled up to summary assessments for Risk Categories that are in-scope for the profile being assessed. Figure 6 below shows an example of doing this for the Figure 5 sub-profile set of Risk Factors and Risk Categories. Note the key to the figure introduces the set of risk range depictions used in Figure 6.
Figure 6 – Hierarchical Scoring Heatmap of Sub-Profile
One of the challenges to assessing against a profile is finding appropriate sources for the data needed to evaluate the Risk Measures within the scope of the profile. Figure 7 below shows an assessment of the primary data source we used in our proof-of-concept assessment using Harvey Balls to indicate whether a Risk Factor’s Risk Measures were able to be evaluated with the data from the data source for the four companies we assessed.
Figure 7 – Depicting Data Source Coverage
The preliminary results for Pilot 1 are illustrated below in Figure 8, an unweighted bar chart depicting the overall risk scores for the 11 companies reviewed in the pilot, and in Figure 9, which presents radar plots of five data-driven scores from the supplier risk categories (leveraging 52 questions in those areas) for three of the 11 companies of interest.
Figure 10 offers a deeper look into the risk scores that generated the radar plot for one of the companies involved in the pilot (Company 10).
All of the pilots use data sources that the SoT leveraged to generate the analytical assessments, which clearly show a larger risk profile for Company 10 compared to the others. This pilot provided a proof of concept that offers early evidence of this tool’s utility, with deeper and broader analysis to follow as the SoT is completed. The other three pilots had similar insights.
Figure 8. Risk scorecard based on the preliminary System of Trust scoring methodology for 5 top-level categories and 52 risk measure questions for 26 risk factors
Figure 9. Radar plots of 5 data-driven scores for 3 of the 11 companies reviewed in Pilot 1
Figure 10. Specific risk scores for one company involved in Pilot 1, in the form of a radar plot
In the next phase of the SoT effort, we will use the full array of data sources envisioned and tailor weighting and score contributions to fine-tune the emphasis on specific sub-risk areas used in any given assessment. Although the pilots only used a subset of the public, private, and restricted access data sources the SoT is anticipated to leverage, we are cataloging and capturing the numerous sources of potential utility in conducting such analyses.
