Football, probably more than any other sport, is a game of opinions.
No matter whether you are a player, coach, scout, manager, chairman or supporter, there is every chance that you are going to interpret key incidents in a game or the performance of a certain player differently based on what you see with the naked eye, which is undoubtedly going to be influenced by pre-existing views and your previous experiences.
However, when you are working in the professional business of scouting and recruitment, assessing the performance of hundreds if not thousands of players every month, what steps can you, as a collective department, take to ensure that every stakeholder in the decision-making process applies the same level of consistency to their assessments, taking into account the club’s key requirements?
Given the significant financial investments that are necessary in the recruitment of a player, the one thing every club must ensure is that there aren’t any individuals involved in the process who apply a different value, whether positive or negative, to certain criteria in an individual assessment. If one scout is very quick to praise, whilst another is more reluctant to give a positive assessment, there is a danger that there will be inconsistencies in the interpretation of reports, which could affect a key decision.
In amongst all of this, the structure of the assessment template is absolutely crucial to ensure you get the desired outcome.
In a wide variety of business sectors, the recruitment candidate selection criteria can incorporate a blend of numerical-based quantitative assessments and subjective qualitative factors, such as a candidate’s performance at a job interview. Whilst the job interview may be less of a factor in football, assessment templates may still comprise a combination the quantitative, for example a numerical questionnaire which position-specific criteria, along with some kind of qualitative written assessment or verdict.
From a quantitative standpoint, it is far easier to make direct comparisons with score-based matrices across several different reports and identify discrepancies in the weighting of individual scores, however is it possible to do the same and identify discrepancies in the words and phrases which are used frequently by scouts when they submit their written assessments?
To try to find out, I have recently completed a project for OptaPro, which has comprised a text mining study of old scouting reports compiled by the company’s Consultancy operation. These reports were written several years ago and are now of little value, as all of the players assessed are no longer active in the professional game.
Some of the findings were highly significant and highlight the value of regularly checking the written testimonies entered into free text boxes in assessment templates.
In this Blog, I am going to highlight just a small number of the report findings, which I believe could be the first of its kind to be applied to professional football.
A General Summary of the Study
In total I analysed over 140 reports and each report was encoded so that a scout’s comments were broken down into five categories based on recurring words and phrases:
Findings
Positive vs Negative Mentions, by Category
The first definitive conclusion I found, following the text mining, was a higher proportion of positive mentions in reports, in all categories, compared to negative ones. On average, scouts commented positively three times as much as they did negatively, as highlighted by the table below.
| Coding Category | Positive Mentions | Negative Mentions | Positivity Ratio |
|---|---|---|---|
| Confidence & Composure | 35 | 10 | 78% |
| Creativity | 7 | 6 | 54% |
| Dynamism & Work Rate | 46 | 13 | 78% |
| Pace | 30 | 18 | 63% |
| Passing | 47 | 18 | 72% |
| Overall | 165 | 65 | 72% |
Positive vs Negative Mentions, by Scout
In terms of making direct comparisons by scout, we were able to unearth some highly informative findings, which suggested large discrepancies in the number of positive and negative comments made by each scout (note that all the names in the table are aliases).
What makes these findings even more significant is that we also found a direct link between the number of positive mentions and the final subjective ‘rating’ the scout gave a player (ranging from A-D, with A being high). We found that the more positive statements a scout made, the higher the rating he gave the player.
| Name | No of Reports | Positive Mentions | Negative Mentions | Positivity Ratio |
|---|---|---|---|---|
| Alex Cole | 23 | 21 | 0 | 100% |
| Chris Hoburn | 20 | 36 | 5 | 88% |
| Terry Dean | 12 | 14 | 4 | 78% |
| Jon Robinson | 8 | 6 | 2 | 75% |
| David Whitehouse | 9 | 10 | 6 | 63% |
| Chris Cunliffe | 12 | 14 | 9 | 61% |
| Alex Dempsey | 20 | 24 | 17 | 59% |
| Simon Timmis | 16 | 12 | 10 | 55% |
| Andrew Armstrong | 8 | 6 | 5 | 55% |
Category Mentions, Broken Down by Position
As well as comparing the positive with the negative, we also studied how often each category was mentioned based on the position of the player each scout was assessing. The results make it clear that the scouts mentioned certain criteria for some positions more than they did for others:
As you can see, the scouts referred to pace and passing for defenders and midfielders far more regularly than they did for forwards, whilst dynamism and work rate was mentioned more in midfielder assessments compared to defenders and forwards. Of course these findings may reflect the key attributes of these positions, however from a management perspective it may suggest that scouts need to consciously look into attributes that they have previously neglected in reports.
| Coding Category | Defender | Forward | Midfielder |
|---|---|---|---|
| Composure & Confidence | 33% | 16% | 33% |
| Creativity | 3% | 6% | 12% |
| Dynamism & Work Rate | 22% | 33% | 50% |
| Pace | 53% | 28% | 56% |
| Passing | 42% | 11% | 52% |
Scouting Terms Used, by Rating
In addition to categorising the various terms used by scouts, we have also analysed the number of times specific words appeared in reports and how they can varied based on the final A-D rating given by the scout.
Once again, you can see there is a marked increase in the number of times terms are used in reports where the scout awards a higher rating. Some terms, notably ‘either foot’ and ‘ability’ are hardly ever referred to in players awarded a ‘C’ grade or below.
Bars represent the % of documents using the term
Scouting Terms Used by Position
As part of the study, we were keen to see if certain terms were position specific. The following table highlights how frequently terms were used based on the position of the player being assessed.
One of the stand out conclusions was how terms relating to possession and movement dominated when scouts were assessing midfielders, whilst several other terms used regularly for defenders and forwards were omitted.
When assessing forwards, understandably the term ‘goal’ scored highly, however a large proportion of all-round terms were also used too. With defenders, key attributes specific to the position appear highly, notably ‘defend’, ‘position’, ‘aerial’, ‘pace’ and the possession based terms.
Bars represent the % of documents using the term
Conclusion
From studying just a very small sample of written reports, we were able to see fluctuations in scouting terminology being used, both in a positive and negative context, based on the position being assessed, the final rating and the author of the report. We also found differences in the word count from each scout and a correlation between the number of words used and the final rating being awarded, with higher grades having a bigger number of words.
Of course the outcomes of such an exercise can only be properly measured based on the understanding the very specific instructions that were given to the scouts. For example, we have highlighted how the number of positive mentions vastly outweighed the negative, but that may be a direct result of how the scouts have been briefed on what they have been told to report on. The same goes for the specific terms used by position – again the scout may have been told to report on attributes which are specific to position and that may reflect in the final outcomes.
That said, given the importance of good recruitment and the impact a report can have on decision making, there is clearly a value of regularly mining written assessments as part of a wider scouting audit, if only to confirm that each contributor is submitting reports to their brief.
Another key conclusion, if indirect, also highlights the importance of structuring your assessment templates so that you get your desired outcomes. This may relate to how you structure questionnaires, the choice of criteria, the ranges, the questions asked, the choice of answers available and the guidelines in place if you choose to include a written testimony. If you don’t get that right, it can make it very difficult to interpret the information that comes in.
—-
Dr. Garry Gelade is a Member of the American Psychological Association, and a Charter Member of the British Psychological Society.
He is the founder of Business Analytic, an independent consultancy specializing in organizational research, sports analytics and applied statistics. He is an Associate of the Centre for Research in Applied Creativity, Ontario, a past Visiting Researcher at City University (Cass) Business School, and the author of twenty-seven peer-reviewed scientific articles.
