The Post Dives Into the Refs’ Collective Bargaining Agreement Part 2
Grayson
Diving Into the Refs’ Collective Bargaining Agreement Part 1
Evaluating Referees
As promised, The Post is back with part two of its review of the collective bargaining agreement between MLS referees (represented by the Professional Soccer Referees Association, or the PSRA) and Professional Referee Organization, Inc. (“PRO”), the company that employs those referees. This installment touches on sections of the agreement relating to performance standards and evaluation, release from employment for substandard performance, and part of how referees are monitored during games.
Performance Standard and Evaluation System
Article VII of the agreement sets out the “Performance Standard and Evaluation System,” which the agreement describes as “intended to serve as a platform for the continued training and development of Officials, as well as a vehicle for evaluating Officials.” That Article describes a Joint Performance and Evaluation Committee (“Joint Committee”) that contains no more than three members from each of PRO and the PSRA. The Joint Committee is “charged with establishing any modifications to [the Performance Standard and Evaluation System] prior to the commencement of each year of this Agreement.” If the Joint Committee does not reach agreement on modifications, then no modifications may be made, except those based on changes to the Laws of the Game put out by the International Football Association Board (“IFAB”).
What that long-winded explanation means is that the Performance Standard and Evaluation System described here (as well as other information below) is what was in place when the agreement was adopted in 2019. It is possible that the Joint Committee has tweaked the system somewhat since then, but it seems likely that the framework is basically the same.
Match Day Records and Feedback
For each MLS Regular Season Match, four kinds of assessment records are created: (1) a record of Key Match Incidents (“KMIs”) for each Referee or Assistant Referee (the “KMI Record”); (2) a Match Day VAR Record, listing all interventions or non-interventions applicable to the Video Assistant Referee (“VAR”); (3) Match Day General Performance Feedback, for Referees and Assistant Referees; and (4) Match-Day General Performance Metric, KMI Metric, and freeform feedback for fourth officials, for development and training purposes only.
These records are prepared by a pool of “assessors.” The agreement provides for two pools of assessors, a 17-member “core assessor pool” and a 13-member “alternate assessor pool.” PRO selects the assessors for each pool, but the PSRA has the right to raise concerns about particular assessors at the end of each Season. Members of the core assessor pool are offered assignments first, and if necessary assignments will be offered to the alternative pool. The agreement limits the alternate assessor pool to performing no more than 20 percent of assessment assignments.
Assessments are prepared for each match week. A match assessor can request a post-match conference call or meeting with the Official being assessed, and the agreement says that a Referee or Assistant Referee who does not attend the post-match call loses his or her right to appeal the KMI Record or General Performance Metric. The agreement entitles Officials to receive their assessments no later than 5:00 p.m. (Eastern) on the Friday of the week following the week of the match, except assessments for Sunday matches are due by 5:00 p.m. Eastern that same Friday. (So assessments for matches occurring from June 12 to June 18 will be due under the agreement no later than Friday, June 23.)
KMI Records are also reviewed by a KMI Review Panel, which is composed of the PRO Director of Senior Match Officials, the PRO Manager of Education, Coaching and Evaluation, and the PRO Manager of Video Review Operations. If any of those three individuals is unavailable for any reason, the agreement sets forth a priority list for which PRO employees are next in line to fill in.
KMI Records
Preparing the KMI Record starts, obviously, with identifying what exactly KMIs are. The agreement defines them differently depending on whether the Official is a Referee, Assistant Referee, or Fourth Official:
*OGSO means “Obvious Goal Scoring Opportunity”
In the KMI Record, each KMI is given an action, accuracy, and difficulty rating, and the Official receives a score as to whether he or she was correct or incorrect as to the call on that KMI. The agreement includes the following chart showing how these scores are applied:
Referees or Assistant Referees with a rolling average KMI Metric or -4 or worse after at least five MLS Regular Season Matches in a Season may, at PRO’s option, cease being assigned to matches.
General Performance Metric
Referees and Assistant Referees are also assessed based on a General Performance Metric. Fourth officials receive a General Performance Metric assessment, but only for training and development purposes.
The General Performance Metric includes a series of weighted categories (meaning some factor into the Official’s evaluation more than others), assessed on the following scale:
1 – Significant areas for improvement
2 – Minor areas for improvement
3 – Good
4 – Best Practice
For Referees, the General Performance Metrics described in the agreement are:
Assistant Referees have their own criteria:
Finally, fourth officials receive training assessments on two categories:
Under the agreement, PRO has the right to cease assigning Referees and Assistant Referees who have worked at least five MLS Regular Season Matches in a season and have a rolling weighted average General Performance Metric of less than 2.
Match Day VAR Record
The agreement specifies that VARs are also assessed using KMIs, as follows:
+1 point if
VAR correctly does not recommend a review to the Referee and the KMI Review Panel determines the Official’s decision was incorrect
VAR correctly recommends a review to the referee
0 points if
VAR correctly does not recommend a review to the Referee and the KMI Review Panel determines the Official’s decision was correct
Inconclusive
Video Review technology failure
Video Review Operator failure
-2 points if
VAR incorrectly recommends a review to the referee
VAR incorrectly does not recommend a review to the referee
AVARs are evaluated by the PRO Manager of Video Review Operations or someone designated by that person. The agreement says that AVARs are assessed based on whether they provided appropriate involvement and assistance within the Video Review Protocols, followed AVAR protocols, and correctly interpreted and applied the IFAB Laws of the Game. The AVAR is then rated as “acceptable,” “unacceptable,” or “grossly unacceptable.” AVARs can be released by PRO if they receive two grossly unacceptable performance ratings in one Season.
Ranking and Retention
In addition to the Match assessments, referees are rated relative to each other. The agreement requires PRO to provide midyear written evaluation summaries (no later than July 15) and end-of-season written evaluation summaries (no later than December 15) to each Official. These summaries include each Official’s decile ranking on the relative ranking tables.
The ranking tables are based on the Officials’ Regular Season Match scores, but they include only those Officials with a minimum of five MLS Regular Season Matches officiated in a category during that Season. In addition, Official rankings do not include assessments they received before they joined the PSRA bargaining unit. Referees and Assistant Referees are given two separate rankings – one for the KMIs and one for the General Performance Metric. VARs are ranked in only one table, based on the Match Day VAR Record.
It would certainly be interesting to see those performance rankings. Unfortunately for those curious, the agreement requires that the rankings be kept confidential, except for two authorized disclosures: (1) “to official PSRA representatives solely for the purposes of dispute resolution, collective bargaining and/or in the context of a release of an Official from employment,” and (2) in response to requests from governing bodies, such as the Canadian Soccer Association or the United States Soccer Federation, in relation to assessments, such as for certification or decisions relating to international referee panels.
Referees who do poorly on the end-of-season rankings can be released under the agreement if they fall below certain standards:
Probationary Referees may be released for substandard performance if they rank below the 40th percentile on both rankings at the end of their first full category year of service, or if they rank below the 40th percentile in either ranking in two consecutive seasons;
Probationary Assistant Referees may be released if they rank below the 20th percentile on both rankings at the end of their first full category year of service, or if they rank below the 50th percentile of either ranking at the end of two consecutive seasons;
Probationary VARs may be released if they rank below the 40th percentile of the ranking at the end of any Season;
Non-Probationary Referees may be released if they rank below the 25th percentile of either ranking at the end of the season, or if they rank below the 25th percentile of either ranking at the end of two consecutive seasons;
Non-Probationary Assistant Referees may be released if they rank below the 15th percentile of both rankings at the end of any season, or if they rank below the 15th percentile of either ranking at the end of two consecutive seasons;
Non-Probationary VARs may be released if they rank below the 25th percentile at the end of any Season.
Officials have the right to appeal releases for substandard performance. They also have the right to appeal their KMI Records and General Performance Metrics.
Monitoring Officials During Matches
The agreement contains detailed procedures for officials on match days, which will be discussed in a future post. However, most relevant to performance evaluation is that Officials’ on-field communications may be recorded by PRO, and the recordings can be used to clarify on-field decisions, develop standard phrases, protect Officials in disciplinary proceedings (if, say, misconduct is alleged by a player or coach), reduce the amount of on-field dissent, and assess and evaluate Officials in limited ways. The agreement says that the recordings can be used to discipline an Official only for misconduct that has come to light through other means or if PRO learns of serious misconduct (such as racial epithets or admission to wagering on games) while listening to the recordings for purposes unrelated to discipline.
Video and audio recordings are made of the VAR work area, but notably the agreement says that only video of the work area may be used during live broadcasts, but not audio. This is likely one reason why the VAR discussions are not aired during the game broadcast. The agreement does allow PRO to use the audio and video recordings “for purposes of educating the public, broadcasters and/or other key stakeholders, provided that PRO shall not use audio and/or video recordings to portray an Official in a negative light.” This carveout would apply, for example, to the Inside Video Review segment that PRO publishes on YouTube every week during the season.
Earlier this year, IFAB announced a pilot program for broadcasting VAR decisions live, both in-stadium and on broadcast. The U-20 World Cup is currently being used as part of this pilot. If IFAB approves a permanent change, it will be interesting to see how that might be implemented in MLS, given the language in the agreement (and the fact that, unless the agreement has been extended – which has not been announced publicly, as far as The Post can determine – it will be expiring early next year and up for renegotiation).
What’s Next?
The next story will discuss what the agreement says about Officials’ training requirements, including annual and periodic courses, physical fitness requirements, and other required education required by Officials to keep their employment with PRO, as well as other optional resources that might be offered.