“A lot of people had a lot of opinions about my defense, which is fine; you can have your opinion. But a lot of people were basing it off these metrics… and no one ever watched you play. So that’s what kind of bothered me, was that everything was tied into this one number, and people hadn’t put eyes on you or seen it and just kind of labeled you.”In an interview posted on Metsblog a few days ago, Jason Bay provided this juicy little nugget about fielding metrics. While he does not outright claim that advanced defensive statistics are useless, he does echo the strong sentiment of many less-saber-inclined baseball fans: you cannot judge a player’s performance by his statistics. You have to watch him play and that will you tell you what you need to know. And this feeling holds doubly-true for defense. At least when batting there are countable events such as a home run or stolen base that make nice, easy-to-understand statistics, but with fielding that is not the case. Basically, the only numbers we have are fielding percentage, putouts and assists, and those numbers just do not tell us a whole lot. Thus, to judge a player’s fielding, you have to watch him play, and those fancy schmancy numbers like Ultimate Zone Rating (UZR) or Defensive Runs Saved (DRS) really mean nothing.Well, there’s another defensive metric, created by the immortal Tom Tango, called the Fans Scouting Report (FSR). Fans like you and me who regularly watch baseball games are asked to rate players on their favorite team in a variety of fielding categories: reaction/instincts, acceleration/first few steps, velocity/sprint speed, hands/catching, release/footwork, throwing strength, and throwing accuracy. Tango’s system then combines all the different scouting reports into one number with the player’s overall defensive rating.Tango writes this on the introduction page to the FSR:“And, most importantly, do not, absolutely do not, look at any numbers. Don't look at his fielding percentage, range factor, zone rating, UZR, or anything else that someone else is telling you. I just want you to rely on your eyes. You are the scout. I need you to rely completely on your own observations.We know the fans know what they are talking about. We know they know how to observe good and bad fielding. Now, I just want to know what the fans know.”So Jason, you want a defensive metric that does not rely on 64 separate “zones” on the field or run expectancy charts like UZR and DRS do, you got it. I decided to look at Jason’s FSR numbers over the past four seasons and see what the fans have to say about him, as well as how they compare to his UZR and DRS. First, the FSR numbers (50 is average, and the ratings are compared to the average fielder, not the average player at a certain position. FSR numbers from Tangotiger.net for 2007-08, and fangraphs.com for 09-10):FSRReaction/ InstinctsAccel/First StepsSpeedHandsRelease/ FootworkThrowing StrengthThrowing AccuracyOverall2007343449502721933200858565957574957562009313244444034423820104239454644334642It seems that even the people who watched Jason play rate Jason as a below average defensive player. The FSR ratings, just as UZR and DRS, do not tell the whole story, but they do provide a very different type of defensive statistic, the type that scouts and more traditional fans might be more receptive to. For reference, Bay’s score of 38 in 2009 rated as 3 runs below average.Now let’s take a look at his UZR and DRS numbers:DRSrARMrHRrPMTotal DRS200712-11-82008-32-7-8200930-5-2201010-10UZRArmRngRErrRUZR2007-1.7-16-1.2-192008-4.8-14.51.1-18.220091-1.12.1220100.6-2.60.1-1.9While DRS and FSR both have Jason being below average over the four seasons, UZR blows them out of the water in its Jason Bay negativity. According to UZR, Bay cost his teams nearly four wins due to his sub-par defense in 07-08. If Jason was referring to those UZR numbers, he would be correct in claiming that those stats are not indicative of his defensive talent. Whenever one of these metrics strays that far from the other two or career norms, you know you need to double-check your numbers, and it would be unfair to take merely one stat as gospel.A few interesting tidbits from the numbers:Bay massively improved in 09-10 according to both UZR and DRS, and he checked out as average. The fans, however, actually saw his skills decline these last two seasons. If people merely looked at UZR and DRS to gauge Bay’s defense the last two years, they would actually have found him to be a better defender than if they had watched him play.An interesting disagreement is how the three systems view Bay’s arm. UZR and DRS both have Bay’s arm right around average, whereas FSR sees Bay’s arm as notably below average (outside of that insanely flukish outlier performance in all respects in 2008). If UZR/DRS disagree with FSR on arm quality, I would tend to stick with the FSR, the fans’ subjective views. It is hard to judge range because positioning, immediate reactions, and route selection are often hard to observe while watching a game. But when a defender throws the ball, everyone in the stadium is focused on that player and that throw.And I am very curious as to what happened in 2008. FSR had Bay as about 3-5 runs above average, while UZR had him at -18.2 runs and DRS at -8 runs. There’s a one win to 2.5 win difference in those evaluations. In other words, if Bay was an average starter with his FSR, he would be below replacement level with UZR. That is just a massive difference over a whole season, and it’s discrepancies like this that keep people away from defensive numbers.But while it is right for us to take a defensive metric with a shaker of salt, we cannot ignore them completely. The fact is that none of these metrics rate Jason as a good defender, while they all have shown him at times to be below average, perhaps horrible, in the field. While I would not feel comfortable saying Jason is a -7.5 run defender over a whole season, or something equally as specific, I feel confident saying that he is somewhere in between bad to average, probably a little bit below-average range. At a certain point we can draw some conclusions, and I think with Jason we have enough data in rough agreement to make an assessment. And much of the data is in fact just our own personal opinions.That is the beauty of the Fans Scouting Report. It can provide a number of runs saved that can be incorporated into WAR (can’t you just feel Francesa squirming?), but the numbers are entirely subjective, and based purely on scouting. FSR can be a very useful tool, especially if you carefully consider when it would best be used. As far as Jason goes though, if people waited to watch him to judge his defense, he might actually be worse off.