Pitfall-Proof Your Claude Dashboards

Ben Jones: Hello, hello, hello. Welcome to the Powered by Data podcast. I'm Ben Jones, your host. Well, I hope your fall is going well. Back to school time, flu time. Just getting over it myself. But I thought I would just tune in real quick here to give you a quick update on something I've been working on. So, you know, many of you who have been connected to me, following me for a while, know that my second book, Avoiding Data Pitfalls, came out quite a while ago now, 2019.

So cutting to the chase, I wanted to create a version of this book that you all could access and use where you're using data these days, which of course maybe happens to be Claude. I know I'm exploring a lot. Claude just released their dashboards artifact earlier this week. And so we've all been trying to see what Claude can do with data lately, right? So I thought, well, you know, I wrote this book half a decade ago, more than that now.

In fact, I was working on it 10 years ago, and I was trying to help human beings that are working with data avoid some of the pitfalls that I was familiar with and that I had been making. But the thought occurred to me, well, you know, AI is going to make plenty of mistakes with data as it analyzes it and visualizes it. So, you know, it's not perfect. Neither are we. So why not create a version of the content and all the pitfalls and the types in the book, to be able to let Claude evaluate itself, see whether or not it's falling into pitfalls, and if so, maybe to correct some of those?

So let me kind of show you what I've got going on. So basically I created, you know, a Claude connector, an MCP server called Data Pitfalls, that allows anyone to directly connect their Claude Code or Claude chat session to the server we're running, so they can compare their analysis with all the pitfalls we've identified, you know, over 70 of them now, up to 78, I think. We keep adding more and more beyond just what's in the book as we see more examples out there. Okay, so that's the idea.

So let me sort of show you how you could do that. Okay, so I'm going to share my screen with you right now. All right, there we go. So the data set I want to use here is from the US Department of Energy. This is basically showing, you know, vehicle registration counts by fuel type. So it's kind of neat. You get to see, hey, you know, is one state using more hybrids or electric vehicles, or what's kind of the more common, and is it growing? We can look at previous years and such.

Okay. So I went ahead and I created here in Claude Artifacts... Okay, so how did I get there? I went to Claude and I went to Artifacts and I went to Dashboard. Okay, that's what I did there. And then since I have the Data Pitfalls connector already connected to my Claude account, I can actually just ask Claude to take a look at it and compare what it did. So how do I get that? Well, in the settings in the bottom left corner, you would just go down here to your connectors, okay. And you could just type in here "data pitfalls." And there it is. You'll find it and you can connect to it.

It doesn't actually look at any of your data at all. So, you know, you're not sending your data anywhere. It's just really allowing the given chat session or Claude Code session to access the taxonomy and the rules and the descriptions and some instructions so that it can kind of evaluate what it's looking at and how it's doing. Okay, so that's how I pulled Data Pitfalls into Claude right there, just using this community connector that we created and published.

So now that that's all in there, I went to Claude and the dashboards, and I actually added all of this data from the US Department of Energy. Now I started off by just asking it to make me a dashboard. And I gave a nice little description here to try to get a decent dashboard. I think that's what everyone's trying to figure out, right? Is how do I make Claude dashboards create something great?

I think it created something pretty good. We've got some filters across the top here and some ways to swap between different measures. I can take a look at different fuel types and see maybe how those are changing, you know, over time. Looking at that by region, perhaps. Okay. I think I can just reset this here, just look at the entire United States, look at a regional view. We can see by states in this nice tile map here to see which one's using more hybrid electric vehicles.

Okay, we've got these little sparklines down here to see different vehicle types. Maybe I'm looking for electric or plug-in hybrid electric. So, you know, overall I think a pretty well-designed dashboard, all right? Just for me to see kind of how it's been changing over time.

Well, then you can see what I did is I said, okay, this is nice, but can you run this through the Data Pitfalls MCP connector? What pitfalls might this dashboard be falling into? And how could we change it to avoid them? Okay, so then, because I have the connector in there, of course it goes and checks it out, and it says it's gone through all 78 rules before writing findings. "Let me check a few data patterns I suspect matter." Okay. It's going to look at some interesting jumps it's seeing and some tiny percent changes it's noticing.

And then it came back, right? Over here, and it said, "Overall: serious pitfalls found." Okay, so the default view, electric vehicles, holds up. The biggest problem is that the 2025 hybrid numbers look like a change in how the Alternative Fuels Data Center classifies vehicles. So it looks like there's maybe a data definition change or a classification change, perhaps. And it's saying, hey, you know, the dashboard's presenting it as real growth and there's no kind of caveat or warning. So this is really what happened: it's noticing that the 2025 hybrid numbers all seem to jump quite a bit.

And so it thinks it might be, you know, falling into this pitfall called measurement system drift, which comes from our chapter on epistemic errors. That's the first chapter in the book. Okay. So then it's taking a look at what evidence it sees for that. You know, it sees this big jump. It also noticed that in the same year, gasoline registrations kind of went down by a similar amount. And it looks like it's thinking it possibly could just be a change in the counting, as opposed to a surge in adoption.

Okay. So it doesn't know for sure, but it wants to mark a break there and to put a note in there. It also recommends asking the data source owner whether its methods changed. So actually I did that. So that's actually kind of good advice, you know. Data journalists always go back to the source and check that. So that's useful. Okay. That's one thing it noticed.

It's also noticing that there's a break, but the summary sentence at the top up here, you can see this one right here, is comparing the year to 2016. But there was actually a break in a specific vehicle type, E85, and it still goes back to the year before that, the first year in the data set, 2016, and everything else as well. So it's going to suggest maybe instead switching to comparing the value here to 2021 instead of 2016, and adding the comment there about when AFDC's method changed. Okay. So it's going to recommend that. That's maybe a caveat.

Okay. This one's actually kind of interesting. The third one it's noticing is that percent changes in the dashboard are calculated on tiny rounded counts. You see that here? So basically, you know, we've got this small baseline of comparison, and we've got a one-year change happening there. Okay. And then it's noticing that for compressed natural gas, CNG, there are some large percent changes for Tennessee and also a large percent decrease for DC, but it's just because it goes from 100 to 200, right? Could this be a rounding error? It goes from 100 to zero in the case of DC. So these are small, small numbers here, and we're giving large percent changes on top of tiny, tiny numbers.

So if you want to see what I mean, if you go down here to this table, well, let's switch from hybrid electric vehicles to compressed natural gas. And let's come down here and take a look and sort by one-year change. And you see right away that, you know, Tennessee has this huge change, but it's only because there's a hundred, you know, 200 vehicles. We go down a little lower, and you can see that we've got a negative 100% for DC. Went from 100 to zero, I guess, that year, right?

So anyway, the point is those are very small counts. And not just small, but chunky, because they have to fit into the nearest 100. So reporting these huge percent changes on top of these tiny deltas, or maybe uncertain deltas given the rounding, is a little misleading. So it's going to suggest that we hide the percent change when the count is under about a thousand, and even show something like "too few to compare" there. So that's probably a useful change. Sometimes these small values lead to these crazy percent changes that stick out like a sore thumb, and that can be a bit misleading in a dashboard.

All right, number four, it keeps going. This one's a big one for me, a pet peeve I talk about a lot in the book. It notices that, hey, one color means three different things in the same dashboard. So what is it talking about? Well, take a look here at these line charts, right? On the left, we've got the US here and it's blue. Okay. But on the right-hand side, right next to it, we have another line chart with four colors, including the exact same blue, but on the right-hand side, the blue is for the Northeast. So the same color means totally different things in adjacent charts, which could be visually confusing.

It's even more confusing when, let's say, you change from the United States to, say, California. Right. And now California is blue on the left-hand side, even though on the right-hand side, West is orange. So that's confusing. All right. So that's what it notices. And then it's going to say, well, let's gray out the regions in the regions chart. And maybe I could even say, hey, why don't we just change this one to gray, or have the state color be represented by the region color on the right? That kind of thing. So a way to make those colors more consistent, maybe.

In these same two charts, in the fifth pitfall it noticed, it even says, hey, here's a problem that maybe is one to be aware of. And that is that on the left-hand side, this y-axis is calculated on its own compared to the one on the right. So on the left, you can see it goes up to 0.25%, whereas on the right, it goes to 0.35%. In other words, you know, these y-axes are different, even though our eyes can look to the side and compare the same line left to right. But they're not lined up, and even the baselines aren't aligned. So I might want to put some extra comments in here, perhaps, to make sure that these baselines are aligned. You can even see the headlines in the charts have different numbers of lines. So in other words, I might give it a comment here to say, hey, make sure the axis lines up from baseline to the top, not just in the values themselves.

And then I like this one too, number six. It notices the dashboard describes registrations as vehicles. That's what we call the data-reality gap. So it's mistaking these registration counts for actual vehicles on the road, right? The summary says "electric vehicles." But actually, this is just an estimate of registrations. It isn't even registrations from, like, the Department of Licensing or DMV or whatever. It isn't even that specific. It's actually derived from Experian data in the case of this source. If you look closely at the data itself, where it comes from, you can see it's coming from Experian. All right. So in other words, that's not vehicles necessarily. It isn't even registrations. It's Experian data that's based on registration estimates. Okay. So it's going to say, hey, let's change the wording here so that it doesn't mislead people into thinking that's exactly how many cars were counted on the road. So some of those little details can be important.

It even notices that there's no color legend. That's the seventh one. All right. So then, I love this part. It actually does tell you which pitfalls you avoided. So it isn't only giving you the bad news, it gives you the good news too. So it notices that, you know, we avoided the pitfall of not weighting the averages. We avoided the pitfall of percentage points versus percent change. Okay. So anyway, it's saying, hey, kudos for at least avoiding some of those.

So then it says, hey, do you want me to apply all seven fixes? I'll say sure. "Apply all seven fixes." And I'll say, "For the y-axis comparison, make sure the axes fully align from left to right," and I say that because the titles of the charts take up different spaces, and the horizontal axis baseline isn't at the same height in the boxes, right? So we want these perfectly lined up.

Now, would I say that these seven recommendations are describing what is, like, a totally broken dashboard? In this case, I would say no. I would say these are really fine-tuning adjustments to make to go from good to great. And a lot of times that's what you want. A lot of times that's the goal. And a lot of times, you know, that's the difference between a dashboard that works and is adopted versus one that gets people confused, and they just maybe, who knows, stop using it.

But I love the fact that, you know, Claude here is able to check itself and its own work, not just against something random, but really against our known and specific set of pitfalls we've been compiling over the years, one at a time, as we've noticed all of these mistakes out there in the wild, and even ones I've made myself. Compiling all of that has been a labor of love. And anyway, there you have it, right? So we'll let it make its fixes.

While it's doing that, I'll show you a few other things you can do. Well, you know, this is out there for free also as a command line tool, a CLI. So we have released an npm package. Meaning if you are a terminal lover, you can go ahead and use the npm package here and install it. It's called datapitfalls, and you can scan an image, let's say, something like that. That's one way to do it.

We also have on the website itself, dataliteracy.com, this new area called the Lab, which I'm having so much fun working with, right? But here it is. This one is free, wide open for everyone. It's a SQL scratch pad, so you can write a query against your own little data table you upload there. It stays in your browser. But we also have now, for our Pro account holders, this Pitfall Detector page. So when you come to this page, it actually lets you upload charts or slides or even Python scripts or a full report.

So for example, let's take a chart, maybe an old-school one here. Let's just grab this one. I'm just going to take a screenshot of good old Andrew Yang. Remember him? We'll take this screenshot. You can see he had this bar chart trying to show his polling lead there. Okay, well, all politicians maybe are guilty of these chart sins, but let's drop that right here into our little Pitfall Detector in the browser. Okay. And we can say check for it.

So now it's going to go ahead and actually run a full scan of this single chart against all of our 78 pitfalls. It's going to come back right here and say, hey, you know, we've got some disproportionate encoding on there. The bar lengths don't fit one linear scale from zero. Okay. Yang's 22.5% is about one and a half times Bernie's 21% in length, and so forth, right? So it caught the issue there. And maybe even beyond just that kind of chart faux pas, it goes further and notices that it's missing some context, and could provide some additional details, maybe about the poll itself. All right. So that's another place you can use this same exact pitfall detector tool we've created.

All right. So if we go back now, well, it's still working on it, so we'll let it crunch away. But anyway, there you go. I hope you all use it and like it and enjoy it, whether you're using the command line app, whether you're pulling Data Pitfalls right into Claude itself and working with it that way, or whether you're a user of our site and leveraging this tool we're giving you directly on our platform. Hey, any way you check for pitfalls, that's a good way in my book, literally.

So anyway, hopefully you all enjoy it. Everyone, y'all take care. Well, we're coming up on the holidays here real soon, but I'll have some more exciting episodes coming up soon. I'll get a chance to talk to some people doing some amazing things in the data world. So looking forward to being able to put those episodes out there soon. I'm just in the process of lining up some of those interviews right now to get other people here on the show. All right. So you all take care. We'll talk to you soon. Bye now.