Data Shame, Big Data, and the Need to Tell a Story

Final edits are just going in on a Database Foundation Plan for the Oregon Department of Land Conservation and Development (DLCD). We are tentatively emerging from the thickets of statutory law, plan amendment procedures, urban growth boundary expansions, appeals, hearings…. Before burrowing into the next project and it’s challenges, it’s refreshing to take a moment above ground and reflect on our work.
If you go with the marketing, ours is the age of ‘big data,’ and discovering patterns, trends, and critical relationships in behemoth and organization-straddling data sets is our grail.  In this context, I find that ‘data shame’ is a rampant if little acknowledged condition that extends into the recesses of many companies and agencies.  I don’t have statistics to share, but I know that this vexing malady is experienced in varying degrees by many people who must rely upon particular data to inform decision making and to direct actions that have real consequences.  Expectations are high, hype is unchecked, and many people endure the daily angst of working with data that doesn’t measure up to their needs.

In projects where our firm has been brought in to help people lasso their data, we have come to expect the moment of pause, then followed by apologies and looks of mild embarrassment, just before our clients reveal the uncomfortable secret of their data’s deficiency… and the workarounds, assumptions, and wild guessing that they are forced to do in order to get work done despite imperfect information.  Most people tend to think their data is the worst we will have seen.  “Can you believe this?”  “Have you ever seen anything like this?”

The answer is “yes.”  The day is yet to come when someone sits us down and shows us a perfect data system with perfect content.  The thing is, data could always be better.  And helping define and get to better is what we do.

One unique element of the DLCD engagement has been the bright light that project members were readily willing to focus on their data issues from the get-go.  This project involved designing a new data platform for the organization, and we were able to start at a jog, because there was never any need to coax or cajole information about data challenges from the stakeholders with whom we met. Perhaps this group had already been through the data shame therapy?  Or maybe they were so ready for change that they just didn’t care anymore ;-)? Whatever the reason, it was a little unusual to so promptly and directly get to identifying the problems that needed fixing.  As the end of the project has coincided with the New Year, I’ve been thinking that, for this client, this work has a sort of New Year’s resolution aspect to it.  There is a strong sense of commitment to change, and I’m gathering that it is now being accompanied by a growing sense of optimism that the changes they seek are within reach.

While the DLCD’s data may not qualify as ‘big data’ in terms of shear volume, it streams in from a broad variety of sources and has a reach extending to all the coves and corners of Oregon. As we have learned more about the work of the agency and its partners, we have come to understand that time-informed analysis of patterns and trends in land use, and exploring relationships among complex, amalgamated data is critical for the agency to be able to fulfill its mission.  One stakeholder concluded her comment about a database by saying, “we need a better way to analyze and tell the story of land use in Oregon.”

We feel good about the data platform that the DLCD will be implementing based on the design work we have done together.  The design features a rich integration of time and place. The designs resulting from this project aren’t just garden variety databases — they’re uniquely designed with sensitivity to the client’s current and envisioned needs and goals, and they include some novel elements created in response to the specific and unique challenges that stakeholders brought to light. The design will provide a great foundation from which modern, efficient, and user-focused information management tools will arise.  These folks will have the data infrastructure to capture, curate, store, search, share, analyze, and visualize their information.  This is going to be the platform to support the DLCD’s own brand and style of storytelling.

So, this has been one of those jobs where the promise of meaningful change seems real.  Things can be different.  They will be different. These folks made their resolutions and stuck to them. There’s no shame in that!