Data Strategy - 201: Requirements Gathering

Wed, Jul 27, 2022

Tags:

• data-strategy
• data-and-the-biz
• short-short-course
• analytics

“Short Short Courses” are questions meant to prompt the reader to think and reflect. In my time as a tech lead I’ve found that asking questions is far more important than laying out answers. Imagine you’re in a meeting with your team and someone asks these questions of you, would you have a good answer? And if you don’t have good answers, now is your chance to start asking these questions.

Interrogating Your Metrics

In my opinion the following questions should always be asked and considered of any metric you’re trying to measure i.e. ask them of metrics for system performance, business metrics, product metrics, etc.

1. What am I trying to measure and how can I describe & communicate it?

   • Ad Opportunities - what does that mean to you and be explict?

2. What does this metric translate to from the perspectives of others e.g. the user’s view, your Buisness team’s view, Product, Web, and other Data team members view, etc.?

   • Will the Biz use this quantity differently than Product? Why or why not?

3. Can I measure this metric accurately and how can I assert accuracy?

   • Proove to your Buisness team you’re not inflating numbers.

4. Are noisy approximations to meausrements okay or is it worth spending time (and money) to measure this metric more accurately?

   • What assumptions of the data are being made, is this reasonable based on the volume of data affected by this assumption?

5. Can I drive this metric or control it with interventions?

   • Will a vanilla A/B test give me trustworthy insights? If not, what system level factors need to be controlled for to ensure trustworthy experiments?

6. How is this metric going to be used by the business?

   • This one is suuuper important to understand because it’ll inform whether you should even spend time on it at all.

Concluding Thoughts

The more important the measurement is to the business the more tech effort you should put into aligning all your teams on the metric, the data it’s derived from, and all expectation surrounding the quantity being generated. This includes coming to agreements on how the data and metrics should be accessed, what’s the expected quality, how far back is data kept, what are the assumptions, and maybe a few other things.

Think about it, how many past decisions would have been different if our current measures were as accurate then as they are today?… 🤯