A compass, not a map
The Analytics Development Lifecycle (ADLC), featuring Captain Jack Sparrow
This week’s Roundup comes from Alex Welch, head of data at dbt Labs.
It’s 2003, and you’ve just watched Captain Jack Sparrow reclaim his beloved Black Pearl. You rode the edge of your seat as he evaded the English navy, outsmarted an undead pirate crew, and navigated treacherous waters with a compass that doesn’t point north.
Fast forward to 2024. You are navigating the equally unpredictable seas of modern analytics. Your treasure? It's not buried gold on a remote island, but the invaluable insights hidden within your data. Your crew? A talented team of data professionals. And your compass? That's where the Analytics Development Lifecycle (ADLC) comes in, ready to guide you through the challenges ahead.
Talk ADLC all day long—and anything else important to you—at Coalesce 2024. Join data practitioners and data leaders in Las Vegas October 7-10 at the analytics engineering conference built by data people, for data people. Use the code Coalesce30 for a 30% discount.
But here's the catch: just as Jack's compass behaves differently based on his deepest wants, the ADLC's guidance changes with your team's unique circumstances, skills, and business needs. It's not about following a predetermined route; it's about navigating the waters of data analytics using eight interconnected stages as your guide: Plan, Develop, Test, Deploy, Operate, Observe, Discover, and Analyze.
Understanding your compass: ADLC basics
Imagine you’re holding Captain Jack Sparrow’s compass. It’s not your ordinary navigational tool—it’s a bit eccentric, incredibly powerful, and, most importantly, personalized. That’s the ADLC in a nutshell.
The ADLC isn’t just another rigid framework to implement and forget. It’s a versatile guide comprising of eight interconnected stages: Plan, Develop, Test, Deploy, Operate, Observe, Discover and Analyze.
But here’s the kicker—unlike a traditional compass, the ADLC doesn’t always point in the same direction for everyone. Your team’s skills, your organization’s needs, and the maturity of your data practice all influence which way the needle swings.
For a scrappy startup, the compass might pull strongly towards the Develop and Deploy stages, emphasizing quick iterations and rapid value delivery. A heavily regulated organization might find the needle frequently pointing towards Test and Observe, prioritizing data governance and reliability.
The ADLC's strength lies in its adaptability. It’s not about rigidly following a predefined path, as you would a map, but about using these stages as guideposts to chart your own course. Sometimes you’ll move sequentially through the stages. Other times you’ll jump back and forth as needed. The key is to let your compass guide you based on what your organization truly needs at any given moment.
Remember, just as Jack's compass led him to what he desired most, the ADLC is designed to guide you towards your most valuable treasure—a mature, efficient, and impactful data practice.
Assessing your current position: data practice maturity
Before setting sail with the ADLC, it’s crucial to understand where your data practice currently stands. You need to know your starting point to chart an effective course. Data practice maturity isn’t just about having the latest tools or the biggest team. It’s about how effectively you use what you have, how robust your processes are, how well you can adapt to changing business needs, and, most importantly, what value to drive.
Your data journey might begin with scattered information across systems, manual processes, and a small team. At this point, your ADLC compass likely points towards Discover and Develop, focusing on understanding your data landscape and building foundational pipelines.
As you progress, you’ll establish more robust pipelines and begin to deliver regular insights. Here, your focus might shift to Test and Deploy, improving reliability and speed.
Further along, your processes become largely automated, delivering significant business value. You might concentrate on Operate and Observe, fine-tuning systems and implementing proactive problem-solving.
At later, more advanced stages, data becomes core to decision-making and you begin to explore cutting techniques. Your compass might guide you towards Analyze and Plan while you navigate complex cross-functional challenges.
These aren’t distinct categories, but a continuum. Your position on this journey isn’t a judgment—it’s a reality check. The key is honesty about your current state to guide your ADLC implementation effectively.
Some questions to ask to help triangulate:
How is your data currently stored and managed? Is it scattered across systems? Is it centralized but with limited accessibility? Or is it well-organized and layered with governance?
What’s the state of your data process? Is it a mix of automated and manual processes? No automation? Or is it highly automated with reliable workflows?
How does your organization use data in decision-making? Is it being used to drive the strategic plan or only for operational decisions? Do they trust it at all?
What’s your team’s data expertise level? Are you learning as you go? Do you have a solid foundation with some specialized skills? Or do you have advanced skills across multiple domains?
How do you approach data quality and reliability? How reactive are you to issues? Do you have adequate test coverage?
Before you plan your next move, take a good look at your data practice. How reliable are your data pipelines? How comprehensive is your documentation? How aligned is your team with business objectives? Your answers will help you understand which way your ADLC compass should be pointing next.
Charting your course: aligning with business needs
Now that you’ve got your ADLC compass in hand and you’ve assessed your current position, it’s time to chart your course. But here’s the catch: your destination isn’t a fixed point on a map—it’s the ever-shifting landscape of your organization’s needs and priorities.
The ADLC isn’t just a technical framework—it is a business tool. Each step should solve a stakeholder’s pain point or support a key initiative. Your compass might be pointing in one direction, but if there’s no business value in that direction it’s time to recalibrate your understanding of data’s place within the organization.
To keep your ADLC journey on course:
Start with the “why”: Ask yourself, “Why does this matter to the business?” No clear benefit? Recalibrate.
Engage your stakeholders: Regular check-ins with business leaders help adjust your course. Their perspectives, goals, and pain points can help you adjust your course. Sometimes the most valuable data projects aren’t the most technically exciting ones.
Prioritize flexibility: Business needs can change overnight. Maybe a competitor launches a new product or a global event disrupts your supply chain. Ensure your ADLC implementation can pivot when needed.
Balance short-term wins and long-term goals: Deliver quick wins to build trust, but don’t lose sight of the long-term strategy. Your ADLC compass should help you navigate both.
Practical Tip: Use a “Business Value Checklist” for each ADLC stage. Ask: Which pain point does this address? How does it align with top priorities? Can we quantify the impact? What’s the cost of not doing this now?
This checklist is just one tool in your toolkit. We’ll explore more later, including regular health checks and feedback loops.
Your ADLC compass is pointing towards value, not just completion. If you can’t articulate the business value of a step, it might be time to change course.
Navigating the winds of change: adapting the ADLC
Implementing the ADLC isn’t always smooth sailing. Your team’s skills will evolve, business priorities will shift, and new technologies will emerge. The key to success? Adaptability.
Adjusting based on team skills:
Skill gap analysis: Regularly assess your team’s strengths and weaknesses. Maybe you’ve got a strong analytics engineer team but lack data visualization experts. Your ADLC compass might point towards up-skilling in the Analyze stage.
Cross-training: Encourage versatility. A data engineer dabbling in analysis can bring fresh perspectives to both roles.
Strategic hiring: Use the ADLC to guide recruitment. Identified a bottleneck in the Test stage? It might be time to bring a QA specialist on board.
Leverage external expertise: Don’t be afraid to call in reinforcements. Consultants or temporary hires can help you navigate the tricky parts of your ADLC journey.
Adapting to changing business priorities:
Agile ADLC sprints: Break your ADLC journey into short sprints. This allows you to reassess and pivot more frequently based on business needs.
Priority mapping: Regularly map your ADLC activities the current business focus. If the business suddenly pivots to customer retention, your Discover and Analyze shift to churn prediction.
Feedback loops: Establish regular check-ins with key stakeholders. Their input can help you adjust your ADLC compass before you veer off course.
Modular implementation: Treat the ADLC stages as building blocks, not a fixed sequence. You might need to jump back to Plan midway through Develop if business requirements change.
The ADLC is meant to be a flexible guide. For teams early in their evolution, this may mean focusing on basic skill development across all ADLC stages. A team further along might emphasize cross-training in specific areas. Your ability to adapt to your team’s evolving skills and your organization’s changing needs is what will ultimately determine your success.
Avoiding the sirens: common pitfalls
Even with the best compass, treacherous waters remain. Beware of these common challenges that can derail your efforts:
Perfectionism: Striving to perfect each stage can stall progress, especially as you begin to scale. This is when the desire to create robust processes can overshadow the need for quick wins and iterative improvement. Solution: Embrace iterative improvement. Start with a minimum-viable process and refine as you go.
Technology fixation: Implementing new tools without clear business justification can lead to unnecessary complexity and cost. Teams early in their journey are particularly susceptible to this, often believing that the latest tool will solve all their data challenges. Be wary! Even teams on the bleeding edge can get distracted by the shiny object. Solution: Tie tool adoption to specific business needs or ADLC improvement. Prioritize solving real problems over acquiring new technology.
Departmental silos: Isolated ADLC implementation leads to fragmentation. This becomes particularly problematic once your team has grown beyond the initial centralized team. The increased specialization may inadvertently lead to isolation between teams. Solution: Foster cross-functional collaboration. Ensure the Plan stage involves stakeholders from across the data team and the broader organization.
Misaligned metrics: Focusing on vanity metrics can lead to misguided efforts. This is a pitfall that teams of all shapes and sizes run the risk of falling victim to. Don’t overemphasize complex metrics that fail to tie directly to business outcomes. Solution: Align your ADLC metrics directly with business impact. Prioritize quality and value over quantity.
Lack of continuous improvement: Treating the ADLC implementation as a one-time project can quickly lead to outdated and ineffective processes. This is particularly dangerous in more mature practices, which might become complacent with their existing processes. Solution: Schedule regular ADLC reviews. Continuously assess if all stages are serving their purpose and incorporate new best practices.
Over-engineering: Building complex systems for hypothetical future needs wastes resources and creates unnecessary complexity. This is particularly dangerous for early and young data teams. Solution: Start simple. Let actual business needs drive increased complexity over time.
Insufficient documentation: Neglecting documentation can turn your ADLC into tribal knowledge and hinder your ability to scale. This is something all stages are likely to encounter. Solution: Make comprehensive documentation a required part of each ADLC stage. This supports consistency, scalability, and knowledge transfer.
By being aware of these potential pitfalls, you can proactively address them in your ADLC implementation. Remember, encountering challenges is part of the process. The key is to learn from them and continuously improve your approach.
Calibrating your compass: continuous improvement
Implementing the ADLC is ’t a one-and-done process. Just like how Jack Sparrow’s compass changes course depending on what he wanted most in the moment, so too will your ADLC compass shift. The key is understanding why that happened. Implementing the ADLC isn't a one-time process. Understanding why is key to keeping your implementation relevant.
Regular health checks: Schedule quarterly reviews of your ADLC implementation. Ask key questions like:
Are all stages still aligned with our business objectives?
Are we collecting the right data?
Which stages create the most value? Which create bottlenecks?
How has our data maturity evolved, and does our ADLC approach reflect that?
Feedback loops: Establish mechanisms to gather ongoing feedback and use it to make incremental improvements to your process:
Data team members: Frontline insights can identify practical issues quickly.
Business users: Are they getting timely, needed insights?
Executive leadership: Is the ADLC supporting strategic decisions?
Metrics that matter: Develop and track metrics that reflect the effectiveness of your ADLC implementation. Regularly review these and adjust your approach accordingly. Below are some metrics to get started with:
Cycle time from data ingestion to insight delivery
Number and severity of data quality issues
Team productivity and satisfaction.
Stay informed: The data world evolves rapidly. Stay current with the following and consider how these developments might enhance or impact your ADLC implementation:
Industry best practices
New tools and technologies
Regulatory changes
Emerging data ethics considerations.
Cross-pollination: Encourage knowledge sharing within your organization.
Rotate team members across different ADLC stages
Host internal workshops to share learnings and challenges
Create a best practices knowledge base.
Flexibility in implementation: Be prepared to adapt your ADLC approach as circumstances change. Your ADLC should be robust enough to handle change, yet flexible enough to embrace it.
Business priorities shift
Team composition evolves
New data sources become available
Technological capabilities advance
The goal isn’t perfection—it’s progress. Each small adjustment to your ADLC implementation can lead to significant gains in efficiency, effectiveness, and business impact over time.
Your true north
The ADLC isn’t a rigid map with a predetermined route. Instead, it guides you toward what you truly desire: a data practice that delivers real, measurable value to your organization. And just as Jack’s compass behaved differently based on his changing needs, your ADLC implementation will be unique to your organization’s needs, maturity, and aspirations.
Your current position (data maturity) influences how you interpret the compass readings.
Your desired destination (business needs) determines which direction you should sail.
The winds of change (evolving team skills, new technologies, and shifting priorities) require constant course adjustments.
Treacherous waters (common pitfalls) demand vigilance and strategic navigation.
Your unique voyage (company stage, needs, and goals) shapes how you apply the ADLC principles.
Regular calibration (continuous improvement) ensures your compass remains accurate over time.
The ADLC’s power lies not in blindly following its stages, but in thoughtfully applying its principles to guide your data journey. It’s about striking the balance between structure and flexibility, between best practices and practical realities.
As you embark on your own ADLC journey, remember that there’s no “perfect” implementation. What matters is that you’re moving in the right direction, continuously learning and improving along the way. Your North Star isn’t a flawless data practice—it’s one that consistently delivers value to your organization while adapting to new challenges and opportunities.
It’s time to grab your ADLC compass and chart your course. Where will your data journey take you?
 |
|