Common Pitfalls of Data Analytics Projects
Have you ever been part of a data or software project that seems stuck in a loop?
Three weeks have passed, and although you arrive at work daily, exhausted, having tackled numerous issues, the project remains stagnant.
Why?
Then, suddenly, a new engineer or project manager steps in, reorganizes and prioritizes tasks, and just like that, the project picks up pace again. This exact scenario once unfolded in a project I was involved in, and it was a pivotal moment for me.
It taught me that possessing deep tech knowledge and coding skills isn’t sufficient on its own.
To truly excel in technology, you must also master the art of driving a project to its completion. Let’s explore how you can effectively manage and deliver your data projects.
Start With the End in Mind
Let’s assume your team has gone through the process of getting buy-in from leadership to invest 500k into building several new products(of course one of them has to have AI involved).
Great!
Now how do you start?
You likely pitched the project from a high level for leadership, clear phases, and general delivery dates but now you’ve got to get down into the nitty gritty and actually plan out the project.
So how do you create this more granular timeline?
One way I’ve learned to approach this is to start from the end and think through the tasks backward. I have also heard people reference it as “reverse planning.”
What I like about this is it forces you to think through dependencies, not just technical, but in some cases people.
For example, on one project I was on we knew that a key person was going to be on vacation for two weeks and we had to get a task across the finish line prior to them leaving. Otherwise, the project would likely get delayed two weeks. We were able to see this months prior to the person leaving because we had planned the project from end to start. In these cases, you will either need to find someone else during that period to deal with the issue or you need to ensure you handle the blocker prior to the individual leaving or risk having your project delayed by months.
Overall, I have personally found this method reliable in terms of walking through what tasks need to get done.
Keep Things Moving
One of the biggest reasons I have seen projects stall is because someone doesn’t keep it moving. Now in a perfect world, this means each person would reach out to make sure everyone knew that the task they were responsible for was done and they were handing it off.
This seems to rarely be the case. Instead, for one reason or another, tasks get stuck, people don’t communicate, and suddenly what should have taken a week takes three.
That’s why someone needs to own the actual project; someone needs to keep things moving. Generally, this means someone will check-in. It doesn’t have to be daily), but at a reasonable cadence to keep the flow moving.
- Is there a blocker keeping two teams from delivering. This can often be caused by ambiguity, lack of ownership or different priorities. In these cases I have generally set-up a meeting between the two parties ASAP to drill down to the blocker (I swear 80% of the time a quick call gets teams unblocked almost within the first 5 minutes of the meeting).
- Is there someone who has decided not to push the task because they are juggling too many tasks? Help them prioritize.
- Sometimes just setting a date to meet with a clear expectation that a tasks should be completed is a method of keeping projects going. I do this for myself all the time, because if you don’t meet for 3 weeks another project with a squeakier wheel will likely take priority.
Everyone working on a project might have other responsibilities, so although in a perfect world, they’d make sure to keep the project running independently, having a senior data engineer, tech lead, or project manager who owns the project generally is required.
Define “Done Done”
Along with keeping the process moving, there must be a clear done stage of each task. That is to say, it can be tempting for engineers to go back and try to rework an old task, perhaps gold-plate a feature before the project itself is done.
That’s why it’s good to have a done-done step or something that represents a “don’t touch this again” step, but not until the current set of tasks or features are delivered. Otherwise, you can easily get stuck in that cycle I referenced before, where for some reason, you’ve done a lot of work during the past few weeks but nothing has happened.
One of the reasons is that sometimes there seems to be a rehashing of the same few tasks over and over again, even though the product itself still hasn’t been seen by end-users.
And no one knows when the task is really done-done.
So make it clear, commit it, and get to the next task.
Don’t Just Go Through the Motions
I assume everyone has an opinion on tools like Jira or Monday.com when it comes to trying to manage projects. Some engineers despise it.
Others swear by it.
My only bit of advice is, if you’re going to use it, use it. Don’t just go through the motions.
I have had projects in the past led by PMs who have a stand-up, and ask for what tasks are being worked on but somehow the project never really moves forward.
On one project, we’d meet 3x a week. We’d talk about our tasks, but no one really had a vision for what was next. No one owned the process (truthfully, it was one of those projects where everyone wanted someone else to own it).
So if you’re going through the trouble of setting up all your tasks on a Kanban board, please use it to drive your project forward!
Have Clear Landmarks
When projects start, your team will be excited. Everyone is all hopped up on dopamine and is excited for the future. The problem is, that initial excitement fades and then the hard work begins.
I have been 4-5 months into a migration, losing hope that the project will ever get finished as we translate the 1,000th SSIS package into dbt or T-SQL to PL/SQL (because although there are tools to help migrate between the two, they always miss things).
Thus, you must have clear landmarks or phases–whatever you want to call them–that you can celebrate with your team so they can gain a second, third, or fourth wind. I said 4-5 months earlier that many migrations take years.
So you need to keep your team engaged, excited, and with a clear view of the end of the tunnel.
Don’t Be Shy – Publicize Your Project
One tip I have heard over and over again is that you need to clearly highlight the work you have delivered.
Personally, when I first started in the data world, I thought that was silly. I thought good work should be sufficient, and sadly, at most companies, good work doesn’t always get noticed; not because it’s unsatisfactory, but someone else who has done good work is more willing to tell everyone about it than the person who just waited for others to notice.
I believe a good saying a consultant I once talked to said was:
“If you’re working on a project that no one knows exists, it’s not worth working on.”
Now, let me be clear:there are thousands of projects that are being worked on right now that the company has no idea is likely keeping it afloat. But their point was from the standpoint of getting promoted.
Besides the promotion aspect, communicating your project’s status is key to keep your stakeholders bought in. Especially if the project is longer than 3 months. You can use the template below to ensure you let your team know where your project is going.
📅 Project Update Email Or Post
At the end of the day, whether people know about your project or not doesn’t take its value away. But it does likely impact your likelihood of getting promoted.
Clearly Define the End Of The Project
Similar to having a clear “done-done” step, your project needs a clear end.
Why?
Otherwise, when the project is 98% finished, you’ll quickly notice individuals start to disappear. They’ll all think the project is done enough; they’ll leave loose ends and not wrap up their work.
This could cause a whole host of issues.
Engineers may leave tech debt or commented-out codes in their modules.
A front-end engineer may have not made sure to fix that one final bug.
I have seen this happen early on, especially at large companies where I’d often be assigned to multiple projects. I’d develop the tables and dashboard to display some marketing or operations metrics. We’d be going through the final revisions, then the VP of marketing solutions would get pulled into another project.
So we’d have most of the dashboard finished but we’d now have to wait 5 weeks before hearing back for any final revisions. By that time I had been reassigned to a new project with new goals and by the time the VP came back, they’d still know the dashboard wasn’t 100% of the way there and would have to try to tear everyone away from their new projects. Which doesn’t always end successfully. In turn, the VP will use the dashboard for a while but will likely stop engaging pretty quickly.
Thus, leading the project to failing in the end.
Thanks for reading.