Implementing This Miracle: If I were to start all over again, I'd tell you this:
A) Get cozy with Git, simulate real-world scenarios as a team during a demo session, and document/record everything.
To better prepare for collaborative development, we crafted a scenario in which multiple developers would work simultaneously on the same file, aiming to rehearse a range of actions we had previously encountered. This included creating different measures at the same time and then merging these into a single file, adding, removing, or modifying visuals and integrating those changes, reverting to a previous version to serve as the basis for new features, and managing version transitions between DEV, TEST, and PROD branches. Through this exercise, we sought to simulate and navigate the complexities of real-world collaboration, enhancing our team's ability to handle various project management tasks efficiently.
- Commit early and often, and
- Treat your commit notes like a diary of your project's life. Writing comprehensive commit descriptions will enhance clarity and facilitates easier navigation through the project's history for both you and your teammates.
- Branch Carefully! Always verify you're working on the correct branch to avoid unintended changes in the wrong areas of your project.
- And never forget the power of communication. Despite the technical tools at your disposal, effective teamwork hinges on constant communication to ensure alignment and collaboration.
These practices didn't just make our projects better; they made us better developers and teammates.
B) Bumps Along the Way: Sure, we faced our fair share of surprises.
Challenges Encountered Since Adoption:
Despite the myriad benefits, several challenges have emerged since adopting source control in Power BI, particularly given the preview status of this feature. I'm going to highlight the few most common ones.
- Custom Visuals Compatibility: Custom visuals used in reports need to be installed separately by each collaborator. Don't forget to let your colleagues know which custom visual you added to your dashboard as its name won't show up only the following error message which is a bit hard to dechypre.
2. Merge Conflicts and Code Loss: Situations have arisen where accepting both changes during a merge resulted in lost code, highlighting the need for careful conflict resolution (screenshot from another blogpost describing the issue).
3. File Opening Issues After Merge: Conflicts within the data model, such as incorrect relationship settings, can prevent files from opening, necessitating reversion to previous versions.
4. Infinite Semantic Model Refresh Loops in PBI Desktop: Unexplained delays in model refreshes post-merge, extending for hours, indicate potential issues with large semantic models.
+1) Data is not stored in repos: It’s good to keep in mind that data is not stored in the Repo, only semantic model and report related code is stored, you still need to refresh your data on Power BI Service.
Conclusion: This journey of integrating CI/CD and source control into our Power BI workflows has been one of the most rewarding experiences of my professional life. It wasn’t just about making our projects more efficient; it was about transforming our team into a more cohesive, capable, and confident unit. As we look to the future, I’m excited about the possibilities that lie ahead, ready to tackle whatever comes our way with a smile and a git commit.
Hello, friends! Today, I'm diving deep into my own adventure with Continuous Integration (CI) and Continuous Deployment (CD) in the realm of Power BI—a journey marked by trials, triumphs, and a lot of learning along the way. For so long, CI/CD in Power BI felt like trying to fit a square peg in a round hole. With a mix of third-party tools and makeshift solutions, my team and I navigated through a maze of compliance and administrative hurdles, often feeling lost in a sea of approvals and support tickets to try introduce a potential CI/CD solution or workaround. Then came May 2023, and with it, a beacon of hope: Power BI project files. This was more than just a feature; it was a revolution that promised to redefine how we approached our projects.
The Before Times: Rewind to before this pivotal change. My team, like many others working in data analytics within complex, multinational landscapes, struggled immensely with the lack of source control in Power BI. Our attempts at version control felt archaic—think "save as" with date and version stamps—and far from agile (shared folders on SharePoints or OneDrive). We were a team longing for simplicity and efficiency but found ourselves bogged down by the limitations of our tools.
A Game Changer: Power BI Project Files: Then, the game changed. The introduction of project files (.PBIP) wasn't just an update; it was a lifeline. This wasn't about saving our projects as mere files; it was about evolving them into living, breathing entities, organized within folders that spoke the language of our data stories through JSON (not ideal, but still an enormous improvement) .
Getting Our Hands Dirty: Embracing this new world, we ventured into the depths of Git with VS Code and Azure DevOps.
Each Power BI file became its own project, its own repository. This structure, while logical, required us to rewire our brains, to rethink our approach to collaboration and version control. Our workflow transformed, becoming more streamlined yet demanding a new level of diligence and precision.
The Rocky Road: It wasn't all smooth sailing. Adopting Power BI project files and integrating Git introduced us to a host of challenges. Merge conflicts became our nemesis, and the occasional quirks of a preview feature tested our patience. But with every stumble, our resolve grew stronger. The benefits—oh, the benefits!—far outweighed the occasional headaches. We were building something resilient, transparent, and infinitely more manageable.
Looking Ahead with TMDL: The next big step ahead is the introduction of TMDL. Storing your semantic model in TMDL (preview feature since the March 2024 update) transforms the code into a format that's not only human-readable but also intuitively organized, making every element—be it measures, tables, or columns—distinctly easy to identify in separate blocks. This clarity significantly simplifies the process of resolving merge conflicts and tracking changes, making the whole experience more straightforward.
Why This Matters: The move to source control in Power BI has been nothing short of transformative. It brought clarity to our development process, made our end products more reliable, and, most importantly, it strengthened our team's bond. We learned to communicate more effectively, to trust in our collective skills, and to embrace the inevitable learning curve together.
To be continued...
Every quarter we have three Objectives that will help us improve as an organisation. This can vary from Software Development practices to improving (or rather: re-hauling) our finance reports, but more will be written on these Objectives later.
This quarter, one of our Objectives is to start and cement having a company-blog. This has been a recurring topic since the beginning of the organisation - although I've consciously put it to rest in the past years as we've been so focused on cementing the core team.
Last year, we started focusing on our marketing efforts and naturally, the topic of the blog has arisen. Initially, I thought that with Judit, our marketing and sales assistant, we can just put our heads together and churn out a bunch of posts, but then realised we need to get the "troops on the ground" to provide inputs - at least the backbone of a post to stand a chance of sounding novel and not just being "yet another corporate blog."
I knew it's not going to be easy to get a bunch of consultants knee-deep in project-work to get their focus and time allocated to writing blogs. What a perfect opportunity to make this a Q1 objective!
A reality check
I naively thought it'd be sufficient to call out the Q1 objective and have a brainstorming session about our social media usage and post ideas and off we go: blog posts will be sprouting left and right.
Six weeks later, the end of the quarter is nearing, and we have 0 posts.
So what can I do to get our smart and eager colleagues to exert discretionary effort to create a blog post - let alone, get into the habit of writing and maintaining a blog.
I resorted to the following basic management (leadership?) tactics in yet another Dojo agenda point. (Dojo: The forum held weekly for the development of our internal operations.)
Start with why
We talked with the team about what's the purpose of writing a company blog and why it matters in the professional services industry:
It's so easy for anyone to say they are experts in a particular topic but what speaks volumes is if we provide insight into our expertise.
Plus, it also makes it possible to create more meaningful "noise" on LinkedIn and social media.
Not the least, we maybe help fellow Power Platformers, consultants and end-users alike by publicly sharing knowledge bites that proved to be useful for us.
Provide context and address concerns
After having talked about the why, I showed an example of what good looks like (eg. https://www.ycombinator.com/blog/how-to-maintain-engineering-velocity-as-you-scale or https://www.faire.com/blog/) and asked the team if they have any questions, concerns.
The main message here was:
don't worry too much, just get started, we are not writing War and Peace here
Make it easy to start
Next, we talked about how we "don't need to start from scratch", a lot of what we do is documenting, so why not use those as a starting point for blog posts.
We have the following areas to go to for inspiration and starting points:
Wiki
Last year we started making good use of our Azure DevOps wiki sites and it's increasingly becoming an internal go-to resource. Why not just use some of the more useful entries to turn them into external-facing material?
Internal Comms
Not as often, but we do rely on written internal comms - such as our strategic goals, one-pager vision statement, competencies. We can turn these into blog posts that talk about us as an organisation, how we work and where we are heading.
Our clients are increasingly looking for "cultural fitness" when selecting a partner, so I'll be happy to share this openly. This may even help with recruitment and on-boarding.
Ways of Working
We are extensively using Azure DevOps, GitHub and generally the whole Microsoft eco-system to manage our work. I appreciate it's not straight-forward to design the toolkit at our disposal in a way that makes it ergonomic for every member of an organisation. We have lots of lessons learned in this place and our current Ways of Working is pretty well documented.
When we built our internal systems, it would have been great to understand like-minded companies' architecture. The toolkit we use definitely shapes us as an organisation (think: socio-technical complex systems), hopefully gives us a competitive edge, but sharing our good (and sometimes not-so-good) practices may benefit others with similar challenges.
Knowledge Sharing Sessions
We regularly hold knowledge sharing sessions, internal demos - lot of which is already documented - with a little effort and obfuscation, these can be easily turned into external-facing materials.
These will provide great insight into what are the areas that interest us and what challenges we face as a team in our projects.
Brainstorm
Prior to the Dojo, I asked that each PO is present or delegates a team member so we have the right coverage. I asked each team to gather post ideas based on the above and take them back to the team and select one which they will work into a full-fledged post in the coming two weeks.
Lead with example
I had to admit, I wasn't comfortable writing blogs myself (I had one attempt when taking a gap-year and travelled to India). Those that worked closely with me, know I'm not much of a writer.
But I had to be the "tip of spear that will break the ice", so here I am writing the inaugural blog post.. :)
Where to next?
Accountability and Follow-up
Hopefully, by now the message has landed with the team that this is something, we'll crack on with and asked the team whether they can get behind this. The jury's still out to see how many posts will be written by the end of the month.
We've put in a check-in for next to see how the teams are progressing with the single selected topic and see if we can help them overcome and doubts or concerns, they may have.
What we didn't do
Carrots or sticks
We quite consciously avoid tying performance evaluation or bonuses against our quarterly objects (very much in line with OKR principles) - hence we will not mandate a blog quota or tie it to any bonuses. I'd much rather have a few of our team members produce quality posts, then have everyone write poor ones out of sheer necessity
Emphasize how writing improves structured thinking and deep understanding
In hindsight, we could have talked about how writing helps improve one's understanding, and this is a great way to deepen one's knowledge.
So you may ask, why blogs? Why not YouTube videos and/or Podcasts? Don't worry, we'll get there eventually :)
