Simetrique #8: State of Data
The one about "State of Data" report, engineering culture and dbt testing
Hey!
Over the past few weeks, I have been busy with my new secret project (I’ll make an announcement soon) and did not have time to write a proper newsletter post. Now I am back and ready to share some interesting bits about the world of data!
We’ll start with exploring yearly poll called “The State of Data 2023”. It’s a poll that helps us understand the current trends in the data world.
Next, I suggest revisiting an old article about Spotify's engineering culture. Although it was released in 2014, many of its ideas are still relevant today.
Finally, my favorite topic of the past couple of months has been data quality and testing. I recently came across an article on "7 dbt Testing Best Practices" from Datafold and it’s a blast!
State of Data 2023
State of Data 2023 is a very comprehensive survey of data specialists about their roles, tools, and basically their day-to-day challenges.
A couple of interesting highlights:
A growing number of tools in many categories: ingestion, storing, visualization, data quality, reverse ETL, and many more. Sure, there is no need to know all of them, but at least to understand their purpose and distinguish who-is-who.
Every tool tries to be “as a code”: “orchestration as code”, “visualization as code”, you name it. Frankly, I like this declarative approaches to everything, it keeps my internal perfectionist cool.
Semantic Layer and Data Modelling are in trend. It’s also a good sign, because these things keep our data structured and much accessible to end-users.
Lastly, more experience correlates with more pay. Keep learning for a better compensation.
To sum up, "The State of Data 2023" survey shows that the data industry is constantly evolving with new tools and technologies emerging every year. It's important to stay up to date with these changes and understand their purposes. Additionally, the trend towards declarative approaches to data management and the focus on data quality and accessibility are great signs for the industry's future.
Spotify engineering culture
I like Spotify a lot. Not only they make an awesome product, but also they have an awesome company.
Just watch this presentation of their Engineering Culture. Despite that it was done almost 10 years ago, many principles and approaches are still relevant even today. How to build an engineering culture, how to align the whole team, how to deliver features and great product. Everything is there.
P.S. Also can recommend a TV Show on Netflix called “The Playlist” about the history of Spotify. In some ways it’s rather a show than documentary, yet still very interesting.
Bonus: dbt testing best practices
I’m obsessed with data quality testing and everything connected to it. Also, I’m pretty much obsessed with building good and reliable data moleds in dbt. So the combination of these two makes me feel really good :)
In this article from Datafold you can learn about 7 best practices doing data testing with dbt. I like that is recommends starting slowly from built-in tests and capabilities and then transition into more advanced topics.
That’s it.
If you liked this issue please share it with your colleagues, this greatly helps me developing this newsletter!
And see you next time 👋