Hey you. It's been a while! The newsletter is back. So, expect Data News to land in your inbox every week between Friday and Sunday. Same recipe as before: a bunch of links about data and AI, topped with my usual spicy opinions.

Below a best of the last months Data News, mainly the best articles about the AI and data ecosystem that I've came across, it's a great reading list.

AI News 🤖

• The consumer AI companies are working on changing the way we browse and consume internet
  • OpenAI brought to ChatGPT new integrations like shopping and courses. This is a new way to consume the web, ChatGPT shopping will be a way to monetise but also a paradigm shift in how we use internet, OpenAI is trying to rebuild a web from within a chat, shortcutting the browser. But they also release a browser this week, named Atlas.
  • Browsers are getting more and more AI capabilities, whereas it's Dia or Comet, the goal is to give AI browsing capabilities like if it's human. Might be a transitional phase until the whole web gets destroyed because of bots and usual websites disappears?
• From GPT-2 to gpt-oss: analysing the architectural advances — This is a great architectural deep-dive to understand the architecture behind all the GPTs. What layers are used, how gpt-oss compares with Qwen etc. If you're French, Defend Intelligence redeveloped a GPT for a YouTube video.
• The upcoming GPT-3 moment for RL — Small essay about the current state of reinforcement learning which need to move from being task-specific to scale. Scaling RL would require something like replication training as a set of specs to reproduce complex RL scenarii.
• AI vs Gen Z — How AI has changed the career pathway for junior developers. It has been posted on Stack Overflow blog, which ironically has been also very impacted by AI in the last 2 years. It describes well the current situation, being a junior developper was already difficult and AI made it worse (25% decrease in junior job posting in 2024), and the employment for Software engineers has decreased nearly 20% since 2022 peak.
  
  After years of considering SE like a promising career, AI is changing everything, we don't learn as mush as before, we don't need intern or juniors, salaries might decrease if the job become less complex. But if you don’t hire junior developers, you’ll someday never have senior developers.
• Use LLMs to analyse postmortems at Zalando — Large companies often have a large number of postmortems (memo written after incidents) and it might be a great use of AI. They designed a multi-stage pipeline with: summarisation, classification, analyse, patterns and opportunity.
• How I got the highest score on ARC-AGI again swapping Python for English — ARC-AGI is a benchmark doing an intelligence test designed to measure pattern recognition over puzzles that humans can easily solve.
  
  Currently an human panel score 98%, while GPT-5 Pro scores 18%. The author of the article successfully score 29% when switching from code to English.
• An unusual consequence of AI coding — "What AI coding has taken away is the time where you know exactly what you want to implement and have a rough mental model of how to do it [...] There was a beauty and joy to this part that I miss, a flow state you can hit with a nice linear progression". Probably what factory workers might have said when their factory got automated? We don't have to think the way we were thinking before.
  
  Related: Dumb Cursor is the best Cursor.
• Basic facts about GPUs — Explains how GPUs compute and memory work and the different performance regimes: memory-bound, compute-bound and overhead.
• Prompt injection attacks through images — Hide a text in a image that might be readable when the image gets downsampled or filtered. If a LLM interpret this text it's an attack surface when people are adding images to their chat conversation.
• Context Engineering: How RAG, agents, and memory make LLMs actually useful and Learn Agentic AI: A Beginner’s Guide to RAG, MCPs, and AI Agents — Two guides to explore agentic concepts.
• Use GEPA automated prompt optimisation to surpass Claude Opus 4.1 — Databricks achieved great performances after doing prompt optimisation on gpt-oss-120b.
• [study] Using ChatGPT is not bad for the environment — A cheat sheet about carbon emissions related to LLMs.
• [paper] Large Language Models often know when they are being evaluated.
• [podcast] How GPT-5 thinks — From OpenAI’s VP of Research Jerry Tworek. He explains how reasoning works.
• [paper] How people use ChatGPT — OpenAI ran a classifier on 1.1m sample conversations to understand how their 800m+ weekly active chatters are using the AI. It shows how widely the AI can be use to do people stuff.

Fast News ⚡️

• Python, the documentary — A great documentary about Python and the origin, how the initial community has been build and what it takes to create such a piece of open-source software that is widely used. Python is scoring 25% popularity index (TIOBE), when top 2 and 3 are C and C++ with 9% each.
• Apache Airflow Summit took place a few weeks ago, videos are not yet out but Marc Lamberti shared a few takeaways on LinkedIn like how Duolingo is using Airflow. Airflow 3 has also been released.
• Python 3.14 is out — it gets a natural performance uplift and pave the way for the GIL changes.
• Astral innovations to Python ecosystem — Astral is changing Python tooling forever with great crafted product. Recently they released:
  • astral/ty — A Python type checker, written in Rust (obv) that runs faster than anything else.
  • uv — uv format (might replace black). And funny thing someone solved wordle using uv dependency resolver.
  • pyx — If you need a private package registry Astral created pyx. Might be their way to make money at the Enterprise level to keep them working on this great tooling.
• How not to partition data in S3 and what to do instead — When you need to partition by data on S3 you should partition using the YYYY-MM-DD format.
• Does OLAP need an ORM? — Great question. ORM can bring type-safety to SQL generation because database objects are translated in the native programming language. This way AI when generating objects knows types and might know if something will fail before it hits the database. As chat with your data is becoming more and more tried at companies, this is maybe a requirement we actually need.
• Some news about the Iceberg / lake house ecosystem.
  • ClickHouse and DuckLake now support write to Iceberg.
  • Thoughts on DuckLake — Max explains why DuckLake might be a big thing when it comes to improving the local developper experience. As DuckLake can make DuckDB function as a data warehouse. Imagine if while developing you could run you usual BigQuery pipelines but locally on the production data (that is available on GCS).
  • Cloudflare data platform — Cloudflare announced their lake house platform based on R2 (S3 compatible storage). They released R2 catalog (a fully manage Iceberg catalog) and R2 SQL. R2 SQL relies on Apache Datafusion.
  • The age of the 10$ lakehouse — A great deep-dive of the combination of the 2 previous bullet points. This is awesome to see this new kind data platforms. Back then they moved away from Fivetran + Snowflake to CDC with Debezium + Hudi (an Iceberg alternative).
• Building and scaling Notion data lake — An old article about how Notion structured their data lake.
• The minimalist data stack — 5 parts article describing a dltHub + dbt + BigQuery data stack.
• If you missed it Fivetran and dbt Labs are merging, here are my thoughts.
• Redefining analytics roles and aligning skills and practices for future-ready insights — How to rebalance the skills and responsibilities when analytics engineering becomes a bottleneck.
• Data modeling framework + revisiting medaillon architecture — Would you take a bit of data modeling content?
• Analytics at scale — How to do product analytics at scale when tens of new features are released every week and product teams wants to understand what's happening. The article shares the organisation Doctolib implemented and the data modeling that was put in place to make it work.
• Scaling Success: The dbt ecosystem at BlaBlaCar — What a team of 45+ engineers had to put in place to make their dbt setup work for everyone: dev containers + extensions + a few dbt packages. If you want the same setup but without doing anything you can use nao.
• Data as a product, applying a product mindset to data at Netflix.
• RIP Tableau — 2 months ago Voi killed Tableau and switched to LLM as a bridge in Slack and Sheets to accomplished what was possible in Tableau before. It required an effort of metrics definitions tho.
• Why BI in the AI age — "Great analytics isn’t about generating charts quickly, it’s about building confidence in decisions through rigorous investigation of data. Every discovery, design choice, and contextual annotation represents a human analyst’s business intelligence."
• Vibe Analysis — The other side of the coin.
• Doing SQL work with LLM aids as a SQL addict.

I'll be speaking at OSDC AI next Tuesday about Building AI Agents is data engineering.

See you next week!

Hey here. I hope this email finds you well. My dear Data News has been a bit neglected these past few months—I’ve been busy with my other gig (that you might nao). But don’t think I forgot you. Every Friday, I thought of you and this little corner of data passion we share.

I’ve decided the weekly write-ups are coming back—they have to. Earlier this year, I went through YC, and the aftermath took up way more of my time than I expected (especially the unplanned detour into hiring). But that’s over now. It’s time to get back to basics.

So, expect Data News to land in your inbox every week between Friday and Sunday. Same recipe as before: a bunch of links about data and AI, topped with my usual spicy opinions.

To reboot the machine this week, I’m sharing my take on dbt Coalesce and the dbt Labs + Fivetran merger that caught everyone by surprise over the past few weeks. Next week will be a “best of” edition—a curated collection of the most interesting articles from the last six months. It’s a great one, don't miss it.

A bit of history

If you were living in a cave last month, you might have missed some big news. But before jumping in, a bit of history.

I started my journey in data back in 2014, at the height of the Big Data era—when Hadoop was on everyone’s lips and companies were throwing hundreds of thousands of euros at infrastructure, teams, and software. It feels like another lifetime, when building a recommendation system was a multi-month, six-figure project

But that wasn’t even the beginning. The story starts in the late ’70s and early ’80s when the term data warehouse was coined (did you know Excel was created in 1985??). From early Oracle data warehouses to Hadoop, one pattern stayed the same: these tools were painful to use. Getting into data required obscure knowledge you couldn’t find in school, and the technology itself was… well, a bit of a nightmare (or a Java nightmare).

Then came AWS and the cloud, which simplified a lot of what we were doing. BigQuery made it even easier, just throw in your data, query it, and pay per query. In 2018, I migrated a 3 To exploding Postgres warehouse to BigQuery, cutting query times from hours to seconds. Everything ran through Airflow, orchestrating extraction and transformation. Like thousands of others, I had unknowingly built a simili-dbt.

At that time, Airflow was the glue. Every issue, every new need meant extending our internal Airflow framework — even reverse ETL was just another DAG in our dynamic DAG factory. Then, after being laid off following an acquisition, I went freelance and worked on my first dbt project. At first, I wasn’t convinced — then it clicked. It was exactly what I’d built internally, but open-source, standardised, and ready to become the industry norm. It empowered less-technical users while letting data engineers focus on keeping the platform running.

dbt also helped make SQL-first thinking mainstream. For years, it was SQL data engineers vs. JVM data engineers. The former chilling, the latter raging about our pipelines not being type-safe. Then came the “Modern Data Stack”: ingest with a paid tool (and a few custom scripts when it fails), transform with dbt on your warehouse, and visualize with two BI tools — Tableau for execs, Metabase for everyone else. We’d unbundled Airflow into a set of specialized tools.

It was a great run. But the fun had to end sometime. After a decade of building the foundations of the modern data stack, did we finally get it right? So that when AI arrived, we could just plug it in and have it magically work? Read clean metric definitions from a central warehouse and deliver a single, shared version of “revenue”?

We did nail that… right? Right?

Nobody got fired for choosing dbt

I’d say AI has arrived and it’s forcing companies to move faster than ever. This is the era of building AI apps on top of AI, not the time to pour more resources into data pipelines. That’s why we’re seeing consolidation: companies want bundled services, a single invoice, fancy certifications, and the comfort of validation from expensive tools.

All of this sets the stage for where we are now. The Modern Data Stack has become the useful idiot of the moment. Replaced by Analytics and AI Stack. Because, of course, AI runs on data, data runs on dbt.

Every year since 2020 dbt Labs is doing an annual conference called Coalesce—to not mix up with Coalesce.io (one of their Enterprise and lawsuit competitor). I've covered 2021 and 2022 Coalesce from abroad and this year I had to chance to go in-person to live the hype. Here my main takeaways:

• The merge — Fivetran and dbt Labs are merging (stocks operation) and will provide a first-of-his-kind open data infrastructure (see below). Pay attention, the merged company will do a lot of things open, because "open means opens". In their word open means open standards and not necessarily open-source.

• The vision —The follow-up of the vision of the open data infra is: all their biggest competitors are all selling storage and compute (Databricks, Snowflake, Fabric, BigQuery) but dbt Labs/Cloud/Platform is the only valid and trustworthy platform to do data because they are not vendor-locking you with a compute engine and providing you open standards on all other parts of the stack. So you can switch over something else.
• dbt Fusion — There is an economic reality, dbt acquired SDF Labs to develop Fusion. 2 mains selling points of Fusion are: cost cutting and better developer experience. I guess it's schizophrenic to sell cost cutting (~50%) while setting compute.
• Open standards — dbt Labs is rooting and supporting open standards
  • Iceberg and lake — with Fivetran EL destination as datalake and dbt capabilities to support Iceberg catalog / adapter. Fivetran is a data lake company now also.
  • OSI — the Open Semantic Interchange to create a unified way to define metrics with other big companies (Snowflake, Salesforce etc.) and (re)open-source MetricFlow (after taking it private after Transform acquisition)
  • SQL and ADBC — dbt relies obv. heavily on this
  • MCPs — because it's AI, the dbt MCP provides new capabilities to do stuff with AI.
• But what about open-source — This is our friend who stayed out of the party. They tried to do the magic trick to make us believe Fivetran is a true open-source contributor with more than 100+ open-source repos (they have 278+ but the one with the most starred has 184, even my yato thing has more).
  • dbt Language — it was the best demonstration of the Community keynote, what makes dbt dbt is the language and the fact that everyone speaks the same with a bunch of SQL and YAML file. It unified the way we define transformations. dbt language is an open standard. It a way to organise files and this can be picked up by whatever engines, today we have dbt Core and dbt Fusion as engines.
  • They announced dbt Core will be maintained for the foreseeable future while maintaining dbt Fusion at the same time, so spending twice the effort to support language evolution in a codebase that wasn't meant for this.
  • Just to remember dbt Core is stupid as fuck as it's just a templating processing engine organising files in a DAG thanks to the manual declaration of relationships, whereas Fusion is understanding SQL by parsing it.
  • dbt is the common language of 90.000 data teams around the world. That's a lot.
• Coalesce — I was a bit disappointed by the quality of the talks at Coalesce this year. Some speakers didn’t seem to even be using dbt, while others gave entry-level presentations aimed at… well, I’m not sure who. In the past, I always learned something new from Coalesce, but this year felt like a turning point. The tool has gone mainstream, reaching Enterprise™ levels and drifting away from its original community of people hacking around dbt Core. That said, the people I met and the conversations I had were great.

Conclusion

I’ve been a dbt (Core) advocate for years. If you look through this blog, it’s probably the most mentioned technology here—alongside Airflow and DuckDB. Those three tools share something fundamental: they’re open-source and community-driven. In France, I helped run the Airflow community for a few years, later became known as a dbt expert, and at one point people even thought I worked for the ducks.

The reason I’ve spent the past eight years sharing and writing about these tools is simple: they were open-source. I was happy to give my time with no direct return because it felt like my own way of contributing back. But lately, something feels broken in my relationship with dbt. It’s not the merger itself—it’s the direction, the shift in strategy.

dbt Labs now seems focused on the Fortune 500. The new features aren’t made for someone like me anymore. Why would I need a drag-and-drop UI when that’s exactly what I tried to escape early in my career (hello, Talend)? Why would I pay $10,000 to run a simple SQL-only DAG? The new company’s focus just doesn’t speak to me as a data engineer.

Of course, as a founder, I understand why they’re doing it. They have to make money eventually, and I don’t have a solution to this. This is just my perspective.

And we shouldn’t forget SQLMesh—the only real open-source alternative to dbt Core—which quietly disappeared after an acquisition not long before all this. I can’t help but think that was part of a larger chess game, by Fivetran, to smooth the path for the dbt Labs deal and remove the one viable option that could have welcomed dbt users looking for an exil.

I bet than the consolidation is not yet finished, it's either a bigger fish acquiring the new venture or dbtran acquiring a catalog and/or an orchestrator. Dagster being, I think, a match in heaven.

If you're looking for an exil there are some alternatives when it comes to transformation: bauplan, bruin, lea and for ingestion: dltHub (and all the tools based on it).

Other writers

• You can read Hugo views on the matter here — he enters a bit deeper in the Iceberg / Open compute topic that I squeezed a bit because the post is already too long.
• On my views on LinkedIn after Day 1 of Coalesce.
• Poetic Benn views.

See you next week ❤️ (and this time it's real).

Small head's up, I'm organising the Forward Data Conference (2nd edition) on November 24th in Paris and we are cooking a great program!

The call for talk proposal is ending this Sunday (July 6th), so make sure to propose a talk this week if you wanna join this awesome moment! We are welcoming all level of speakers for everything about data, English and French submissions are welcome.

We are also looking for sponsors to make this event awesome and unforgettable! We have announced Omni as our first platinium sponsor.

And last thing. Big news, starting this Friday Data News are coming back! Be ready, I miss you.

Hey here. What's up? While you're all data vibing I'm sliding into your inbox with the fresh Data News of the last month.

I have moved to San Francisco for the next 3 months, so if you're in town and wanna talk data or go for a run, you know where to find me. It's been a week since we arrive with nao Labs team in SF and it has been a blast. We will be at the Data Council pitching at the AI Launchpad on the 22nd.

I'm planning to create content for you to follow the Data Council from the inside as it has always been great to write takeaways about the talks these last year (2023 and 2024).

AI News 🤖

• New OpenAI text-to-speech model — OpenAI released a new text-to-speech model, available through their API, it looks like better than the Whisper baseline. There is a demo website which is quite impressive.
• Llama 4 is out — Meta has released the new iteration of their open models. This time it includes 4 models:
  • Llama 4 Scout, a small 17B model. Natively multimodal, it achieves an industry leading 10M+ token context window and can also run on a single GPU.
  • Llama 4 Maverick a multimodal model, beating GPT-4o and Gemini 2.0 Flash across a broad range of widely reported benchmarks. It can also run on a single host.
  • And soon they will release Llama 4 Behemoth (outperforming GPT4.5, Claude Sonnet 3.7, and Gemini 2.0 Pro on several STEM benchmarks) and Reasoning.
  • Cool part is that they partnered from the beginning with Databricks and Snowflake to bring LLM to your data.
• Reasoning models don't always say what they think — A summary of a research about the faithfulness of reasoning models. Anthropic discover that these models, that are not really "reasoning" but more doing a Chain-of-Thoughts (CoT) are not always honest when they are given a hint (or hacked) about the the answer they should provide.
• How to roll-out a data conversational agent? — Gorgias engineering team released a conversational agent to the company using Dot which is a AI Slack bot that answer data warehouses questions. The article explains super well how Dot fits in the data team and how you can evaluate the AI.
• Google announces at Next'25 AI in their cloud products
  • AI.GENERATE_TABLE in BigQuery — The setup looks a bit weird because in order for everything to be accessible you have to have a model in a dataset and prompts in a table but this is a great way to extract informations in strings as per the example.
  • Talk to your data with Looker —BI tools are already widely deployed in most organizations, and conversational analytics serves as another interface to access the same insights. That's why embedding this "talk to your data" capability within existing BI tools is likely the best path for adoption. However, this approach might also introduce additional complexity, given that many BI tools are already somewhat cumbersome, cluttered repositories of charts.
  • Google announced their Agent Development Kit (ADK) and an open protocol to enable communication between agentic apps called Agent2Agent (A2A).
  • Also announced BigQuery AI engine that is doing something I don't understand completely: analysts could ask to extract info in a image and match it to a product catalog, and, a copilot for Jupyter Notebooks (in Collab) — more about analytics announcements.
• Lessons learned from building agent that can code like Composer — Petr wrote a great article that explains how coding agents are working (which is dead simple). In order to build the most simple version of a coding agent you need to give 3 capabilities list files, read file and write file. With this and a few prompt you can have a working demo in a few minutes, but how to move forward? He gave the 5 important lessons he got out of this.
• Oxy, an open-source agentic analytics framework — Robert and Joseph co-founded Hyperquery back in the days (at the peak of the notebook hype) and they are back with a new journey: Oxy. An open-source framework to create analytics workflows in a friendly way. Looks promising.

Navigate the MCP hype

If you've been on the internet lately you should have seen the massive MCP hype, everyone is either building a MCP server or a MCP registry or even a registry of registries.

But what's a MCP?

MCP means Model Context Protocol and is an open protocol created by folks at Anthropic. A MCP is a most of the time referenced as a server that encapsulate discoverable tools, prompts and data to be used by an LLM. MCP clients are on the LLM side and make requests to MCP server.

For instance there are a few Snowflake MCP servers, if you add them to Claude, you will be able to query Snowflake from a Claude prompt for instance, or get a table metadata.

• Everything a Developer needs to know about the MCP — If you want to deep-dive more there is this article about it.
• OpenAI supports MCP in the agents SDK — The biggest news recently is OpenAI validating the protocol and supporting it (not yet in ChatGPT tho).

Fast News ⚡️

• Overclocking dbt, Discord custom solution — Discord data platform is huge and they reached few dbt limitations, especially on the backfilling side (which is not really a dbt strength), so they built their own way to overcome this leveraging the meta tag. They also managed to create isolated environments for the whole team and have a bunch of CI/CD jobs running on each PR validating their own internal rules.
• Current state of Databricks SQL warehouse — Does Databricks SQL outperforms Snowflake?
• Deduplication in BigQuery, 7 ways to do it — If you're usually do deduplication in BigQuery (or elsewhere) here 7 patterns to achieve it.
• Ensuring data contracts adoption across an organization.
• The shift left data Manifesto — I did not read it because it's too long, but Chad have been a shift left advocate for a long time. Which mean "Shifting Left means moving ownership, accountability, quality and governance from reactive downstream teams, to proactive upstream teams". Putting in another way give software engineers the responsibility of the data.
• BI is dead, change my mind — It's Clickhouse director of engineering turn to say BI is dead, he got the light while chatting with Clickhouse using LibreChat + a Clickhouse and a Github MCP servers. Looking at how chat for everything is taking all over the place, it's only a few months until stakeholders asks for data interfaces using chat.
• Local data transformation with dbt and DuckDB — Great article showcasing how you can locally do all your transformations today with dbt and DuckDB, and we even got a great DuckDB local UI.
• Software is now content — I really liked this week Benoit's post.
• How we built a robust ecosystem for dataset development — Duolingo process to apply software engineering practices to data modeling, in a sense of datasets are assets that could be treated like APIs.

Navigating Iceberg landscape

Over the last month a lot of things happened also in the data engineering space, especially around Iceberg which is taking over a lot of discussion when it comes to data storage.

Why is Iceberg so important right now?

Iceberg is a way to escape the data warehouses to build your own warehouses in kit on-top of bucket storages. Iceberg being open-source it will allow us to build interoperability between all systems while supporting some kind of transactional systems on-top of Parquet files.

• The Iceberg Summit took place in San Francisco (but I could not go), tho Neelesh publish a small recap of the Summit. I guess the videos will be on the YouTube channel soon.
• I personally think that DuckDB might be the easiest developper interface to interact with Iceberg ecosystem, as it's dead simple to spin a Duck instance. Recently we got DuckDB to attach to Iceberg and the capabilities preview Amazon S3 Tables.
• Athena vs. Snowflake on Iceberg, performance comparison. In the end Snowflake won being 2x less expensive, the tests uses the engine on top of Iceberg datasets to see how they handle working with Iceberg. Would have been cool to compare it to the same using native tables.
• Data wants to be free: fast data exchange with Apache Arrow — How Arrow compares to Postgres when it comes to serialisation and why is it so fast?
• Cloudflare R2 data catalog — Cloudflare R2 is a global object storage (like S3) with free egress (meaning free data read from external system) which is a paradise for Iceberg as Lakehouse really heavily on data reads on buckets and most of the engine are living elsewhere. So Cloudflare announced a Iceberg catalog that can live close to your tables.
• Amazon reduces prices for S3 Express One Zone — Following Cloudflare announcement Amazon decided to reduce the price of the Iceberg offering.
• xorq, declarative, multi-engine pipelines — This new world opened by Iceberg brings us to the multi-engine data stack. Where we use different engines (Snowflake, DuckDB, BigQuery) for instance for what they are great about and store the underlying data in bucket using Iceberg unifying everything in a catalog. xorq is one of the first multi-engine pipeline system for ML use-cases.

Because we have to unify all the trends there is a Iceberg MCP server that has been developed.

Examples and thoughts

Just to go further and connect everything, a few post about the relathionship with Iceberg and the lakehouse and where all this fuzz is going, and what it could mean for your actual data stack.

• Iceberg?? Give it a REST!.
• We built a data lakehouse to help dogs live longer.
• Towards Composable Data Infrastructure.
• Roadmap: data 3.0 in the lakehouse era — 4 possible thesis of what could be the next revolution about your data stack.

My two cents about this: this is mainly experimental and this is not relevant yet for the scale most of the companies are. Warehouse + native tables is the easiest user experience you can find, and as data engineers what we want it users using our platforms, right?

Data Economy 💰

• OpenAI raises $40b at $300b valuation.
• Omni raises $69m Series B.

See you soon ❤️

Hello here ☀️

I feel ashamed for not posting any Data News for the last 2 months, a lot was going on and I did not manage to find time every Friday to write news, I'm so sorry about it.

Hello to all the new subscribers who arrived since January, I want to warmly welcome you ❤️. This is your first Data News ever, enjoy the moment, read whatever you feel curious about, at your own rhythm.

At the moment, I don't want to promise something about being back to our regular weekly schedule, but I'm trying as hard as I can to organise my new routines/life as a content creator and company founder.

I've always worked on multiple projects at the same time, but since I've started nao things changed. There's a truth you only grasp when you've lived it: you are thinking about your company all the damn time.

Events 🪭

While being less present online, I've done a lot of things in real life in the last weeks and I'll continue to do in the weeks to come. I was at the DuckCon #6 in Amsterdam to talk about yato, the smallest DuckDB SQL orchestrator and Robin published 3 podcasts episodes—in French—that I hope you'll listen while running this weekend 🤭:

• 3 data trends to follow in 2025
• A comparison between SQLMesh and dbt
• The 3 priorities of a VP data in 2025

At the end of the month, on March 31st, I'll co-organise the AI Product Day in Paris, we are sold-out, but we still have slots for sponsors if you want to help us organise the event and get massive visibility to AI and product teams.

I'm going to Barcelona 🇪🇸 (from March 19 to 22)—I'd love to hangout with data people there. I'll give a talk to the French Tech Barcelona on March 20, you can register here. I might plan a day-trip to Madrid (?).

The biggest news as a data fan is also that I'll be at Data Council this year as your news reporter on duty 🤓. So if you plan to got, or if you're in San Francisco around April, let's have a coffee.

AI News 🤖

The pace of change is nothing short of extraordinary. I haven't published in two months, and it feels like two years. Here's a recap.

• Timeline of the major models releases. When I say major it's obviously subjective. It's mainly related to the noise they made online.
  • phi-4 — Microsoft continues to release their small open models. I never came across someone using it, however.
  • DeepSeek R1 — DeepSeek is a Chinese startup building foundational models, they released v3 previously, then R1 which is a reasoning model that made OpenAI and American AI companies panicking because they claimed major cost reduction in model training. More, DeepSeek code and models are open-source with MIT license. R1 is built on top of v3 using reinforcement learning combined with chain-of-thoughts (CoT) to "reason".
    
    HuggingFace created open-r1 a fully open (for what it means) version of r1, in Python where every step is detailed.
    
    There is also a good analyse of DeepSeek vs. the world.
• Mistral Small 3 — A small model that can be use to do CoT, under Apache Licence.
• Google Gemini Flash 2.0 — Multimodal reasoning.
• Anthropic Claude 3.7 — Claude 3.5 has been by far the most used model for code generation for the last 6 months. 3.7 should be a uplift, and to be honest I feel it's not. They also release Claude Code, an agentic coding tool that you can use in command line to make changes, commit and fix issues. Simon, created the pelican bicycle test, which is fairly good to evaluate models.
• Alibaba Qwen2.5 — Nothing much to say to be honest.
• Open AI o3-mini — OpenAI fast reasoning model series.
• Grok 3 — Musk fans says it's the best model.
• OpenAI GPT 4.5 — One month later OpenAI released GPT 4.5, and I feel like a teleshopping presenter. Now we have a selector with 6 models in ChatGPT, I'm kinda lost to be honest. There is also the pelican test.
• OpenAI deep research — OpenAI mode to replace McKinsey consultant or Phd people, because why not.
• Mistral OCR — The promise is crazy, you can have a quick look at their examples, from a pdf or a photo it can extract information so you can use it. It's even "multimodal" because it keeps the figure in the output.
• ChatGPT for MacOS can interact with your code (in the IDE) — in the demo it works with XCode or VS Code and directly changes the files on disk so they changed in your editor.
• Claude 3.7 plays Pokemon on Twitch — finally something useful.
• I'm not a Perplexity user but I see more and more people switching their Google search usage to Perplexity, which announce a new web browser called Comet and deep research. Deep research has been built to generate ideas, summaries or takeways.
• Streaming AI agents: why Kafka and Flink are foundations — A small bond with the data engineering world.
• Building effective AI agents — This is a great article from Anthropic if you wanna learn how to build AI agents. It explains well the flow between the user, the UI and the LLM.
• 📺 Deep dive into LLMs — 1.5m views and have been recommended by a lot of people, should be good (did not see it). Goes from GPT-2 to DeepSeek R1 and give a mental model of what it is.
• RAG stuff
  • Forget RAG, welcome Agentic RAG
  • RAG is dead long live RAG.
  • From RAG to richness: how Ramp revamped industry classification.

dbt Core and SQLMesh, wat 🧭

dbt Core has become one of the most used tool across data teams all around the world. Because of its success, companies might feel the dbt fatigue which happens when your dbt project has been a success but widely spread within the company leading to A LOT of tables—we call it models, the dbt way.

When you have a lot of tables, dbt projects tend to become less manageable, the CLI becomes slow, the local development experience isn't great and more and more features are going into the Cloud version. SQLMesh has been created to fix dbt Core issues and to compete with dbt Cloud.

A few weeks ago, dbt Labs acquired SDF—which I had been watching closely for more than a year, see DN#24.07. SDF is a Rust binary that understands dbt projects and speeds up everything, making up to 100x gain in performance. Under the hood SDF, parses the SQL queries, gets syntax trees, compile and executes them to find issues even before they hit the data warehouse.

We will know very soon what this acquisition will bring to dbt and we all praise for the best improvements to be in the open-source codebase, (spoiler: not sure).

On the other side, SQLMesh answered with the acquisition of Quary, a Rust knowledgeable that made significant improvement to SQLGlot the underlying SQL parser of SQLMesh.

There is a fierce competition between the 2 companies and shots are fired. SQLMesh team is also organised GROUP BY their annual conference in a few days, any resemblance to another event is fortuitous. This week Tobiko also published a benchmark claiming that SQLMesh on top of Databricks delivers 9x cost saving.

Time will tell where this leads, but ultimately it will benefit data professionals as they strive to build the best SQL orchestrators. However, I believe there are still unresolved issues with the developer experience in the age of AI—challenges that I'm actively working to address with nao 🤭.

Fast News ⚡️

• Benchmark your data team — Mikkel has been a great contributor to metrics about data teams worldwide: size, ratio to software engineering, team composition, salaries. This time it's a dynamic website where you can explore all these metrics to compare your team with what's out there.
• The myth of measuring data team ROI — ROI of a data team is one of the most difficult thing to measure. Hex view on this is to ask other to tell the ROI for you, especially via a NPS of your users.
• Fivetran and Airbyte pricing changes — The 2 data ingestion services changes their method of billing. Fivetran is doing thing I don't understand but they have charts explaining and Airbyte switched to capacity-based pricing—which means it's based on the number of pipelines your run rather than the volume you move. Benjamin analysed the pricing changes on LinkedIn, it's a competitor perspective.
• What is a flat file? — A large article explaining all the flat files format. I do not miss fixed-width files.
• Query engines: gatekeepers of the Parquet file format — DuckDB team is unhappy because most of the query engines are not supporting the latest Parquet advancement forcing the duck to write old spec, which lower performance.
• How we migrated to Iceberg using Athena, Trino and Spark — How you can plan a migration to Iceberg. It lasted 4 months and reduced the data volume from 70TB to 40TB.
• How not to partition data in S3 — You should partition by folder/date=2025-03-08, rather than with subfolders (sorry American readers, we put the months before the day 🙃).
• Polars launches Polars Cloud — Run stuff remotely, why not 🤷‍♂️, on your own Polars cluster. Looks like a 2025 Spark.
• Tobi launched a DuckDB newsletter: learning DuckDB by example.
• DuckDB goes distributed — DeepSeek released smallpond a lightweight data processing framework built on DuckDB and 3FS—their distributed storage tech. smallpond is an alternative to Daft or Spark. I'm skeptical, distributed processing is not really the initial purpose of DuckDB which is made to remove the communication burden between client <> server through single-node processing.
  
  📅 Mehdi is organising an online event about Scaling DuckDB.
• Count.co combines DuckDB processing in-browser and on the server.
• uv is becoming a thing, how to use it in PyCharm — uv is a Python package manager written in Rust, that aims to fix all the issues we all faced one day. uv brings also on-the-fly package management for script, which is freaking cool.
• Building robust CI/CD pipeline for dbt — Ideas of things you can put in your CI to test your dbt projects before production. Even though I'm personally convince that the CI arrives too late in the process and that it should be done even before you push, this is a great start.
• Minimising the runtime of a SQL DAG — What if you could theoretically save time in your SQL DAG by looking at the duration and the dependencies? This is what Max did and he found 26% uplift in performance. This guy never stop to amaze me.
• VS Code extension for Google Dataform users — For the first time in 2025 I've met Dataform users, this is cool if it brings another alternative to the table. Tho, it's strictly coupled to BigQuery. Dataform, is dbt but for BigQuery with another syntax (SQLX).
• Looks like BigQuery is getting a git integration — BigQuery is getting a lot of new features over the last months: notebooks, lineage, data profiling, etc.
• A head of data take on AI code editors — Cursor and Windsurf as AI code editor are everywhere and a lot of engineering teams are starting to use them, but what's the equivalence for data teams? How can we as data works benefits these innovations?
  
  the post has been written by my co-founder Claire—it's her first post, send her love ✨
• How dlt enters the AI code generated pipelines world — dlt is becoming the best Python toolki to ingest data into whatever destination. In this AI assisted code writing era, because dlt is just code, it means MCP or other LLMs can really shine into help data engineers writing ingestion pipelines.
• SQL related stuff
  • Does Ibis understand SQL?.
  • Beyond SQL as a pure database syntax.
  • SQL is all you need.
  • Detecting similar SQL queries with vector search.
• 📺 Graph databases after 15 years?

🕵️ What if you could become a SQL detective: SQL Noir. It's a funny game to practice you SQL skills to solve mysteries.

Data Economy 💰

Become it's already too long, only headlines.

• Anthropic has raised $3.5b Series E.
• Upsolver have been acquired by Qlik.
• fal.ai raised $49m Series B.
• Groq gets $1.5b funding from Saudi Arabia.
• dbt Labs acquired SDF and dbt Labs reached $100m ARR.
• Tobiko acquired Quary.
• Databricks raised $10b Series J, and $5b more in debt.
• Hightouch raised $80m Series C.
• Eleven Labs $180m Series C, at the same time they released Scribe, their new cloud-base speech to text model.

Sorry for the large edition, I also feel a bit rusty after 2 months not writing. See you soon folks ❤️.

Happy new year ✨. I wish you the best for 2025. There are multiple ways to start a new year, either with new projects, new ideas, new resolutions or by just keeping doing the same music. I hope you will enjoy 2025.

The Data News are here to stay, the format might vary during the year, but here we are for another year. Thank you so much for your support through the years.

Some personal news:

• I will be in Amsterdam for the DuckCon on Jan 31, I'll give a 5 minutes talk about yato, if you're also going or living there, reach out so we can chat!
• We announced the AI Product Day, a 1-day conference that will take place in Paris on March 31. It will be a day dedicated to product teams who want to fully exploit the potential of AI. We are looking for sponsors and the ticketing is open. I have a 15% discount code if you're interested BLEF_AIProductDay25.
• We published videos about the Forward Data Conference, you can watch Hannes, DuckDB co-creator, keynote about Changing Large Tables.
• Over the past four weeks, I took a break from blogging and LinkedIn to focus on building nao. It was refreshing to recharge and kick off the year, there’s nothing quite like diving back into the joy of hacking and creating.

Let's jump to the news, and have fun reading, it's a large wrap of everything that happened at the end of the year + how 2025 started.

AI News 🤖

The current economic uncertainties are affecting the tech and data worlds. Meanwhile, the AI landscape remains unpredictable. AI companies are aiming for the moon—AGI—promising it will arrive once OpenAI develops a system capable of generating at least $100 billion in profits.

• Why AI progress is increasingly invisible.
• It's happening, leaders wants AI agents to take the job of human employees. NVidia CEO said "IT department of every company is going to be the HR department of AI agents in the future" (cf. Keynote video) and Firecrawl, a tool for turning websites into LLMs, posted a $15K job for AI agents. Actually a modern Kaggle for Agentic AI, in the end it's a mechanism to lower human labor cost, because spoiler human will code to create these agents.
• Agents — Chip Huyen wrote a very large guide about AI agents, this is very detailed, it covers the necessary tooling, how planning works with agents and how you evaluate them. This is great quality material to be honest. There is also a Google introduction video about AI Agents.
• Don't do RAG, use CAG — A paper about another way to think about the information retrieval for AI knowledge tasks. The goal is to use a key-value (KV) cache that eliminates latencies issues traditional RAG might occur.
• Does size matter? — Paper written by Gael Varoquaux (sklearn), Meredith Whittaker (Signal) and Alexandra Sasha Luccioni (HuggingFace) about the negative impact of the bigger-is-better paradigm. It's easily readable (mildly large ~10 pages) and gives metrics about the performance plateau that we start to see at scale.
• A large international scientist collaboration released The Well: 2 massive datasets from physics simulation (15TB) to astronomical scientific data (100TB). They aim produce the same innovation as ImageNet produced for image recognition.
• Models news and tour
  • DeepSeek-v3 — It entered the space with a bang. DeepSeek is a model trained by the Chinese company with the same name, they directly compete with OpenAI and all to build foundational models. They released v3 in open-source and it outperforms every other models.
  • OpenAI o3 — OpenAI announced their advanced reasoning model called o3, that can achieve large task. o3 is kinda a waste of energy when you look at the numbers estimated carbon impacts (estimated via kWh), based out of François Chollet research on ARC-AGI benchmarks.
  • smolagents — HuggingFace released a barebones library for agents. Agents write python code to call tools and orchestrate other agents.
• ❤️ Transformers laid out — The best article out there to understand Transformers (which are key to understand LLMs).
• Things we learned about LLMs in 2024 — I discovered Simon content during Christmas break and to be honest it's one of the best. He compiled a list of things we learned in 2024 about LLMs. This is a must-read.

Fast News ⚡️

• O'Reilly 2025 tech trend report — Every year O'Reilly releases a report based on their skills platforms search. From the traffic they get they draw market trends. A few things to notice:
  • Interest in AI grew by 190%, Prompt Engineering by 456%.
  • Python and Java still leads the programming language interest, but with a decrease in interest (-5% and -13%) while Rust gaining traction (+13%), not sure it's related, tho.
• Read the pdf version directly. Not really digest.
• The limits of data — The articles argues that while data's universality offers significant power, it often sacrifices contextual nuances, leading to oversimplified representations of complex human experiences. (Generated by AI as the article is too large to read atm for me).
• The hidden cost of over-abstraction in data teams — It's an interesting take that about the layer of abstractions we tend to create to build data platforms, whereas it's dbt Macros, CLI wrappers, etc. in the end it often adds more complexity than it removes.
• Designing a declarative data stack: from theory to practice — Related to the previous article Simon wrote a great article about the things to have in mind when we build a proprietary DSL for a declarative data stack. Meaning: a YAML configuration system for ingestion and transformations, and now, visualisation with BI-as-code.
• Lessons learned implementing Metrics Trees — An article in a form of an interview about someone who implemented Metrics Tree, mainly it's not about the visual representation but more about the process to translate business needs into an equation (I mainly see the tree as an equation).
• The evolution of OLAP — What is OLAP in the modern data stack? As Cube is preaching for their tooling, obviously the semantic layer is the OLAP layer.
• Hybrid Kimball & OBT data modeling approach — This is maybe the most common setup I've seen the last 3 years. A combination of Star schema with OBT in the marts for ease of consumption.
• Analytics engineering at Netflix — (and part 2). A internal survey of analytics engineering practices at Netflix. They developed a /data command internally that answer questions about everything and structured the analytics around a foundational data platform with company-wide analytics data layer that provides time series efficiency metrics across various business use cases.
• The future of data querying with Natural Language — What are all the architecture block needed to make natural language query working with data (esp. when you have a semantic layer).
• Hard data integration problems — As always Max describes the best way the reality. He listed 4 things that are the most difficult data integration tasks: from mutable data to IT migrations, everything adds complexity to ingestion systems.
• Materialization of data warehouse layers — What are the consideration for every materialisation you should pick in your data warehouse layer: view, tables, schema vs. databases, etc.
• The software development lifecycle within a modern data engineering framework — A great deep-dive about a data platform using dltHub, dbt and Dagster.
• The best code is the code you never wrote — Every line of code is a form of debt—a liability that must be maintained and understood. As we move toward a future dominated by AI-generated code, the balance will shift dramatically, making human-written code an increasingly scarce and valuable resource.

• Canva Snowflake journey:
  • Our journey to Snowflake monitoring mastery.
  • Continuous data platform with Snowpipe Streaming.
• DuckDB vs. coreutils — A side by side comparison of DuckDB counting features with grep and wc.
• Amazon S3 Tables — Amazon released S3 Tables, an out-of-the-box support of Iceberg within S3 a few weeks ago. It came with a bang but with a few concerns.
• Database in 2024, a year in review — Mainly last year was about licensing issues, Databricks vs. Snowflake and DuckDB trying to decrown pandas as the default.
• How Uber is saving 140,000 hours/month using text-to-SQL — They developed within their tooling QueryGPT that helps Uber employees find the best stuff according to their needs. They have a table agent that list the best tables according to the user intent.
• A tool to make data stack diagram — Great tool to do stack diagram.
• Staff Engineer vs Engineering Manager — One of the best article about the topic, when it comes to expertise staff engineers are still rare. How should we structure the career ladder to make it work.
• Claude Sonnet 3.5 within Snowflake Cortex.
• Journey with Apache Flink & Flink CDC.

I still don’t think companies serve you ads based on spying through your microphone.

Economy 💰

• Boomi acquires Rivery in a $100m deal.
• Snowflake acquires Datavolo.
• Databricks raises $10B towards an IPO.
• Cursor raised $100m in Series B.

I missed you ❤️ — and see you next week.

Hey, it's been a few weeks since something has been published here—I hope you haven’t forgotten about me 😊.

In the last weeks I've been all over the place and worked on a lot of topics except this newsletter, I've decided to take a break from the newsletter to catchup the rhythm in January!

The Forward Data Conference was a huge success and I want to thanks again all the attendees, speakers, sponsors and my co-organisers. I can't wait to work on next year edition. I also organised the first dlt Paris community meetup and I plan to do some more events like this next year because I really like this IRL part of the content sharing.

I'm also switching from freelancing to building a company and I can't wait to share more about it.

About the Data News, I'd like to use the blog as a place for other to share stories, so if you wanna write or co-write plain articles with me, hmu.

I wish you all the best in advance, and see you next year ✨.

It's Data News time. Time really flies on my side, and apart from the bad news from across the Atlantic, all is well on my side. To be honest, I miss you folks. Writing here has been my little thing for the last 3 years and because I haven't been able to get back to my previous frequency since July, I feel empty every Friday.

I'm back in Paris and, wow, the way I live my life in Paris is so different from Berlin, Paris demands speed at every level. I've only been back 6 weeks and I feel like I haven't even left the last 2 years. I haven't yet settled back into my routines by jumping between all the hats I've decided to accumulate over the last few years: content, freelancing, founder and conference organiser.

I don't know when I'll be able to start writing here once a week again, but I'm doing my best to do it as soon as possible.

Enough.

On November 19th, I'm organising with dltHub folks the first dlt Paris community meetup, the event will take place at 42 and will start a 16h. It will feature:

• Navigating the complexities of enterprise ELT, towards data democracy and cost efficiency
• Me — Towards a simple future (dlt, DuckDB, yato and more)
• dltHub CTO, Marcin Rudolf — A teaser of the upcoming dltHub "Portable Data Lake"
• Lightning community talks (reach out if you want to present something)

It will be only a few days before the Forward Data Conference that we sold out a few weeks ago, the program is out—the official schedule is coming soon. We are very proud and honoured, along with the organising committee, that around 300 people took a paid ticket to the event. I'm sorry for those who weren't able to get a ticket, we've set up a waiting list and we're doing our best to find a way to push the walls.

I would also like to thank the sponsors who are accompanying us on this exciting adventure: Castordoc, Omni and Corail Analytics, Mirakl, SYNQ, nibble, Sparkline, Monte Carlo.

Whether at the dlt community meetup or at Forward, I look forward (no pun intended) to meeting you all.

AI News 🤖

I'm a bit offended, AI news is not what it used to be 🙃, we were used to more exciting news, competition and drama by the space.

• ChatGPT Search — OpenAI finally plugged ChatGPT to internet and live data. You can now switch on the web logo and ask for model to search on the web alongside his training knowledge. When you mix it to the new Canvas UI ChatGPT look more and more like Google Search results.
• HubSpot co-founder bought chat.com earlier this year and sold it to OpenAI for shares [via Oliver].
• Claude Computer Use — Like a soufflé, everyone was hyped when Claude released Computer Use, a chat interacting with an operating system, but a few days after it looks like almost everyone forgot it, like the 01 interpreter. If you wanna try Computer Use (at your own risk), there is a repo with a Docker image launching a VNC and a Streamlit app—it works fine.
• New Mistral APIs — a batch API, for batching calls rather than doing it synchronously lowering by 50% the costs and a moderation API which is a 0-categories classifier scoring text into intent.
• IBM released lightweight open-foundation models — 2 sets of "small" models: 2B / 8B dense models and mixture-of-experts 1B / 3B models. IBM has proudly shared the datasets they used to train their model.
• Skrub: Less data wrangling, more machine learning — skrub is a preprocessing / feature engineering library for tabular machine learning. The video emphasis something critical, even if we often talk about training impact—time, carbon footprint—we tend to forget that inference is also a critical part and more importantly because of the preprocessing. So your preprocessing matters.
• Standford, "Building Large Language Models (LLMs)" — 1h44 of a Stanford class about building LLMs. I did not watch it, but I bet you gonna learn at least a thing watching it.
• Generative AI’s Act o1 — Sequoia essay deep dives into the current state of LLM apps and infra which has been stabilised around Microsoft/OpenAI, AWS/Anthropic, Meta and Google/DeepMind. The text touches fast vs. slow reasoning, System 1 and System 2 thinking, while in the awaiting the all-mighty AGI to come plunging us in a new technology era.
• Apple proved LLMs do not reason (at least Mathematically) — Did we actually needed Apple for this? Because it's actually a design nor a bug or a feature. Apple researchers published a paper saying: "our work underscores significant limitations in the ability of LLMs to perform genuine mathematical reasoning". No way.
• The future is robot — In the recent weeks a lot of new robots were (re)-announced with eventually new features like Humanoid robots, Tesla Optimus, Boston dynamics. Happy to learn that we really need robots to fix the jobs market.

Fast News ⚡️

• Russia fines Google $20,000,000,000,000,000,000,000,000,000,000,000 — because a few Russian YouTube channels that have been blocked 2 years ago are still blocked.
• Why is Kafka always late? — A great large article about Kafka concepts about what Time is, and which configuration lever can be moved to fix eventual issues.
• The path to vendor-agnostic data platforms — The Iceberg trend and the competition between Snowflake and Databricks might be scary for the future, thinking about vendor-agnostic data platforms based on open-source technology is becoming a thing again. dlt + DuckDB is a combo that bridges a lot of gabs.
• Snowflake and Databricks finger pointing like kids — Recently both CEO played the card of "we are better than them and we are so different". Someone wrote a comparison side-by-side with numbers. Regarding performance if you're going the Iceberg way here a benchmark with Snowflake.
• Analytics-optimized concurrent transactions in DuckDB — Technical writeup about the concurrency concepts at stake within DuckDB and a good Hannes interview.
• DuckDB inside Postgres!!?? — Daniel Beach really showcased what's pg_duckdb and out-of-the-box performance. He noticed that pg_duckdb performances are slower than raw Postgres. Main reason is because the extension does not support indexing yet.
• Data warehousing is dead — Actually the article is saying the opposite, with all the fuzz around lakes, real big data warehouse with HUGE DATA are still out there. Funny article.
• Why Microsoft Excel won’t die — A small reminder for me (and my co-founder).
• Python becomes the most popular language on Github — Who would have guessed. According to Github Python overtook Javascript as #1 language. A lot of metrics are shared in the post like the number of 5B of contributions in 2024 in Github 🤯.
• Github announced Github Spark — Can we enable anyone to create or adapt software for themselves, using AI and a fully-managed runtime? With a natural language based editor, everyone can spark and application ✨.
• The road ahead: What’s coming in Airflow 3 and beyond? — Keynote from Airflow Summit about Airflow moving towards assets and more.
• Managed Polaris by Snowflake — The open source Iceberg catalog can be managed with Snowflake UI.
• Fivetran is now in Snowflake marketplace — Tomorrow, the warehouses will ingest themselves their data, Fivetran can be used from Snowflake marketplace.
• dbt runs in Fivetran becomes a paid feature — you start to pay when you run more than 5000 models a month. In my local community in France I don't know a lot of company orchestrating dbt this way.
• Ingestion performance comparison — dlt compared the performance of themselves against other tools. Tbh, I'm quite impressed by dlt perf.
• dbt Coalesce 2024 — Sorry, I wasn't able this year to pull a large recap of all the Coalesce talks, still in my todo tho. Gleb wrote about it on LinkedIn ; mainly the dbt Explorer + mesh becomes more complete, a new visual experience (like Alteryx or Knime) to write models and Iceberg. All of this illustrates dbt Labs strategy : targeting large companies (visual editor for non-tech people) and a move towards emancipation of the warehouse with Iceberg.
• dbt: incremental but incomplete — A critic of the new dbt microbatch incremental feature, it works with Postgres, BigQuery, Spark and Snowflake. If I understand it right microbatch are a way to batch your large incremental queries by a time dimension that you specify—see it as chunking.
• Loading data into Redshift with dbt — lmao it's been ages I did not heard about Redshift, so I wanted to do a shout out.
• dbt-column-lineage-extraction — A Python CLI tool from Canva team.
• BigQuery's AI-assisted data preparation — Do data preparation in the fresh new BigQuery editor with natural language description of the transformations.
• Enabling and disabling Studio in Looker  — Looker Studio will become a feature that you activate / deactivate in your Looker setup.
• ❤️ In the era of GenBI — one of the best post of this week.
• Lightdash raises $11m — I really like Lightdash, it's a promising great mix between a BI tool, a semantic layer, as code.

Food for thoughts

• Docker for data products.
• Why Agile and Product Management fail with Data & AI?
• Who’s really responsible for team failures?
• Empowering Engineers with AI at Slack.

See you soon ❤️

PS: for our product research with nao we are looking for analytics engineers working on modeling daily. If you fall into this bucket, answer by saying "hi i'm an analytics engineer" and we will follow-up on this 🤗.

Hey, hey, hey. I'm so sorry for this small break about the news. I was in middle of starting my new company, nao, and moving back from Berlin to Paris. Still I hope this edition finds you well, it will be a mix of personal news, OpenAI saga and usual data engineering stuff that I enjoy reading.

First things first, yes, I'm co-founding a company. We called the company nao and you can see it as a no-code semantic layer. Still I keep a post about it for later, but if you're interested, hmu.

Then, with my girlfriend we decided to move back from Berlin to Paris after 2 years there. It's a professional move for both of us, we will miss Berlin to be honest but a big part of our social life is in Paris. Being in Paris will ease all the events and IRL stuff I go / organise.

Forward Data Conference ✨

As a reminder, on November 25th I'm organising the Forward Data Conference. It will be a day to shape the future of the data community, where teams can come to learn and grow together. There are still tickets left—we sold around 80% of the tickets.

This week we announced the program, you can find it on our website. I really like the program we put in place, it a mix of Engineering and Strategic / Vision talks.

The conference will be held in French + English, a few talks will be given in French but we will subtitle them live and we will also find a way to always have something in English in parallel for all English native speakers.

You can use BLEF_FWD24 promo-code to get 15% reduction on your ticket.

PS: dear readers, if you proposed a talk to the FDC which has been rejected, I'm so sorry you did not get a detailed explanation, we received a lot of talks and I wasn't able to write a personal message to every talk that has been rejected. Tho, if you're wondering why, reach me and I will explain you.

AI News 🤖

• OpenAI is our best saga about drama and tech, when the Netflix show is going out?
  • DevDay recap — OpenAI DevDay was the developer conference to announce features, models and stuff about their product. The "biggest" announcement was around Realtime API targeting the speech-to-speech applications.
    
    In addition they introduced prompt caching to save tokens costs, the possibility to fine-tune vision for GPT-4o. Last thing is Canvas, which is a new way to interact with the models, I'd say it's a mix of Notion and Anthropic better UI. This is mandatory for OpenAI to improve and diversify their public UI/UX in order to compete with large apps ecosystems.
• Whisper large v3 turbo — New turbo version of Whisper has been released on Hugging Face (announcement). Following Realtime voice API, it's great to see improvements in Whisper, the voice model.
• OpenAI to remove non-profit control and give Sam Altman equity — After a magic trick, Sam could receive equity worth around $150b. The important note is also that OpenAI is moving it's core business to for-profit which will not be controlled anymore by the non-profit board.
• Advanced Voice not available in EU — Advanced voice is a Siri interface on top of Chat-GPT capabilities. The unavailability in EU is lobbying at it's finest, fearing AI Act or GDPR could harm innovation. Explain to me why companies with the best engineers in the world can't find a way to make things legal.
• They raised $6.6b at $157b valuation (and $4b in debt). Another $10b after the first in Jan 2023.
• Meta, if there was a race, Meta would be well positioned, who would have thought after Metaverse choices?
  • Meta Movie Gen — Meta announce new research for movie generation models. Let's be honest for the moment it just feels unreal, like a video game or something in virtual reality. But in the end, this is maybe what we need?
  • New hardware (powered with AI) — Two promising product have been demonstrated a pair of glasses and a wristband that allows you to interact with virtual interfaces with your finger movements.
  • SAM 2, Segment Anything Model 2 can run on-device on Apple CoreML — A demo of image segmentation that run 100% offline and on-device. Industrial application might easily follow out of this.
  • Mark Zuckerberg says leaders should have technical skills if they want to call themselves a tech company. Yes, but technical leaders are also sometimes not the best ones, maybe the crazy ones, so other skills are required.
• Introducing contextual retrieval — Anthropic introduced a new way to do RAG with more context, that performs better than standard.
• Meta and Google announced automatic dubbing for resp. Reels and YouTube videos, this is something. Translation looks like a use-case that is almost solved with LLMs. It unlocks a world where languages are not anymore barriers, giving us access to instantly content and discussions all around the world, especially if it can run on-device, cheaply.
• Web browser automation through agentic workflows — A Github repo with a demo using Gemini and Selenium to automate browser actions.
• New AutoGen architecture — AutoGen is an open-source programming framework for agentic workflows, they designed a new architecture (to be honest I don't know what it means).
• Klarna drama — Klarna CEO announced he will shutdown Salesforce and Workday to replace it with internal initiatives + AI. Let's see where it goes.
• Paris police wants to keep AI surveillance in place post-Olympics — Who could have predicted?
• Malt AI report — Malt is a French / European freelance marketplace and they dropped their new AI report. A few things I can note going through the report below.
  • Snowflake demand has largely increased and it's close to Databricks in volume, tho Hadoop demand is still larger 🙃
  • The biggest demand concern stuff around AI like LLM, Deep Learning, Machine Learning, scikit-learn, etc. — in 2024 there are 16k AI freelancer profiles
  • dbt pops out as a specific skill on freelancer profile
  • AI engineers and scientists have an average daily rate around 500€, which is 100€ more than tech and data general category.
  • AI supply is half data scientists half all other tech positions (DA, DE, Back-end, SE, DevOps).

Fast News ⚡️

• CfP for DuckCon in Amsterdam on January 31, 2025 — In January next week, the DuckCon will take place, the call for paper is still open until Oct 18th. I might propose something about yato (?).
• dlt goes 1.0.0 — dlt announced their 1.0.0 version, as well as 1000 open-source customers in production. This version brings stability and marks a new milestone for the library.
  
  Side note, I'm a dltHub investor.
• Airbyte is also going 1.0 — Following dlt (?), Airbyte is also going 1.0 with 3 objectives more use-cases, reliability and better throughput performance.
• ❤️ NO SLIDES conference — Be careful before clicking on this link you might loose yourself in a loophole. Recently Timo organised a NO SLIDES conference, a conference where people would only share their screen and no slides. I participated to demo nao, but the demo failed, so the recording does not exist anymore (oups), still I've watched other few talks and really enjoyed.
• ELT with Kestra, DuckDB, dbt, Neon and Resend — How with Kestra you can create a declarative data pipelines to move data using the trendy libraries.
• DuckDB is the foundation.
• Fast feedback when SQL writing — A nice experiment showcasing how writing SQL tomorrow would look like. Imagine getting results directly while typing to have a faster iteration loop.
• BigQuery jobs explorer refreshed — Google team released a fresh new explorer for BigQuery Jobs.
• Coursera and Joe Reis launched a Data Engineering Professional Certificate — I can't recommend Joe enough, he's one of the best when it comes to capture date engineering job and the syllabus is great.
• Current state of Databricks SQL — "The best data warehouse is a lakehouse", lmao. Episode 21425325 in the competition between Snowflake and Databricks.
• The data death cycle — 5 traps you wanna avoid to deliver value with Data & AI products: the tech trap, the doing trap, the project trap, the silo trap and the performance-first trap. And follow-up about silos by Hugo.

No comments

Mainly because of time and length of this issue.

• Is Excel immortal? — Benn
• Evolution of catwalk: model serving platform at Grab.
• How HomeToGo improved our Superset monitoring framework.
• The importance of clear software requirements.

Data Economy 💰

• OpenAI raises $6.6b at $157b valuation. Softbank goes in with half a billion.
• Supabase raises $80m Series C. It's an open-source Firebase based on-top of Postgres.
• Kestra raises $8m Series A. Kestra is an open-source orchestration engine, written in Java, and you create workflows using a declarative model. Ludovic the CTO wrote about turning an open-source project into a viable business.
• fal.ai raises $14m Series A. For the readers that are here for a long time you might remember fal.ai, they were the first to propose a way to mix Python and dbt models with a specific tooling, and they pivoted into a super-faster GenAI inference platform.
• NVidia acquires OctoAI.
• BlackRock and Microsoft plan $30bn fund to invest in AI infrastructure.
• Voltron Data laid off 50+ employees recently. Voltron engineers are one of the best when it comes to under the hood engines powering our modern data platforms.
• :probabl. raised €5.5m Seed round. probabl is the official operator of the scikit-learn brand and will develop products and services around the library. Because we need the data science tooling to be and stay open-source.
  
  Side note, I'm a :probabl investor.

See you soon ❤️

Hey you, can you believe it's already September? This year has been flying. It feels like I just blinked, and here we are. In August, I've been focusing mainly on my next big journey—if you follow me on LinkedIn, you might have caught a sneak peek! I'll be making a full announcement next week. I want to take the time to explain my thought process and ideas behind it. I hope you will like it.

Below are the Data News wrapping summer and the first two weeks of Sept.

AI News 🤖

• OpenAI released 2 new models OpenAI o1-preview and o1-mini — These models brings changes and a breakpoint in the models naming. OpenAI decide to give up on the GPT naming, which means GPT-5 will never be plugged in. GPT paper has been co-authored by 4 person and 3 are not anymore at OpenAI, leaving GPTs also mark a change in paradigm.
  
  The o1 series brings more “reasoning”, it looks like a pre-prompt that does a chain of thoughts on top of what they already did best. Lots of stories about exceptional things the model can do have been published today—e.g. in the OpenAI system card explained that the model was able during a cybersecurity challenge (a CTF) to understand a failing Docker environment (due to infra) and still be able to find the flag.
  
  Here a YouTube playlist demonstrating o1 capacities.
  
  As clem mentioned on Twitter, it's always important to pay attention to words, even if the “reason” model, it doesn't think, it processes.
• More news about OpenAI
  • New models are already available on Azure ; but be careful Microsoft open-source Phi-3.5-mini is out.
  • Ilya Sutskever, previously Chef Scientist at OpenAI, raised 1b$ to co-found Safe Superintelligence with a manifesto.
  • Alexis Conneau, Her ex-research lead at OpenAI, decided to create a new company and got a lot of Tweet impressions. Previous OpenAI members are quite popular when it comes to founding.
  • Bloomberg reported that OpenAI seeks to raise $11,5b more at $150b valuation, making it the third private company in terms on valuation [paywall article].
  • NEO Beta, a humanoid company backed by OpenAI, released a first video demo. And it's impressive (🙃), the robot is able to handover a bag to a human!
  • We hope next OpenAI model is not o7. /s
• OpenAI and Anthropic will give their model first to US gov (NIST) to help advance safe and trustworthy AI innovation for all. But they cry when in Europe the AI Act is voted threatening innovation.
• NVidia released Eagle a vision-centric multimodal LLM — Look at the example in the Github repo, given an image and a user input the LLM is able to answer things like "Describe the image in detail" or "Which car in the picture is more aerodynamic" based on a drawing.
• Aleph Alpha introduced Pharia-1-LLM — it's a 7B model and the license is explicitly targets non-commercial and research usages. Aleph Alpha is a German company, funded by German VCs (with $500m), was trying to compete with US companies (like Mistral and OpenAI 🤭) in the models race but gave up this competition to pivot to a AI-support company for public sector.

Fast News ⚡️

• EU and China launch cross-border data flow communication mechanism — It's an official statement saying that EU and China will re-discuss the policy about the data transfer out of China for European company, which is difficult.
• Building a cost-effective analytics stack with Modal, dlt, and dbt — A great example of how you can built a small analytics stack in today's world with dlt, dbt and Modal, a serverless platform to run Python stuff. The articles contains a lot of code snippets to understand what's under the hood.
• Data teams survey 2024 — Jesse Anderson released the results of his survey about the state of data teams in 2024.
• Daily cache implementation in Python — A highly effective approach for caching when working with large datasets stored in distant buckets is to implement a local cache. It avoids the need to repeatedly download the data.
• Safe composable Python — A good article about function composition and testing in Python and how it articulates together.
• What are the key part of data engineering — Simple way to present what are the key part of data engineering.
• Automation strategies for monitoring and self-healing of data pipelines — I like the concept of self-healing pipelines, tho, not sure it's really idempotent and not sure it leads to a great management of data assets, still the article is related also to data contracts and might be solved another way.
• Medium feature store, how do they store lists — Medium built a feature store powering their recommendation system (which could work better tbh), in this blog they explain how did they decided to store features of type list.
• How the UK football rely heavily on data? — It's common knowledge that Liverpool FC won multiple titles recently by being data driven. This article shows how data teams are becoming larger and larger in the clubs. In the Premier League top 6, data team headcount average is 14.
• Spotify data science personas — Data science role is evolving and Spotify proposed multiple persona among their data teams.
• Unlocking insights with high-quality dashboards at scale — A checklist of stuff that you should have a look at to build high-quality dashboard. They even developed a Dashboard portal to improve dashboard usage and discovery.
• Ibis drops pandas backend and embrace fully DuckDB — It's a big choice moving forward Ibis, a multi backend dataframe library, decided to drop pandas support using by default DuckDB. The article says that DuckDB is way faster, covers the feature gap and pandas was mostly annoying because of NaN for nulls values, whereas it's NULL for all the other backends.
• DuckCon #5 videos — All the videos from the 5th DuckCon in Seattle are on YouTube. I did not had the time yet to look at it but I think awesome things are waiting us behind a click.
• Should you be migrating to an Iceberg Lakehouse? — This is an excellent question, and a good starting point for considering whether you should change all your assets to Iceberg.
• Amazon S3 now supports conditional writes — I think it's a great start for table formats.
• The analytics development lifecycle by dbt — Tristan Handy proposed a framework (very Enterprise) to rethink how the analytics workflow should work as of today. The article is 37 minutes long so I did not read but I saw the holy DevOps/DataOps infini sign that gave me instant headache. Toby from SQLMesh answered on LinkedIn, drama time.
• Making SQLMesh faster — the road for SQLMesh is crystal clear, they want to be the faster alternative to unmanageable large dbt projects, so they work of the speed of execution.
• What's your question?

See you next week ❤️

It's been 3 weeks.

Summer continues and I hope this new edition finds you well, having had a great vacation and a nice break before getting back to business in September. Content and articles have been a little slow over the last few weeks and that's to be expected, but I feel it gonna get back to business as usual soon.

Some personal news, in September things will be changing professionally on my side, I'll be slowly leaving the freelancing world. More details soon in a 2-parts articles I'm writing about it. I can't wait to tell you more to be honest. Still the newsletter gonna be the same formula.

Events ✨

As you may know I'm co-organising the Forward Data Conference on November 25th in Paris. The Forward Data Conference will be a day to shape the future of the data community, where teams can come to learn and grow together. There are still tickets left—we sold around 60% of the tickets.

We have started to announce a few guest speakers for the conference that I can't wait listen on stage. At the moment we have announced:

• Joe Reis, best-seller author, data engineer, he will speak about the new art of data modeling
• Hannes Mühleisen, co-creator of DuckDB
• Claire Lebarz, Chief Data and AI at Malt, who was working at Airbnb previously
• Virginie Cornu, Co-founder / CTPO and previously VP data at Jellysmack

You can use BLEF_FWD24 promo-code to get 15% reduction on your ticket.

Transition to another event. Demetrios Brinkmann is organising on Sep 12—in 18 days—a free online conference called Data Engineering for AI/ML, the agenda is pretty packed and the lineup is full of awesome speakers (Joe and Hannes will be there ☺️). The idea of the conference is to go deeper in the AI/ML current state and how we do data engineering in 2024 that services ML and AI teams.

AI News 🤖

• Drama at OpenAI people leaving (again) — Only 3 of the 11 original co-founders are still at OpenAI. In July reports were saying that OpenAI could be on the track to make a $5b loss.
• OpenAI structured outputs — You can now force OpenAI APIs to return you a specific enforced JSON schema when calling.
• New image generation capabilities — The German lab Black Forest created more realistic than ever images and with their model Flux.
• How Meta animates AI-generated images at scale.
• Meta segment anything model (SAM v2) is impressive in order to identify anything in images.
• ML in content moderation at Etsy.
• Semantics in the LinkedIn search engine.
• StackOverflow AI survey — a few insights quoted from the survey
  • 76% of all respondents are using or are planning to use AI tools in their development process this year, an increase from last year (+70%)
  • 81% agree increasing productivity is the biggest benefit that developers identify for AI tools.
  • 70% of professional developers do not perceive AI as a threat to their job
• Watermarking Generative AI — I believe in watermarking and I root for users app in the future to identify watermarks and to inform users if its AI, not AI or unknown.
• The open-source AI definition (v0.0.9) — A try to put words in order to define, in open, what's AI: "an AI system is a machine-based system that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments".
• What nobody tells you about RAG — Large deep-dive about RAGs.

Fast News ⚡️

• Timeless skills for data engineers and analysts — Benjamin proposes 4 non-technical skills that are useful when doing the data work, the 2 first one are "thinking in systems" and "data intuition". With the current resurgence of data modeling I think it both are critical part of the modeling work and need to be looked at.
• How top data teams are structured — Mikkel had a look at 40 data teams and analysed the way they are structured. As an output we get the roles ratio understanding better team composition.
• Mainframes finds new life with AI — Obviously we will never get rid of mainframes, and IBM says clients wants to run AI on it. I think this is time to learn machine learning in COBOL.
• Foursquare modern data platform — A classic modern data stack, but still interesting to see that Foursquare—like large companies—is going multi-technology by having multiple storage, processing engines and notebooks technologies.
• Amazon migration from Spark to Ray — Exabyte-scale migration and in the end Amazon saves a lot of processing time when it comes to files compaction.
• Design a text-to-SQL solution — An Indian food delivery company, called Swiggy, developed an internal text-to-SQL solution that user can interact with via Slack. The article describe all their thoughts and the challenges they faced while working on it.
• In browser WASM Postgres — Recently a company ported Postgres to WASM and now you can run Postgres within your browser without any server, you can test it at postgres.new (only desktop for the moment). You can see more on pglite website.
• Why we replaced Airflow in our experiment platform — Eppo reached a job concurrency limit with Airflow (wanted to run 50k concurrent experiments) and decided to switch to something else. After trying a few different things they decided to developed their own tech.
• Airflow Summit in San Francisco — Airflow gonna turn 10 this year. There is Airflow Summit in San Francisco on 10-12 of September. This year will also mark the start of Airflow 3.0 aiming to be release in March next year. If we look at the confluence page Airflow 3.0 will feature: a new web UI, DAG versioning, remote execution (from cloud to on-prem), data assets (like Dagster, a renaming of Datasets) and more but that's a great milestone.
• Airflow and dbt ; in Astronomer — Orchestrating dbt within an orchestrator is one of the most discussed topic among data teams using dbt. It's also an large adoption lever for companies like Dagster from the private discussion that I have (Dagster-dbt integration is top-notch). Astronomer has to go the same direction and through cosmos and a better integration now they propose it as well.
• No, data engineers don't need dbt — Common sense but good reminders. If you do ETL instead of ELT and you don't have a warehouse, dbt might not be the best fit.
• Sparkle write Spark pipelines in YAML at Uber — Uber doing uber things. Still declarative platforms in YAML are the way to go to standardise data work.
• ETL development life-cycle with Dataflow — How Netflix is also doing YAML development for Dataflow job writing.
• StarRocks usage at Pinterest for faster analytics.
• Table format comparisons — Honest linear review of 4 formats (it includes the 3 main ones and Apache Paimon). This first part details how formats manage physical files, the second part is about append-only and incremental tables.
• Polaris catalog is open-source — Snowflake released the expected Polaris catalog. On this matter, it has been reported that Databricks finally acquired Tabular for $2b while Snowflake tried to get them for $600m.
• BigTable supports SQL (and this is something) and GCP can run managed Kafka.
• The revolving doors of BI — I frequently observe companies switching BI tools every 2-3 years, hoping that the latest solution will resolve all their issues. While these migrations often provide short-term relief, they inevitably lead to another dead end. The initial success of the migration is typically due to the fact that, during the transition, companies address their technical debt and apply the lessons learned from previous mistakes. However, without addressing the underlying issues, the cycle is likely to repeat itself.
• Metrics management at Duolingo — I should go back to learning German in order to improve Duolingo metrics.
• How we test Airbyte and marketplace connectors — Exhaustive tests suite Airbyte put in place to check if connectors are behaving correctly.
• Slack summary ELT pipeline — It showcases how you can create an ELT pipeline with dlt, Ibis and Hamilton—which is a Python library to create transformation DAGs.
• snowflakecli — a DuckDB-powered command line interface for Snowflake security, governance, operations, and cost optimization.
• A Postgres extension (pg_duckdb) that brings DuckDB as an analytical engine within Postgres has been backed by Microsoft and fostered a bit of discussions in the community.
• Genealogy of databases — like a subway map that depicts all databases changes since Prehistory.
• Martin Kleppmann is working on a version 2 of Designing Data-Intensive Applications.

Final stuff

• dbt Generic Tests in Sessions Validation at Yelp.
• LETSQL inference for DataFrames — I don't understand the blog but looks cool.

Dear members, it's Summer Data News, the only news you can consume by the pool, the beach or at the office—if you're not lucky. This week, I'm writing from the Baltics, nomading a bit in Eastern and Northern Europe.

I'm pleased to announce that we have successfully closed the CfP for Forward Data Conf, we received nearly 100 submissions and the program committee is currently reviewing all submissions. Many thanks to all people who trusted us and submitted a talk for the conference (especially the DN members!).

We also announced our first guest speaker, Joe Reis. Joe is a great speaker, he wrote Fundamentals of Data Engineering, which is one of the bibles in data engineering and I can't wait to hear him at Forward Data. He his currently writing his second book about data modeling.

Forward Data is a 1-day conference I will co-organise on November 25th, in Paris. It will be a day to shape the future of the data community, where teams can come to learn and grow together.

AI News 🤖

Some days, AI News is like a TV shopping show. Over the past two weeks, a few dozen models have been released, and I'd like to introduce them to you.

New models & stuff

• OpenAI — OpenAI is trying to continue leading the charge by releasing models like Apple products.
  • GPT-4o mini: advancing cost-efficient intelligence — After GPT-4o which brought great performance and became the new flagship model, available in the free tier, OpenAI released a smaller version of it, the mini. According to the benchmark GPT-4o mini is close to GPT-4o performance but best in class among the small models. Even if OpenAI did not disclose how small it is, a few people are claiming it's a 8B.
  • Fine-tune GPT-4o for free — Until September 23, 2024 GPT-4o mini is free to fine-tune. This means each organization will get 2M tokens per 24 hour period to train the model and any overage will be charged at $3.00/1M tokens. Worth trying [docs].
  • SearchGPT, new OpenAI product —Yesterday, OpenAI unveiled their latest product, SearchGPT, a prototype AI search application. The system generates answers while providing reliable sources. This announcement coincides with Google Search's recent report of a 11% increase in revenue for the last quarter, reaching $64 billion. It shows that search did not disappeared with the advent of GPTs.
• Meta — This is crazy how Meta who suffered an unintentionally leak of LLaMA weights on torrents 1 year ago is now the company advocating for open models and leading this part of the ecosystem.
  • LLaMA 3.1 is out — The model is out in 3 versions, the largest one with 405B and 2 smaller ones (70B and 8B). They even released a 92 -pages whitepaper explaining how they trained it, the expected performances and what you can do with it. How dear fried Mark even wrote an ode to the open source with some kind of manifesto. The announcement is like a summary if you want it short.
  • Meta won’t release its multimodal Llama AI model in the EU — It would have been perfect if Meta was complying with the rules but in the end Meta is Meta. Doing lobbying and like a crying kid we punish for being bad they announce they will not release they super multimodal AI in Europe because regulatory environment is too "unpredictable". How to say that they use training data they should not have used.
    
    Last point related to tech giants (Apple, Nvidia, Salesforce) stealing YouTube subtitles to train foundational models.
  • Meta Chameleon — Finally Chameleon is available on HuggingFace, it's Meta Mixed-Modal Early-Fusion Foundation, which actually means it can understand and generate text and images.
• Mistral — The US-French company is keeping the rhythm with other giants going back on open models.
  • MathΣtral — A 7B model for math reasoning and scientific discovery under Apache license. I'll try it soon for something I'm cooking.
  • Codestral Mamba — A 7B model for code generation under Apache license.
  • Mistral Large 2 — Competing directly with the large LLaMA 3.1, Mistral Large 2 is 123B parameters and is at the moment the closest model to GPT-4o which still sets the benchmark.
• Microsoft Phi-3 models — Microsoft continues to try hard at the game with their Phi-3 models available in Azure. But who cares?
• SmolLM - blazingly fast and remarkably powerful — HuggingFace released new state-of-the-art small models (135M, 360M and 1.7B parameters) trained on a open corpus.

Articles

Because AI and GenAI is not only about models, a few great articles have been written as well.

• Twitter uses your data to train xAI Grok — Twitter recently added a opt-in to utilise your X posts as well as your user interactions, inputs and results with Grok for training and fine-tuning purposes. The opt-in is only available on desktop.
• AI Lab: The secrets to keeping machine learning engineers moving fast — About the AI Lab Meta put in place to keep ML engineers velocity giving them possibilities to A/B tests models and avoid regressions.
• Building Pinterest Canvas, a text-to-image foundation model — Great article about creating a image generation model for product backgrounds.
• Multimodal RAG pipeline — A notebook explaining how you can index and build a RAG on deck slides.
• Watermarking Generative AI: ensuring ownership and transparency — This is the future, being able to watermark all the generated content for ensure trust and transparency for end consumers.
• Germany (in German, sorry) and Switzerland both added into their law some kind of preference for open-source software (even required in Switzerland).
• Understand Vector database in Google Sheets — Playful. This is a Google Sheet template that you can copy that will explain you how a vector database works as well as the search.
• Infinite dataset hub — A Generative app that generates datasets for you out of a few words.

Fast News ⚡

Because the fast news are always the best.

• Why it feels impossible to get a data science job — Data science market became highly competitive in the last years, even more with everyone rushing to AI and jobs switching from being great at machine learning to being good at maintaining APIs orchestration. This articles tries to explain why and what to do.
• Snowflake brings seamless PostgreSQL and MySQL — It has been announce at the Snowflake summit, now you can directly ingest Postgres and MySQL from Snowflake UI, removing the need of any other tool for these sources. The way they did it requires you to run a Docker container, which is kinda meh.
• Query Snowflake Iceberg tables with DuckDB & Spark to save costs — That's what Iceberg tables on Snowflake unlocks. The capabilities to offload compute in DuckDB or Spark to save costs (or to move costs actually).
• BigQuery team is on fire and released a lot of cool new stuff
  • Table explorer — an automated way to visually explore table data and create queries based on your selection of table fields.
  • Continuous queries — A way to answer to Snowflake dynamic tables, continuous queries are SQL statements that run continuously. Google announce that CQ can be used for low-latency tasks.
  • On the same topic as CQ, they released the changes function, which is a SQL function that returns all rows that have changed in a table for a given time range. I think it will unlock a lot of use-cases in BigQuery.
• How Mixpanel delivers funnels up to 7x faster than the data warehouse — Mixpanel team is proud to say that they have better performance than Snowflake.
• You can run Clickhouse functions in DuckDB with the chsql extension and there is a great post about how DuckDB manages memory.
• Querying 1TB on a laptop with Python dataframes — A benchmark on a 96 GB of memory laptop using DuckDB, DataFusion and Polars. Crazy what we can do nowadays.
• Data modeling techniques for the post-modern data stack — A great recap of all the techniques that exists out there for modeling (medallion and dimensional).
• Parquet File Format: everything you need to know — How parquet file are written.
• Encoding and Compression Guide for Parquet String Data Using RAPIDS — For Nvidia geeks.
• Hudi merge-on-read — Iceberg has been all over the place, but there is Hudi as well, and the merge-on-read is a great feature tbh.
• Maestro: Netflix’s workflow orchestrator — 5 years later Netflix finally open-source their orchestrator. Curious to see if it will pick up. It's written in Java and it does what other orchestrator are already doing.
• The Analytics Development Lifecycle — dbt Labs needs to reinvent itself, now that dbt is everywhere, they need to define the next vision as they have pure gold in their hands. Tristan, CEO of dbt Labs, is aiming for a new acronym, ADLC (Analytics Development Lifecycle), and provides a draft manifesto with user stories of what AE should be able to do tomorrow.
• Deliver your data as a product, but not as an application.

Data economy 💰

• Google in negotiations to acquire Wiz in $23 billion deal, actually no. Wiz is a cloud security firm.

See you next week ❤️

Dear members, it's been a few weeks since I did not catch you on a proper Data News with a collection of links. Here we are.

This week, I attended EuroPython in Prague. While I spent most of my time at the dltHub booth in the sponsors hall, I didn't attend many talks. However, I did give a few presentations on my SQL orchestration library, yato, which pairs well with dlt. A YouTube video might come out soon.

Additionally, I attended an interesting talk by a Data News reader about the rise of YAML engineers, Matthieu also wrote in the past an article about this. I'm so happy to have met a few of you there 😊.

This is a great transition as a reminder to say that I'm co-organising a 1-day conference on Nov 25th in Paris. The Forward Data Conference will be a day to shape the future of the data community, where teams can come to learn and grow together. The Call for Paper (CfP) closes in a few hours, on Sunday 23h59. So propose a talk. Submissions are welcome in French and English.

AI News 🤖

• OpenAI — Always the biggest news provider, announcements or dramas
  • OpenAI was hacked, revealing internal secrets and raising national security concerns — The hacker reached OpenAI’s internal messaging systems early last year, stealing details of how OpenAI's technologies work from employees.
  • Microsoft quits OpenAI board — Microsoft said that they are not needed anymore because the governance has improved and at the same might want to avoid antitrust issues raised by gouv around the world. Apple also will not join the board as it was also expected.
  • How good is GPT at coding, really? — A research team evaluated the capabilities of GPT-3.5 in solving LeetCode problems. Although GPT-3.5 might be outdated, the findings are still somewhat relevant. The team discovered that GPT-3.5 performed significantly better on problems that existed before its training cut-off date. However, the model struggled with correcting its own mistakes.
  • OpenAI’s acquisition of Rockset, what it means — I announced it a few weeks ago, OpenAI bought Rockset a real-time analytical vector database. Customers have 2 months left to migrate away from the database that will probably because the core of the OpenAI architecture.
• kyutai released Moshi — Moshi is a "voice-enabled AI". The team as kyutai developed the model with an audio interface-first with an audio language model, which make the conversation with the AI more real (demo at 5:00 min) as it can interrupt you or kinda "think" (meaning for predict the next audio segment) while it speaks. Moshi will be part of kyutai open-source released and is purely local.
• Claude 3.5 Sonnet — To end the tour of the last model, if you missed it Claude 3.5 went out in June and featured great performances when "reasoning", Claude is capable to split the screen and develop some kind of Codepen playground with a React app applying what you're asking. There is a demo where Sonnet transformed a research paper into a simulator app about the paper in one prompt.
• pandas-ai — Give a dataframe to pandas-ai and configure a model, then you'll be able to chat with your data to get answers or chat about your questions. Nothing new I'd say, the only difference is the fact that the API to use it is fairly simple.
• Meta’s approach to machine learning prediction robustness — Principles Meta applies to bring robustness to ML.
• Bringing AI-powered answers and summaries to file previews on the web — Dropbox has developed a feature that generates summaries for every file in your storage. This process involves converting files, regardless of their format, into text. The text is then transformed into embeddings, which make the content easily summarisable and queryable for Q&A purposes.
• Recommendation system using a vector database — How Malt (a freelancer platform) built a recommendation engine using current vector database technologies (with Qdrant).
• DevOps for data science — An open-source and free book covering what data scientists need to know about DevOps. It has been written by someone at Posit (the company behind RStudio). It covers general knowledge about infra + code snippets.

Fast News ⚡️

• End-to-end data lineage in AWS — AWS announced DataZone to bring lineage to your data assets, from the picture it can mixes datasets (?), Glue table and jobs while giving you a greed/red vision of what's up to date. They mention column lineage, but from the picture it looks like they track columns but not proper column-level lineage. UI is AWS-tier.
• A brief history of modern data stack — Ananth from data engineering weekly wrote a few weeks after the modern data stack debate his views on it (read my opinion on this), he considers that we are in the post-modern data stack era with a few a points that will (or are) be implemented everywhere. Especially the interoperability.
• Apache XTable — XTable is a new layer that provides a cross-table interoperability, you don't need to choose only one table format out of Hudi, Delta and Iceberg. It provides abstractions and tools for the translation of lakehouse table format metadata.
• Create dynamic Iceberg tables — Snowflake added support for dynamic tables in a Iceberg format. Dynamic tables are tables based on "real-time data" (or streams, or continuous pipelines). So it means now Snowflake can just be used as an engine writing continuous tables in a blob storage in a open format like Iceberg — future is coming.
• Creating a file format in Rust — Experiment about what you need to create a new file format. That's super interesting to understand what's under the hood of popular stuff we often use.
• Data pipelines and SCDs — Slowly changing dimensions are an important pattern to know when it comes to data engineering. Julien wrote a great article about it and explaining the 3 possible form and the snapshot way. His chart are great. You can also read Timo detailed post on Mixpanel blog and why SCD are the best thing for product analytics.
• How to build a Semantic Layer — An great small guide that gives you the things to consider when going the Semantic Layer road. Gwen gives a step-by-step method to migrate from marts to a dbt Semantic Layer.
• How dlt uses Apache Arrow — A great post explaining why the next generation of data tooling need to use Arrow and how it impact the performances. This article explains how dlt (extract and load) then leverages Arrow.
• nanoarrow, a way to technically understand Arrow — Slides about a re-implementation of the Arrow framework (it's highly technical to be honest without a video).
• DuckDB and dbt — How with DuckDB and dbt you can build the transformation layer of a BI application (e.g. a Pokemon dashboard).
• DuckDB extension mechanism — DuckDB wants to provide a repository for community extension. This way the community will be able to extend DuckDB easily and it will also reduce the minimal size of the DuckDB allowing use to have an even more portable database/engine.
• Don’t lead a data team before reading this — 5 important points you should consider when leading a data team. I really like the "The business doesn’t care about how you solve the problem" because it's a good reminder for my technical audience that your role as a data person is to empower others with data, so boring tech is often the best.
• Apache Kafka overview — If you're not familiar with Kafka this is a great overview.
• Spark-connect, what's this — Very detailed post about what is spark-connect and why it will change the way we do Spark. It highlights how it simplifies and enhances the development process, particularly through its compatibility with various languages and the potential it unlocks for creating a data quality process.
• How Discord uses Dagster — 2000 dbt tables, covered by over 12000 dbt tests. Discord uses the dbt <> Dagster integration to power they whole data assets management.

Stories

• How Canva collects 25 billion events per day — Protobuf + Amazon Kinesis.
• In-memory analytics for Kafka using DuckDB — The author develop kwack a small utility that allows you to run SQL queries on top of Kafka streams (in-memory).
• 40 AI prompts to boost your marketing team’s creativity — Atlassian collection of 40 prompts to do marketing stuff. I'm not sure I'm happy to see this on Atlassian blog.
• A Recap of the Data Engineering Open Forum at Netflix — Video about the Netflix data engineering open forum are out on YouTube and this post is a recap + takeaways.
• Senior engineer fatigue — When you gain in experience as your career progress, you will start to feel fatigue as an engineer. Senior fatigue is characterised not by a decline in productivity but by a deliberate deceleration. I find the first part about the paradox of slowing down to speed up so true, that I recommend you to read it warmly.

See you next week (probably) ❤️ — I'll take random breaks this summer in order to prepare for the changes coming in my professional and personal life in September.

Every year, the competition between Snowflake and Databricks intensifies, using their annual conferences as a platform for demonstrating their power. This year, the Snowflake Summit was held in San Francisco from June 2 to 5, while the Databricks Data+AI Summit took place 5 days later, from June 10 to 13, also in San Francisco. The conferences were expecting 20,000 and 16,000 participants respectively.

Snowflake is listed and had annual revenue of $2.8 billion, while Databricks achieved $2.4 billion—Databricks figures are not public and are therefore projected. Snowflake was founded in 2012 around its data warehouse product, which is still its core offering, and Databricks was founded in 2013 from academia with Spark co-creator researchers, becoming Apache Spark in 2014.

Snowflake and Databricks have the same goal, both are selling a cloud on top of classic¹ cloud vendors. In the data world Snowflake and Databricks are our dedicated platforms, we consider them big, but when we take the whole tech ecosystem they are (so) small: AWS revenue is $80b, Azure is $62b and GCP is $37b.

The Google search results give an idea of the market both tools are trying to reach. Using a quick semantic analysis, "The" means both want to be THE platform you need when you're doing data. Both companies have added Data and AI to their slogan, Snowflake used to be The Data Cloud and now they're The AI Data Cloud.

Below a diagram describing what I think schematises data platforms:

• Data storage — you need to store data in an efficient manner, interoperable, from the fresh to the old one, with the metadata.
• Data engine — you need to make computations on data, the computation can be volatile or be materialised back to the storage
• Programmable — you need to run code on your platform, whatever the language or the technology at some point you need to translate your business logic into a programmatic logic
• Visualisation — you need to visualise the output of the computed data because charts are often better than table
• AI — you need to be proactive or predictive, that's when machine learning or deep learning enters, more generally today AI.
• In order to make all of this work data flows, going IN and OUT.
• Edge stuff — and then everything else that goes with it like privacy, observability, orchestration, scheduling, governance, etc. which might be required or not depending on the company maturity.

When we look at platforms history what characterises evolution is the separation (or not) between the engine and the storage. Good old data warehouses like Oracle were engine + storage, then Hadoop arrived and was almost the same you had an engine (MapReduce, Pig, Hive, Spark) and HDFS, everything in the same cluster, with data co-location. Then cloud changed everything and created a new way separating the storage from the engine leading to ephemeral Spark clusters with S3 and then, Cambrian explosion, engines and storage multiplied.

This is the fundamental difference between Snowflake and Databricks.

Snowflake sells a warehouse, but it's really more of a UX. A UX where you buy a single tool combining engine and storage, where all you have to do is flow data in, write SQL, and it's done. Databricks sells a toolbox, you don't buy any UX. Databricks is terribly designed, it's an amalgam of tools, they have a lot of products doing the same thing—e.g. you could write the same pipeline in Java, in Scala, in Python, in SQL, etc.—with Databricks you buy an engine.

At least, that's what the two platforms are all about. Ultimately, they both want to become everything between the left and the right arrows.

Now that I've introduced the two competitors, let's get down to business. In this article I'll cover what Snowflake and Databricks announced at their respective summits and why Apache Iceberg in the middle crystallised all the hype.

Snowflake Summit

Snowflake took the lead, setting the tone. I won't delve into every announcement here, but for more details, SELECT has written a blog covering the 28 announcements and takeaways from the Summit. If you're a Snowflake customer, I recommend reading Ian's insights. His business is centered on Snowflake, and he always offers the best perspectives.

Here what I think summarises well the summit:

• Apache Iceberg support — it means Snowflake engine is now able to read Iceberg files. In order to read Iceberg files you need a catalog, Snowflake support external catalogs—like AWS Glue—and they will open-source Polaris, in the next 90 days, their own Apache Iceberg catalog.
  
  If you're not familiar with Iceberg, it's an open-source table format built on top of Parquet. It adds metadata, read, write and transactions that allow you to treat a Parquet file as a table. For a comprehensive introduction to Iceberg, I recommend reading my friend Julien's Iceberg guide.
• Native CDC for Postgres and MySQL — Snowflake will be able to connect to Postgres and MySQL to natively move data from your databases to the warehouse. This could be a significant blow to Fivetran and Airbyte's business. While the exact pricing hasn't been revealed yet, the announcement emphasises cost-effectiveness.
• Store and run whatever you want on Snowflake — They bring a serverless / container philosophy to Snowflake as you will be able to store your AI models, run pandas code or any container.
• Dark mode interface — Ironically it was their closing announcement, their most asked feature and their Reddit most liked post following the announcement. I found it a bit ridiculous but it showcases how much Snowflake is a UX first platform.

From the start, Snowflake has been a straightforward platform: load data, write SQL, period. This approach has always appealed analysts, analytics engineers, and pragmatic data engineers. However, to capture a larger market and address AI use-cases, Snowflake needed to break through its glass ceiling. To me, that's what these major announcements are. Snowflake becomes Databricks.

Databricks Data+AI

I didn't attend either summit in person. While I enjoy these events, I prefer avoid flying for ecological reasons, and large gatherings can be challenging for an introvert like me. Watching the Data+AI Summit from home did give me a bit of FOMO, but the Snowflake Summit did not. Databricks successfully built hype during the event, announcements after announcements.

Once again it boils down to the nature of the platform, Snowflake is insanely boring, even if use-cases are different Snowflake solution standardise everything, when it comes to Databricks, creativity arise—or we can call it tech debt. By the multiplicity of products or ways to handle data shiny stuff can appeal everyone.

Here what Databricks brought this year:

• Spark 4.0 — (1) PySpark erases the differences with the Scala version, creating a first class experience for Python users. (2) Spark versions will become even easier to manage with Spark Connect, allowing other languages to run Spark code—because Spark Connect decouple the client and the server. (3) Spark 4.0 will support ANSI SQL and many other things.
• Databricks AI/BI — Databricks has introduced AI/BI, a smart business intelligence tool that blends AI-powered low-code dashboarding solution with Genie, a conversational interface. AI/BI will be able to semantically understand and use all the objects you have in your Databricks instance. Visually the dashboarding solution looks like a mix between Tableau and Preset.
• Serverless compute — This is something that keeps bridging the gap in term of user experience, because manage Spark cluster is painful, Spark serverless let's you run a Spark job without worrying about the execution. Still, serverless compute does not support SQL.
• Buying Tabular — Before the last bullet point, it was already something big. Databricks bought Tabular for $1b. Tabular was founded in 2021, had less than 50 employees and raised $37m. Jackpot. Accordingly to the press Snowflake and Confluent (Kafka) were also trying to buy Tabular.
  
  But what is doing Tabular? Tabular is building a catalog for Apache Iceberg and Tabular employs a few part of Iceberg open-source contributors. By getting Tabular Databricks gets all the intellectual knowledge about Iceberg and how to build a catalog around it.
• Open-sourcing Unity Catalog — Finally, on stage, Databricks' CEO hit the button to open-source Unity Catalog, directly responding to Snowflake’s open-sourcing of Polaris. Unity Catalog, previously a closed product, is now a key part of Databricks' strategy to become a the data platform. This move, combined with Tabular acquisition, will help Databricks achieve top-notch support for Iceberg.

If you've made it this far, you probably understand the story. Databricks is focusing on simplification (serverless, auto BI², improved PySpark) while evolving into a data warehouse. With the open-sourcing of Unity Catalog and the adoption of Iceberg, Databricks is equipping users with the toolbox to build their own data warehouses.

Apache Iceberg and the catalogs

We finally get down to Iceberg. What's Iceberg? Why catalogs are so important? How do they differ to data catalog we are used to?

So Iceberg has been started at Netflix by Ryan Blue and Dan Weeks around 2017. Both co-founded later Tabular (which got acquired by Databricks). Iceberg has been designed to fix the flaws of Hive around table management, especially about ACID transactions. The project became a top-level Apache project in Nov 2018.

Currently Apache Iceberg competes with Delta Lake and Apache Hudi and became the leading format in the community when looking at all metrics. Newcomers are also arriving late to the party like nimble or DuckDB table format which could be a thing in the future.

What is Iceberg?

The community decided these last year and Parquet became the go-to file format when it comes to storing data. Parquet has many advantages like being columnar, the compression, can pushdown predicates, own the schema at file level and more. But there are a few issues with Parquet. Parquet is a storage format, except for a few metadata and the schema Parquet has lack of information about the table.

A table format creates an abstraction layer between you and the storage format, allowing you to interact with files in storage as if they were tables. This enables easier data management and query operations, making it possible to perform SQL-like operations and transactions directly on data files.

Iceberg is composed of 2 layers but has sublayers, like a onion:

• the data layer — contains the raw data in Parquet, Iceberg manages the way the Parquet files are partitioned, etc.
• the metadata layer
  • manifest file — A manifest is an immutable Avro file that lists data files or delete files, along with each file’s partition data tuple, metrics, and tracking information.
  • manifest list (or snapshot) — A new manifest list is written for each attempt to commit a snapshot because the list of manifests always changes to produce a new snapshot. This is simply a collection of manifests describing a state or a partial state of the table.
  • metadata file — Table metadata is stored as JSON. Each table metadata change creates a new table metadata file that is committed by an atomic operation.

That's what it is, if you have to understand something, Iceberg creates table on-top of raw Parquet files.

So once you have Iceberg you're capable to create multiple tables, but you need a place to store all the metadata about your tables. Because Iceberg each table atomically but obviously you need more than one table. That's why you need a catalog. This catalog is like the Hive Metastore. I've read somewhere that we should call it a super metastore rather than a catalog which is already used to describe another product in the data community.

Still that's why we need a place to keep a track all our Iceberg tables. That's what is Unity Catalog, AWS Glue Data Catalog, Polaris, Iceberg Rest Catalog and Tabular (RIP). Actually all of these catalog are implementing the Iceberg REST Open API specification.

Conclusion

Databricks and Snowflake embracing Iceberg by open-sourcing a compatible catalog and opening their engines to Iceberg show how far ahead is Iceberg. I don't think Databricks or Snowflake really won the competition.

On Snowflake's side, they mitigated the impact by open-sourcing Polaris and embracing the Iceberg format. However, most Snowflake end-users won't be concerned with these changes ; they simply want to write SQL queries on their data. These format details are more relevant to data engineers. Snowflake finds itself between Databricks' innovation and BigQuery's simplicity³ (ingest data, query). To grow, Snowflake needs to expand in both directions.

With this move Databricks will finally provide a data warehouse to their customers, it will be a data warehouse in kit, but a data warehouse. Because this is what it is, the Iceberg + catalog combo just create a data warehouse. It mimics what a database is already doing for ages, but more in the open with you pulling all the levers rather than something hidden in a black box written in database compiled language like C.

Wait, Iceberg is written in Java, and honestly, PyIceberg is lagging significantly behind the Java version... Here we go again.

1 — I don't like the classic term to qualify AWS, Google and Microsoft but actually that's what they are right now. Leaders and commodities.

2 — I just made this term, looks like it does not exist for data really but I like it a lot.

3 — Actually recently BigQuery added a lot of features to extend the compute and with more way to interact with data (notebooks, canvas, etc.)

🥹It's been a long time since I've put words down on paper or hit the keyboard to send bytes across the network. We're in the age of AI, and my lords computer science have evolved over the last 30 years. I'm writing this edition from my child's home, and it brings back memories. I got my first computer at the age of 6 and spent my days installing Windows 98 over and over again, getting lost between the BIOS and the Windows installation pages, playing with Word, Dreamweaver and Adobe Premiere.

Who would have thought that 25 years later, I'd be celebrating 10 years working with computers? June also marks the third anniversary of this newsletter. 3 years ago I started the newsletter in order to share my expertise with people, I'm so happy how it turned out. More than 5000 members subscribed to the newsletter and the blog generated almost 100k unique visitors.

Recently a lot of people subscribed and never received a Data News, I want to give you a warm welcome and this edition marks the journey we embark together, you will enjoy what's coming next, I'm sure.

I've taken a little forced break because I've been overwhelmed with work lately, juggling a lot of requests and my customers' work. In order to deliver I had to reclaim my Fridays. Around the newsletter there are unfinished projects with the Recommendations page and Qrators and I'll get back to them starting July once I'm done with the rest.

Forward data conference ⏩

I'm excited to announce that I am co-organising the Forward Data Conference, a one-day event in Paris. Join us on November 25th as we bring together around 350 attendees and an impressive lineup of speakers. It's going to be an incredible opportunity to connect, learn, and explore the latest in data. We will make our best to make the conference friendly for English natives.

Forward Data aims to be a hub for knowledge sharing and best practices, offering you the chance to expand your horizons, explore new facets of the data ecosystem, and connect with key international community leaders.

AI News 🤖

A lot of AI news and changes were made in the last 3 weeks. This is a small recap.

• OpenAI
  • The super-alignement team was fired — The goal of the super-alignement team was to research all related topics to AGI security. But it seems priorities reshuffled. Then OpenAI appointed former NSA leader (nominated by Donald Trump), he will probably work with Safety and Security committee.
  • Annualised revenue projected to be $3.4b — This is crazy how the company successfully reached this amount, mainly by selling to Enterprise customers. By comparison Snowflake revenue was $2.8b in 2023.
  • Extracting concepts from GPT-4 —
• Apple announced iOS 18 and their own AI — AI will stand for Apple Intelligence. With great ego Apple appropriates the letters AI at their annual developer conference (the WWDC) they showcase how AI will be integrated everywhere in iOS:
  • Siri has been revamped — now looking like a Microsoft AI copilot, Siri will be able to sort notifications, to help you writing better or to give better contextualised answer. Siri will also integrate with OpenAI through ChatGPT when needed.
  • At the same time they announced their model will run on-device (keeping your data safe and private) and when more compute will be required they will use a private cloud.
• Writing tools — to bring a few of the best GenAI features: proofreading and rewriting. When selecting a text you will be able to ask the model to rewrite it more professionally, etc.
• Genmoji — a way for your parents to be even cringer in their emoji usage by generating emoji from a sentence.
• Finally, with new Siri and Writing tools they reworked one of the worst Apple application: Mail. Giving a better look and new capabilities in email writing.
• It joins other features for which Apple will introduce AI (and GenAI) throughout its products (audio transcription, image generation from tags, better natural language search on photos, etc.). But this anchors Apple in a consumer products company, not an AI company like Google, Microsoft or Meta. Apple has decided for years to keep its users' data safe and private, which means they don't have a pool of data to train large language models.
• How to rethink the recommendation for social networks — This is a small video about Jack Dorsey (Twitter co-founder) about recommendation algorithm and how platforms today should give the choice back to users, this is about free will and building biais / filter bubbles. Why should we have transparency on what rules the recommendations and why should platforms propose multiple algorithms and let the users decide, like a marketplace.
• Changing the GPU is changing the behaviour of your LLM — A cool experimentation that shows how GPU impact the inference.
• MLOps coding course — Great MLOps course! It contains 6 chapters and covers all the needed topics to put models in production with the correct choics.
• RAG in BigQuery — When you do RAG in database it's often correlated to embedding functions and being able to query these vector with performance. BigQuery has all the toolkit to do it and this article showcase it well (and let's be honest all the competition does the same).
• What makes a Gen AI system open? — A paper that survey 45 models across 14 elements that could define them open. OLMo 7B Instruct is the most open according to the paper and ChatGPT the least one. On the same topic Mozialle released a paper about a framework for Openness in Foundation Models.

Fast News ⚡️

• Solving probabilistic Tic-Tac-Toe — Probabilistic tic-tac-toe is like tic-tac-toe but each cell is given a probability distribution. So when you make a play randomly you can x, o or do nothing. Someone develop a Unity version of the game and someone else wrote a math solver giving the best play at every turn.
• Amphi ETL — Amphi is a low-code visual ETL that you can run in JupyterLab. This is super clever. This is the first time I see this kind of application that can run as an extension of Jupyter. Worth watching it in the future, this is still early.
• Compare Airbyte and dlt ways to create custom sources — A large article that compares Airbyte and dlt when it comes to creating custom sources. Both extract and load tools can create custom source via either Airbyte low code CDK or dlt REST API Source toolkit.
• trip.com migrated from 50PB Elastic to ClickHouse — I've never been fan of NoSQL platform like ES for data work. This article on ClickHouse blog showcase how a client migrated their ES cluster to ClickHouse to improve their logs querying capabilities. More, the article focus once at scale with multiple CH clusters how to correctly route the queries.
• Hunting non optimised queries in ClickHouse — The talk is about ClickHouse but can apply to every engine. In the talk Yohann explains the mechanism he put in place to find non-optimised SELECT. He did it with a machine learning model, which means that he identified the features slowing the queries like nesting, subqueries, join and wheres.
• BigTesty — a framework that allows to create integration tests with BigQuery on a real and short lived infrastructure. It uses Pulumi (a infra-as-code tool) and requires you to give inputs, SQL queries and outputs and tests it against a dedicated BigQuery project.
• Data platform explained part II — Part 2 of the Spotify article about data platforms. Their name 3 different steps: data collection, management and processing (and they even mention GDPR) and finally explain how they treat data culture.
• What is really Apache Iceberg? — Iceberg has been at the center of the discussion this week. Julien wrote the greatest deep dive you can find on the topic.
• Cron expressions with DuckDB — An handy function in DuckDB that can generate time arrays when given a cron syntax, it's more understandable than generate_series().
• Serverless Jupyter notebooks at Meta — They develop a system called Bento which allows notebooks to either run with classic kernels or with in-browser kernel (being really serverless) using Pyodide. They have handy functions to get sql, googlesheet or graphql data in the browser memory to then work on it.
• Airflow new youth — If you stayed with Airflow 1.x or previous 2.6 you might have missed Airflow new youth. This presentation from Jarek showcase all the recent improvements: data aware scheduling, deferrable operators, object storage, etc.
• A hybrid information retriever with DuckDB — how can you fusion semantic and lexical search with DuckDB. Looks neat.
• dbt-score, lint metadata and get max score — Lint you dbt metadata, gets a score and be happy in the CI/CD.
• Automatically detecting breaking changes in SQL queries — Use SQLGlot diff function (on AST) and gets what changed on a SQL query and act accordingly.
• How I failed to implement dbt — Benoit explains why he failed implemented dbt in his previous role. He identifies 5 errors that led to a failure. As always this is not about a technical issue.
• 250 European data infrastructure startups and what we learned from them — Another perspective about data infrastructure that greatly complete the MAD landscape. At the end of the page it gives great definition about every part of a data platform.
• The rise of medium code — Between low-code practitioners and software engineers there are medium code practitioners like analytics engineers and data scientists. This code often lies into Python orchestrators and has to be treated correctly because it's production code as well.
• Write-Audit-Publish pattern — Once again a great article about this pattern.
• How Monzo uses incremental modelling to handle billions of events every day.

Data Economy 💰

• Mistral raises €600m — Mistral has never been a French company from the first rounds but raises again a lot of cash to go faster.
• xAI raises $6b — Late to the party and it seemed no one care about but Musk tries to fight.
• Cube raises $25m — Cube has the most advanced piece of technology today when it comes to semantic layer and they raised enough money to continue going into this direction.
• Snowflake invests in Omni — Omni is a refreshed version of Looker with a fresher LookML.
• Databricks acquires Tabular — It created waves last week in the data community. I'll write more about it on Monday.
• Tobiko raises $17.3m — The company behind SQLMesh and SQLGlot raises cash to create a suite of tool to invent the data development of tomorrow.
• Redpanda acquires Benthos — In the streaming world it was big.

I want to address something weighing on my mind. We've all seen the results of recent European elections and how the far right has influenced public debate and opinion. I strongly believe we should not fall for their tactics or their so-called solutions. In the tech community, many of us are privileged, often due to our financial stability. However, we cannot build a society with only people like us. Because of our privilege, we (1) should vote, (2) should use our vote to support those marginalised by the system.

For my French readers, there are parliamentary elections in France in 15 days. I urge you to vote and to vote against the far right. Hate and division are not solutions. Cutting public services through tax reductions is not a solution. Pushing for more productivity when AI is on the rise is not a solution. Individualism is not a solution. They don't bring any solution.

Consider what the tech ecosystem would look like under far-right principles: diversity stifled, innovation hindered, and global collaboration restricted. These ideologies could limit talent flow, reduce educational programs, and promote censorship and surveillance (which is almost already here, we work in big data face the reality), undermining our core values of privacy and open access.

If you feel this message doesn't belong in a tech newsletter or professional sphere, I don't care and you can unsubscribe. However, I believe that advocating for openness and tolerance is essential, and accepting hate speech is unacceptable.

See you next week ❤️.

Hello you. The sun is out, the days are getting longer and Data News is still here. Next week marks 3 years of this newsletter/blog (yay 🎉 ). It'll be a time for looking back, reflecting and celebrating, but next week. This week, we reached 5000 members.

Yes, 5000 of you read my content periodically. Just thank you ❤️.

In the recent days I've been working on a new side project. What if you could search in video content and get the exact timestamp of what you're looking for?

Let me introduce an application of this on the Data Council 2024 80 videos.

Data Council 2024 ✨

Data Council Austin is according to me one of the best conference when it comes to think about the future of data. Every year the talk that are given at DC are always full of quality content. There is a main drawback of this which is: it 80 videos of ~30 minutes and not everyone have the time to watch everything or search among the videos.

So I developed an app which allows you to search for words in the Data Council video playlist and we've curated highlights with Julien in order for you to watch only what we've curated.

It's available on qrators (can be pronounced curators / creators). And it works for the moment only on desktop.

The search is working greatly, for instance you can full-text or exact-term search. For instance Airflow, dbt, backfill, "data mesh" or "SQL Glot". Quotes means an exact-term search.

I'll write another post later about the behind the scene and how this app has been built, but because I'm your humble servant, this app uses DuckDB WASM and requires no backend to work (except a bucket with the data).

Still, I want you to get as always a few takeaways of the conference so here are my favourite talks with a few highlights:

• Data culture as a product— Abhi already had one of my favourite talk from Data Council 2023 about Metrics tree. Following on from his work on metrics, this time he attempts to give advice on creating a good data culture in order to create a good decision culture in companies. After all, companies need to make decisions, and these decisions need to be informed by data. [highlights]
• Processing trillions of records at Okta with DuckDB instead of Snowflake — it was one of my most expected talk from the council because a few months ago Jake posted on LinkedIn his team successfully reduced Snowflake billing by hundred of thousands by shifting to DuckDB. In the talk he explained what was the issue with Snowflake and how a multi-engine data stack built on-top of S3 + Lambda drastically reduced dollars spent. [highlights]

I like a few of the others talks, but I'll do a dedicated post for this I think because the Data News is already super dense.

AI News 🤖

• OpenAI recents announcements — The company behind ChatGPT announced a few things hyping everyone recently. Especially their GPT-4o (it's 4-o the letter and not 40 the number) model, this model adds new capabilities to ChatGPT around photos, videos and audio. The model can talk or understand what's in a image/video and answer questions about it. They also released a MacOS app that you can call on Option+Space to ask ChatGPT. OpenAI also detailed a bit their model specs and what principles they implemented to put guardrails around answers.
• Why Microsoft invested in OpenAI in 2019 — Emails explaining why Satya Nadella (CEO) and Kevin Scott (CTO) pushed Microsoft to invest in OpenAI have been made public, and are worth a look. It mainly reads that Microsoft was "several years behind the competition in terms of ML scale" (compared to Google, in search / ML in applications) and that to get there, they needed someone with gigantic ambition, from silicon chips to high-level programming abstractions. And the OpenAI team was someone.
• OpenAI is offering $10m packages to top AI researchers. There is a paywall I can't say more.
• AI lobbyists are everywhere now — A bit more politic but with the stakes around AI (money, power, content moderation and generation, privacy, etc.) lobbying around it is through the roof.
• Google I/O keynote — Google I/O was the response from Google to OpenAI announcement around models. They showcased agents that can help you do more in your favourite Google Apps, then DeepMind showcases new capabilities around image and music processing / generation. But one of the most important announcement was only a few seconds about search that might change forever (paywall can be avoided with a page reader). Google introduced AI overview that will be presented first in search answer pushing traditional results far below.
• LLMs with Keras — Keras team demoed various workflows around LLMs (Gemma) with Keras.
• Opt-out to avoid Slack training LLM models on your private data — Slack (acquired by Salesforce) could train their LLM models on your data. Still their answered in the Twitter thread but it's legal stuff I don't understand.
• HuggingFace releases Idefics2 — An open multimodal model that accepts arbitrary sequences of image and text inputs and produces text outputs. Works with multiple images as well to create stories.
• Building DoorDash’s product knowledge graph with LLMs — A good graph is like good wine and DoorDash used LLMs capabilities in information extraction to improve their product catalog graph.

Fast News ⚡️

• Apache Arrow DataFusion becomes Apache DataFusion — DataFusion, a query engine built in Rust and uses Arrow for in-memory structures, has been promoted as a top-level Apache project. DataFusion is one of the most important alternative to DuckDB when it comes to engine (not mentioning polars here). On that topic this week I've met people from SDF who are betting on DataFusion as their core execution engine.
• facebook/nimble, a new columnar file format — A new columnar file format is out. They announce it as "a replacement for file formats such as Apache Parquet". Ok but why?
• Unexpected tips for data managers — A comprehensive and pragmatic list of tips to be a great data manager. This is pure gold.
• Data about data from 1,000 conversations with data teams — Mikkel output of his interviews with a lot of data teams and what topics are important.
• How to save 90% of BigQuery’s storage cost and how to reduce your Snowflake cost. On the same topic if you don't know about GROUP BY ROLLUP you should use it in Snowflake or BigQuery.
• Reverse engineering exercice — Awesome idea. In order to learn concepts the author decided to reverse engineered a NYT game and he documented the process and what he understood. I find this exercice super insightful and I'd love to do something similar.
• Initial thoughts on SQLMesh — A post describing what are the key concepts of SQLMesh (esp. around envs, plans and projects), this is a great introduction. Last week SQLMesh team also released stuff around testing, similar to dbt unit tests you can define input and outputs to test your models.
• dltHub REST API source toolkit — dlt released a toolkit to build extract and load pipelines on top of custom APIs. With the toolkit you can declare your endpoints resources and auth and then you'll be able to extract and load your data.
• Cube releases their AI API — Now you can query in natural language your semantic layer and get answers (it uses OpenAI). This is close to what I had demoed last year in a talk.
• MotherDuck pricing page — Great pricing page, competitors should get inspired from this. It's fun to play with it to see how many hundreds of thousands of dollars you would have spent.
• Uber, auto-categorizing an exabyte of data at field level through AI/ML — Reminds me SDF article about end-to-end classification of your data models but at Uber scale.
• great_tables — A great tool to create nice looking tables in Python on top of your dataframes.

Food for thought to end (because it's already too long)

• On Orchestrators: you are all right, but you are all wrong too
• Do you model in dbt or BI?
• How women lead data engineering at Slack
• How we migrated from dbt Cloud
• Lyft, build a strong data science team

See you next week for the anniversary 🎂

Hey, new Friday and special Data News this week. This week has been pretty packed in term of work for me so here a joker as a weekly newsletter. This is a compilation of great resources about building a data team.

This is a collection I've created when working on a talk called "How to build a data dream team" that I've made last year. All the articles are different and give a large spectrum of perspectives about creating a data team.

In my experience, building a data team is a mixture of everything, there's no single recipe, but for it to work, you need to adapt the technology you choose to the people you have. These days, it's very easy to technically build a data platform, but building a data team goes further than that, it's about processes, communication and prioritization, how to build trust with stakeholders, etc.

10 great resources to build a data team

• How to build your data team? — This article from Castor team brings all the vocabulary needed. It explains what are the different models: centralised, embedded or federated and what are the pros and cons for each. It also overlook the topic of the size and the roles.
• Net promoter score for data teams — A very important topic I guess. A reminder, one of the most common data team mission is to empower stakeholders. So face the truth and compute a NPS to know what your stakeholders are thinking about.
• Vision for a data team — Probably the most pragmatic and full of handy advice. This blog from Alan data team explain what a data team should do.
• Building a data team at mid-stage startup: a short story — Views of all the journey your data team will go through in a startup. From the first day to one year later.
• Data team, how we work — The Gitlab handbook which is a big bible of resources when it comes to data. Everything is detailed, how do they work, how they do triage, prioritisation, etc.
• How Typeform build a data team in under 6 months — 5 key insights and 7 top advices about what to do.
• Data organisation: why are there so many roles? — A great guide about the role and responsibilities of people in a data team.
• How to build a data analytics dream team — Goes a bit further than the previous article open to new (weird) roles.
• The next big challenge for data is organisational — Yes, technically this is just about alignement. The rest is human collaboration and change management which is quite hard.
• Building a data team, dbt recommendation guide — dbt Labs wrote a great guide about data team building.

Sorry about this intermittence in my Data News, I promise next week I'll be back ❤️.

Hey, new Friday, new Data News. This week, I feel like the selection is smaller than usual, so enjoy the links. I'm a bit late with the Recommendations emails, I'm sorry about that I got a few new leads as a freelancer I had to take in priority changing a bit my schedule. But don't worry it gonna be out soon.

AI News 🤖

When do models get the same hype as 2007 iPhone release? I did not get the memo.

• Meta releases Llama 3 — After last week Mistral new models, this week ends with the new Meta open-source models. Llama the Third is online. One thing to analyse is that the model created more hype than Meta AI the new ChatGPT run by Zuck company going all-in into his AI vision. It shows something changed, generative models reaching massive adoption, in my bubble at least, people care more about a new model than an assistant available in Meta ecosystem (Insta, WhatsApp, Facebook and more). Until we reach the model fatigue hype is real.
  
  Personally, I can't comment on the performance of the models, it's like comparing the performance of two cars, as long as I can drive, it's fine. You can try Llama 3 on Modal or HuggingChat.
  
  In order to go further you can read this excellent analysis on Twitter or the model card—they even give the teCO2 estimated in the training phase. In a nutshell it says:
  • Llama is available in 8B and 70B, 400B is coming once training will be completed—and approaching GPT-4 performances.
  • Llama has a larger tokeniser and the context window grew to 8192 tokens as input.
  • It was trained on a large dataset containing 15T tokens (compared to 2T for Llama 2).
• Mistral wants to raise again at $5B valuation.
• Microsoft VASA-1 — Microsoft published a paper about a model generating talking avatars from an image and an audio. This is quite impressive. They did not released the code so I tried the closest solution open-source called SadTalker and tried it on me. This is a bit creepy, but impressive for the low quality of my inputs.
• Structured generative AI — Oren explains how you can constraint generative algorithms to produce structured outputs (like JSON or SQL—seen as an AST). This is super interesting because it details important steps of the generative process.
• Evaluate anything you want with LLMs — I really like how LLMs can be used for tasks that are not the one we are first thinking of. This blog shows how you can use Gen AI to evaluate inputs like translations with added reasons.
• How we build Slack AI to be secure and private — How Slack uses VPC and Amazon SageMaker with your data secured and private.
• OpenAI batches — OpenAI opened a new API endpoint to batches requests.

Fast News ⚡️

• Principal Engineer — Although staffs and principals have been on the career ladder for a long time, there are very few articles on what it takes to become one of the greats. This article covers the whole ladder and the mix of skills needed to reach the top. It's a mix of hard, soft and business skills.
• Data pipeline, incremental vs. full load — A comprehension comparison between 2 mode of ingestion with a decision tree about which one to pick.
• ❤️ Scaling to count billions — An awesome retrospective of the Canva OLAP architecture to count marketplace usage, from MySQL to Snowflake + buckets. It greatly decompose every important part of an OLAP platform with the collection, the deduplication and the aggregation.
• Spark (and Theseus) on GPUs benchmark — A detailed benchmark by Voltron data about running Spark and Theseus (their GPU data processing engine) workloads on GPUs. This is crazy how Theseus outperform Spark. The conclusion look like a great summary for me:
  • For less than 2TBs > use DuckDB, Polars, DataFusion or Arrow backed projects.
  • Up to 30TBs > Cloud warehouse or Spark
  • Over 30TBs > Go Theseus. [Theseus] "prefer to operate when queries exceed 100TBs". 😅
• Polars new benchmarks — Polars released new benchmarks about the TPC-H dataset. Polars and DuckDB are the cool kid and benchmarks show you should stop using pandas and switch to polars to get x10 performance gain.
• Hydra: the Postgres data warehouse — Postgres is one of the most used database, this week I discovered Hydra an open-source columnar port of Postgres aiming to create an open-source Snowflake. To watch.
• Neon GA — Neon, another Postgres fork, is generally available. Neon wants to provide a serverless autoscalable Postgres for devs.
• Clever Cloud offloading 20TB every month — Kestra showcases how one of their client is using the declarative orchestrator to offload TBs of data every month.
• KubeCon + CloudNativeCon Europe’24 notes — A few notes from the 2024 Kube big mass.
• Is SQLMesh the dbt Core 2.0? — A great blog to answer a great question. SQLMesh is bringing fresh ideas to the SQL transformation landscape. The post covers a lot of topics and explains the concept similarities between both tools.
• gwenwindflower/tbd — A code generator for dbt. Winnie developed a great tool to save time in documenting you dbt projects using Gen AI models.
• Snowflake text embeddings for retrieval.
• Distributed dashboarding with DuckDB WASM — Ramon put words into ideas I have in my mind for months: distributed dashboarding. I buy so much this concept, especially with DuckDB WASM and what it unlocks in term of autonomy or privacy for users.
• WAP with dbt, Icerberg or Nessie — Julien showcase how you can achieve WAP pattern with different technologies.
• CSV to DB — What if you could open a CSV, re-order or rename the columns directly in the browser? Without any backend call—with DuckDB obv—. s/o to Théodore, a Data News subscriber, who developed this. This is a great idea.

See you next week ❤️.

I hope this Data News finds you well. In today's edition we have a large selection of links, I think you will enjoy it.

But first I want to welcome all the new members joining this week after my new episode on DataGen with Robin Conquet. This is an episode in French and we talked mainly about the eventual end of the modern data stack. Which I have already condensed in a post a few weeks ago (in English).

MDS Fest 🥳

As announced last week I've participated to the MDS Fest 2.0 this Thursday. I've shared my journey with Apache Superset and why I consider Superset the best open-source alternative when it comes to building BI applications.

Yes because you should stop building dashboard and build BI apps instead. It enters in the productisation of the data, but mainly I think you should consider your BI tools as a way for your users to interact with data and not only to monitor metrics. With customisation possible Superset is the best tool for it.

You can have a look at my slides or watch the replay on YouTube.

In the same conference a lot of other talks took place here a few selection you should check out:

• How to pivot your data team from a service team to a value-generator — Very often data teams struggle in delivering value or in finding what's their real identity. Taylor identified pattern and gives great advice to help you finding it.
• Data contracts: federated data governance — Another talk by Chad about the data contracts, always on point in describing the pains around the "data supply chain".
• Deliver reporting in pure SQL with dbt + Evidence — A great showcase of what you can build with Evidence (a BI as code solution).
• Build analytics at Hive.co — The journey Oleg and his team went through to implement a modern data stack. They used RFC to document where they were heading to.

PS: Apache Superset is going 4.0 this week with a lot of new features.

AI News 🤖

• Open-source LLMs for everyone — A great post from Siemens AI team about open LLMs initiatives that brings new usages in the dev workflow, wether it's about code completion and pull request / crash reporting summarisation, it looks neat.
• Building LLMs for code repair — Replit is a AI-driven workspace for developers , think of a supercharged IDE. They wrote a blog about what they developed to create a LLM driven fix suggestion for LSP (Language Server Protocol), which is a protocol between your IDE and a server that understand and analyse the code to find errors or highlight the code.
• LLM DataGen — A small demo of a LLM based on Gemma that generates JSONL based on a given name. This is not working super well and it would be better if we could specify the columns names and types for instance, but showcase another great usage example of generative algorithms.
• Meta, building an infrastructure for the future — It explains how Meta is partnering with GPUs vendors to design new chips and how it's incredibly hard to connect thousands of GPUs in a cluster where everything can fail at every moment.
• Can Gemini 1.5 actually read all the Harry Potter books at once? — A nice Graphviz chart spotted on Twitter of the whole Harry Potter relationships in a poster. Done by Gemini with the content of all the books. Obviously Gemini already knows a few of the Hogwarts lore by his training, but still this is impressive. Sadly we don't have the complete prompt / code.
• Speaking of prompts, PromptLayer organised a tournament and they blogged about their favourite prompts of the competition. Once again speaking to a LLM is like speaking to children, USE CAPITAL LETTERS TO CAPTURE THEIR ATTENTION.
• OpenAI open-sourced a light library to evaluate language models — you can use 7 different evals and check the results on OpenAI or Claude models.
• Mistral-8x22B is out — a new model that does something probably awesome.

Fast News ⚡️

• Last week I forgot to share the 2024 state of analytics engineering by dbt Labs. By overlooking at it it depicts well the actual trends I also see in my local market and we see that in 2024 more than 50% of the time spent by data practitioners is to maintain or organise data assets.
• Introducing Beam YAML — Apache Beam is a unified processing framework (understand unifying streaming and batch) that runs on many different engines. Today they introduce Beam YAML a way to write pipelines the declarative way. Reminds me Kestra so much.
• How I became an AST convert — I've been an AST convert for a long time, I'm so happy someone writes about this. AST stands for abstract syntax tree and can be an abstract representation of a language, and this is what SQLGlot does (hence SQLMesh) to SQL. Afzal writes in this blog what it means, especially in a diff context.
  
  You can also listen Toby interview in Joe podcast about SQLMesh and SQL transformations.
• Airbnb open-sources Chronon — A data platform for serving for AI/ML applications. In Chronon you defined sources and groupby—a collection of aggregations on keys—which represents in the end features and then the platform handle the downstream management.
• BigQuery releases data canvas — This is a large open canva (like count.co) in which you can write SQL queries, assisted by Gemini, and link them in a DAG fashion.
• Write-Audit-Publish pattern in modern data pipelines — A pattern worth to be known more because it can avoid your pipelines pushing wrong data into your users tools.
• Using Preset (Superset) to explore the dbt Cloud semantic layer — You can configure a sync between both clouds or use a CLI, then you will be able to explore metrics in your BI tool.
• Book review of DuckDB in Action.
• Efficient CSV parsing — A YouTube talk about the DuckDB cvs parser and what it means to parse unstructured files.
• To conclude this edition 2 project walkthrough:
  • Slack summary pipeline with dlt, Ibis, and Hamilton.
  • Local pipeline development with SQLMesh, Airflow, and Postgres.

✨ s/o to Hugo who runs a weekly data round-up and this week he published before me so you can also check his great links selection.

See you next week ❤️

Hey, new Data News edition. I hope you will enjoy this week selection after skipping last week one. I was a bit overwhelmed with the amount of tasks I had on the desk—and I'm still. But here we are.

Before jumping to the news, I want to let you know that I have improved the Recommendations page and the weekly emails with the recommendation should arrive soon. The new page supports better the mobile and give you titles and overviews of links which are GPT4 generated.

I'll speak at the MDS Fest 2.0 next week on April 10. MDS Fest is a free virtual 5 days conference about Modern Data Stack topics, a lot of awesome speakers, there are a few talks I can't wait to watch. On my side I'll talk about Apache Superset and what you can do to build a complete application with it.

AI News 🤖

• LlamaIndex slides, examples with Mistral AI — A few slides with a lot of example on how you can use LlamaIndex with Mistral AI models. I guess there is a video associated with the slides, but I don't have it. It shows a few RAGs, Agents and parsers on documents to retrieve the data you need.
• DBRX, a new state-of-the-art open LLM — Databricks has to be an AI company (bragging vs. Snowflake). This week they released a new open model that performs great.
• How we built Text-to-SQL at Pinterest — Pinterest open-sourced a tool called Querybook that they used to access Pinterest data every day. In order to boost usage they developed a text-to-SQL feature. This article greatly explained how they did it.

Fast News ⚡️

• MAD 2024 landscape — The new edition of the Machine learning, AI and Data Landscape is out, with many logos and obviously changes since last year with the GenAI hype. I did not yet analysed the new map, but I'll try to do it soon.
• Polars on GPU — Polars announced that they are collaborating with Rapids to bring GPU performance to reach another summit with Polars. Looks cool to be able to switch Polars engines like this. If you are using Polars reach out to me I'm curious to know how people are using it.
• Git in Snowflake — Snowflake is getting more and more feature months after months. Becoming a fully complete suite of applications directly reachable from SQL in your warehouse. Reminds me of Oracle and I don't like this centralisation, but the future is always going the bundling way. Now you can read a git repository when creating procedure in the SQL DML.
• DuckDB is the new jq — The author show how you can manipulate a json file with a DuckDB one liner. I really like this take, it gives a great perspective about DuckDB and how you can use it locally to do fast manipulation. But contrary to jq which has a non-trivial syntax, DuckDB is SQL.
• Survey about query engines used by companies — Data Council happened recently. It's a US-based conference that I often really like because talks and ideas that are discussed there often shape what we do in the data industry, at least from what I see in the YouTube videos, I never went there. A talker did a survey during his keynote about the query engines used by the audience and Spark is still leading to BigQuery/Snowflake/Athena.
• How we built a 19 PiB logging platform with ClickHouse — ClickHouse is a tech company and you can see it from the blogpost. They deeply explain in this article why they choose ClickHouse to monitor their ClickHouse Cloud offering saving money on their Datadog bill.
• ❤️ Building open data portals in 2024 — David open-source a end-to-end framework to build open data portals. This is awesome (example), you can easily ingest, transform and share data, looks like yato but with many more features puzzled together to create a local-first data platform.
• Spotify, data platform explained — The beginning of a series explaining the Spotify data platform.
• A path from data mess to data mesh — 5 key principles you should apply to avoid the data mess.
• Semantic layers, a buyers guide — This is a exhaustive comparison between dbt Cloud metrics offering and Cube. In a nutshell, I'd say that both technologies are not yet mature, with a smaller advantage for Cube for being open.
• The data analyst every CEO wants — I really like this blog from Benoit, he gives practical advices about what you have to focus on if you're working as an data analyst for C-level of your company.
• YAML developers and the declarative data platforms — A good introduction to why declarative languages are perfect for creating data platforms. To be honest, I think this is a topic that can be used to compare good and great data engineers. Creating a declarative data platform is easy, but creating the right level of abstraction that describes reality without creating debt and over-engineered solutions is much harder.
• PR review tool for dbt projects — A nice tool creating visual representations comparing 2 dbt artifacts that you can embed in a CI to validate changes before they get merged into production code.
• When is the data model finished? — Spoiler: a data model is never finished. Actually you need to depict your company business and activities, as the time goes, activities grows and obviously you have to manage this asset in time.
• A training set for bike sharing forecasting — Max has created a large dataset of bike sharing providers in ~50 cities around the world. If you want to play with DuckDB and visualisations this is a good start.
• Calculating walking isochrones in Python — Cool way to produce Python viz.

See you next week ❤️

Dont forget to check out the new Recommendations pages (below an overview of mines).

It's Friday and it's Data News. I don't go into too much detail about the magic of Data News, but every Friday is the same. At first, I'm: oh shit, here we go again and 10 minutes later I'm lost in reading the content and picking too many articles to fit into a thousand word edition.

Usually all the process takes me a whole Friday. I organise myself as follows:

• During the whole week I scroll—too much—LinkedIn. Save posts without reading. Sometimes I also save stuff on Twitter by liking. The reason I do this is to avoid context switching—let's be honest, it works for the DN context, but does not work in general in my life.
• Exploration, Friday morning
  • I read the last 7 days of 2 Twitter lists (MDS, Data voices) and I open interesting stuff in tabs.
  • Then I use Feedly which is connected to ~500 websites, Reddit and Medium and opens interesting articles in tabs.
  • Then I opens the elements saved from LinkedIn.
• Reading and writing, Friday afternoon
  • I read the articles and remove what I find irrelevant (context, values, quality, etc.), I create a first connexion between all the links, trying to sort them to have a fluid path between articles ideas
  • I usually go from ~50 links to 25 after the reading part.
  • I write in one-go starting from top to bottom.
• Publication—Once the Data News is ready, I just click publish, I don't proofread much (sorry for the typos). I think I already spend so much time selecting + writing that I can't be stuck in revision mode for long.
• Post-publication—After the publication I do my homework of promoting my own work (mainly on LinkedIn), I run a few post-publication scripts for the Explorer / Recommendations. I also watch the click / opening stats and thats all. But I could do it better I think.

The process works well, but as you can see, because I use fresh news, it's just-in-time. Which puts pressure on my Fridays. I'd like to have a few articles in stocks to remove the pressure of having to write something on selected Fridays and be off.

❤️ I rarely say it, if Data News helps you save time you should consider taking a paid subscription (60€/year) to help me covers the blog fees and my writing Fridays.

Just before I jump to the news, I'll speak at the MDS Fest 2.0 on April 10. MDS Fest is a free virtual 5 days conference about Modern Data Stack topics, a lot of awesome speakers, there are a few talks I can't wait to watch. On my side I'll talk about Apache Superset and what you can do to build a complete application with it.

Ok. Now give me the news.

AI News 🤖

• 01 open interpreter — The 01 light is a small device operable with your voice that controls your home compute. I've been rarely amazed by the latest physical device AI startups produced, but this one is different. It's a small white sphere that understand what you says and then control your compute mouse to execute actions for you wether you're in front of your compute or elsewhere.
  
  The initiative wants to be open(-source) and they provided the code on Github. Actually they trained a "computer LLM". And they are the reason this newsletter was late, because you can build yourself the physical device with Arduino stuff—list of materials—and I wanted to do it today but a piece was not available at the electronic shop 🥲.
  
  Under the hood it uses a big prompt to instruct their LLM because in the end fuck you, show me the prompt. And actually it's always fun to read the prompt companies are using to do specific tasks. Sometimes it looks like you speak to a child. Capslocks and repetitions to make the algorithm understand.
• Finally, xAI released Grok-1 in open — The weights are available in torrent / HF and everything is under Apache License. The repo has been released last Sunday after Musk publicly announced last week the released, I feel bad for the sweaty engineers who worked the whole week on it. I did not see a lot of feedback on it since.
• Apple tries to enter the LLM game (while facing fines) — Rumours says they will partner with Google to use Gemini to power iPhone AI features, at the same time they wrote a paper about MM1, a family of multimodal models up to 30B parameters.
• Microsoft hires DeepMind co-founder — Mustafa Suleyman will lead a new organisation called Microsoft AI. Following the announcement Copilot, Bing, Edge and the GenAI teams will all move to the new organisation. Satya Nadella is going all-in on AI. It's important to say that Mustafa is joining Microsoft from Inflection a LLM company in which Microsoft invested 1 year ago.
• OpenAI closing partnerships with major newspapers in Europe — After Axel Springer in Germany, they sign with Prisma Media which groups Le Monde (France), El País (Spain) and the Huffington Post (worldwide). All these partnership will help OpenAI train GPTs on media corpus to enhance the reliability of the answers in return for a significant source of additional revenue.
• Commun Corpus — A HuggingFace dataset collection including public domain texts, newspapers and books in a lot of languages. TB size.
• Designing RAGs — A super long and detailed article about RAG. It covers the 5 main components: indexing, storing, retrieval, synthesis and evaluation. Let's be honest it contains all you have to know about this new trend and what are the key considerations.
• Vector DB comparison — A table comparing all the different vector technologies on different axes like the search, models, APIs and technical details.
• Python codebase with best practices to support MLOps — This is a Github repository with a lot, I mean a lot, of tools and tips to create a production grade repository.

Fast News ⚡️

• Run Spark procedures in BigQuery — BigQuery released a way to write PySpark code in the web editor and to run / deploy it from there creating a new serverless way to create BigQuery assets. This is a nice way to mix SQL and Python code.
• pip install data-stack — This is a title I could have written myself. In this blog Julien covers the new Pythonic tooling and how far it can bring us in building lightweight programmatic data stacks. He also mentions my baby yato.
• Pretzel notebooks — A new open-source notebook / exploration tool based on-top of DuckDB WASM and PRQL, it allows you to chain operation like upload file, SQL, charting, filter, sort etc. You can explore the demo.
• On the same topic Hashquery launched — this is a Python framework to create semantic data models.
• Williams F1 used Excel to build their car — F1 parts (thousands) were managed in a spreadsheet. These Excels files were unmanageable and explained why Williams had delays in deliveries. That's not surprising because I think during a F1 season there aren't a lot of breaks which you can use to pay technical debt.
• dbt Core unit testing in v1.8 — dbt Core has implemented unit testing and it's coming soon. When unit testing a model you can give input rows and says what you expect as output rows in the YAML definition. dbt will run and validate the model for you. This is game changer.
• Awesome DuckDB snippets — A website that collects cool DuckDB snippets. The most popular is a 4 line bash command that you can add to your bashrc to convert a CSV to Parquet.
• The cost of data incidents — Mikkel is one of my favourite authors, he carefully picks all his titles that they resonate deeply in me. He proposes a formula to compute the costs of your data incidents, changing downtime numbers into $.
• 2023 in reading — This is a great side project idea. This is a visualisation of the hours spent by Erin reading books in 2023. Personally I just finished my first book of 2024 😅.

This newsletter edition is already too long and I have 10 other deep articles that I'll keep for next week ❤️.

Recommendations have been computed this Wed., go check what the algorithm prepared for you. The email notification feature is almost ready so opt-in on the reco page to get your recommended link by email once ready. I know the mobile version of the reco page is buggy, I'll work on it next week as well.

I hope this e-mail finds you well, wherever you are. I'd like to thank you for the excellent comments you sent me last week after the publication of the first version of the Recommendations. This is just the beginning!

This week I've added a subscribe button in the Recommendations page in order for you to opt-in for the weekly recommendation email—every Tuesday. You can subscribe starting today on the page and you'll get emails as soon as I've developed the email sending—expected to be out at the end of the month.

Second point, I passed the 100 stars on Github for yato, which is a crazy amount! I'd like to do a bit of user research about yato, if you consider using it drop me a message please.

yato, is a small Python library that I've developed, yato stands for yet another transformation orchestrator. With yato you give a folder with SQL queries and it guesses the DAG and runs the queries in the right order.

AI News 🤖

• Mira Murati answers the Wall Street Journal about OpenAI Sora — OpenAI CTO has been asked a few questions about the underlying technology in Sora. She revealed a few insights. OpenAI consider for the moment Sora as a research output and might eventually be released later this year, it required "much much more" compute power than DALL-E to generate a video and they have a lot of interrogations regarding impact on elections or film industry. Saying mainly that "Sora is a tool to extend creativity".
  
  Last point Mira has been mocked and criticised online because as a CTO she wasn't able to say on which public / licensed data Sora has been trained on. When she was asked if it was on YouTube videos, Facebook or Instagram she said "I'm actually not sure about that".
  
  I personally really recommend this interview which covers a lot of interesting topics in 10 minutes.
• Elon Musk said out loud that xAI will open-source Grok this week. It's Friday and it seems they are later than me when it comes to release stuff. Just-in-time for a reminder about the fact that open-source ≠ open-weights when it comes to AI licensing but differences in weights licensing are not as important as they seem.
• Databricks invests in Mistral AI — Mistral successfully positioned as the main OpenAI rival by being integrated in all major data platforms (Azure, Snowflake previously).
• A French commission released a 130 pages report untitled "Our AI: our ambition for France". You can download the French version and an English 16 pages summary. Report includes 25 recommendations given by French-speaking AI leaders (Yann LeCun, Arthur Mensch, etc.).
• Assisted AI wars are around the corner — I'm only following the French news, but the government is proudly doubling its budget for "AI defense". From what I know, AI is mainly used as an information companion to find signals in the huge amount of data we generate, creating more efficient agents.
  
  This is related to Paris testing automated video surveillance during Olympics. The technology under this, is, Cityvision.
• Yann LeCun clashed with Elon Musk on Twitter about AI future. Musk thinking that AI will be smarter than any single human next year, while LeCun said "No" taking as en example the false self-driving car promise. More, LeCun believes that human information compression capabilities are still so far ahead of AI that AGI is not even close.
• Cognition AI introduced Devin — Devin is the first AI software engineer, Devin can, unassisted, do software engineering tasks like fixing Github issues (13% of success, previously best was ~5%), apply to jobs on Upwork, train and fine-tune its own models. I'm speechless.
• Building Meta’s GenAI infrastructure — 2x 24k GPU clusters and it's growing. I like how Meta tries to do stuff out in the open (or at least with some kind of transparency) but the number of GPUs is just disconcerting.
• RAG is the new trend — RAG means retrieval-augmented generation, it has been coined in 2020 (see more) and let's you ground AI models with facts fetched from external sources.
  • A real-time RAG chatbot built on Sharepoint and Google Drive
  • RAG on-top of DuckDB
  • RAG on LinkedIn data
• There is an exponential number of technologies in the RAG space, especially re vector databases that I don't even mention them but obviously post are all saying "ours is the best".
• Croissant: a metadata format for ML-ready datasets — In order to move forward, faster in AI and model building we need a interoperable and easy-to-use metadata format for ML datasets. This is Croissant. Starting today it will be supported by 3 majors platforms: Kaggle, HuggingFace and OpenML. Croissant is under mlcommons and you can have a look at the specification.
• The State of competitive machine learning — a study about ML competition platforms. Give a lot of insights on the market.

Fast News ⚡️

• Since the end of Feb. BigQuery supports DELETE to delete partitions in a SQL query.
• How I saved $70k a month in BigQuery — Junaid shared a few techniques he used to saved a bunch of dollars on the BigQuery bills, this is nothing new. this is more common sense but always works. In a nutshell it's: smarter schedules, tables optimisations, incremental, avoiding views and precomputing.
• Attributing Snowflake cost to whom it belongs — Fernando gives ideas about metadata management to attribute better Snowflake cost. Wether it's a dbt model, a Tableau dashboard or a Metabase question it has to be tracked to understand what drives your bills.
• Understand how BigQuery inserts, deletes and updates — Once again Vu took time to deep dive into BigQuery internal, this time to explain how data management is done.
• Pandera, a data validation library for dataframes, now supports Polars.
• PyData is coming to Paris in 2024 — The CFP is open and I submitted a talk there about yato.
• A comparison between Kestra and Airflow — Benoit (who works at Kestra) did a great comparison between the 2 tools, comparing the syntax to write DAGs and the performance in term of scheduling capacities—tasks per seconds. Obviously Benoit prefers Kestra, at the expense of writing YAML and running a Java application.
• New Apache Arrow engines — Arrow has become one of the most used library when it comes to built in-memory engines. Arrow doing a lot of the data operation heavy lifting.
  • Apache Arrow DataFusion Comet — a native Spark SQL accelerator, the idea behind is to improve Spark performance behind replacing Spark executor by delegating it to Comet. On the matter there is also Apache Gluten which is a plugin aiming to double SparkSQL performance.
  • Arroyo, a stream-processing platform, rebuilt their engine using DataFusion.
• Postgres creator launches DBOS, a transactional serverless computing platform — Mike sees DBOS like a cloud-native OS that runs on-top of the database in order to rethink application development and deployment.
• Unlocking Kafka's potential: tackling tail latency with eBPF.

Forward thinking

• Dataviz is hierarchical — Malloy, once again, provides an excellent article about a new way to see data visualisations. It's inspirational.
• Coding data pipelines is faster than renting connector catalogs — This is something I've always believed. The devil is in the details and when it comes to data pipelines there are a lot of details, which often refrain us to buy leading to build (or code). Matthaus gives the dlt vision about creating the foundation for developers to be able to create sources in a wink creating a large ecosystem of APIs datasets easily maintainable.
• Differential storage, a building block for a DuckDB-based data warehouse — It's MotherDuck vision, creating the next data warehouse on-top of DuckDB leveraging DuckDB morphing capacities between a single machine and a production ecosystem. In the article Joseph explains how MotherDuck extended DuckDB to add time travel, zero-copy snapshots opening the door for more collaboration and concurrency.

See you next week ❤️ — recommendations for this week have been computed, go check it out.

When I started writing this newsletter nearly three years ago, I never imagined that the words I write on my keyboard would take such an important place in my life. All the interactions I have with you, whether online or offline, are always amazing and give me wings.

Today I want to introduce a new feature in the Data News galaxy.

I don't talk much about my freelance life in Data News because sometimes I think that's not the contract we have together. Data News promise is to give you, every week, the links I've hand-picked with my spicy opinion about them. Since the beginning of the year balance between content and freelancing has gone from 80/20—80% client stuff and 20% to content—to 30/70. This is mainly due to the fact that I've done my annual University lectures and talked at 7 events since the beginning of the year.

Let's be honest, I'm also a bit stupid. At every event I talk, I decide to do a new presentation. That's great because it helps me innovate and pushes me to new horizons every time, but it takes time to assimilate chunks of work in order to produce creative keynotes.

All of this is made possible thanks to my Data News curation. Thanks to the time I spend reading content, forging ideas and chatting with all of you, I get inspired and my crazy brain invents things. And I want you to have the same superpowers as me. This is what motivates me.

PS: Fast News ⚡️ at the very end if you want to keep this story. Which will makes me sad, but I understand.

There is a problem

Data News have grown so much since the beginning, I currently have 4500 members on blef.fr. I have sent 132 Data News editions which represents 2500 links (~20 links per edition).

But there's a big problem: all my old Data News is dead content.

I mean, there is a big difference between podcast for instance and news blogging like I'm doing. When you subscribe to a new podcast you often scroll over the past episodes of the creator. When someone subscribe to the Data News rarely the person goes over my old news.

All these 2500 links that I've liked and commentated. When I'm looking at all these links for the most of them they are timeless and I think they can still bring a lot of value to all of you.

That's why I want to re-activate my old content.

The Explorer

One year and half ago I had developed the Explorer. The Explorer is a search bar that let's you search over all the links that I have shared in the 132 Data News editions.

It was my first step in this journey to make my handpicked links browsable and usable to everyone. While I'm not good at marketing it there is a few number of you using it every month but I think it could be used way more.

But I want to go further.

Introducing the Recommendation

2500 links is a huge amount and sometimes this is like finding a needle in a haystack. That's why I've developed a new feature: a recommendation module.

Data News recommendation will give you every week a single link that you should have clicked on.

For the moment the recommender will be based on your click history. In every Data News email I send you I know which link you clicked on, so I'm able to leverage this information to recommend you content.

This is just the beginning and for the moment the algorithm is very trivial, this is a collaborative filtering algorithm that recommends you links you did not clicked on that have been clicked by members with the same click behaviour as you.

As you can see in the screenshot of the feature in the Recommendation panel you can see the link that have been recommended to you and the link you've clicked on. In order to for me to get your feedback you have the possibility to like / disliked all the links (wether it's recommendation or clicked links).

Christophe, why did you make this? No one asked for it.

Yes no one asked for it but let me extend deeper on the why

• Frustration — Like I said before, I'm super frustrated by all the content that I've referenced that is "dead". I'm pretty sure that if I successfully reactivate this content I can: generate more traffic on blef.fr, diversify my revenue and bring more knowledge to the data community.
• It's a showcase — It can be an educational project showing others how you can orchestrate and schedule a small-scale AI application.
• It's fun and rewarding — Looking from my side I like the fact that every week members will have a gift coming from me being this recommendation.
• Why not? — Finally I don't run any playbook, why not trying stuff?

Architecture

I said it, while being a new feature to the blog this is as well an educational projet I can use to showcase technologies. See below the global architecture I've used to make this links recommender work.

• Ghost — My blog is hosted on ghost.org, I really like Ghost because it's open-source (but I use the paid hosted version) and give me the possibility to extend the blog with custom code. The main part of the blog is just a bunch of Handlebars templates connected to Ghost Content API. I extended the website by embedding a React application that powers the custome frontend of the Explorer and Recommendation.
• blefapi — In order to make the React apps working I need to have a custom backend that I've developed with Django, this backend connects to Ghost using some kind of SSO (with JWT), which means that I don't need to create another login page, once you're a member you can use all my extended features. The Django app uses a Postgres as a database and a bucket to host a few static files. Everything is hosted on Scaleway (a French cloud company).
• CI/CD — Everything is just deployed from Github Actions, wether it's the React application of the Django API I just need to push and it will deploy a new version.
• newsletter-reco — This is where the recommendation magic happens. This is small pipeline that needs to get the activity data from the blog API, do a bit a feature engineering, recommend an article for every member and then publish the recommendations to the blefapi. Under the hood (see below) it uses dlt, DuckDB / pandas and Github Actions.

The recommendation is fairly simple, it uses dlt to do the extract-load from the Ghost API then dlt loads the data into a DuckDB database then this DuckDB data is transformed using SQL / Python transformations orchestrated by yato. In order to publish the recommendation to the API it uses the DuckDB ATTACH capabilities by directly inserting records to the Postgres database (it's a hack, but works). All of this will run into Github Actions every week to produce a new recommendation for everyone.

Next steps

I'll work incrementally in the next week on the recommendation, I'm open to all suggestion and I'd love to get your feedback on this, you can even do Pull Requests on the code if you feel it. Here what I plan to add in the following weeks:

• Subscribe to an additional email to receive the recommendation on Tuesday (if you really want to receive recommendation by email answer to this email I'll opt you in directly).
• Use GenAI to summarise all the links database to give you a summary of each link that have been recommended to you—saving you one-click maybe
• Improve the recommendation algorithm by using an item-based approach and embeddings
• Taking into account the like / dislike from the Timeline
• Develop public BI-as-code dashboard showing metrics about the content and showcasing Evidence and Observable

Bonus: yato

While working on the recommender I've developed something else called yato. yato stands for yet another transformation orchestrator and is the smallest DuckDB SQL orchestrator on Earth.

The idea behind yato is to provide a Python library (pip install yato-lib) that you can run either with Python code or via CLI that run all the transformations in a given folder against a DuckDB database.

yato uses SQLGlot to guess the underlying DAG and run the transformations in the right order. For the moment yato is tight to DuckDB, philosophically yato has been developed like black (the formatter) you just have one required parameter: a transformation folder and then you can do yato run .

I don't think yato will ever replace dbt Core, SQLMesh or lea, yato is just lighter alternative that you can use with your messy SQL folder.

It was a special announcement for me, I hope you'll understand and receive this news as excited as I'm.

And because I still want you to get a few news below a very fast news.

Very Fast News ⚡️

• Elon Musk decided to sue OpenAI for violating company principles by putting profits and commercial interest first. Funny to see this from Elon Musk the philanthropist.
• Google is slowly loosing the race for (Gen)AI, so people are starting to call for Sundar Pichai to step down.
• Anthropic released Claude 3 — that seems to achieve great results in benchmarks with "sophisticated vision capabilities".
• HuggingFace released Enterprise Hub — A private space to use HF features but in a dedicated space.
• Yann Lecun went on Lex Fridman podcast — He chatted for almost 3h. I did not listen the podcast yet but I guess he chatted about the concept of intelligence like he he used to do.
• Sicara released a tech radar about AI technologies. It includes 4 pillars: algorithm, data, methods and industrialisation. This is funny to see parquet as a technology still to adopt.
• Easy introduction to real-time RAG — Showcase how you can include your Langchain / OpenAI pipeline into a classic Kafka / Pinot infrastructure.
• ClickHouse acquired chDB a DuckDB alternative and achieved the 1 trillion challenge (with classic ClickHouse) in under 3 minutes for $0.56.
• Snowflake now support trailing commas and partners with Mistral AI to bring models to the warehouse, we also learn that Snowflake Ventures also invested in Mistral AI. Long gone are the days when mistral was French.
• Orchestra released a free-tier platform to rapidly build and monitor data products. Orchestra is graphical solution to define DAGs and orchestrates different parts of the Modern Data Stack.
• Use Ibis to load data from other databases to Snowflake. This is similar to the ATTACH I did in my recommender with DuckDB.
• How to measure a data platform — A great article discussion the metrics tree we need to put in place as a data team. I really like it.

See you next week ❤️ — and please give me feedback wether you like it or not.

Hello all, this is the Data News, this week edition might be smaller than usual in term of comments as I'm working on a Data News related project that takes me a bit of time, which will probably lead to a series of articles.

Before I forget I've appeared on The Joe Reis Show, we chatted with Joe about data engineering teaching, why it is hard and about generative AI that will change education for ever. This is a 1h podcast, I hope you will enjoy listening to it.

Final reminder, next week there is La Conférence MLOps which will take place in Paris on March 7th. If you want to register I sill have a 40% promocode: mlops-blef-40. I'll give a talk—in French—about how to put in production machine learning at a small scale. Topic which is related to the Data News project 😬.

AI News 🤖

• Mistral AI announcements
  • Mistral Large, their new flagship model, which outperform other concurrent excepting GPT-4. At the same Microsoft closed a partnership with Mistral to make Large available to Azure, their first distribution partner. It has led to a lot of discussion in French politics about Mistral AI being American more than French. With the partnership Microsoft entered the Series A with a €15m addition joining a16z.
  • They also released a smaller model called Mistral Small.
  • Le Chat, the conversational interface to interact with Mistral models.
  • Final comment, with these 2 announcement Mistral left the open side to go commercial / closed. It led to conversation where people felt betrayed by Mistral which built their differentiator—or should I say marketing—on-top of open-source/weight models. Mistral perdant.
• GitHub Copilot Enterprise is now generally available — This week I've started to use GitHub Copilot (not the Enterprise version). And let's be honest this is a productivity boost, especially when you want to write docstrings and comments. Still there is an annoying interaction in PyCharm where Copilot takes too much space. Copilot Enterprise mainly comes with 3 features: understand your whole org codebase, a chat to ask question about the codebase, summarise pull requests.
• Fast, efficient active speaker detection on videos — This is a great introduction to active speaker detection, which means you are able to detect in video speaker faces and if they are actually speaking or not.
• Klarna's AI customer support agent do the equivalent of 700 agents — Klarna developed an AI agent that interacts automatically with customer driving profit. It has to be put in context.
• Using DuckDB + Ibis for RAG — Handy code snippet to explain why DuckDB is a good solution bringing best of both world when it comes to RAG.

Extract and load, still unsolved 🤭

I've started writing data pipelines in 2014 and the movement from sources to destinations has always been one of the most discussed topic in my data engineering spaces. Personally I'm the kind of guy who likes to build it custom because I think an out-of-box solution does not exist. In the end you finish with a composable solution mixing up 2 or 3 technologies to extract and load you data in your central storage, ready for transformations.

In 2024 we are more than ever tools to move data from sources to destinations. But the field has taken a new direction.

Until now, solutions were mainly full platforms (often in the cloud) with the promise to do everything in search of rebundling the data platform (cf. The unbundling of Airflow). Recently, it has reached new heights: what if the extract and load is just a small library layer that integrates whatever you're doing—for people reading me carefully this is what I was calling for in using Airflow the wrong way, but the fun way.

Enters the new kids on the blocks:

• dlt — it stands for data load tool, it's a Python library installable with pip. It provides a framework to do the extract and load, you need to define sources and resources what are the specificities of the resources you want to load: primary keys, write disposition, incremental mode, etc. and the library does the heavy lifting accordingly.
• PyAirbyte — Airbyte announced their Python library in beta. Currently it support around 250 sources, which is a subset of all Airbyte sources (only the ones written in Python) and it seems it does not support connecting to classic databases. They call a destination a Cache, which is a terrible name. Even if the library is a great idea I feel this is a sad that the interoperability with Airbyte is not 100%.
  
  Adrian from dlt wrote a small post about PyAirbyte.
• CloudQuery — Written in Go, YAML driven configuration to move data.
• ingestr — ingestr is a CLI tool to copy data between any databases with a single command seamlessly. It's built on top of dlt.
• Slings — Sling is a CLI tool that extracts data from a source storage/database and loads it in a target storage/database. Written in Go.
• Let's not forget Meltano.

We see a pattern here, when we talk about extract and load there are 2 kinds of sources: databases and APIs, behind able to do both correctly is the key.

On the other side of the movement there is a new open-source reverse-ETL technology called Multiwoven/multiwoven. This is built in Ruby (haha). At the moment it can sync to Facebook, Salesforce and Slack.

Fast News ⚡️

• Your first 90 days as a head of data — Handbook and roadmap with pragmatic insights on what to do in your new journey as head of data.
• Career pathways of data engineers — IC, manager, being a data engineer or a data full stack. It covers great topics.
• Google Search filetype:pdf was not working for a moment — Internet panicked and believed Google was continuing to downfall. But actually it was a bug.
• BigQuery time series data — BigQuery will now support time series analyses.
• Snowflake access management — I've already share Teej work in the past, but now he launched a company to solve Snowflake access management, using code. I bet it gonna become the best solution for this issue out there.
• Rill dashboards for ClickHouse — Rill works now with ClickHouse.
• Fast Poetry and pre-commit with GitHub Actions — An efficient and useful Github Actions to cache poetry installs in the CI.
• BigQuery cost dashboard app — Hashboard develop a dashboard to help you follow your BigQuery costs, it's free. It uses Hashboard, which is a BI tool. Even if you don't use it, it gives good ideas about what to track.
• Introducing DoorDash’s in-house search engine — Custom searcher built on top of S3.

Tech stuff

• Adidas Lakehouse engine.
• Why Apache Spark RDD is immutable?
• High-order and partially applied functions in Python.

See you next week ❤️

Hey, fresh Data News edition. This week I've participated to a round table about data and did a cool presentation about Engines. The idea was to depict the history of engines over the last 40 years and what leads to polars and DuckDB. Obviously the I forgot a few things and I'll do a more complete v2 soon.

This is my third presentation about DuckDB in the last 3 months and I think I'll slow down a bit until I get new crazy things to share.

There are 3 points that have triggered discussion about the visualisation I done

• What about Arrow? — Apache Arrow is an awesome library that powers a lot of innovations in the data space in the recent years. But UX is where it differs to others, DuckDB user experience is insanely magical. So yeah. But for sure I'll add Arrow in the v2.
• Spark future — I'm convinced that Apache Spark will have to transform itself if it is not to disappear (disappear in the sense of Hadoop, still present but niche). This is already happening, according to the feedback I've had, but Spark requires more infrastructure and investment, which will continue to drive adoption down, whereas the current trend is towards simplification.
• JVM vs. SQL data engineer — There's a big discussion in the community about what real data engineering is. Is it Java/Scala or Python? Is it DataFrames or SQL? Is it lake or warehouse? It's a sterile debate: both are useful and can serve different organisations with different service level for data users and stakeholders. Still, I prefer SQL/Python data engineering, as you know me.

Small reminder, I'm partnering with La Conférence MLOps, a half-day conference on the challenges of industrialising AI. It will take place on March 7 in Paris. The list of speakers includes many important figures from the French data ecosystem, and I'm very excited about it. You can get a ticket with at 40% discount with the following promo code: mlops-blef-40. We have only a few seats left.

AI News 🤖

• Mistral AI will release next week Mistral Next a ChatGPT alternative. We don't have a lot of detail because it has not been announced publicly—I got the news in a French politic newspaper. Still you can test mistral-next on lmsys. Here a first review.
• Google releases Gemma — Gemma is a family of open models. Available in 2 sizes: 2B and 7B it seems to have baseline performance compared to Llama-2.
• The same Google got a backslash after Gemini image generation rollout — Conservative people over social networks have been hurt because Gemini wasn't capable to generate image of white people. Google rolled back Gemini until further improvements.
• Models comparison across key metrics — I found it via Guido on LinkedIn, it shows a lot of cool metrics like for instance the price per token, the speed or the model quality.

Fast News ⚡️

• Is the modern data stack dead? — This is a follow-up podcast of Tristan Handy with Matt Turck—famous VC guy producing the MAD landscape—following last week post about the MDS. In this 40 minutes podcast they chat more in detail of the dynamics behind the end of hype regarding MDS, AI implication and the future of analytics engineering work.
• Is the modern data stack disappearing? — An article I wrote 4 days ago as an answer to the trend. Pragmatic and easy-to-read. Essentially I analyse why the modern semantic is an issue.
• State of the Duck — Introduction Keynote of the DuckCon that gives an overview of how is the current ecosystem and what's to come.
• PyIceberg 0.6.0: Write support — Yeah, finally I'll be able to play a bit more with Iceberg. Still you need a catalog to make it work.
• How you can write a Polars plugin — A dedicated website to explain how to write Polars plugins to extend the library capabilities. In order to do it you'll have to write Rust and Python code. This is a good way to enter the Rust world I guess.
• Unit testing for data engineers — Daniel describes what you need to know as a data engineer to write test. He mainly covers BDD (behavior-driven development) as opposed to TDD (test-driven development).
• Understand the design principles of Snowflake — Someone took a few hours to understand Snowflake internals and this is a great wrap-up.
• Aligning Velox and Apache Arrow — Go deeper into memory management and how you can create open standards across the different librairies.
• Enabling near real-time data analytics on the data lake — Grab showcasing what they did with Flink and Hudi to enable real-time use-cases.
• Retrieve, merge, predict: augmenting tables with data lakes — A paper that explains how you can improve data discovery on data lakes to finally augment a given table with new data. I did not read the paper except the introduction and a the first schema, but it looks like awesome.

• Building a cost effective solution using Fabric — Another look at Fabric. In the end the author creates a workspace and transform the data with pandas and DuckDB in notebooks. Thank you Microsoft.
• A newsletter about the streaming data space — Robin collected a lot of cool articles about the streaming ecosystem.

Cool ideas

• Open-source, current state and future hopes.
• Data will not tell you what to do.
• Turning ideas into AI use cases — the Product Manager point of view.

Data Economy 💰

• MariaDB takeover at $37m. MariaDB is a public company and could be taken over in by an investment company.
• Neurelo raises $5m seed to provide HTTP APIs on top of databases (PostgreSQL, MongoDB and MySQL). We can see it as a semantic layer but on software engineering side.
• Motif Analytics raises $5.7m seed. This is a tool made to analyse sequences, especially useful in web analytics / acquisition. They provide tooling to do without writing awful SQL queries.

See you next week ❤️.

No.

This question generated a lot of content last week, and a lot of words were written. I wanted to keep my answer short so as not to burden you with a few thousand more words to read.

• Modern data stack has been coined by US companies and VCs—mainly Fivetran / dbt Labs—as a word to quickly emphasis a way to build data stack in the cloud related to ELT. It was a well-suited marketing term, let's be honest.
• The time came and everyone took their place at the table to eat a slice of cake.
• A lot of people have issues with the modern word. Probably because it's not an explicit semantic, definition is relating to the present or recent times as opposed to the remote past. In this definition there are 2 issues.
  • is relating to the present — not all companies are in the same present
  • as opposed — the term creates an opposition between 2 worlds, creating something we always like in tech: a debate between 2 kind of technologies.
• Actually modern creates some kind of exclusion between new technologies and old technologies. It was useful as first for Fivetran or dbt Labs to be disruptive, but now that everyone is using the MDS is it still a good idea to create this competition? Especially if you want to enter the Fortune 500 where they actually use old tech?
• And, we should stop being cynical, who in the hell—in my readers at least—wants to work with SAP, IBM or mainframes in 2024? Because they still exists, numbers show that still 50% of companies are on-premise, when it comes to publicly listed companies or government stuff it's probably way higher.
• For these organisations the ideal of a modern data stack still resonate. Employees are stuck in hell regarding data tooling. Data projects are still failing to go in production.
• Personally, down there, my vision of the modern data stack has always changed over the years. As always, I don't blindly apply the principles by the book. The idea of dedicated storage where all the data lies with SQL transformations on top and top-notch CI/CD processes with everything-as-code and a galaxy of convenient tools around to be observable sums up what's modern about our data ecosystem.
• That's why I think modern data stack vision isn't going anywhere.

In my four years of freelancing, I've always said I build data platforms or data stacks, because who am I to judge whether I'm modern?

As a reference read my online friends views:

• Is the "Modern Data Stack" still a useful idea? — Tristan Handy, dbt Labs CEO. He mainly coined the term and whistles the end of the playtime. MDS was previously useful to align practices but now he thinks we should move on to analytics stack. And AI is around the corner to take all the lights while we actually do stuff at the bottom of the pyramid.
• The problem was the product — Benn Stancil, Mode CTO, scroll to mid-article.
• Everything Ends - My journey with the modern data stack — Joe Reis, author of Fundamentals of Data Engineering. Joe depicts his own journey and views and why it became a mess with too many companies on the radar. Creating finally the most fragmented platforms with no coherence at all, negating all the good MDS aspects.

Hey you, time for the Data News. Because I did not send the news last week you will get articles from the 2 last weeks. Last few days have been heavily packed with AI News as well.

Disclaimer, the 2 events below will be in French.

Before jumping to the news there are a few events I want to write about. Next Wednesday I will participate to a Data Night Talk a open discussion about AI & data engineering with other content creators. We will do it online / in-person. So tune-in. I working on a 10-minute light talk about data engines (🦆) and a funny game.

✨ Second, I'm partnering with La Conférence MLOps, a half-day conference on the challenges of industrializing AI. It will take place on March 7 in Paris. The list of speakers includes many important figures from the French data ecosystem, and I'm very excited about it. I may give a talk—but I'm not sure yet.

PS: This is not a paid partnership with nibble—the company behind the conference—but in fact they've been a client of mine for 2 years and have been a huge supporter of the newsletter since day one and they're good humans. I'm happy to partner with them.

AI News 🤖

The last few days have been stacked in term of AI News. Have fun getting through everything 😊, this is really cool how fast things are improving.

• OpenAI Sora, generate 1-minute long videos — OpenAI released yesterday a new generative model that is able to create 1-minute long videos from text prompt. At the moment this is not public and only in the hands of limit testers but the first look shows that OpenAI might be already ahead of the competition. You can see a few videos on their landing page as well as the current limitations.
• ByteDance boximator, create motion on images — Boximator is a friendly method to instruct generative algorithms with boxes. With the boxes you define a motion on an image and the models create a video out of it. ByteDance is the company behinds TikTok. At the end of the page you have a current comparison of generative videos and you can make yourself an opinion compared to Sora.
• Meta V-JEPA, fills the void in videos — V-JEPA is not a generative model. With this model you can "fill in the blanks of what’s missing—not in terms of the actual pixels, but rather as a more abstract description in this representation space".
• Sam Altman wants to raises $7 trillion — It was the WSJ news of last week about OpenAI CEO seeking for 7,000,000,000,000 of dollars. Obviously everyone tried to bet why he wants this money—which could came from UAE government—probably to enter the semi-conductor industry. I'm happy we correctly use money to save our planet /s.
• Canada partners with NVIDIA to bring more computing power.
• Groq, which speeds up LLMs inference — This week I discovered Groq a company that created LPU™ inference engine claiming to be the most efficient cloud-based provider (18x faster) in term of tokens/s.
• Google announced a "new" AI hub in Paris — Techcrunch is mocking US company about this announcement considered as a communication effort because the 300 members of the new AI hub are already working for Google and it's just a new office space.
• Still Google announced Gemini 1.5 — Gemini is the new name of Bard (as of last week) and blablabla Gemini is awesome blablabla. This is crazy how Google feels outdated when you look at smaller AI companies in term of hype or magic they are able to build. More details on Twitter.
• HuggingFace model usage on HuggingChat — HuggingChat is a Chat UI from HF that lets you play with whatever model works with it. It depicts how fluid is the model market. Mainly it shows that Mistral is currently replacing LLaMa. You can also see the models "market share" among the major APIs / Clouds vendors in a nice Sankey diagram.
• NVIDIA Chat with RTX — It's a Windows app (~37GB) that locally runs a GPT model to unlock chat with your files in a secure way out of the cloud. Happy gamers.
• French Finance ministry released a LLM to summarise legislative proposals — Called LLaMandement it's a fine-tuned LLaMa designed to produce neutral summary of law proposal to help the government with preliminary notes. All the data used for the fine-tuning is available on Gitlab and the FastChat command used as well. Here the English paper on arxiv.
• Why you need LLMOps — A great post that encapsulates all the words needed to understand what needs to be done when it comes to put LLM in production.

Fast News ⚡️

• ❤️ Observable 2.0 — Observable has a close place to my heart. Observable has been created by Mike Bostock the creator of D3js which is my Proust madeleine. Today they announced the 2.0 version which is mainly Charts as Code. It goes beyond the notebooks and become a static site generator for building fast, beautiful data apps, dashboards, and reports. I'm so excited to play with it.
• Introducing universal SQL — I have to talk about Evidence now, which is as well a static site generator for building data front-end. They introduce universal SQL as a way to connect to all kind of datasources, adding interaction in the frontend while staying fast. Mainly it means data is exported in Parquet and compute with DuckDB WASM in the browser.
• uv: Python packaging in Rust — I've been using poetry in the last 2 years and I'm quite satisfied about it. Seeing a new kid on the block is good because it renew the ideas because let's be honest we will never have a de facto packaging tool.
• sqruff: SQL linter written in Rust — This is the results of the Rust hype, people are now porting more and more tools in Rust for efficiency. And it's for the better.
• Introducing the column explorer in MotherDuck — A cool feature in MotherDuck (DuckDB in the cloud) to add sparklines and columns distributions when looking at a dataset.
• dbt Labs announced a few new things to their dbt Explorer (that is only available to dbt Cloud). In a nutshell they announced column-level lineage, recommendations and semantic layer exports. This is fair to say that lineage is powered by SQLGlot (and Toby is not happy about it).
• Introducing vector search in BigQuery — RAG is everywhere and BigQuery enters the game.
• Snowflake lowers the cost of task from x1.5 to x1.2.

Engineering ⚙️

• Eventify everything — This is an ode to event modeling and a different way to think data modeling. Timo showcases how you can eventify your data model to think differently your business activity.
• What we learned after running Airflow on Kubernetes for 2 years — Outstanding article with great insights about the journey of running Airflow in production. It breaks down how to handle dynamic DAG generation, multiple DAG repository, configuration fine tuning and observability.
• A dataframe is a bad abstraction — The article is too long for me to read it right now but the title is enough catchy for me to put it in the newsletter. If you read it I'm curious to know what you think about it.
• Back Market’s journey towards data self-service — How to be a data self-service company and what initiatives they tried toward this journey.
• How to leverage Metabase for efficient self-service analytics? — 3 companies joined to shared tips about Metabase governance and monitoring. This is goldmine if you're using Metabase and struggling understanding how users are using Metabase.
• Automating data classification for the 21st Century — How Semantic Data Fabric (SDF) is able to statically infer data types and lineage how of a SQL patrimony. In a sense SDF is a dbt alternative. I really like this article.

Food for thoughts 🍱

• What founders need to know to build a high-performing data team — Centralised, distributed or hybrid data team? This article discuss it.
• Let's elevate the role of analytics.
• Data ownership: A practical guide.
• Is the Modern Data Stack still a useful idea — I'll write more about this later I guess this is too big too write my views in a bullet point. The article is about Tristand Handy views about the future of the modern data stack.

See you next week ❤️.

PS: I'd love to get your feedback about the newsletter either if you recently joined or if you're here since the beginning. More I'd love, as well, to understand who you are and what eventually would make you financially support my content creation activity.

Hello here, this is Christophe from Amsterdam. I hope you're doing good. I'm in Amsterdam for the day for the DuckCon #4. The DuckDB annual conference, and god I like Europe. Being able to travel by train from Berlin to Paris to Amsterdam while going to the west of France for a lecture in a week is something truly awesome.

Anyway this week will be a mixed Data News with links, stuff and ideas and a small wrap-up of the DuckCon + the stuff I presented on Wed. to a Modern Data Stack meetup in Paris about DuckDB WASM. I hope you'll enjoy it.

The text-to-sql problem

Every once in a while the people are trying to give a shot at the text-to-sql problem. Each time a new breaktrough is happening (meaning a new LLM) company launch and people tries. 2 weeks ago TextQL raised $4.1m seed trying to solve this issue.

But what problem are we trying to solve?

In fact, I think we're trying to solve two different problems. The first is self-service, we want our stakeholders to be able to access information on their own and with no errors, once again chasing the dream that our clients can navigate the data jungle on their own, in fact this problem is "text-to-insights". And there's the second part of the problem which is much simpler, a data copilot, which can be a tool that accelerates the productivity of data workers by bootstrapping SQL writing or analysis.

Obviously when it comes to self-service we need a layer that does a text-to-sql conversion. In the current cycle of hype it can be done with LLMs, like DuckDB-NSQL-7B, the one MotherDuck provided recently. Like every model you have to analyse the efficiency of these generation layers.

From my own little experiments in this field here what I can say a generating layer can behave like an analyst but will be way more stupid than an analyst. I mean, on one side a LLM can get a thousands lines queries right the first time, like an analyst, it has to be done incrementally, either with prompt for the LLM or by test and run by the analyst.

But there is something that limits the LLM: his business understanding. Even if you give your LLM access to the database, the codebase and the docs there is something the LLM does not have: the implicit (vocal) business rules that are written nowhere.

I have 2 thing for the conclusion:

• Have a look at what Alan did as a Copilot / Metabase bot to help people getting insight — by people in this case it means the CEO, who is explicitly saying on LinkedIn "It's incredible I don't need to ask anymore my People or Data Analysts team" — 😬
• Having a data catalog does not mean that people knows what to do with the data [they just know it exists] —this is like an aggregate of quotes from my Wed. conference.

State of the French data market

2 benchmarks have been published recently about the French data market.

• The public sector released their salary grid for all tech workers — this is in French but scroll to the last page of the PDF to have the table
  • We have 4 experiences buckets <5, <10, >10 and >20 years. Which is completely relevant for the tech / data field I think, only a few people are +20 years from what I see.
  • This is crazy how bad data engineers are paid compared to all others positions — especially when you know that other positions are doing data engineering when there is no data engineers
  • The comparaison to data scientists is nevertheless not relevant because very often data scientists have PhD. so make sense they start higher that other positions
  • What do you think of it?
• At the same time the Modern Data Network released the annual benchmark of data professionals
  • An Analytics Engineer role enters the chat — this is explained because the MDN is full of startup and roles evolves faster than elsewhere.
• This is complicated to compare the 2 benchmarks because the experiences ranges are not the same still we see trends that are similar between the positions.
• My main takeaway is that Data Analyst role is finally taking the place it should take as a full role and not a transition role before a DS or a DE role — being paid higher than other at entry for instance.

Fast News ⚡️

Not to fragment the news that much because I already wrote too much AI News is blended without the Fast News.

• OLMo, a new open-source LLM — The Allen Institute in Seattle released what they called a truly open LLM. For the first time we have the model, the weights and the training data. I can't wait to see how it compares and used by people.
• hf/moondream1 — This is really awesome, this is a tiny LLM that can answer questions about a given image.
• :probabl. launch — The team behind scikit-learn is joining forces and creates a new venture with the goal to maintain a state-of-the-art data science tooling suite to benefit France, EU and the World.

• github/cybersec-ctf-box — A cybersecurity CTF to train yourself. A friend of mine created this repo to train yourself against a few attacks you might face. The first one is around Chart.js library.
• dbt Labs names a new CTO — He was CTO at MongoDB previously.
• Don't fix bad data, do this instead — This is never a good idea to apply patch on bad data. Always remember to identify root causes before jumping on the fixing wagon.
• Our transformation journey toward an open data platform — Condé Nast data platform walkthrough. The platform is built on-top of Databricks with a lot of other logos revolving around making sense of the lakehouse platform.
• Mastering Airflow variables — All the different techniques to master Airflow variables.
• Grab, rethinking stream processing: data exploration — How do you unlock analyst super-powers by giving them capabilities to analyse real-time data directly on streams and not in the offloaded lake data.
• How to build high-performance engineering teams — Get click baited like me. If I can add a following point, step 0 is important but then you need to give them enough freedom and vision.
• The business-critical data warehouse — Putting back the church at the center of the village.
• Databricks acquires Einblick — It goes back to the text-to-insights problem. Einblick is a drag-n-drop solution to "solve any data problem in one solution". LMAO marketing teams at the finest.

DuckCon + my Duck stuff

Because this Data News is already too long I split the content into 2 articles. Read my DuckCon takeaways 🦆.

Still last Wed. I've presented DuckDB to a French audience during this presentation I've showcased what you can do with DuckDB and DuckDB WASM. WASM is a portable way to run DuckDB in the browser.

You can play with the SQL editor I've worked on here (mobile + desktop), try to run a small group by query after a load tables, everything you do run on your device. This is the wasm magic. There is as well the Firefox extension the let's you hover parquet file in cloud console to get the schema, but more of this later as I plan to push it forward this month.

PS: I'm so happy to met a few readers IRL, it anchors my content and my work into the reality. So once again to the few people who came to me, thank you so much.

See you next week ❤️.

Hey, this is a straightforward post about the ides and the takeaways I got for the DuckCon. I guess the recording will be posted online a in few days / weeks.

It took place in Amsterdam in a wonderful location. The agenda of the afternoon was quite small (because it is still a small conference) but interesting. There is something awesome to meet the DuckDB community at this step. The tool has not yet reached his peak so you meet people that are early adopters and fans of it — it's a nerds (male — diversity might come later I hope) conference actually.

DuckDB announcements

The Duck creators announced that v0.10.1 is coming soon and before end of July we might get the v1.0.0. DuckDB adoption numbers are demonstrating a real trend behind the "hype". DuckDB docs website gets 500k unique visitors per month and DuckDB has a new shiny website.

Soon we will get things like:

• Forward (best efforts) and backward (guaranteed) compatibility between duckdb file formats
• Attach Postgres database to execute Postgres queries from DuckDB prompt
• Fixed lengths arrays new data type
• A new unified memory manager
• A secret manager that can persists between sessions
• A new compression algorithm called ALP that brings faster compression / decompression and higher compression ratio
• v1.0.0 will have no new feature compared to v0.10, focus on stability stuff and bugfixes

Ideas from talks

I'll just throw in the wild ideas and stuff I've seen from talks.

• HuggingFace is using DuckDB in multiples features to power data exploration in the frontend. In their datasets product when looking at a dataset you can full text search or see distributions (with bars at the top of columns) and this is powered with DuckDB. Lastly they pre-compute statistics on datasets with DuckDB.
• Fivetran uses DuckDB as the tech to do file merge in the data lake offering
• Datacamp uses DuckDB to be able in notebooks to query dataframes in SQL and consider it for teaching SQL — I might have something in the making about this on my side.
• dbt Core developer is using DuckDB is pdb to debug what happening in the database pretty easily and can create "debug packages" to send to other people.
• DuckDB feels magical for a few people (Liverpool FC) because it does stuff faster than other technologies with less technical footprint — you just write SQL and it works.
• The pattern might me
  • Get the data out of db
  • Query it with DuckDB
  • Put it the data back into the db

In conclusion

There is something between the lines, even if DuckDB is used differently by everyone it just runs and creates something universal (thanks to SQL). Actually this might be the final tool that will break the wall between tech teams and data teams.

With DuckDB you offload a business logic that would be embarked in a backend app into SQL queries. You can use DuckDB as a library and not a service, which changes everything, what you need to do is import duckdb and not launch a Docker service manage connection strings, etc.

Last point, parquet was the starting point of a lot of use-cases because the Duck is working well with the columnar files. But between all the question and feeling people seems to like the idea of a DuckDB file format that will become the defacto data format.

Let's see.

I'm sorry I've written this as enhanced raw notes, I hope you'll like it.

Hey, new week new email. This is already end of January but I took time to travel and see people I did not see for a long time so I'm super happy how this new year is starting.

Next week, I'll be wrapping up my DataOps lecture by incorporating how to deploy machine learning models. This is a fun part where students learn how to serve a simple classifier in production. Building a custom HTTP API, Docker image and CI/CD processes making it accessible on internet. For the modern part this year, I'm going to integrate an LLM "classifier" part, it might attract their curiosity. We'll see.

Yesterday an interview I did for the podcast Let's talk AI has been published. Available everywhere. We talke about data engineering, freelancing and career stuff.

Data & AI products

Yesterday I went to 5h conference organised in Paris about Data & AI products—in French, the idea of the conference was to mix people coming from data and product ecosystem which is, let's be honest, the key enabler for AI in production. The recording will be online in a few days / weeks and I'll share them once online.

Here a few takeaways in a messy way:

• Data products and organisational impacts
  • Data engineers are still the limiting human resource.
  • Data mesh by the book will not work, if you want to scale you can't just add more people in a central team.
  • Data mesh means decentralisation but more importantly ownership and responsibilities to team (esp. data producers)—if every team has to be responsible you need to have a easy-to-use platform and you have to explicitly give them responsibilities.
• UX for data products — This is a presentation I really enjoyed by Claire Lebarz, VP data at Malt. Without the voice you will miss a lot of things, still it contains great practicals tips.
  • Before jumping to AI projects you need first to start with words and to define metrics reflecting [your] values. You don't want your AI to give bad product experience. So define—as a metric—what you don't want to have.
  • Then Claire schematised human interaction with models (slide 8) via an interface with inputs and outputs. Inputs and outputs can be instrumented with multiple techniques that will empower people in their interaction with AI algorithm.
    • Inputs — This is what you ask from the users to feed your algorithm. It can be done with calibration, implicit or explicit feedback and corrections.
    • Outputs — Product design choices where you give power over the algorithm. It can be done with multiple options (like trips alternative on Google Maps), attributions (why something has been recommended), confidence interval (weather) or limitations.
  • As data people you need to build relationships with designers to converge on common terms about human-AI interactions
• And other bits I got from the others talks
  • OKR means metrics alignement across the company which lead to team autonomy—AI teams should be autonomous in finding solutions to move indicators
  • It critical to have dashboards measuring success when AB testing models
  • "Product is about people crafting together to best solutions and experiences to solve a customer problem" — Anne-Claire Baschet.

AI News 🤖

• This Chinese startup is winning the open source AI race — Thanks to Mistral AI open-source became the new standard among the community. There is a Chinese company called 01.ai who wants to build the first killer app of the Gen AI. (see also open-sourcing the future of AI, which is a HuggingFace praising post at some point).
• Hugging Face and Google partner for open AI — Do not mistake it's open AI and not OpenAI 😬. This partnership will benefit Google Cloud customers with unique hardware to train models and HuggingFace users will have some benefits but I did not understand the corporate sentences from the press release.
• OpenAI new embedding models and API updates — new Turbo models for 4.5 and 3.5 and 2 new embeddings models.
• Unleashing the power of LangChain — From POC to production, it showcases the LangChain expression language that helps developers chaining prompts in a nicer way.

Fast News ⚡️

• Disney Holotile VR floor — Disney developed a "dynamic" floor for VR use-cases. With it you can walk without really moving. This is a bit disturbing but it can unlock the metaverse future.
• ClickHouse and the one billion row challenge — ClickHouse proposed a SQL solution with ClickHouse local to the a challenge consisting in aggregating 1B rows in a text file. Initially this challenge has to be answered in Java. The leader submitted a solution running in less than 2s—have fun—while ClickHouse took 19s.
• SQLGlot switch to Rust — I really like SQLGlot, this is a SQL parser that gives you back the AST to do stuff. They ported the parser from Python to Rust and got 30-40% performance improvement.
• 2024 data engineering trends — We are still in January so it's still valid, Anna captured a few things that will make data teams busy this year. Firstly the reducing in resources leading teams to do more with less (or at least doing the same with less).
• Snowflake batch data loading — A good explanation of the Snowflake kCOPY INTO command and what you need to setup around it to make it work.
• Effective pandas 2 is out — I did not read the book. As pandas is, still, everywhere, it can be a good ressource if you need to learn the 2.0 version.
• Introducing girls to code, one flower at a time — An awesome initiative to introduce girls to code and data visualisation through creative coding projects, there is a Notion guidebook to do data visualisations with p5.js.
• The open-source enterprise data platform in a single portal — Bern local community open-sourced an data platform blueprints to launch an all-in-one data platform with dbt, Airflow and Superset on top of Postgres and K.
• Github/dbt-assertions — A dbt package to write dbt tests at row-level and to save exceptions alongside your failing rows (cf. example).
• PR comment template for dbt data projects — Great stuff. This is a proposal of Github pull request template when modifying dbt models. It includes description, lineage diff, illustration of model changes or impacts and more.

See you next week ❤️.

This is a special edition of the Data News. But right now I'm in holidays finishing a hiking week in Corsica 🥾. So I wrote this special edition about: how to learn data engineering in 2024.

The aim of this post is to create a repository of important links and concepts we should care about when we do data engineering. Obviously I'm full of bias, so if you feel I missed something do not hesitate to ping me with stuff to add. The idea is to create a living reference about Data Engineering.

📬 Subscribe to the excellent weekly newsletter 📬

A bit of context

It's important to take a step back and to understand from where the data engineering is coming from. Data engineering inherits from years of data practices in US big companies. Hadoop initially led the way with Big Data and distributed computing on-premise to finally land on Modern Data Stack — in the cloud — with a data warehouse at the center.

In order to understand today's data engineering I think that this is important to at least know Hadoop concepts and context and computer science basics.

• What is Hadoop? A quick overview of what everyone used for years (and still using it for some of us). It's important to understand the distributed computing concepts, MapReduce, Hadoop distributions, data locality, HDFS.
• Data & Data Engineering — the past, present, and future ; this is a good overlook on data engineering history.
• This one is a gitbook with a lot of content but I recommend you specifically to read the introduction to data engineering.
• In order to become a great data engineer you'll also need to understand computer science. How do computer works? Additionally by understanding how web works — frontend & backend, deployment, etc. This is oversimplified but I did not found a simple resource on this topic, so if you have something, I'm interested.

Who are the data engineers?

Every company out there has his own definition for the data engineer role. In my opinion we can easily say a data engineer is a software engineer working with data. The idea behind is to solve data problem by building software. Obviously as data is different than "traditional product" — in term of users for instance — a data engineer uses other tools.

In order to define the data engineer profile here some resources defining data roles and borders.

• Data Organization: why are there so many roles ? — And why it is important to understand them. This is one of the most synthesized article about data roles. Furcy defined Programming as the core skill for data engineers.
• To complete the picture here are some missions and skills that are expected to be done by data engineers. Warning, the article is from an online bootcamp but they summarize pretty well everything. You can also have a look at the gov.uk data engineer job card, they detail every seniority level expectations.
• We don't need data scientists, we need data engineers — for years companies were hiring data scientists because it was booming, then realized they were in need for data engineers to team up with scientists. This post shows the data job market with numbers.

What is data engineering

As I said it before data engineering is still a young discipline with many different definitions. Still, we can have a common ground when mixing software engineering, DevOps principles, Cloud — or on-prem — systems understanding and data literacy.

If you are new to data engineering you should start by reading the holy trinity from Maxime Beauchemin. He wrote some years ago 3 articles defining data engineering field.

• The Rise of the Data Engineer
• The Downfall of the Data Engineer
• Functional Data Engineering — a modern paradigm for batch data processing

There is a global consensus stating that you need to master a programming language (Python or Java based) and SQL in order to be self-sufficient.

Some concepts

When doing data engineering you can touch a lot of different concepts. Firstly, read the Data Engineering Manifesto, this is not something official in any kind but it greatly depicts all the concepts data engineers daily face.

Then here a list of global resources that can help you navigate through the field:

• The Data Engineer Roadmap — An image with advices and technology names to watch.
• Reddit r/dataengineering wiki a place where some data eng definitions are written.
• This book, 📘 Data Pipelines Pocket Reference, defines everything related to data pipelines and how to treat data movement from source to target.

If we go a bit deeper, I think that every data engineer should have basis in:

• data modeling — this is related to the way the data is stored is a data warehouse and the field has been cracked years ago by Kimball dimensional modeling and also Inmon model. But it recently got challenged because of "infinite" cloud power with OBT (one big table or flat) model. In order to complete your understanding of data modeling you should learn what's an OLAP cube. The cherry on the cake here is the Slowly Changing Dimensions — SCDs — concept.
• formats — This is a huge part of data engineering. Picking the right format for your data storage. Wrong format often means bad querying performance and user-experience. In a nutshell you have: text based formats (CSV, JSON and raw stuff), columnar file formats (Parquet, ORC), memory format (Arrow), transport protocols and format (Protobuf, Thrift, gRPC, Avro), table formats (Hudi, Iceberg, Delta), database and vendor formats (Postgres, Snowflake, BigQuery, etc.). Here a small benchmark between some popular formats.
• batch — Batch processing is at the core of data engineering. One of the major task is to move data from a source storage to a destination storage. In batch. On a regular schedule. Sometime with transformation. This is close to what we also call ETL or ELT. The main difference between both is the fact that your computation resides in your warehouse with SQL rather than outside with a programming language loading data in memory. In this category I recommend also to have a look at data ingestion (Airbyte, Fivetran, etc.), workflows (Airflow, Prefect, Dagster, etc.) and transformation (Spark, dbt, Pandas, etc.) tools.
• stream — Stream processing can be seen as the evolution of the batch. This is not. It addresses different use-cases. This is often linked to real-time. Main technologies around stream are bus messages like Kafka and processing framework like Flink or Spark on top of the bus. Recently all-in-one cloud services appeared to simplify the real-time work. Understand Change Data Capture — CDC.
• infrastructure — When you do data engineering this is important to master data infrastructure concepts. You'll be seen as the most technical person of a data team and you'll need to help regarding "low-level" stuff you team. You'll be also asked to put in place a data infrastructure. It means a data warehouse, a data lake or other concepts starting with data. My advice on this point is to learn from others. Read technical blogs, watch conferences and read 📘 Designing Data-Intensive Applications (even if it could be overkill).
• new concepts — in today's data engineering a lot of new concepts enter the field every year like quality, lineage, metadata management, governance, privacy, sharing, etc.

The modern (and the future) data stack

Coming from Hadoop — also called the old data stack — people are now building modern data stacks. This is a new way to describe data platforms with a warehouse at the core where all the company data and KPIs sit. Below some key articles defining this new paradigm.

• 👍 The Modern Data Stack: Past, Present, and Future
• Emerging Architectures for Modern Data Infrastructure
• The New Generation Data Lake
• Bootstrap a Modern Data Stack in 5 minutes with Terraform
• Modern Data Stack as a Service
• Storm in the stratosphere: how the cloud will be reshuffled
• ❤️ A path towards a data platform that aligns data, value, and people

And now some articles I like that will help you get inspiration.

• Gitlab Data Team Handbook — One of the best data resource. This is a public documentation on how Gitlab data team do stuff.
• Airbnb is great at exposing what they are doing in term of data. For instance with these 2 articles: How Airbnb achieved metric consistency at scale & How Airbnb built “Wall” to prevent data bugs
• Data Engineering patterns are important — Dagster tried to introduce Software-Defined Assets and Prefect spoke about Positive and Negative engineering.
• Scaling data analytics with software engineering best practices
• Jesse Anderson ; Creating a Data Engineering Culture and his book 📘 Data Engineering Teams
• What is MLOps? Some people wrote a white paper detailing Machine Learning Operations (MLOps): Overview, Definition, and Architecture in which they write about rols and missions.

Once again if you feel I forgot something important do not hesitate to tell me. I'll add more and more stuff to this article in the future.

If you enjoyed this article please consider subscribing to my weekly newsletter about data where I demystify all these concepts. I help you save 5 hours of curation per week.

Hey I hope this new edition finds you well. We are deep in the winter, it's time for comfy Data News to read near the fire 🔥.

This week, on Monday, I started my annual university lecture. It's been 9 years since I started teaching and this year something was different. The students were incredibly calm, obviously my course is a bit difficult at the beginning because it touches on concepts that they are not used to—cloud, data in production, data engineering, etc. So it's normal that they don't have any questions at first. But still, even during exercices hands were still down when previous year they were asking me for debugging help.

This year something was off.

On Wednesday I finally understood what changed. It was ChatGPT. Actually the whole class was using ChatGPT—I did a raising hand survey and everyone said yes. So now, the default go to was to ask ChatGPT questions rather than ask me, and then if ChatGPT does not have the answer they might ask me.

I still don't know how to react about this. I think it does not makes sense to ban ChatGPT, like it was stupid to ban Google Search at my time. But still there is something to do, I need to research and think more about it.

I assume that education will be radically transformed. Both ways. The way students learn will be different, but the way teachers teach will have to be different. This will force us to bring something to the class that ChatGPT can't: humanity.

AI News 🤖

• OpenAI opened an Elections Program Manager — The role is to support the "efforts around elections security and integrity for the EMEA region". We have this year the European parliament election. It deeply shows how OpenAI products are—or might be—used in order to win races. I guess they already have people for the US elections.
• Palantir CEO: U.S. eating everyone's lunch on AI — Let's continue on politics, at the World Economic Forum in Davos Palantir CEO said that within 10 years 95% of the world top tech companies will be American. I don't see any difference with today.
• Meta released MAGNeT — MAGNeT is a text-to-music and text-to-sound model capable of generating high-quality audio samples conditioned on text descriptions. It seems that it works well when generating sound effects.
• Zuckerberg teased 2024 Meta AI strategy — In a selfie video on Facebook / Instagram Zucky explained that Llama 3 is coming and that Meta is building a massive 600k H100 NVidia GPU infrastructure. It just represents $27b just in raw GPUs. But luckily for us Meta will open-source everything they do because they love us so much <3. In exchange we can wear the new Ray Ban Meta glasses with AI inside to give Meta more training data. We are just seeing the world shifting and we are all ok with it.
  
  If you want a less salty opinion than mine Oliver as always aced it.
• Prompt engineering with Bard — A recording from a local Google Developers meetup from last November. In this talk we discover a few concepts on how to talk to Bard. Mainly you can do it through the API or the UI and Peter explains that Bard shines in creativity, factuality and reasoning. He greatly explains the concept of grounding and why it matters.
  
  Just after he explains how you can "teach" Bard to reason with a reverse word example in which Bard fails. In order to do it you have to ask Bard to write and execute code in background but to activate the code execution feature "you're at the mercy of the classifier". This is our future, being at the mercy of classifiers.
• Google News is boosting garbage AI-generated articles — This is a paid article. The title speaks by itself.
• Paper, Sleeper agents — A not reassuring paper. Anthropic research team proved that this is possible to insert backdoors in models and the backdoor persists despite safety and adversarial training.

Fast News ⚡️

• It's time to build — Still a big fan of Benn's content. This week he talks retrospectively about why his content shifted from the modern data stack to AI. Then there's all the marketing that goes into selling data tools to unleash the power of your data. Far from trends and the lights, it's actually time to build tools.
• My thoughts going into a New Year — Tristan, dbt CEO, wrote thoughts about 2024. He covers why AI has not yet impacted data jobs and writes about OSS licensing after Snowplow recent changes, renewing a statement saying that dbt Labs does not need to do this change because they have a solid commercial path.
• EU AI Act — A nice looking summary of what matters in the new EU AI Act, from the risk definitions to the potential fines. Then CastorDoc explains why a data catalog can help you overlook how to be compliant.
• Packaging, one year later: a look back at 2023 in Python packaging — Understanding Python packaging is one of the most important skill to master when you want to enter Python world. I've seen too many people struggle in their development workflow because they are not used to pip and all. Chris wrote a follow-up to last year post about the sad state of Python packaging explaining standards and proposing things for the future.
• How lazy imports accelerate machine learning at Meta — Meta developed their own implementation of CPython called Cinder. In order to speedup model training time they switch to Cinder and decided to use lazy imports.
• Measuring data quality: bringing theory into practice — Mikkel is one of the best when it comes to putting the correct words on data quality issues. You should read this article to clarify these concepts.
• Big O — A practical approach — The Big O notation is something taught at school and super important when programming, especially in data when complexity has to be understood to speed up data transformation. This articles gives you what's important. o/
• How DoorDash used a service mesh and saved costs.
• The evolution of a data platform, part 2 — The part 2 of the MatHem’s analytical platform. On GCP, BigQuery at the center with event flowing from PubSub / DataFlow with a great usage of all Google Cloud items.
• Airflow evolution at Snap — Large teams need multi-tenancy and Snap is one of them. This article shows all the different architecture Snap put in place to deploy Airflow at scale.
• Integrating Airbyte with data orchestrators: Airflow, Dagster and Prefect — A orchestrators comparison and how Airbyte can be used as extract-and-load within them.
• Convert your PySpark code to Snowpark code using SnowConvert — Snowflake trying to attract Databricks customers.
• Hey Snowflake, send me a <fancy> HTML email — This is one of the feature I'm the most unsure about. Do I want my warehouse to be able to send emails without my global orchestration system to be aware of it? Yes because it's cool to give freedom to users... but no because as a data engineer I want a platform where flows are controlled. What's your take of this?
  
  If you want to do it with BigQuery, you should take a look at my friend's BigFunctions (see the send_email function).
• bq-lineage-tool — Java code that uses ZetaSQL to build a column-level lineage parser for BigQuery.
• Comparison running dbt-core and dlt-dbt runner on Functions — If you're runnning dbt-core within Cloud Functions check this article integrating dbt with dlt to avoid a weird hacky subprocess.
• Build a personal real estate dashboard with dlt and dbt — Another example where you can chain dlt—to extract and load data—and dbt—to transform data—in order this time to build a real estate dashboard to find your dream property in Portugal.

Data Economy 💰

• Phospho raises €1.7m in pre-seed to build GenAI monitoring applications.
• SKY ENGINE AI raises $7m Series A. With a platform that generates synthetic data for deep learning vision algorithms. It let's you create 3D stuff that you can use to train algorithms.

See you next week ❤️.

Hello you. Back to the usual Data News—with a little delay, I'm sorry.

First of all, I'd like to thank you for your positive comments on last week's article. It's a subject close to my heart and I was very happy to share it with you, because I never thought that Data News would become such a big part of my life.

I'm starting my annual university lectures today. It's always very exciting to go back and teach students, to help them discover the world of data from another perspective. The details: it's a 27-hour course called DataOps. It's quite a broad subject. I actually cover data engineering and how to put data stuff into production.

For years I gave a 30-hour lecture called Python for Data Science in which I covered the basics of Python, pandas and scikit-learn. But I stopped 2 years ago because it was too much and repetitive for me. I'm very happy with this new DataOps lecture because it's much closer to what's really going on in the data world.

Over the years I've accumulated exercices and one day—I hope this year—I'll provide it to everyone in a nice way.

It's funny because in the days leading up to the lecture, I'm always stressed about something: I'm always afraid I'm going to run out of content. The last thing I want to do is give a boring class.

Wish me luck and have fun reading the news.

AI News 🤖

• Bill Gates talks with Sam Altman — An 30 minutes episode of Bill Gates' podcast where he chats with Sam Altman.
• 14 predictions about AI — In a long form article, Vincent shares his predictions about AI and the trends we might see in 2024. Garbage in, garbage out, still one of the most important issue. Personally, I have a question for authors in 2024: when are you going to stop generating images to illustrate articles? They're horrible and destroy the content. If I have to predict something it would be the this trend to stop.
• Meta, from audio to synthesize human in conversation — Do we finally see an outcome of the billions Meta invested in the Metaverse 🙃. To be honest this is impressive, from an audio Meta is capable to generate a photorealistic avatar that behaves like if it was you speaking.
• How Meta is advancing Gen AI — a podcast about Meta GenAI breakthroughs.
• Microsoft will replace the Windows keycap by a Copilot — This might be a major change to Windows computer and keyboards, Microsoft wants to add a physical AI trigger on every keyboard. Might be the best adoption trigger we ever saw.
• I coded exclusively with ChatGPT for 30 Days — Good takeaways about a nice experiment.
• IBM explaining why they invested in HuggingFace — During gold rush sell shovels. It explains NVIDIA 2023 success, but HuggingFace is legendary for the same reason. HF became the defacto platform when it comes to share and showcase AI models.
• Sentence embeddings — After reading this article you will be able to do a PhD in embeddings. Personally I did not read it but if you want to understand embeddings you should.

Fast News ⚡️

• dbt related stuff
  • Download artifacts from you dbt Cloud job runs — a tutorial from a CLI tool to generate ERD diagrams for dbt Cloud projects.
  • Testing dbt macros — A clever pattern to write unit tests on dbt macros with a model computing all the possible macro values and a dbt test checking all the possible cases.
  • Unit testing dbt models — Using a dbt-unit-testing package Matthieu showcases how you can easily test your models.
  • dbt meta tag — A list of the companies habing product features depending on the meta tag. It shows how deeply dbt change the data world.
• What I would do differently getting into Data Engineering — Data engineering has changed a lot in the recent years and Daniel gives 3 advices that you should consider to get into data engineering. Learn SQL, be social and learn to say no.
• Lead Data Engineer career guide — Detailed skillsets needed to be a lead data engineer.
• Effectively managing junior developers on remote teams — In the current state of the ecosystem this is super important to provide a perfect introduction to the data world to juniors.

• Every data transform is technical debt.
• How BigQuery stores semi-structured data? — It relates to Dremel and parquet structures.
• Mixpanel modern data stack fast lane.
• Netflix video processing rebuilt with microservices.
• How Monzo built Year in Monzo.
• A/B Testing at HomeToGo.
• Datadog, scaling self-serve analytics, serving 5000 employees — 🤯.
• 2024: the year of the value-driven data person.
• Transfer data from BigQuery to Fabric with Arrow and Rust.
• Removing egress fees when moving off Google Cloud.
• The evolution of a data platform.
• Fixit, MotherDuck SQL AI error fixer.
• What's next for Malloy in 2024.

Data Economy 💰

• Talend will shutdown Talend Open Studio their open-source version on January 31. As a reminder Talend has been acquired by Qlik 9 months ago. This is probably a strategy to keep money flowing. See you Talend 👋.
• Alteryx to be acquired by private equity firms in $4.4B deal. OK.

See you soon ❤️.

Hello, it's 2024. I hope you're well and that you've ended 2023 on a high note with your loved ones. I wish you a Happy New Year and all the best for 2024. I'm very happy to have the privilege of corresponding with you and it honours me.

This edition of Data News will focus on the end of 2023 with a good retrospective about me and my activities—content and freelancing. Of course, it will also look ahead to 2024, and I'll try to set the vision 2024. But you know how bad I'm with goals.

Let's wrap-up 2023

In technology, we live in a fast-paced environment and when you compare yourself to others or try to keep up with all the news, it's easy to get FOMO. This year was perhaps the key moment when I managed to step away from vanity metrics and take breaks away from work. With the exception of my trip to Japan in 2019, I think this is the first time in my life it's happened in this way. Even my parents noticed I was away for a week from the computer for Christmas.

The next step for me is finally to recognise that even if I have a deep feeling that I haven't achieved anything in 2023, that I'm always behind in my ideas, it's a wrong feeling and I should be proud.

Proud about my content, my professional and my personal life.

Content — I don't do it for fame

When I started content creation, my North Star was to create an international audience while being in France—hence in Europe—aiming, at my level, to balance with everything in data being said/made in the US.

Then, I also stated that I wanted to produce content that I'd like to read, actually all the stuff I produce is something that helps my sharpen my ideas and to save for later. If I like my own content other will do it because I'm a just a normal person.

What 2023 brought:

• Followers — I doubled in followers on my 3 main platforms: I reached 4000 people on the blog, 8000 on LinkedIn and almost 600 on Twitter (even if I don't post that much there). My only target is, to be honest, to grow my blog, having 4000 people who trusted my enough to enter their email and validate the subscription is just crazy.
• The blog — 46 articles published in 2023, this is way less than in 2022 but it's ok. In term of views my blog got an increase of 67% going up to 36k unique visitors, just wtf.
• Video & audio — This year I've published 3 podcasts episode, which is way less than what I initially wanted and I participated in 2 podcast épisode with DataGen. This is something I want to change in 2024.
• Conferences — I've talked to a few meetups and online conferences, my main issue here is that everytime I do a talk I want it to be a unique experience, so it asks me so many prep hours. Something I should change maybe. On the same topic we ran the Paris Airflow meetup for 6 months but took a never-ending break after the summer holidays.

In the end, I've done more things IRL and I've met a lot of people I wouldn't have met if I hadn't been visible online and I think that's the big W of 2023. That's what brings me the most joy in fact. Thank you all for being so nice and supportive with me ❤️.

In conclusion I'm happy with this. But frustrated not to have done more. Still I decided not to focus entirely on content creation so I kept room for freelancing and personal life.

Professional — The limits of freelancing

I started my freelance career almost 4 years ago. In a world affected by COVID, At that time, remote work was the new norm in the tech and people were surprised that I decided to go down the unstable route when COVID was already enough.

And I won't regret it, for the last years I've worked on the most pleasant projects with people I really enjoyed working with. I'm lucky enough to be able to choose the companies I work with, people who understand my requirements. Being able to take time for myself while working wherever I want on projects I choose is something I wish everyone could do. It changed the way I see work.

It's time for a review of 2023.

• Revenue — My freelancing activity is stable, last year I billed almost the same revenue as 2022 around 140k€, while taking more big breaks. In term of clients I had in number less clients than in 2022 because my main client kept me busy.
• Projects— The few noticeable projects I've worked on
  • I designed, developed and deployed a reporting application with Apache Superset. This is for the French gov, for more than 60k users (+12k weekly), it contains more than 10 dashboards with 5 custom visualisations in React—you can see an example screenshot here.
  • At the same time for the gov I've worked on a larger project to develop a private datalake to work datasets with on-demand RStudio and Jupyter containers. For this I deployed a private Kubernetes with on-top MinIO, Keycloak, LDAP (for auth) and Onyxia to deploy containers.
  • Then I've worked to implement a few small data stacks (revolving around ELT and a warehouse) and helped 3 companies migrating from something to dbt.
• Partnerships — I had a few discussion with people about partnerships in 2023 and did not really push it forward, but I should next year.
• Angel investment — I did my two first investments recently, it continues my content North Star, putting light on stuff made in Europe. I'm so happy to finally open this path. Welcome blef ventures. More about on this soon.
• Data engineering — Data engineering is changing and my work is changing. When I started data engineering in 2014, the term wasn't even existing. Moving data from A to B has always been something fun for me. But with the years something changed and I might want something else as evidence by my work at gov, I do engineering. Data engineering is moving towards the left creating a deeper gap between data users and underlying layers. More on my views about this in a coming article.
• Other freelancers — When I started, I France we were only a few doing data engineering in freelance. Now, because of all the layoffs, the way the work is changing and the promise of money a lot of people entered the game. I've met a lot a tried to give advices but it probably means for me that I need to renew my offering as well.

I'm happy about 2023, but it brought a few big issues in my daily routine that I want to fix next year:

• I feel alone in my daily work — working partially and remotely for companies isolated me a bit and after 4 years it's time for a social boost
• Fuck dopamine — all the attention business distracts me so much and I lose focus so fast, especially when I open Twitch. I changed my phone routine and spend way less time on it but I have to change something on my computer as well.
• Get things done — When it comes to finishing task, I'm good when it's for client, but when it's for myself, there is a huge improvement.
• Administrative tasks — ...

Personal — Catch me in a train

In 2023 I've achieved a great Work-Life-Balance. I'm so happy and in love with my girlfriend, who is freelancing as well, so the rhythm is kinda the same for us and we have the same kind of issue even if we are not working in the same ares—she's in the movies industry.

• Travel — We still mainly live in Berlin, and I travelled a lot between Paris—where my business is—and Berlin. These ~10 trips represent all-together around 80kg in carbon emissions. In comparaison I went once to Malaga last year to follow my gf for work and it's just insanely more (300kg eC02).
• Sport — Since August I started running again, 550 kms since. Twice a week for 2 months then 4 times a week and I've never been happier. Some people might remember but it was my 2022 goals to run once a week. It took me 1,5 years to reach the goal. The target is 45 mins for a 10k next year. I also started bouldering, unexpected and I like it. Still in 2023 I did less bike than the previous years.
• Friendships — Met a few new friends and I'm so happy with this because when you pass 30 I feel that creating new relationships become way more difficult.

A few articles

That's a wrap for 2023, and because it's the Data News, here a few articles that people (and me) liked in 2023 that you might find interesting:

• Count.co SQL guide — A infinite canvas with content to discover and improve in SQL.
• Answering "Why did the KPI change?" using decomposition — An excellent article by Max, I really enjoyed both the content and the form.
• Malloy 4.0 announcement — Malloy is a new language created to transform and analyse data, it transpiles to SQL. I did not had the time to play with it but I'll in 2024.
• Coding patterns in Python — a great list of Python to know in Python.

2024

2024 marks the tenth anniversary of my entry into working life. My 6-month internship started in April 2014 and I was developing a custom drag-n-drop dashboard application with Django and D3 as a project. Fast-forward 10 years later projects haven't changed 😅.

Once again I wish you the best for 2024.

If you are following me for a long time you know that I'm super bad at having resolution. Let's more make ideas and stuff I'll be proud about in January 2025 when writing the 2024 post.

• Keeping the habits — I often repeat this is not about motivation but discipline. Let's continue having the habits I have, continuing running and the newsletter.
• Adding new habits — I'd like to add at least 2 habits, especially in the content creation, this year I want to reboot my YouTube channel and stick to podcast publication.
• Create courses — For all my pro career I've written courses—I teach since 2015—but I've never really created something for people online. It's time.
• Release 2 products — I want to release 2 products that can live by themselves next year, one around the blog, the other one we will see.
• I want to find my new pro journey. If you have ideas, hit me up.
• Invest in 4 companies — If you are in this journey, same, hit me up.

Let's go and see you this Friday for a more traditional newsletter. I'm sorry for this long format but changing years is something enough noticeable.

Thanks again ❤️. I wish you a good end of Sunday.

And that's the problem. I want to do so many things at once. It's quite funny because when I'm coaching someone, one of the first pieces of advice I give them is to stay focused and avoid multitasking, but when it comes to me... Yeah, you know.

However, some excellent articles have been written and I want to end 2023 with one last big wrap on these December articles. I'd also like to say hello to all the newcomers who arrived in December, thank you for your trust. We're going to get to know each other.

Before moving on to the Data News, a bit of personal news, in December, I took part in the MotherDuck meetup in Berlin. I presented what I believe to be the future from my DuckDB experiments. I've especially been amazed by DuckDB in the browser with WASM. I'll also go to the DuckCon in Amsterdam on February 2nd—pm me if you're going.

End of January, on the 31st I'll speak at a Modern Data Stack conf in Paris, still about DuckDB, but this time in French. I also took part in my friend's podcast where we discussed 3 trends in data: data modeling, real-time analytics and DataOps.

My retroprojective—a retro 2023 with a projection into 2024—will soon be written. It will talk about my search for a new spicy adventure, the fact that I've finally taken up running again, my new journey as an angel investor, and so on.

Enjoy this last 2023 Data News.

AI News 🤖

• An interactive 3D explaination of LLMs — Explaining complex things the visual way is the best. In this one it details all the components in a LLM—a big part explain what's a Transformer.
• LLMs for builders: jargons, theory & history — Mehdi, compiled in a large article all the necessary vocab to understand the basic conversation when it comes to generative. He even quickly explain how you can run a model on your computer.
• Cocorico 🐓. Mistral AI, one of the French "OpenAI" startup, entered the field setting new standards and with recognition. They released their first AI endpoints: generative and embedding. When it comes to generative they currently have 3 models: tiny, small and medium, which are performing well against GPT-3.5. At the same time they released Mixtral 8x7B, the first open-source model of this calibre under an Apache Licence. And the weight are open-source as well.
• What I wish someone had told me — It's borderline AI news, but as the author is Sam Altman, I think it belongs here. After the whole Hollywood thing around Sam being pushed out and then coming back, Sam clickbaited us. He's written 17 great HR / team building tips—but they have nothing to do with the drama we're all living for.
• Building applications with real-time stable diffusion APIs — fal, has written a great article about how you can use WebSockets in Javascript to interact in real-time with a Python backend and stable diffusion. There is a great article with an image generation from a sketch. It gives so many ideas.
• People underestimate how impactful Scikit-learn continues to be — The year is coming to an end and LinkedIn is playing at the 2024 predictions game. Obviously no one will get it right. At the same time one of the Scikit-learn confounder, put the church back at the city center—this is a French expression poorly translated. Scikit is still the most used library when you look at some numbers and LLMs have still to bridge the gap in usage.
• OpenAI prompt engineering guide — Wow, an official guide to become a prompt engineer /s. Seriously, it seems it contains good tips to communicate to the algorithm.
• Google announced Gemini, their new multimodal model "beating" GPT-4, but fooled us with an edited video.

Fast News ⚡️

• Airflow 2.8 is out — The Airflow rhythm of release is crazy, I can't keep up with the awesome feature that have been added this year. To finish the year Airflow team have released improvements to Datasets and a major step forward with the new Object Storage API that provides a generic abstraction over Cloud Storage to transfer data from one to another.
• The EU AI Act has passed — After many years working on the text the EU has voted for the AI Act to regulate usage of AI when usage European citizen data. It points to a cheat sheet that summarises what you need to know. In a few words: the AI Act provides the glossary to define what's an AI and define the boundaries of prohibited and high-risks AIs.
• BigQuery now integrates DuetAI — to help you generate or complete SQL queries.
• AWS announced S3 Express — S3 Express is a new zone with 10x better performance (latency and parallelisation). Paul, wrote a few speculations about the new S3 tier, this is highly detailed and explains very well what to expect, DateEngineeringWeekly also wrote thoughts about it. S3 will still be the king, or the GOAT.
• Idempotence — Matt wrote an article to explain what's the idempotence and why it matters in data engineering. Idempotence can be mathematically summarise to f(f(x)) = f(x), it's important in data engineering because for the same input you want a pipeline to produce the same output. Never forget to have it in mind when thinking of a pipeline it leads to great questions.
• Have I Resolved the Pie Chart Debate? — We all know pie chart are terrible. Nick proposes how to fix the pie chart dilemma.
• How to know if your data team is successful? — Reflexions around team performance and how to measure it.

Engineering stuff ⚙️

• Netflix internal data engineering Summit — Netflix team organised an internal conference about DE topics. And they recorded it. 8 videos are on YouTube and to be honest this is awesome content to learn patterns and get ideas from the best. They still use technologies around JVM (Spark and Flink), but with no surprise everything resolves around Iceberg—which has been created at Netflix.
• Using Netflix Maestro and Apache Iceberg — Going deeper into incremental processing the engineering team details how they implemented it.
• Introducing WAP pattern support with Apache Iceberg (with SQLMesh) — Small article about a important pattern to avoid putting bad data in production. The WAP pattern—Write-Audit-Publish—let's you first write the data in a staging layer in which the data is audited, if the audit is green then the data is published in the production layer. This article is just an entry point to SQLMesh—a dbt alt—that enables you to do it.
• Use Databricks to read Iceberg tables in Snowflake 🙃 — This post have been written by Snowflake team, but reflect a strategy from Snowflake to attract customers by being open, and Iceberg do the glue here, winning the table format. Still, don't do it and try to avoid spaghetti data platform.
• Efficient ELT refreshes — Max detailed how he designed his ELT pipelines
• Run dlt on Lambda to save on extract and load costs — dlt is an open-source Python library to do extract-load in Python, if you want to save cost out of different cloud services that moves data, it might be an alternative.
• Druid deprecation and ClickHouse adoption at Lyft — Data engineer loves migration. They prefer even more speaking about the migrations they have done. Moving from Druid to ClickHouse looks like a good improvement.
• Data Quality Score: next chapter of data quality at Airbnb — After all the data cataloging vision and trends Airbnb launched, this time they explained how they see dataset quality and how they score it.

Other reads

• The state of SQL-based observability, on ClickHouse blog.
• Designing OBT and comparing OBT with Star Schema.
• Code review best practices for Analytics Engineers.
• Self-Service data analytics as a hierarchy of needs.
• Easy GCP cost anomaly detection.
• An elegant platform, Grab.
• A guide to MLOps with Airflow and MLflow, TheFork.
• What made Apollo a success — A Nasa PDF with 8 articles reprinted from March 1970 issue of Astronautics & Aeronautics.

Data Economy 💰

• Mistral AI raised another €415m at $2B valuation. Mainly from US based capitals, it will probably change the governance of the company, is it still French?
• Elon Musk’s generative AI startup xAI looks to raise $1bn.
• AssemblyAI raises $50m. API endpoints to convert voice data to text in all his forms (transcript, chapters, summaries, etc.).
• Keboola raises $32m in Series A. This is a all-in-one data platform for non-technical data users.
• London-based Harriet raises €1.4 million pre-seed. An AI assistant using HR data to help employees.
• AI data platform VAST Data raises $118m. All-in-one platform for big corp to do AI and engineering at the same place.
• Octolis has been acquired by Brevo (ex-SendinBlue). Octolis is a CDP / reverse-ETL solution and Brevo is a CRM, the join makes total sense.

See you this Friday with a post opening 2024 🎊.

Hey, it's been a few weeks since I've not written any news. It was a necessary break for me and a blank page syndrome at the same time. Still I've accumulated a lot of articles that I think should fit in the Data News so this week might be a huge recap of content that has been produce in the last month.

I hope you will enjoy the selection.

On Monday I'll also give a talk at Berlin MotherDuck meetup: DuckDB experiments, a glimpse of the future. I think it will not be live but the recording will be published after the event on YouTube I think.

AI News 🤖

• Sam Altman has been fired as CEO of OpenAI.
  • OpenAI announced this leadership transition yesterday. At the same time Greg Brockman (actual President and co-founder) will step down from the chairman of board and Mira Murati (actual CTO) will become interim-CEO. It was a brutal decision.
  • The public official given reason was "[Sam] was not consistently candid in his communications with the board, hindering its ability to exercise its responsibilities. The board no longer has confidence.".
  • The Internet has spent the last 15 hours guessing what this really meant. Here are a few theories I've read: a security leak occurred and Sam/Greg hid it from the board, Sam is publicly accused of sexual abuse by his sister, Sam has different views about company vision which doesn't please the board—esp. regarding profits or AI regulations, Sam invested in an OpenAI competitor. Either way, we'll see in a few days.
• People are mostly saddened by the news because Sam was a publicly-beloved and transparent CEO who changed AI. Comparisons with the coup that overthrew Steve Jobs back in the days are many.
• The news arrived a few day after OpenAI dev-day, a public conference announcing new products and features. Mainly they announced GPTs, a no-code UI to create custom versions of ChatGPT.
• Other AI announcements
  • Github Universe was the moment to announce more Copilot everywhere in Github ecosystem. The most interesting thing was the fact that Github will introduce M1 and GPU runners.
  • xAI—the company founded by Musk after quitting OpenAI—announced Grok. It's a 33B parameters LLM.
  • Germany wants to build the European OpenAI competitor and invested $500m in Aleph Alpha, a startup. On the landing page it's clear that the focus is to build safe AI.
  • Kyutai has been announcement at a AI Pulse event in Station F, Paris. Kyutai is an open science lab to build and democratize AGI—artificial general intelligence—through open science. They carefully picked open science rather than open-source. The team looks great.
  • The GPU availability competition is on. Y Combinator announced a Microsoft partnership and priority access to compute resources. This is linked as well to Microsoft making custom AI chips.
  • Biden issues executive order on safe, secure, and trustworthy AI.
• 2 reports with hundreds of pages about AI were published — The State of AI report and AI: The Coming Revolution. Both looks full of interesting things to say, but I did not read them.
• Google team wrote a paper "demonstrating various failure modes of transformers and degradation of their generalization for even simple extrapolation tasks". In a nutshell, LLM can't generalize.

Now that I gave you the general news, let's jump to a few use-cases about AI.

• Towards a real-time decoding of images from brain activity — This is crazy, Meta researchers have been able to create a system that predicts an image seen by a person from the brain magnetoencephalography.
• LLM-powered data classification for data entities at scale — Grab explains how you can use LLMs to do classification, in this case identifying PII in the database. They explain the real-time architecture the system is using and give an example of the prompt they are using.
• Generative AI, the intern you can’t trust — A small post from Atlassian blog, it gives 3 ways to improve LLMs accuracy.
• Summarizing post incident reviews with GPT-4 — Canva has so many incidents that they need a LLM to summarize them for reporting purposes 🙃. Obviously it's a joke, but while the use-case is interesting I question myself about real need behind.
• Building in-video search at Netflix — What if you could prompt for a specific situation and get all the movies—at the relevant timecodes—presenting the situation. This is so cool.
• Cost analysis of deploying LLMs — All of this is cool, but pricey, this post do a good exploration of the costs.

Fast News ⚡️

Because the AI News is pretty packed and I still want you to enjoy this newsletter articles will be less commented than usual. But still spicy opinion, because you know, it's me.

• Data contracts is undoubtedly a new growth lever for data observability companies and data VCs. Soda announced their open-source data contracts engine. It's done in YAML. Here another example of contracts with msgspec.
• NVidia research has been able to supercharge pandas with cuDF to run pandas on GPUs.
• Wes McKinney, pandas and Arrow creator will join Posit—the company behind RStudio—as a Principal Architect. His new role will probably ease the integration in the Posit ecosystem of all the Python tooling, even if it has already been the case for months.
• dbt Labs hired Brandon Sweeney as new President and COO. Brandon was previously dealing with Revenue at Hashicorp. The same company which recently changed licensing to BSL getting backslashed by the tech community for it. Our prayers goes to dbt Core.
• Onehouse , Microsoft and Google are working on table format standard called Onetable. This isn't a new format but a way to create interoperability between Delta, Iceberg and Hudi.
• If you are curious about Iceberg and Hudi ACID guarantees read the article.
• Code faster with Ruff, a Python formatter written in Rust. All the time wasted for black to reformat your code will be used for good purpose now.

Taking other companies as example is often a good way to get ideas

• Gusto, data platform to generate HR insights — All data send to OneModel—a paid HR tool, in a Redshift with Tableau for visualisation.
• Criteo, how to compute data lineage — Criteo has a homemade application for data document called... Datadoc in which they compute their cross assets lineage.
• Picnic, master data management — Creating MDM for retailers is like the one-piece.
• LinkedIn, how to use 4 trillion events daily — Leveraging Apache Beam and Samza.
• Netflix, streaming SQL — Flink architecture in a data mesh organisation.
• Zalando, how to patch Postgres and fix WAL — Zalando team explains what they patched to Postgres JDBC driver that was growth in the write-ahead log.
• GoDaddy, layered architecture for a data lake — Naming conventions ideas and 5 data layers: source, raw, clean, enterprise and analytical.

A few food for thought articles about data concepts and roles.

• From data platform to ML platform — How incrementally data platforms are built, first for analytical use-cases and then adding ML capabilities.
• Why you should not build apps directly on the data warehouse.
• SQL is not designed for analytics and why Malloy is a paving the future.
• Would you become a data strategist? — Great post from Marie about a key analytical role shaping companies strategies.
• Two archetypes of data engineers — Closer to business or to the tech. Best data engineering teams successfully blend the 2 archetypes.
• The Economics team at Instacart — Or how economists and PhDs become more tech-savy enabling more and more relevant usage of data.

Data Economy 💰

• ZenML raises $3.7m additional Seed. A MLOps platform that works with all cloud and tools.
• Snowflake acquire Sisu and Ponder. The first one is an engine to monitor business metrics while the second is a tool to run pandas at scale.
• Yahoo spin-out Vespa and raises $31m. Vespa is a search engine and a vector database. This is the good timing to open-source is for AI use-cases.
• Aleph Alpha raises $500m Series B to build the German OpenAI.
• Kyutai is funded with $330m from 2 French billionaires and Eric Schmidt—ex-Google CEO. Kyutai is a open science lab that wants to build the AGI. The team as a good resume and the science committee looks awesome (Yejin Choi, Yann Lecun and Bernhard Schölkopf).

Ghost implemented a recommendation feature recently so I've added a few folks I like to read on internet.

See you next week ❤️.

Hey, this week Coalesce—the dbt Labs annual conference—took place. During 3 days, people shared how they used dbt around the world. I'll, as usual, write a takeaway post after binge watching all keynotes, but this is for next week. Still dbt Labs announcements were mainly towards dbt Cloud with great features to drive adoption of the paid product.

They announced dbt Mesh a product enabling cross-project dependencies for teams with multiple dbt projects. In addition they also released an Explorer view that lets you navigate through all you project and see models, macros and more directly in one nice graph.

Does this mean that you have to use dbt Cloud to have a multi-project setup? No, you can activate multi-project collaboration with dbt Core. I've written a guide that helps you do it.

📺 On the content side I'll also present next week the Fancy Data Stack project at the Data Engineering And Machine Learning Summit 2023 organised by Seattle Data Guy. I'll be online on Thursday 26 at 5PM CEST. Add it to your calendar and sign up for the conference—the list of speakers is insane.

Data News is packed this week, take time to enjoy it, rainy times are coming, you can see it as a gift 🎁.

Enough dbt use lea 🥰

Max—the first Data News member 🤗—open-sourced carbonfact/lea this week. lea aims to be a minimalist alternative to dbt by fixing a few flaws that comes with dbt. You can even see the traditional Jaffle shop example done in lea.

What are the main differences?

• You configure lea with env variables.
• a lea prepare command that creates database objects that needs to be created (dataset, schema, etc.). Schema are interpreted from the folder structure (with DuckDB).
• lea understand the views relationships, you don't need a ref. Jinja templating is still supported tho.
• Tests are directly added in the SQL code at the column that is target. For instance if you need to test unicity on a column you add the @UNIQUE decorator. Singular tests are still supported.
• lea generates documentation as Markdown in the workdir.
• Other cool features: lea teardown delete database objects, lea diff shows table schema differences and you can write Python model as long as they return a DataFrame.

Max also wrote a nice post about data downstream issues—which is the main problem leading the data contracts space: Sh*t flows downhill, but not at Carbonfact. You should read it because it gives another perspective of solution to fix it.

Gen AI 🤖

• Can you run it? — There is a HuggingFace app that tells you by taking your specs what you need to run a LLM model for inference or training.
• 25 million Creative Commons image dataset released — Fondant, an open-source processing framework, released publicly available images from web crawling with their associated license.
• New Vertex AI Feature Store — GCP Vertex AI is the place to do "serverless" AI. This is awesome to see this directly integrated within BigQuery as it obviously brings simplicity. In public preview.

Fast News ⚡️

• Meta banned a creator for selling Python and Pandas courses — The automated AI filters identified the ads as violating wildlife protection rules. We are therefore thinking with our feet these algorithms are probably written in Python. Do we still want a future where AI decide for us?
• At the same time, luckily for us, Meta is creating custom silicon for AI.
• Disney new intelligent robot / toy — The entertainment company showcased a new toy with impressive capabilities opening doors for a fun future for kids.
• 11 lessons learned managing a platform team within a data mesh — BlaBlaCar, a carpooling company, is well-known France for recently adopting a data mesh organisation. This post gives great insights about the impact on the data platform team.
• The need for an open standard for the semantic layer — Following news about dbt's semantic layer, this post from cube opens the door to defining a standard when it comes to semantics. What should be the main entity type at the center of the semantics: metrics or datasets?
• Why data integration will never be fully solved — Anna covers a few data integration tools and tries to explain why this is such a tricky field that have issue to be resolved with only one cloud tool.
• Popsink a real-time ingestion and processing platform released their self-service offering this week. They are French and they built a great platform on top of Redpanda and Flink claiming to be 4x cheaper than Fivetran to do data replication. As an echo of last bullet point.
• Following Popsink kind of stuff, an example of how Fortis Games, a game editor, developed real-time platform with the same technologies.
• Rise of the data generalist: smaller teams, bigger impact — You don't need to convince me. In all experience and talks I have with people smaller teams obviously drives bigger impact.
• La stratégie nationale pour l'intelligence artificielle — In French. This is about what France wants to until 2025 to drive IA adoption.
  • 3500 new students and at least 200 additional thesis on AI topics
  • Do between 10% and 15% of the world market share when it comes to embarked AI
  • and more stuff in order to attract foreigners and help companies

Engineering stuff

• Dagster released their internal data platform in open — Surprise they use Dagster as an orchestrator.
• dbt related stuff
  • To dbt or not to dbt — A few lessons learned while implementing dbt at Intercom.
  • dbt MetricFlow, semantic layer 2.0 — An quick analyse of the new semantic layer vision.
  • Data contracts and schema enforcement with dbt — It comes with dbt Mesh and gives a lot of new metadata over your models to bring more software engineering practices to dbt development.
• Pros and cons of One Big Table data modeling — I really like OBT, it brings a lot of simplicity, especially in the downstream usage, but obviously it has known issues.
• What is data versioning and 3 ways to implement it — A comparison between change data capture (CDC), dimensional modeling and slowly changing dimension (SCD).
• Mage, BigQuery and bundled-up bike trips — A homemade project where Patrick used Montreal public data of bike counting sensors.
• Replace Dockerfile with Buildpacks.

Data Economy 💰

• OpenAI is near $90b valuation. With a product that has been launched in late 2022. It seems OpenAI is doing more than $100m in revenue per month. The numbers are just crazy.
• Lonestar raises additional $825k in Seed. Lonestar provides immutable storage to be sent on the moon as a backup service. Yep on the moon 🌕.
• Aindo raises €6m Series A. Aindo is a synthetic data solution, it provides a platform to generate synthetic data from your real data in order to preserve statistical relevance while removing sensible information. With synthetic data you can then publicly seek for help among the world's data scientists.
• ScyllaDB raises $43M Series C. It's NoSQL database that is compliant with Apache Cassandra interfaces, and open-source.
• Pantomath raises $14m Series A. A new data pipelines observability solution enters the game.
• Prophecy raises $35m Series B. This is a drag-n-drop data transformation product that I never heard of.

See you next week ❤️.

Over the last few years, dbt has become a de facto standard enabling companies to collaborate easily on data transformations. With dbt, you can apply software engineering practices to SQL development. Managing your SQL patrimony has never been easier.

So, yes, dbt is cool but there is a common pattern with it: you accumulate SQL queries. If your implementation of dbt is successful, many teams will use it, many business use cases will result in SQL queries in your warehouse. Fast forward to 2 years later, you find yourself with hundreds or thousands of SQL queries. Whatever the number, there will be a critical point at which a single project no longer scale.

Having too many models in a single repository will become unmanageable:

• Governance — many data owners
• Data domains — a lot of different concepts that you would like to isolate as single units
• Name clashes — you can't have 2 models with the same name in a project
• and more 😅

This is when you consider a multi-project configuration for your dbt implementation. With a multi-project configuration, you can imagine isolated dbt projects with possible connections between them. We can draw a parallel with microservice architecture. Each dbt project is like a microservice and instead of exposing an HTTP API, it exposes tables with enforced contracts.

Initially cross-project references was a feature aimed to be released in dbt Core (cf. roadmaps 2022-08 and 2023-02). But after research and first developments it was decided by dbt Labs that multi-project collaboration will become a feature of dbt Cloud. Which I understand perfectly. It's the best feature for creating a differentiating commercial offering. What's more, multi-project collaboration is by its very nature an Enterprise—with a big E—feature, which makes it relevant for a paid-for solution.

Hence dbt Mesh, which has been announced this week at Coalesce—dbt Labs annual conference. dbt Mesh is the dbt Cloud solution to manage cross-project references, a multi-project node explorer and all the governance.

Cross-project references is a key enabler to data team decentralisation. Let's imagine you have a core project, managed by the central data team. In this core model you have an orders model. On the other side the finance data team wants to build a revenue model on top of the core.orders model. With cross-project references you can declare a model to be public on core to use it elsewhere.

All this is possible natively with dbt Cloud. But dbt Cloud multi-project is expensive. At the very least $100/month per project—Enterprise pricing, so it's not possible to have actual figures. But from what I know, it's expensive.

What if we could do it with dbt Core?

Enters dbt-loom

Obviously the community did not welcome well this announcement as it converged with the new pricing. It's a bit frustrating to see a product you truly love and I which you believe keeping awesome features behind closed-doors. But dbt is still open-source, so it's up to the community to adapt.

And the community adapted.

On my side I tried to fork dbt-core to inject in the what was need to make the multi-project working, but it was a burden. It was not very successful. On the other side Nicholas Yager worked on dbt-loom which leverages new dbt Plugins mechanism that was introduced with v1.6. Nicholas wrote a great explanation of the plugin API.

Under the hood, you need to write a Plugin class, inheriting from DbtPlugin, and implementing one of the 2 hooks available—or both: get_nodes and get_manifest_artifacts . The first hook is called every time dbt needs to get nodes and nodes are injected as external nodes, this is the one that interest us. Actually if we want to implement cross-project dependencies we need to add to a dbt project context the external nodes it depends on.

Here what you can do with dbt-loom.

Multi-project collaboration example

In order to help you understand what it really means here a working example with dbt-loom on a 2 projects setup—core and finance. First in the core project. In order to have reproducibility I use dbt-duckdb connector so everyone can try it at home. I have 1 seed that loads a few rows and 2 models: stg_orders and orders.

Obviously orders depends on stg_orders and respectively the first one is public and the second one is private.

-- raw_orders.csv (dbt seed)
order_id,order_date,amount,customer_id
1,2023-01-01,340,c1
2,2023-01-02,13,c2
3,2023-01-03,1456,c1
4,2023-01-04,765,c3

-- stg_orders.sql
WITH raw AS (
    SELECT
        order_id,
        order_date::DATE AS order_date,
        customer_id,
        amount
    FROM {{ ref('raw_orders') }}
)

SELECT *
FROM raw

-- orders.sql
SELECT
    order_id,
    order_date,
    customer_id,
    amount::DECIMAL(8,2) AS amount_incl_vat,
    (amount / 1.2)::DECIMAL(8,2) AS amount_excl_vat
FROM {{ ref("stg_orders") }}

In order to declare these models as available for cross-project dependencies you need to specify it in the YAML. In our case stg_orders will be protected and orders will be public with an enforced contract. The contract is super important because as soon as you expose a model, you have to potential downstream consumers that are building stuff on your models, you can't delete a column or change a type without notifying. Or even more, versioning models.

version: 2

models:
  - name: stg_orders
    access: protected
  - name: orders
    access: public
    config:
      contract:
        enforced: true
    columns:
      - name: order_id
        data_type: int
        constraints:
          - type: not_null
      - name: order_date
        data_type: date
      - name: customer_id
        data_type: string
        constraints:
          - type: not_null
      - name: amount_incl_vat
        data_type: numeric(8,2)
      - name: amount_excl_vat
        data_type: numeric(8,2)

That's all for the core project. Once you have dbt build the core project a manifest.json will be generated and tables will be created in the database. On the finance project, with dbt-loom install—pip install dbt-loom— you need to declare the core project as a dependant manifest.

manifests:
  - name: core
    type: file
    config:
      path: ../core/target/manifest.json

Then you can write a few models that are using cross-project references.

-- stg_revenue.sql
WITH orders AS (
    SELECT *
    FROM {{ ref('core', 'orders') }} -- this is cross-project reference
)

SELECT *
FROM orders
LEFT JOIN {{ ref('margins') }} ON 1 = 1

-- revenue.sql
SELECT
    order_date,
    SUM(amount_excl_vat * margin) AS revenue
FROM {{ ref('stg_revenue') }}
GROUP BY order_date

Now you can dbt build this project as well and dbt-loom will extend dbt models list thanks to the plugin by adding the core.orders model.

In order for you to try it at home I've created a Github repository with a working example using DuckDB as database. You can try it yourself.

Conclusion

Multi-project collaboration is probably the best feature dbt Labs introduced in recent times. This feature has a huge potential to structure dbt projects and avoid chaos.

As a data engineer who loves open-source and community stuff, dbt-loom is a great workaround, but be aware that it's all experimental at the moment and if large workflows rely on this functionality, you should consider using the paid version with dbt Mesh.

In order to go further you can watch a Coalesce 2023 talk about dbt-meshify a tool that helps you automating your journey to a multi-project dbt setup from a monolith—here the direct link to the video.

Hello, dear Data News reader, I hope you'll enjoy this new edition. It's amazing how quickly time flies and this summer I passed the 3-year mark since I started my freelance adventure. I'm so happy with what it's brought me. But I've got this internal alarm that goes off every 3 years asking me for new things. It's time for me to search for my future paths.

Don't worry the newsletter and the content stuff I do is something I enjoy so it will probably stay as an invariant in this quest.

Also, this week I wrote R code for the first time. It's not an experience I'd recommend. I tried using ChatGPT to help me with this task and every answer it gave me was always wrong. In 20 attempts, it never gave me a correct snippet. On the other hand, I asked the AI to help me write a TCP proxy in Python and it worked first time. Probably a training bias.

Going further I've looked at StackOverflow trends to see if there is a reason Python is better covered by ChatGPT than R—more than the obvious one—and Python was 6 to 7 times more popular than R at the time of training. The graph also shows that Python has been losing popularity since 2022, although I don't really know why and it stays on top. Only C# get massive increase in TIOBE index.

This week, the videos from the Airflow Summit 2023 have been released and as always, I'd like to provide you with a list of the talks I found interesting. You can also watch the YouTube playlist and show support to other talkers.

Airflow Summit 2023 🌬️

For the sake of reading I've sorted the few talks I've selected in 3 categories: general stuff, Airflow internals and feedbacks from companies.

General — Get Airflow ideas

• The Summit opened with a panel about the past and the future of Airflow. It was also the time for the panelist to give a huge shoutout to all Airflow contributors. I personally join the shoutout because Airflow has been in my professional journey for the last 5 years and it helped me grow and achieve so much.
• Then Marc Lamberti gave a huge update about Airflow but done differently — It wasn't about slides with a list of new features but rather about how you can write, in 2023, a data pipeline with Airflow. It's a presentation that silences critics about Airflow's rigidity and complexity.
• Airflow operators need to die — This is a funny topic. Airflow operators are often criticised because they don't work, so people just use Python or Bash operators to orchestrate their own stuff, which leave us with useless operator code. So, Airflow needs a new vision. This talk from Bolke is probably the beginning of operators rebirth. Bolke proposed new storage and dataframe APIs to remove hardcoded operators and decouple source from destinations.
• Airflow can also be at the center of data mesh discussion with companies using multiple Airflow instance to give power to many teams. Kiwi.com showcases how they moved from a monolith to several smaller envs while Delivery Hero explained how they run 500 Airflow instances with a lot of unique specificities.
• A microservice approach for DAG authoring using datasets — The idea is to apply SE patterns to pipelines like migration, broadcast and aggregate. In addition you should create micropipelines which we can define as small, loosely coupled DAG which operates on one input Dataset and produces one output Dataset. And then each micropipelines will implement a unique pattern with defined input and output.
• Dynamic task mapping to orchestrate dbt — dbt has changed the data world and is immensely popular, but dbt orchestration is sill a problem. Many of Airflow users have to integrate dbt within Airflow. This time Xebia team propose an usage of dynamic task mapping to do it (link to Github repo with multiple solutions).
• Astro team also showcased how you can deploy LLM with Airflow — following a16z infra guide.

Understand Airflow internals

3 talks you should watch to learn things you don't know about Airflow internals.

• Airflow is made of 3 main components interacting together: the webserver, the scheduler, the executor and they use a database to communicate. Within the scheduler there is a DAG parser process reading files to understand what needs to be scheduled.
  • This DAG parsing step has flaws.
    • By default you have to wait 5 minutes to have a new DAG displayed in the UI.
    • If you have 300 DAGs coming from a single file (forloop) it works way better than if you have 300 DAGs in 300 files.
  • That's why we should probably move to event-based DAG parsing — In the presentation Bas explains the 4 steps in the DAG parser and what configuration you can change to have better performance. He also demo a event-based DAG parsing that instantaneously display DAGs in the UI.
  • Then John also explained what he did to improve parsing performance — Especially around Python import. Because parsing DAG means running Python DAG code (and import) and import fucks the import time.
  • ➡️ In conclusion you should consider running the dag processor in standalone to remove the impact it could have on the scheduler and follow latest community improvements.
• Niko also discussed about the executor decoupling to unlock the development of third-party executors like an ECS Executor.

Companies feedback

To finish this newsletter 3 companies presentation about their Airflow that gave me inspiration.

• Bloomberg, leveraging dynamic DAGs for data ingestion — I'm a huge fan of dynamic DAGs, I think this is the way to go in Airflow because as a data engineer your role is to create a standardisation layer when it comes to data work rather than doing the actual data work, especially in a mesh concept. Here Bloomberg team create a nice categorisation of data tasks to provide DAGs as a config.
• Reddit, How we migrated from Airflow 1 to Airflow 2 — If there are still people out there on Airflow 1, you should migrate, new Airflow are way much simpler and funnier. But to be honest Reddit presentation can be generalised to every team that want to migrate from a old software to a fresher one. Migration recipes can apply whatever the software you use.
• Monzo, Evolving our data platform as the bank scales — This presentation is full of awesome ideas. It talks about dbt integration within Airflow (using a custom DAGBuilder), monitoring, alerting and Slack interaction with the data stack.

See you next week ❤️ — this week other articles will be blended in next week Data News!

Hey, I'm a bit late once again. I hope this newsletter edition finds you well. This is almost a raw edition, I had quite a big amount of links, I hope you will like this selection.

Gen AI 🤖

• OpenAI’s plan to build the "iPhone of artificial intelligence" — Obviously this is one of the main struggle for OpenAI. In order to stay forever in the B2C market they need more than chat interface, they need hardware, they need to enter in users everyday life. Still not sure we need a need addictive device.
• ❤️ Generative AI exists because of the transformer — A scroll story by the Financial Times explaining what's Generative AI. Good for everyone.
• Evaluating LLMs is a minefield — Slides from Princeton Uni about LLMs evaluation and why it's hard to understand how it evolves. You can find the video version on Princeton website, named "Societal Impact of AI".
• JPMorgan CEO says AI could bring a 3½-day workweek — blablabla, AI is awesome, we want AI everywhere and pay people less blablala /s.
• Language Modeling is compression — Paper from Deepmind. Title looks cool. To be honest this is not the first time I see LLMs and compression in the same paper and it opens, at least, to funny experiments.
• Decoding speech perception from non-invasive brain recordings — Even more crazier. This article describes the state-of-the-art in decoding speech from brain activity. There are multiples models and shows what we can achieve by just looking at electro or magnetic recordings of the brain.

Fast News ⚡️

• Microsoft Fabric: should Databricks be worried? — Vantage did a price analysis between Microsoft Fabric and Databricks. Generally Fabric pricing is simpler because fresh and new but for more complex stuff Databricks still shine.
• Introducing Python and Jinja in Cube — Cube, an open source semantic layer, has released a new writing capabilities in Python with Jinja in the YAML definitions. Something that reminds dbt. You can now write macros to generate YAML.
• Confluent announced Kafka roadmap and Flink as a Cloud service — this is the result of Confluent acquisition of Immerok. Confluent is still growing but struggle to become a real competitor to Databricks or Snowflake.
• Python 3.12 is out — Every year a new Python minor version is released and this year it brings a few cool features. Mainly you get a new generic type parameter, better f-strings with multilines in curly brackets and quote reuse, the per-interpreter GIL—where Meta is proud to say it contributed to it.
• Announcing Malloy 4.0 — This is the tool I have to try soon. Malloy is out with a new version and a lot of new features. As a reminder Malloy is a new analytical language meant to generate SQL to query databases.
• 4 tips to save warehouse money — Paul posted on LinkedIn about search index, avoiding rerunning dbt tests when possible or just deleting tables. Ian also proposed you identify and remove things you don't need anymore.

Tech and data engineering stuff ⚙️

• CRON jobs at Slack scale — Why do you need an orchestrator when you can run CRONs? Slack engineering team details how they wrapped CRON jobs on top of Kubernetes with database table to get monitoring.
• Using ML to identify date formats in file names — Dropbox developed a classifier to identify date formats in file names. This is the backbone of a naming convention feature. It gives ideas. Based on DistilRoberta this is something to look at to fix the mess of a datalake.
• Data modeling is dead! Long live data modeling! — Joe Reis Keynote at Big Data London about his next book topic: data modeling. Joe covers why data modeling was put on the side in the recent years and why we need it back today, showcasing a few useful patterns and definitions.
• Chat with BigQuery data — this is a recycling of all the chatbot use-cases, once again. It's an example where you can use natural language to access BigQuery data. There is also a walkthrough example on Airbyte with LLamaindex.
• Creating an Airflow custom hook for API calls — A guide showing you how you can extend Airflow hooks to have a custom way to call APIs.
• Goodbye Spark. Hello Polars + Delta Lake — Spark is under attack. In the last years Spark has been powering a lot of data use cases but with the modern data stack and more recently with DuckDB, Polars and smaller size OLAP technologies it allows a new way to do data processing.
• Ensuring data quality with Verity — Lyft definition of data quality and a tour of the in-house product to address data quality in the data platform: Verity. This is a must-read and a good showcase of what you can do.
• Database file format optimization: per column dictionary — Mixpanel developed a proprietary columnar database and this article shows what they did to improve compaction and increase performance.
• Exploring the power of graph databases.

Data Economy 💰

• Yahoo spins out Vespa. Vespa is the tech behind Yahoo search engine, it's a search engine and a vector database. In the current Gen AI times, it looks like a good time to do it.
• Contentsquare acquires Heap. Heap is a product analytics solution to understand better you funnel acquisition performance.
• Kestra raises $3m Seed funding. Kestra is the new kid in the open-source orchestration space but disrupting the Python status quo because it's written in Java and requires you to writes pipelines in a declarative way, in YAML. IF you want to know more you can watch this YouTube live I did with Kestra's CTO demonstrating capabilities.

See you soon ❤️.

Hello, another edition of Data News. This week, we're going to take a step back and look at the current state of data platforms. What are the current trends and why are people fighting around the concept of the modern data stack.

Early September is usually conference season. All over the world, people gather in huge venues to attend conferences.. Last week it was Big Data London, this week it was Big Data & AI Paris. I wasn't able to go. But every time I went to a conference in the past, I came back with ideas to change everything because someone introduced me to a new fancy stuff.

This feeling is right. But you should temper your excitation. Let's go through the current state of data to understand what you should do next.

Big Data is really dead

Although the term Big Data is no longer very popular, London probably counted over 10,000 visitors and more than 160 vendors (2022 figures). Big Data London exists since 2016 and when we look at sponsors it's like an history book. Over the years Cloudera logo has been replaced by Snowflake and Databricks ones. Microsoft logo still standing over the years. When everybody is digging for gold, it’s good to be in the pick and shovel business.

The era of Big Data was characterised by Hadoop, HDFS, distributed computing (Spark), above the JVM. This era was necessary and opened doors to the future, fostering innovation. But there was a big problem: it was hard to manage.

That's why big data technologies got swooshed by the modern data stack when it arrived on the market—excepting Spark. We jumped from HDFS to Cloud Storage (S3, GCS) for storage and from Hadoop, Spark to Cloud warehouses (Redshift, BigQuery, Snowflake) for processing.

In fact, we're still doing the same thing we did 10 or 20 years ago. We need to store, process and visualise data, everything else is just marketing. I often say that data engineering is boring, insanely boring. When you are a data engineer you're getting paid to build systems that people can rely on. By nature it should be simple—to maintain, to develop—it should be stable, it should be proven. Something boring.

Big data technologies are dead—bye Zookeeper 👋—but data generated by systems are still massive and is the modern data stack relevant to answer this need in storage and processing?

Is the modern data stack dying?

The modern data stack has always been nice words to bundle a philosophy used to build data platform. Cloud-first. With a handy warehouse at the center and multiple SaaS tools revolving around to answer useful—sometimes not—use-cases. Following an E(T)LT approach.

Historically, data pipelines were designed with an ETL approach, storage was expensive and we had to transform the data before using it. With the cloud, we got the—false—impression that resources were infinite and cheap, so we switched to ELT by pushing everything into a central data storage.

If we summarise the initial modern data stack vision, this is something like:

• move data with Fivetran
• store data in Snowflake
• transform data with dbt
• visualise with Looker
• document with a catalog, prevent with data observability, orchestrate

So what's left of the original vision of the modern data stack that can be applied in 2023 and beyond? An easy-to-manage central storage and querying and transforming layer in SQL. When you put the things like this it opens the doors and does not limit the modern data stack to 4 vendors.

The central storage can be cloud storage, a warehouse, a real-time system, while the SQL engine can be a data warehouse or a dedicated processing engine. It can go further than that, you can—in fact you should—compose storages and engines, there are too many use cases for any one solution to address. More importantly, the modern 4-vendor data stack is too expensive to scale.

The modern data stack is not about to disappear, it's so simple to use in the first place and it's the core of too many data stacks and practices today. But it needs to adapt to today's needs, hence its incremental evolution.

I believe in incremental evolution

What do you need to do? Well, it all depends on whether you're a newcomer and want to start building your data platform, or whether you already have a stack and are wondering what to do next. If you're starting your data stack in 2023, simply choose the solution that will be the quickest to implement to discover your business use cases, you'll build something later. A lot of companies started with Postgres + dbt + Metabase, don't be ashamed.

When it comes to incrementally change a data platform this is a bit different, you need to find what is going wrong and what could be improved. Like

• data workflows are always failing, are always late—Identify why workflows fails, data contracts might help to bring consensus as code if it fails because of upstream producers, create metrics about failure or latency aim for a 30-days streak with no issues. Define SLAs, critically and ownership. For downstream data quality there are also a lot of tools.
• data stack is too expensive—With the current economic situation a lot of data team were in need to stop spending crazy amount of compute and introspect storage to remove useless data archives. DuckDB can help saving tons of money.
• developer experience to add new workflows—This is something often neglected by data engineers, you need to build the best dev experience for other data people not everyone is fluent with the CLI.
• data debt—You might have too many dashboards or tables, workflows spaghetti. For this you need to do recurrent data cleaning. Find, tag and remove what is useless, what can be factorised. Only healthy routines can prevent this.
• poor data modeling—This topic might be too large to handle in one bullet. Data modeling is the part that really don't scale in data stacks. Because of the growth your SQL queries patrimony will inflate and only data modeling will prevent data from being unusable, repetitive or false. Good data layers are a good start.
• there is no data documentation—Rare are the people who are happy to document what they are doing. Best to do is to defined what is a good documentation and then enforce the requirements before going to production. Think the documentation for your readers.
• data is not easily accessible for humans or AI—We build data platforms in order to be used. You should create usage metrics over of your platform either about business users conversion in the downstream tools, about SQL query writers but also about how AI is using the data. How the AI platform combines with the analytics platform?

This list is probably not exhaustive, but it's a good start. If you think you're good on all counts, you've probably finished the game and that means your data team has built something that works. Don't forget the stakeholders though, as it's probably more useful to have a platform that barely works but serves users perfectly than the other way around.

Conclusion

This post is a reflection on the changes in the data ecosystem. Marketing would have you believe that your data infrastructure may be obsolete but you shouldn't worry about it, if you're still using a crontab to run your jobs that's fine. Just use the right tool for the right job and identify what are your data needs. Tip: data needs are rarely a technology name.

I hope you like this different Data News edition, I'm curious to know what you think about it, I wanted to keep it short while giving a few practical links and ideas.

Your data stack won't explode if you don't use dbt.

PS: I wanted to write also about interoperability of data storage and file formats but that's for another time.

Fast News ⚡️

• Motherduck has announced their pricing — The model simplicity reminds me a lot BigQuery in the early ages. You pay for the cold and hot storages. Respectively $0.04 per GB per month and $0.02 per GB per hour. But it looks like way more expensive than BigQuery.
• Announcing BigQuery Omni cross-cloud joins — Join datasets located in BigQuery with datasets located in AWS or Azure. This is part of BigQuery Omni offering, which is 37% more expensive (in EU).
• 3 lessons to learn before creating your own data team — Christelle wrote 3 lessons learned about a survey that has been run in a private French data community. Mainly it shows that the first hires in a data team have to be picked cautiously.
• How to prioritise projects and scale your Data Science team efficiently — A nice article about how to understand an OKR and make it your own to lead data science projects.
• Mistral 7B, the best 7B model so far and open-source — Mistral AI is the French company that want to compete with OpenAI and they released under Apache license a first 7B model.
• A selection of SQL tutorials — a long list.

Data Economy 💰

• Rollstack raises $1.8m Seed. This is a YC company and they propose a product that automates slide deck with data coming from your data stack. Without engineering or manual work. This is an awesome idea the young myself would have love 8 years ago when I was generating Powerpoint in Python.
• Kolena raises $15m Series A. Kolena proposes an end-to-end framework to test and debug ML models to identify failures and regressions.

See you next week ❤️.

Hey. This is a super late Data News, I wanted to send it earlier but I was travelling then enjoying time with friends and family. I'm still struggling a bit to write as fast as I would like, but 🤷‍♂️.

So, sorry for the late edition and enjoy.

Gen AI 🤖

• Announcing Microsoft Copilot — Having everything under a common brand is great and Copilot is a great name. Microsoft announced that your AI companion called Copilot will be everywhere in the next Windows 11 update. For instance in Paint, Photos and in your web search (Edge and Bing).
• At the same time Microsoft leaked 38To of data — through a Github repository containing a link to an Azure storage with public access open.
• OpenAI announced DALL·E 3 — natively built with ChatGPT to create more impressive image from user prompts.
• I recommend you to follow Oliver on LinkedIn if you don't want to miss anything related to Gen AI. He's writes the best takeaways multiples times a week.

Fast News ⚡️

• Postgres 16 has been released — featuring a few performance improvements in parallel executions (string_agg and array_agg) but also with SELECT DISTINCT and COPY command.
• Astronomer released Ask Astro — A LLM application that is able to understand Astro docs to answer most of the Apache Airflow questions. The source code is on Github.
• The implications of scaling Airflow — Sarah, who's working at Prefect, wrote a post about Airflow downsides at scale and how Prefect mitigates them. I'd not say that all the downsides are relevant blockers but still it outlines on of the biggest Airflow issue: everything is implicit. Airflow is a framework allowing a wide range of code easily leading to debt.
• dbt pattern, test-transform-publish — Often called staging pattern. The idea is to publish the data once tests have validated that the is valid. What Leo proposes is an incremental transformation with tests on top. If the tests are valid then an view runs and select the last update.
• A guide to the Snowflake results cache — Cache is a critical piece to every data warehouse either for reusing data between runs or between stages in the same run. This article details what you have to understand to optimise your Snowflake query writing.
• Use the new SQL commands MERGE and QUALIFY in Redshift — Redshift still exists and tries to catches with the competition. Merge allows you to deduplicate data by writing what you want to keep when rows matches and qualify filters results of a previously computed window function.
• Real-time analytics with Snowflake dynamic tables & Redpanda — A good showcase of Snowflake dynamic tables with Wikipedia data.

Data Economy 💰

• Cisco acquired Splunk for $28b in cash. Crazy amount. Splunk has been here for a while providing a all-in-one platform for tech observability by ingesting logs and events to provide insights on a tech stack.
• Secoda raises a $14m Series A. Secoda is a data catalog tool with lineage and monitoring capabilities. Fresh money will help them to add AI capabilities to the product and increasing monitoring capabilities.
• Motherduck raises $52.5m Series B. In total they raised $100m and announced that Motherduck product is open for everyone and not anymore behind a waitlist. Mainly Motherduck is the company providing DuckDB as a Cloud product but they are not developing DuckDB, their product is quite young but works like expected: with a simple string you can get an analytical cloud database that just works and that can be instantly replaced by a local one if needed.
• Tabular raised $26m Series B. Tabular is the company providing a cloud platform on top of Apache Iceberg—developed by Iceberg founders. I'd say that Iceberg (or table formats) are probably one of the technology that will incrementally change for the better the way we write data pipelines. Providing more control over data storage. Yet I think Iceberg is not yet ready to be widely used (Python write support still missing, you need Spark).
• Anthropic could get $4b from Amazon. Amazon did a first $1.3b in a corporate round to bring a lot of money to one of the biggest OpenAI. The ChatGPT alternative, Claude, is already out there.

See you on Friday ✨.

Hello Data News readers. I'm still struggling to get back into my usual work rhythm. If you add the fact that last week I came up with fewer articles than I expected, this has led me to another blank page. Anyway, after 2 years of work, I have to accept and let go when necessary. But don't worry I don't forget you.

Let's quickly jump to the news, because it's rather busy.

(Gen) AI News 🤖

• Reinforcement Learning: an easy introduction to value iteration — Title says easy, but the article contains maths formula. RL is always something magical and this article explains it well through golf concepts.
• Falcon 180B has been released on HF — This is interesting to note that Falcon has been developed at Technology Innovation Institute (TII) in Abu Dhabi. It brings diversity to Foundation models usually coming from US. But despite of the number of parameters (180B) can it run on your computer? Spoiler, according to Benjamin it needs 100GB of RAM to run and a good GPUs to be able to fine tune.
• If you're late to the party and you need fresh views on LLMs Daniel wrote an introduction demystifying the Large Language Models and Jesse wrote about LLMs impact from a Data Engineering perspective.
• At the same time Github Research quantified GitHub Copilot’s impact on developer productivity and happiness — Developer productivity is a difficult measure to compute. Also productivity ≠ speed, but speed is important. The research also shown that people using Github Copilot feel more 88% more productive and are more efficient and less frustrated.
• HuggingFace CEO and co-founder opening statement at AI insight forum — This week US AI giants went to a 6-hours private meeting with 60 US senators to explore AI regulation. Clement Delangue transparently shared his speech on Twitter. Mainly he treats about openness, risks measurements—like mis-information, elections manipulation or carbon emission increase—and finally safeguards implementation.
• Meta developed an end-to-end AI system performance simulator called Arcadia. From what I understand this performance simulator unlock capabilities in finding what are the best parameters for training.

Fast News ⚡️

• Birmingham City Council has to pay 5x the initial price of the new ERP Oracle project. From £20 million to around £100 million. Crazy amounts.
• I just discovered this week that in June BigQuery introduced primary keys and foreign keys.
• How to reduce warehouse costs? — Hugo propose 7 hacks to optimise data warehouse cost. And if you can read French (🇫🇷) there is the super post by a French data collective about comment réduire ses coûts Google BigQuery?.
• * * * * * schedule Snowflake queries — If you want to live dangerously you can use Snowflake table schedule to compute tables periodically. I don't recommend it, it's a Pandora's box we don't want to open.
• Dimensional data modeling with dbt — A great 6-steps process to create a simple dim-fact model with dbt. It also uses the dbt_utils macro to generate a surrogate key.
• Head-to-head comparison of 3 dbt SQL engines — A comparison between DuckDB, Spark and Trino where DuckDB wins almost every fight. Obviously biased by the fact that the comparison is done on a mono node and DuckDB is built for this.
• Scrape & analyse football data — Benoit nicely put in perspective how to use Kestra, Malloy and DuckDB to analyse data.
• Factory Patterns in Python — It remembers me Java design patterns classes at the engineering school. A bittersweet feeling. Still I think that the Factory pattern is probably the one that I've used the most since the beginning of my career and this post explains it well.
• When charts looks like spaghetti, try these saucy solutions — Great tips to enhance your dashboards.
• ❤️ The key to building a high-performing data team is structured onboarding — The title say it all. Still in the article it mentions 2 key piece. First you need a great onboarding doc and then you need to successfully pass the "bootcamp" phase, which matches the 2 first weeks.

Of course, great onboarding isn’t the only thing necessary to build a high performing team, but it’s almost impossible to build one without great onboarding

Github gems 💎

• nike-inc/brickflow — Nike engineering team released a Python framework to orchestrate jobs in Databricks workflows. Mainly it maps to Airflow concepts to have a declarative interface over Databricks objects like Cluster, Workflows or Notebooks in order to orchestrate them.
• sourcegraph/cody — Cody is a free, open-source AI coding assistant that can write and fix code, provide AI-generated autocomplete, and answer your coding questions. Under the hood it uses either Anthropic or OpenAI LLMs to work and requires a free cody.dev account.
• teej/titan — Titan is a Python library to manage data warehouse infrastructure. Titan allows you to create Snowflake Databases, Warehouses, Role and RoleGrant in a programmatic manner.

Data Economy 💰

• SQream raises $45m Series C. SQream is a GPU-based SQL database that can act as a data warehouse promising performance peaks at PB scale because of the GPU architecture. It also works well for machine learning use-cases.
• Gable raises $7m in seed funding. Chad Sanderson launched his data contracts product / platform in association with 2 other co-founders. Chad produced a lot of content around contracts in the last 2 years. It seems Gable is here to fix upstream data quality with contracts. Alerts will be sent in Github to alert owners when something breaks enforced rules.
• Databricks raises, another, $500m in Series I. Soon there will be no letter in the alphabet to associate with Databricks fundraising. Since the beginning they raised $4b and are today valued at $43b. Nothing to say except than they love to burn cash. Be ready for a downhill in 2025 if you have picked Databricks.
• Treefera raises $2.2m in pre-seed to develop a data platform that monitors forests built for carbon offsetting and reforestation. I really like their "data products" approach and the geo visuals over forests risks.
• Collibra acquires SQL data notebook Husprey. Husprey is a Notion-like directly in the warehouse to write stories on top of each interesting tables or facts. It will become a nice product in the Collibra data governance ecosystem.

See you next week ❤️.

Hey, I'm back.

I've taken an unplanned 3-week break since the last Data News, let's be honest, it was necessary! I spent a few hours working on the fancy data stack project and articles are in the works, but it was idealistic to produce quality code and content while enjoying the summer. Like wine, it takes time to get it right. If you want a first glimpse of the Dagster code, you can look at it on Github, not yet documented but commits messages are clean.

On September 1, I'm still getting used to the school rhythm. A new year starts in September, new friends, new classes and new things. Even if, as an adult, things are different now. Data News is back, but with the same recipe: a weekly newsletter to let you catch up on the previous weeks' articles. I make the selection myself, I choose things I like while being under the others influence. But I'm not an influencer. I just create content.

This week features what happened in August, even if it was summer holidays, news, features and drama got the data world. Enjoy the news recap.

dbt tests 🧪

dbt Core proposition has been to bring software engineering practices to SQL development. Obviously testing is invited to the party, but tests are hard and everyone does and understands tests differently. There are unit, integration, functional and end-to-end tests.

This summer a lot of people wrote about testing with dbt.

• dbt tests: How to write fewer and better data tests? — Ari catalogs the kind of tests you can write with dbt. Do you want to test data or code changes? (this is the more important question tbh) Do you want to test schema changes, missing data, volume or value anomalies? He covers everything.
• An overview of testing options for dbt — Another exhaustive and less opinionated list about the options out there to write test on the data.
• A simple approach to implementing unit tests for dbt Models — Mahdi proposes a CTE nomenclature to create input and output in dbt models to unit tests them.
• dbt Core unit tests are coming — A discussion on Github about unit tests and fixtures definitions in YAML to tests models. If implement within dbt Core it would be the most awesome feature. Because hacking with seeds and custom macros looks nasty.

Generative AI 🤖

I haven't really been keeping up with the news because it moves too fast, but here are a few things that have stood out:

• Meta releasing models faster than before — Expanding DINOv2 a computer vision model (on X), releasing SeamlessM4T a multilingual multimodal translation model (on X), releasing Code Llama a LLM for coding.
• Snowflake fine-tuning Code Llama for SQL generation — With these fine-tuning it seems they are close to GPT-4 accuracy in text-to-SQL.
• Llama 2 is about as factually accurate as GPT-4 for summaries and is 30X cheaper —
• A French Youtuber released on Twitch a 24/7 AI deep-faking French presidents (Macron, De Gaulle, Chirac) answering the Twitch chat questions, but his channel got banned by a Twitch bot after AI-Macron said something illegal while answering a question about worst french cities. AI fights this is the future we want.

Fast News ⚡️

• Python into Excel — Microsoft and Anaconda announced Python coming into Excel. I'm bitter-sweet about it, on one side I don't think Excel is a good platform for software development, on the other side, let's be honest a face the truth Excel is the only data platform business users wants. Still the big winner of this is Microsoft, because Python code will run on Azure.
• After Excel, Notebooks get a second youth — Meta explained how they schedule Jupyter Notebooks in production, Google announced the BigQuery studio with embedded Notebooks in the UI and Jupyter released Jupyter AI (you call it with %ai) to bring Gen AI to the notebook.
• New features in Airflow — with 2.7 you get a Cluster Activity UI and with airflowctl new CLI you can spin up Airflow instances in a wink.
• Introducing the revamped dbt Semantic Layer — dbt Labs announced the Beta of the Semantic Layer which will be a paid product in dbt Cloud. I've already wrote a lot about the semantic layer and more is to come. So let's see where it goes.
• Introducing SOL: Sequence Operations Language — A new dedicated to to sequence analyses, which can be useful when working with web traffic data.
• Answering "Why did the KPI change?" using decomposition — If you are an analyst who needs to explains everyday why a metric increased or decreased, this article is for you. Max explores metrics decomposition for sum and ratio. This is brillant.
• Apache Hudi: From Zero To One (1/10).

Drama

• Instacart's Snowflake bills — When public companies publish results numbers are looked at. This time Instacart bills have been overlooked. While the company said it has spent $13m, $28m and $51m respectively for 2020, 2021 and 2022 in Snowflake spending and plan to spend $15m in 2023.
  
  People supposed Instacart found the magic solution to reduce costs, others said it migrated to Databricks. But the main reason is: prepaid credits. The Snowflake press team even wrote a post.
  
  Still you can watch the perfectly timed video about How Instacart Optimized Snowflake Costs by 50% or Snowflake optimisation at HelloFresh.
• Hashicorp changed Terraform license model — Hashicorp decided to move from Mozilla Public License to Business Source License (BSL). BSL is source-available and not really open-source. Following the announcement OpenTF forked the repo.

Data platform stuff

4 articles that gives food for thoughts about the future of the data field.

• The uses and abuses of cloud data warehouses — A streaming database saying to a batch database: "you're not suited for operational use-cases, only analytical". The batch database answers one day later.
• Level-up with a Medallion architecture — bronze, silver, gold are the structuring layers of the Medallion architecture. Matt explains it for you.
• The data contract pivot in data engineering — It's a fancy name, but it aims to solves upstream data problems with a technical + process solution.
• After the modern data stack: welcome back, data platforms.

Data Economy 💰

• Hugging Face raises $235m in Series D. You can see Hugging Face like the Github of machine learning models, but it's much more today, this is a global platform to distribute AI—in every form possible. Obviously with the new popularity over Generative AI models HF is playing a key distribution role.
• Stemma has been acquired by Teradata. Stemma is a company that has been founded by ex-Lyft employees working on the company data catalog Amundsen. Mainly Stemma is built on top of Amundsen with Enterprise features. Consolidation.
• Rockset raises $44m in Series B. Rockset is a real-time search (and analytics) database aiming to replace Elastic. Like Elastic but in the cloud.
• Ikigai Labs raises $25m in Series A. Ikigai provides a web platform to do data transformations in a visual way on top of tabular data. You can do entity resolution or forecasting for instance.
• Elementl becomes Dagster Labs, to make it clear. I'm announcing soon blef Labs.
• The Information reported that Open AI will pass $1b in annual revenue "over the next 12 months".

Feels good to be back, see you next week ❤️. I hope you enjoyed your summer.

This is the first article of the Data News Summer Edition: how to build a data platform. I tried to be as short as possible in this first article, details will come in the following ones.

The modern data stack has been criticised a lot, a few are saying it's dead other are saying we are in the post-modern era. The modern data stack as a collection of tools which interacts altogether to serve data to consumers is still relevant. Personally I think that the modern data stack characterises by having a central data storage in which everything happens.

Let's design the most complete modern data stack, or rather the fancy data stack.

In this article we will try to design the fancy data stack for a batch usage. A lot of logos and products will be mentioned. This is not a paid article. However over the years I've met people working at these companies so I might have a few biais.

As a disclaimer, this may not quite make sense in a corporate context, but since this is my blog, I'll do what I want. Still, the idea of this post is to give you an overview of existing tools and how everything fits together.

A few requirements

• The source data lies in Postgres database, in flat CSV and in Google Sheets.
• I want something cloud agnostic—when possible.
• I want to use open-source tooling.
• Everything I do should be production-ready and public. At the end of the experiment you should be able to access the tools—when possible.

Source data

When I was looking for data, I was looking for a bit of volume, something geographical and without PII. I personally like NYC Taxi trip data but sadly it has been used many times so it removes a bit the fun. At the same time the Tour de France was ongoing and I found a "way" to get Strava data for Tour athletes on Strava. So I thought it was the perfect data to build a data platform.

Mainly there are 3 datasets:

• Athletes — all the data about the athletes like their race ids, teams, their profile but also their body size. It will be a Google Sheets.
• Stages — le Tour de France is a 3-weeks race, it contains 21 stages, every stage is a GPS path with a few checkpoints. It will be 21 CSVs.
• Race — the actual race data which is a GPS data point every second for each athletes on Strava + other data points sometimes. It represents almost the half of the peloton. It will be a table in Postgres. Postgres is not the best solution for this, but as I want to mimic enterprise context, having a Postgres database is kinda mandatory.

Race data will be partitioned per day, but as the Tour is already done, it will be a bit different than in real life environment. Still, this is something I keep in mind for future trainings. Because I'm convinced that to learn data engineering you need to experiment real life pipelines running every day to experiment the morning firefighting.

I'll delve into the data in the next article, but I won't detail how I got the data because, you know.... 🏴‍☠️. Actually, it's just a few Python scripts and a bit of F12, but that's not the point of this article.

The fancy data platform

In order to have a complete data platform we will need to move the data from source to consumption. But what the consumption will look like?

I want to answer multiple use-cases:

• Create a dashboard to explore stages results
• Give a LLM driven bot that answer common questions about the race
• Compare 2 athletes performance on a specific segment and generate a GIF

In order to answer this we will need to ingest data from the multiple sources, then transform and model the data in the chosen data storage and finally develop consumers apps to answer the business needs.

Let's try to throw a first design of our application—with logos. Obviously this can be subject to change. Either because it's too complicated either because I want to change. Once again this is fiction so I can afford to change stuff.

Actually as a one of my main advice is that you should never be strict about tech choices because you can't plan the unexpected. So do yourself a favour and accept to throw away something that does not work for you.

Just for the sake of being open, there are a lot of alternatives and my choices could have been different. Here what you can also consider if you're doing your own platform.

• Extraction
  Open-source — Dagster, Airbyte, Airflow, Prefect, Mage, Kestra, dltHub
  SaaS ($) — Stitch, Portable, Orchestra and the cloud versions of the OS tools
  SaaS ($$) — Fivetran
• Transformation
  SQL — dbt, SQLMesh
  Python — pandas, polars
  Distributed — Spark, Pathway
• Datalake
  Open-source — MinIO, Ceph, LakeFS, OpenIO
  SaaS ($) — S3, Google Cloud Storage, Azure Blog Storage
  Table format — Apache Iceberg, Apache Hudi, Delta
• Warehouse
  Open-source — DuckDB, ClickHouse, Apache Pinot, Apache Kylin, Apache Doris
  SaaS ($) — BigQuery, Snowflake
• Semantic Layer
  Open-source — Cube, Malloy, sqler
  SaaS ($) — dbt Cloud, LookML
• Governance
  Open-source — Datahub, OpenLineage, OpenMetadata
  SaaS ($) — CastorDoc, Atlan
• Analytics
  Open-source — Superset, Metabase, Lightdash
  SaaS ($) — Tableau, Looker, PowerBI, Whaly and the cloud version of the open-source tools
• Exploration
  Open-source — Streamlit, Jupyter
  SaaS — Hex, Graphext, Husprey, Count (etc. this list can become infinite)

Conclusion

After this design exercice I have mix feeling. I'm in between. I think this is a fancy stack because I tried to put everything inside, but as the same time is find it quite boring. Like this is just stuff that works. This is linear, I'll move data from A to B to C in to order to use it with D. Actually this is just modern data engineering.

In the following part of this series you'll follow my adventure in the extraction, the transformation and in the serving for analytics and Gen AI usage.

I hope you'll enjoy this Data News Summer Edition.

FAQ and remarks

• Why do you use Google Cloud?
  Because my credit card is already in place and I'll be much faster. My opinion on the matter is this: all clouds are born equal, you just have to find the one you're most comfortable with, or suffer your company's choices.
• DuckDB is not really a data warehouse.
  I pick DuckDB because it's fancy. I think I'm gonna hit some limitation especially in Geo compute, so I might switch to ClickHouse or BigQuery if I lack of time.
• I hate Github actions, but I prefer putting code in public in Github.
• I used the way to visualise data platform Gitlab data team is using.
• What about the performance of the platform?
  I don't really care about performance, because this is not large data and I don't want to spend hour optimising for performance.
• Do you have a budget?
  Something reasonable. I think ~100€ / month is ok for this experience.
• What will you do in LLM category?
  I don't know yet. If you have ideas about what I can do reach me.
• Why Dagster?
  I've been building things with Airflow for almost 5 years, I love trying new things and in the list of orchestrators that have hyped me the most, Dagster is number one. Software-defined assets are something I wanted to play with.

Small Fast News ⚡️

If you want dont care about this, here a few articles you might want to read by the pool.

• How to model: Kimball vs One Big Table — This is one of the main topic of discussion in the data space. Should you go for dimensional modeling or go for OBT or even go for query-driven data modeling (coined by Joe Reis—who's writing a book about data modeling).
• Costwiz, Saving cost for LinkedIn enterprise on Azure — LinkedIn developed a complete data platform on Azure to save costs on Azure.
• Confidence — An experimentation platform from Spotify — After years of experience in building experimentations, Spotify decided to release a product for others to do it. This is in private beta and the move is interesting.
• DuckDB vs. Spark, ElasticSearch and MongoDB — Even if this is not really relevant to compare it to NoSQL databases, tests are showing that DuckDB looks better.
• Overview of JupyterHub ecosystem — Just saving this for me because I do stuff on it.
• Read Data Engineering Weekly.

See you next week ❤️.

Hey, this is a mid-2023 edition with some of my favourite articles and the popular articles that have been shared this year in the newsletter. There isn't any fancy calculation on how to find the popular articles. Here how it's done.

Every link sent in each newsletter is tracked in 2 ways:

• when you click on a link it first redirect you to my blog so I know that you've clicked on it
• it adds ref=blef.fr to the url, so the original articles knows that the traffic comes from me, mainly it's a great way to support me by being discoverable to others

I've used the click data to sort articles by popularity. Obviously it has a few biais like recent editions get more clicks because I have more subscribers but impact is minimal.

A few numbers. Since the beginning of the year I've shared around 500 articles, which generated at least 22k views on creators articles. I say at least because this is an low estimated number, from a projected experience I think that in the reality it's twice this number.

If you have travel time I also recommend you the first episode of Data Minds, my podcast, with Joe Reis.

Popular articles

I have sorted the articles by bucket. The order does not really makes sense, they were all popular.

General

• 💰 Because we all love money, Mikkel's Europe data salary benchmark was the most viewed. In the article he shares salaries extracted from job listings, using dimensions like seniority, location and companies.
• 📃 In every data team this is super important to write documentation, Marie wrote an awesome 101 about data documentation. The article gives best practices for establishing complete and reliable data documentation.
• 🎰 Reducing the lottery factor, also named the bus factor is risk measurement about knowledge sharing. In data teams a lot of work have to be done in the early days to avoid knowledge to be lost later on. The article gives ~10 advices to apply to lower the risks. Among them I like the changelog, the pair-programming, the pre-recorded video and the stable credentials.
• 🌎 The data journey manifesto is a manifesto to put principles on the data journey to avoid the mess in production. There are 11 principles and 11 new ideas to create an healthy platform. For instance you should not trust your data providers and what worked last week will not work today.

Modern data stack

• 🔮 The future of data by Pedram. 3 takes on the future of data teams. I really like Pedram, he tweets a lot—or we should way xs—a gives great advices with humour. Mainly the articles says finally we address ops teams, the semantic layer is the next big battle and business logic management is a mess. He also recently joined Dagster team in DevRel.
• 🔥 Matt gives 5 hot takes on the modern data stack. I don’t totally agree with everything. This is about Redshift, Airflow, Airbyte, dbt and production.
• 🧱 A good summary of the required blocks composing the modern data stack.

Technical deep-dive

• 🏗️ Simon wrote an excellent 3 parts data modeling deep-dive. An introduction to data modeling, the different techniques and the tools and future.
• 📑 Data contracts were very trendy this year. I also think they are quite useful. PayPal released their template for data contract. This is a exhaustive list of what you can expect in contract: schema, quality, SLAs, security and custom properties.
• 👨‍🏫 Count.co designed 2 amazing boards. You can learn SQL or follow a guide to hire your data team.
• 🐍 Finally a few useful code patterns in Python.

See you next week ❤️ and I wish you great holidays.

Hey, I hope this newsletter finds you well. This is a small blogpost to give you a few reads while waiting for your next travel. We can already feel summer, I found less articles to enter the selection this week.

Also be ready for the Data News: Summer Edition. For the next 5 releases it will be a bit different than usual, less curation and more original articles written in advance to allow me to take a break.

You'll—probably—get:

• A 2023 must-read articles
• How to create a batch data platform—using Tour de France data—from ingestion to visualisation using all the fancy tools the data world can offer (in 2 or 3 parts)
• Docker for data people
• The disparition of the data engineer

Fast News ⚡️

• For the love of the game — Winnie from dbt Labs wrote a great post about seeing data a as game, analytics being the game design. What if we conceive data as a game for our consumers and not as a linear tool to do boring actions. In the article the author also shares dbt is jQuery, not Terraform, which awesomely describes how dbt helps you enter flow state for data work.
• How an acquisition fails — It's been a long time since I've shared Benn's articles, but as always I can't recommend him enough. This time it's about tech acquisitions and what can be done to fail—or succeed.
• Microsoft Fabric: An end to end implementation — A first—blurred—glimpse of Microsoft Fabric capabilities, Jordan reads data from Sharepoint and Azure Storage, then transform it using PySpark to visualise stuff in PowerBI. Classically boring stuff.
• How to chat with data in Snowflake using ChatGPT, dbt, and Streamlit — Less boring, obviously when you put ChatGPT and dbt in the same sentence it creates buzz instantly. This is an interesting demo of how you can quickly build a chat experience—using OpenAI—on top of you data models.
• LLM based pipelines with PostgresML and dbt — Mainly for me this is a discovery of the PostgresML an open-source extension that brings ML functions to the database. As cloud databases like Snowflake and BigQuery brought it years ago, this was mandatory for the Postgres stack. In the article it shows you that you can than run transformers or embeddings directly from dbt.
• Taking charge of tables: introducing OpenHouse for big data management — New data product at LinkedIn: OpenHouse. OpenHouse sits on top of the LakeHouse to bring a control plane to managed Iceberg files. It reminds me something... We used to call it warehouse back in the days.
• Models on HuggingFace — Clement, the CEO of HuggingFace, congratulates the community and himself because a lot of public models are hosted on HuggingFace, it shows how fast and deep things are going.
• Plot Gallery on Observable — I'm not often a fan, but Mike Bostock is different. He created d3.js while at the New York Time, he brought something unique to digital data visualisation. More recently he co-founder Observable, which is an awesome tool to do visualisations, and the plot gallery makes me envious—while quite simplistic.

Data Economy 💰

• Hightouch raises $38m in a Venture round. Hightouch has been primarily known for his reverse ETL solution. With the money the team announced a new suite of tools to activate customers in the warehouse. You can see it as a CDP—customer data platform—in your warehouse. It means you get a unified view of customers across all your tables.
• Polar Analytics raises $9m Series A. Polar Analytics is a vertical SaaS to provide analytics for Shopify vendors. This is less data engineering oriented but still I find interesting to see a "reporting" product raising money. Also vertical product like this can give ideas to marketplace on what can be great reportings.
• Unstructured raises $25m Series A to build ETL for LLMs. Unstructured wants to give you the ETL toolkit to use company complex data like HTML, PDF, CSV, PNG, PPTX, as they say on their site. Personally I did not know that CSV was a complex source of data but ok. To be honest at the moment it looks like a fancy text extractor.

See you next week ❤️.

Hey, it's Saturday I hope you're enjoying July, taking deserve break, reading data engineering articles while at the beach or traveling to unknown places. Sometimes there are Fridays when I don't find any glue between articles for the newsletter and I have an idea of something to compensate but it takes me the whole Friday of exploration.

And here we are on Saturday. Yesterday I found a way to get sensor data of half of the Tour de France peloton, I was sure it was a good dataset to explore new tools with. And it's honestly a great dataset but it's a bit hard to download and format all the data for exploration. So it will be for later.

Anyway, here a quick press roundup about a few news and articles.

Gen AI 🤖

• Elon Musk announced xAI, his new company, to show that he's better than the rest. He hired alumni from all the AI companies (e.g. Deep Mind, Google, OpenAI, etc.). They held a 2-hour Twitter Space in which they detailed the vision a little. It's mainly about building an AGI capable of understanding the universe. They say we are a few weeks away from their first release. Here a great summary of the space.
• Associated Press sign with OpenAI to share AP's text archive — Interesting to say as it's one of the first deal like this. It reminds me when press gave up years ago on their own platform writing for Google and Facebook news platform. At least this time we will know what OpenAI uses for training.
• Shopify introduce Sidekick — Once again Gen AI is a Copilot. Shopify introduced a right panel in the UI to help vendors in any way. Gen AI used as a Copilot. In the video we see the Sidekick generating a chart to answer a sales question.
• Hollywood actors taking a strike action — They don't want AI and computer-generated faces and voices to replace actors.
• Clibrain, a Spanish startup, launches to build LLMs models for Spanish. They released LINCE-ZERO. Spanish is the second most spoken language by native speakers and the fourth most spoken by all speakers.

Fast News ⚡️

• How we cut BigQuery costs 80% by hunting down costly queries — Mixpanel team hugely reduced their BigQuery spending. They use Fivetran, dbt and Census. In order to get started they first built a cost dashboard using information_schema.jobs tables. Then they took actions, mainly: avoiding SELECT *, materialising intermediate result, adding partition and going incremental. Nothing new but good reminder.
• Data Contracts in the Modern Data Stack — Whatnot is one of the company who embraced Data Contracts last year. This article details what they shared in they excellent Data Council talk. Mainly their implementation is a Protobuf Schema Registry and interface at event production and consumption.
• Introduction to dimensionality reduction — I've gave up on machine learning a few years ago, so I really like every article explaining with visual machine learning concepts. This article explains the dimensionality reduction that is often mandatory when datasets grows. There is a part two with live Python examples.
• Make Python free-threading — This is how open-source is made. In a community discussion about removing Python GIL. Someone from Meta said they can dedicate 3 CPython internals engineers to work 2 years+ in breaking the barriers. Python GIL stands for Global Interpreter Lock, which is a lock that allows Python to use only one thread. Interesting to see.

See you next week ❤️

Hey you, this is the Saturday Data News edition 🥲. Time flies. I'm working for the Series of articles in advance for August about "creating data platforms" and I'm looking for ideas about the data I could use for this. Having some kind of simulated real-time data would be the best. But it requires to write a simulation. Which is enough complicated. What would you use?

Small French aside 🇫🇷

(A small part in French, jump to next section)

Cette semaine j'ai lancé mon podcast en français nommé À l'heure des données. Dans ce podcast, qui sera mensuel, je vais discuter avec des experts francophones qui font l'écosystème. On discutera du présent mais aussi du futur.

Dans le premier épisode j'ai discuté avec Benoit Pimpaud qui a été data scientist à l'Olympique de Marseille et qui s'est reconverti plus tard chez Deezer en data engineer. Aujourd'hui il s'occupe du produit chez Kestra, un orchestrateur open-source développé en France.

🎧 Pour nous écouter : Apple — Spotify — Deezer — Amazon

Sue un tout autre sujet, Stéphane Bortzmeyer a participé au colloque du CNRS sur Penser et Créer avec les IA génératives et il a écrit un rapport sur ces 2 jours.

PS : est-ce qu'une version française de mon contenu t'intéresse ?

The new dbt Semantic Layer

Following the acquisition of Transform by dbt Labs a few months ago, dbt Core integrates MetricsFlow. MetricsFlow was the semantic layer of the acquired company. This week, Nick Handel, co-founder of ex-Transform, wrote about how dbt Core specs will adapt.

As a reminder a semantic layer is a definition on top of your models meant to be reusable. The idea, is then, to use the semantics to generate SQL queries. You can read my article on the semantic layer.

In the new vision it will be possible to define multiple things:

• entities —It defines the nodes of your business models. In a dbt model, you can define primary and foreign entities. A foreign entity defines an edge between models, hence a join in the final query.
• measures — A value aggregation.
• dimensions — A categorical or a time field than can be used either in a group by either in a filter.
• metrics — A pre-defined object that combines entities, measures and dimensions.

Just ahead I gave you an precise example of how the new nomenclature will behave for a simple case with a fact_transaction model. This is important to notice that the semantic layer is something that sits on top of you current dbt models definitions.

To complete the picture this is important to notice that the revenue_usd metrics can be queried at the moment either with a CLI, either via the API dbt Labs will release through their dbt Cloud offering.

As an extension I've seen 2 things this week that I feel makes sense here:

• VulcanSQL — A data API framework for DuckDB, Snowflake, BigQuery, PostgreSQL. Actually Vulcan let's you define in a blink parametrise SQL that you can expose through an API. It comes then with a catalog, a documentation and a way to connect downstream consumers tools (e.g. CSV exports, Excel, Sheets, etc.)
• A Rill Data dashboard about DuckDB commits — DuckDB commits is just an example. What I want to show here is Rill Data UI, while being relatively simple offers a standardise way to explore a dataset. On the left you get the metrics, on the right the dimensions, everything can be clickable and allows you to drill down. Under the hood it's "BI-as-code", YAML defining this dashboard can be found on Github.

These two examples are not really semantic layers in the strict sense, but revolve around the concept.

Gen AI 🤖

• Deploying Falcon-7B into production — If you want to launch your own open-source model on Kubernetes, this is a tutorial to do it.
• Langchain: explained and getting started — Langchain is a toolkit that lets you chain—what a surprise—components. Actually it's some kind of pipelines, every component as inputs and outputs and Langchain do the glue. Components includes stuff like prompts, LLMs, agents or memory.
• Langchain integrates Cube (the semantic layer) — Wrapping-up with previous category, Langchain can use Cube as a data loader.
• The rise of Vertical AI — Verticality in business always existed because it brings contextualisation. This articles described what will arrive on the market on top of Foundations and horizontal models that tries to be generic.
• Introducing NSQL: Open-source SQL Copilot Foundation models — This is a Foundation models that generates SQL, claiming to outperform others.
• Introducing Superalignment — Some stuff OpenAI wrote about the future (I did not read).
• CodeGen2.5: Small, but mighty — Salesforce released a new version of the CodeGen model. I hope they did not trained it on their internal code 🫠

Fast News ⚡️

• Career advice for aspiring progressive data professionals — Brittany has been working in progressive data for years and she's giving advices for people who wants to follow her path.
• Declarative data pipelines with Hoptimator — After trying to bring self-service for data pipelines at LinkedIn, they decided to go for declarative data pipelines supporting only a specific data movements. With YAML. We were visionary when we designed and developed this at Kapten 5 years ago.
• Airflow: scalable and cost-effective architecture — Hussein, an Airflow committer and PMC member, proposes an ideal architecture for big Airflow projects.
• Scaling data teams: 5 learnings — BlaBlaCar data team is well known in France now and recently embraced a data mesh organisation. Manu, the VP shares 5 learnings you should as a manager be aware of.
• Measuring the carbon footprint of pizzas 🍕 — Shit I've eaten a pizza yesterday. Max includes in the study 4 axes: agriculture, transformation, packaging, and transport. With this Margharita obviously is the less emitting one. 4x less than a Calzone with meat.
• Parquet file format explained — and how it compares with Avro & ORC.
• Iceberg won the table format war — Don't be click baited by the title, the article has been written by a dev rel at the company who mainly maintains Iceberg.
• Reducing data platform cost by $2m — How Razorpay optimised (mainly) their S3 storage (deletion, relocation) to save a lot of money.
• Every major announcement at Snowflake Summit — Another view than the one I shared last week by someone who actually was at the Summit.
• An intro video to open lineage, which is a important topic to give visibility over your data platform.
• Makefile tricks for Python projects — One of the best data magical trick. We repurposed old good Makefile to create simpler CLI on top of our daily tool. This is an article giving tips to make your best Makefiles.
• You can now GROUP BY ALL in Snowflake.

Data Economy 💰

• DigitalOcean acquires Paperspace. Paperspace is an all-in-one SaaS product to develop, train and deploy AI applications. With a custom Notebook UI based on Jupyter you can develop your models while checking at ressources, when the models is reading you can deploy it within containers.
• Redpanda raises $100m in Series C. Redpanda is a great product for developers. The best way to describe it is: this is a Kafka alternative. Built for modern times it removes most of the Kafka complexity by implementing all Kafka APIs.

See you next week ❤️

Hey, since I said I should try to send the newsletter at a specific schedule I did not. Haha. Still here the newsletter for last week. This is a small wrap-up from the Snowflake and Databricks Data + AI summits which have taken place last week.

There are so many sessions at both summits that this is impossible to watch everything, more Databricks and Snowflake do not put in free access online everything so I can't wait everything. I'll try to recap the major announcements by reading between the lines and through social network posts.

Snowflake Summit ❄️

The marketing tagline of Snowflake have always been "the Data Cloud", with this year announcement we can feel it really accelerated to achieve this vision. Snowflake wants you to send whatever data on their cloud and then now you can use a lot of different features to do stuff on it. They announced:

• Document AI — A new integrated product where you can ask questions in natural language on documents (PDF, etc.). With LLMs they will try to answer questions. Once you are happy with the quality of answer you'll be able to publish the model and use it in SQL queries and write pipelines on top of it to infer on new documents and send emails when needed.
• Snowflake Native App framework — Via the Snowflake marketplace vendors and developers will be able to create apps that you can run on your data. In the UI you pick the tables you want the app to run on. Here the native apps marketplace, there are only 25 apps and it only works on AWS at the moment.
• Container Services & Nvidia partnership — Snowflake is slowly becoming a one-stop shop, with container services you will be able to run your own apps in a Kubernetes cluster managed by Snowflake. For instance tomorrow you'll be able to launch Airflow (via Astronomer) within Snowflake. On the same topic Nvidia partnership will bring GPUs to Snowflake offering for users in need of large compute for AI training. Thanks to this data do not move out of Snowflake, or if we say the truth, out of your underlying cloud.
• Dynamic Tables — Dynamic tables are streaming tables. With Snowflake you can send real time data coming from Kafka, for instance, with dynamic tables you can create a table on top of the real time data that refreshes in real time, using only what's needed to compute the new state. Dynamic tables has been announced last year, but looks finally in preview. In the demo there is also how the SQL UI integrates LLMs generating SQL from a comment.

PS: s/o to David who also covered Snowflake changes.

Data + AI Summit 🗻

The theme of the Databricks summit is Generation AI, it's a well found title regarding the current state of data. I watched the 3 keynotes to find announcements but it looks like less structured that Snowflake still here a few takeaways:

• Microsoft and Databricks are still best friends, even after Fabric. In a quick Skype call Satya Nadella, Microsoft CEO said that discussions about responsible AI while developing it is a good thing. We should explore 3 parallel tracks at the same time: misinformation, real world harms (incl. bias), AI takeoff.
• The CEO of Databricks was on stage and use words that I like, he says
• • data should be democratise to every employee
  • AI should be democratise in every product

• LakehouseIQ — Matei Zaharia presented it on stage. LakehouseIQ is a way to use your Enterprise signals (org charts, lineage, docs, queries, catalog, etc.) to contextualise LLMs used in UI assistants. In the demo LakehouseIQ is asked to "get revenue for Europe" but understand that Europe is not the exact name of the region for this company but EMEA. Here a demo of LakehouseIQ. In the demo we also sees that you can generate SQL from a comment in the UI.
  
  This is their way to democratise data to every employee.
• Databricks acquires MosaicML for $1.3b— It should land in data economy category but you know. I've shared MosaicML last week because they are the ones behind the first open-source LLMs, the MPT models, on Apache License. This is a great move from Databricks to set themselves in the AI ecosystem for real. As a side note Naveen Rao, Mosaic CEO, said that to train MPT-30B from scratch you need around 12 days and less than $1m.
• LakehouseAI — Research shown that 25% of the queries get their costs misestimated by the query optimisers and the error can be 10⁶. Databricks built a new way to do I/O with AI, they promise that you don't have to do any kind of indexes and the engine can "triangulate" where the data is to be faster than before. Mainly you have to see LakehouseAI like an AI DBA that does magical stuff to your engine by learning on all your queries telemetry.
• They also announced a lot of stuff around Spark.

As you can see Lakehouse is becoming more than ever a marketing brand around Databricks. In the end what we want is a place to store data and an engine to query data. That's all.

Data Economy 💰

• ThoughtSpot acquires Mode analytics for $200m — This is consolidation at work. ThoughtSpot is a company who tries to bring AI in the analytics domain. With TS you can define insights and access to it, with Mode they gain a end-user application that people are already using. Also you might know Mode through Benn Stancil blog.
• Hopsworks raises $6.5m — Hopsworks is a feature store.
• Inflection AI raises $1.3b from Bill Gates, Eric Schmidt, Microsoft and Nvidia. They developed a personal AI called Pi who's designed to be supportive, smart and here for you at anytime. Let's see where it goes.

See you soon ❤️.

Hey, this is the Data News. It's super hard to change habits, but it's how it is, the newsletter is going out on Saturday. I hope this edition finds you well. Summer is coming ☀️.

Thank you all because we crossed the 3000 subscribers mark last week. Let's go for the 4000 before the end of the year 🤗.

This is a almost-raw edition for this week.

Gen AI 🤖

• MPT-30B-Chat — This is a chat interface hosted on HuggingFace on the MPT-30B model. The MPT models are interesting because they are on Apache Licence, which can means true open-source, unlikely others.
• In the continuity to licence topic you can watch this great video about laptop-sized ML for text, with Open Source where Nick Burch explore what you can do today on a laptop and introduce greatly the Gen AI field.
• New approaches for detecting AI-Generated profile photos — This is the era we're going to live in. We'll be writing models moderating generative models. Am I the only one who thinks this is a waste of energy?
• Crypto collapse? Get in loser, we’re pivoting to AI — It's a rant that begins with the fact that many opportunists are getting into AI after VC have left crypto. ChatGPT "is a stupendously scaled-up autocomplete", which lead to question about intelligence in AI. I really like the conclusion: "The real threat of AI is the bozos promoting AI doom who want to use it as an excuse to ignore real-world problems — like the risk of climate change to humanity (...) The VCs’ actual use case for AI is treating workers badly".

Fast News ⚡️

• MotherDuck announcing DuckDB in the cloud — First, context. DuckDB is an in-memory analytics database. So it's single server. DuckDB has been open-source by DuckDB Labs. Then comes MotherDuck, a commercial company, with a partnership with DuckDB Labs aiming to to build a modern serverless cloud analytics platform based on DuckDB. That's for the context.
  
  So this week MotherDuck finally announced their cloud offering. It's invite only for the moment — and I did not get my invite yet. In a nutshell the announcement is: you can connect to remote DuckDB by doing md: in the connection string and you can join local and remote data (also seen on Twitter).
• Iceberg in the clouds — Last week BigQuery announced Iceberg support in GA. At the same time James from Snowflake wrote a blog post helping you to chose between Snowflake or Iceberg table format. Mainly he says, pick Iceberg if you know what you're doing.
• An introductory video about Iceberg — If you want a great Iceberg introduction, go watch Fokko's talk from Berlin Buzzwords.
• Understanding dbt runtime environment — Leo takes the time to explicit what are the messages dbt CLI is telling you.
• Replacing Apache Hive, Elasticsearch and PostgreSQL with Apache Doris — This is a technology bingo. You can replace 3 technologies with only one! This post details the choices behind a migration to Apache Doris. Doris is a real time analytical database.
• How data engineers drive data culture and empower users — This article reminds all data engineers that you're part of the team that brings data culture to a company, so you need to play your part.
• How to become a valuable data engineer — A post thats aggregates great ressources and advices to become a data engineer. I mention also that I have a similar one on the blog: how to learn data engineering.
• Dealing with missing weight data — Carbonfact tries to measure the environmental footprint of a clothing. This is not an easy task and ask you to work with missing data.
• Conceptual vs logical vs physical data models — The author mentions that there are 3 ways to model data with different layers of understanding. In the end he says that you should model your data in the 3 layers: conceptual, logical and physical.

Data Economy 💰

• Acryl Data raises $21m Series A. Acryl Data is the company behind DataHub, the data catalog that has been open-sourced out of LinkedIn.

See you next week ❤️

Hello, after the good weather comes the storm. I'm now under the Berlin rain with 20°. When I write in these conditions I feel like a tortured author writing a depressing novel while actually today I'll speak about the AI Act, Python, SQL and data platforms. Casual day at the office finally.

Some personal news, next Monday and Tuesday I'll be at Berlin Buzzwords, if you're ping me, it would be a pleasure to meet and hang together.

There are still seats for the June Airflow Paris Meetup (in French).

AI 🤖

• AI Act 🇪🇺 has been voted in the European parliament. Also called GDPR 2.0 the AI Act is meant to regulate the usage of AI in tomorrow's world. It has been widely criticised by lobbyists, companies or developers. I'm not informed enough so I'll wait before giving my opinion on it.
• I-JEPA: The first AI model based on Yann LeCun’s vision for more human-like AI — Meta is in a frenzy to release new models. Yann's vision goes toward AI systems learning and reasoning like animals and humans.
• Deep multi-task learning and real-time personalisation for closeup recommendations — Pinterest still doing deep learning.
• Last week I shared nice QR Code generated with ControlNet, this week someone released a model on HuggingFace to do it QR Code Conditioned ControlNet (not related to the Chinese original paper) and you can even use the generator web UI.
• DashQL – Complete analysis workflows with SQL — A crazy paper about a new language that mixes SQL with analyses and graphs. It looks sexy but my brain can't read 9 pages PDF without overheating.
• Model and Data Versioning: An Introduction to mlflow and DVC — If you want to understand model versioning this is for you.

Data and Analytics Engineering 🧑‍🔧

• Testing frameworks in dbt — Robbert developed a small framework to do tests in dbt. Mainly he unit tests macros (the logic) with his framework and test data with soda and dbt contracts.
• The data journey manifesto — DataKitchen wrote a manifesto to put principles on the data journey to avoid the mess in production. There are 11 principles and 11 new ideas to create an healthy platform. For instance you should not trust your data providers and what worked last week will not work today.
• Why data consumers do not trust your reporting — It is a good illustration of the data journey manifesto. Stakeholders often notice data issues before the data team does. This destroys any confidence they may have in the numbers. Data warehouses are mutable, this is one of the many root causes proposed by Lucas. The past often changes, whether because of code or data. This is metrics drift.
• Data Documentation 101: Why? How? For Whom? — Marie wrote best practices for establishing complete and reliable data documentation. The first advice is about the documentation readers: data team, business users or other stakeholders.
• Change Data Capture (CDC) with PostgreSQL and ClickHouse — This is a nice vendor post about CDC with Kafka as movement layer (using Debezium). The post explains well the architecture you need to make it work.
• A deep dive into graph analytics — Petrica tries and showcases Memgraph in a long-form post. I'm a fond of graph visualisations and analytics—as well as maps.
• Experimenting at Scale, the Spotify Home way — Simple principles to run a good old' experiment at Spotify scale.
• The ultimate SQL guide — After the last canva on data interviews, here's a canva to learn SQL. From databases introduction to SQL writing. It covers simple SELECT and advanced concepts. This is neat.
• The power of pre-commit and SQLFluff —SQL is a query programming language used to retrieve information from data storages, and like any other programming language, you need to enforce checks at all times. This is where you should use pre-commit and SQLFluff.
• Metis: building Airbnb’s next generation data management platform — The new manifesto for every data governance company /S.

PS: I just split the Fast News to have a smaller one. Fast News contains lighter news and broad articles.

Fast News ⚡️

• Stack Overflow developer survey 2023 — Every year SO sends a survey to developers and it gives a great overview of the technology usage across the space. This year ~90k people answered, they also integrate a small AI category to measure impact on dev work.
  
  What we see related to data engineering is mainly: Python and SQL are still shining at the top of technology popularity—around 50% use them. Thanks to AI hype Python is the second most desired technology behind Javascript, which augurs well for the future. They also share salary figures and data engineer / science are well situated in the ecosystem, best-paid job in Germany after management position but less-paid in the US.
• Generating income from open source — Vadim shares how he makes money from all the different open-source projects he has. He shares what works and what does not work. In the post he also shares the journey of Sidekiq founder who's making $10m ARR alone.
• You can put space in BigQuery column names — The editors of blef.fr (me) have no comment. In fact, yes, you are all crazy?
• Malloy's Near Term Roadmap — I've shared recently Malloy demo, which was awesome. The article shares the recent features and says also something I will never forget: "Malloy aims to be syntactically the same no matter what database contains the data".
• The Astro Cloud IDE — Astronomer released a bunch of Airflow operators to their Cloud IDE (which was released in Dec. but I missed it). I get the point why companies wants us to go in their Cloud IDE, but I hate this trend. Let me alone in my PyCharm.
• Cube announcements ; Data Graph and Orchestration API — This is 2 announcement from Cube. I really like following them because they are thoughts leader in the semantic layer space. Data graph create an entity diagram from the semantic definition with the API offers you an endpoint to launch pre-aggregations jobs from your scheduler.

Data Economy 🤖

• Graphext raises $4.6m in seed round (second to continue develop a data analysis platform build for exploration. The Spanish startup develop a tool where you quickly explore datasets and then build charts or AI models on top of it. Last year they build a graph with Data News links, we clearly see the different content categories I share.
• Telmai raises $5.5m seed round. A new data observability platform enters the space, it looks like they propose the same features as the competition: add your datasources, get automated alerts on data drifts.
• At the same time Masthead raises raises $1.3m also as a data observability platform, but done differently. Masthead does not run SQL on your data—which generate costs uplift—but reading logs and metadata to identify anomalies.
• Informatica acquires Privitar. This consolidation will bring new features to Informatica. As a reminder Informatica has been funded in 1993 and is one of the dinosaurs in the ETL space. Privitar will bring "data security" stuff.

See you next week ❤️

Here's a new edition of the Data News newsletter. Since my 2-year anniversary post, I've been struggling to find the right writing rhythm. I've been sick and I've been stuck on a client project. Writing the newsletter was not an easy exercise. Even though I keep telling myself "it's not a question of motivation, it's a question of discipline" like a LinkedIn guy. I do things because I enjoy the process of doing things, not for the results.

That's why I'll try to change a bit the way the things are done for the next 3 months. As of today I do the newsletter every Friday. I search and read articles first and then I write. Starting next week I'll do it on Thursday, to schedule the sending at the same hour every Friday, at 2PM.

This way, I'll dedicate my Fridays to write original articles, explore ideas and preparing articles stock for the summer holidays. I plan to do a 1-month break during August. But at the same time I have the FOMO—fear of missing out. So I need to schedule articles in advance. I can tease you that I'll create content about "Create a data platform in 2023", with live examples.

In September I will do a retro and decide if this is the right way to continue or not.

In term of content I've recorded a new podcast episode (in French) that will be out next week. The French version will be a bit different than Minds of data. It'll be more round tables and discussions about the present and the future of our ecosystem.

We also scheduled the next Paris Airflow Meetup in Mirakl offices. Pierre, an Airflow committer and PMC member, will present his Airflow journey. Join us!

Data contracts, dbt and modeling

Back to the roots, it's been a long time since I did not share dedicated stuff about dbt. This week a natural cluster of articles have emerged. A few people already implemented things with the new model governance dbt introduced last month in v1.5.

Julian shared a nice way to use dbt model governance when you have 1000+ models. In a nutshell you can add new characteristics to models that will give more context to dbt. Models can have group, access, contract and versions. In the article Julian greatly explains the software dev comparison when managing programatic APIs with public or private visibility with models management. Finally he also proposes 6 logical data layers to sort your models: source, base, cleanse, core, business and marts.

This structure gives also more visibility to the team because you can draw clear boundaries like: data engineers are responsible for the 3 first layers, analytics engineers for the others.

In order to go more in depth in the data contracts concepts applied to the warehouse and dbt you can activate ownership with dbt data contracts. Mikkel also showcases his tool synq.io that runs tests and alerts on top of dbt.

In addition there are 2 awesome articles about related topics:

• Simplicity or efficiency: how dbt makes you choose — This is a side-by-side comparison of dbt and SQLMesh, a growing alternative to dbt. The comparison is done using a project with 50 models on 3 aspects: make a change, deploy in dev and deploy in prod. In the end the article is obviously biased towards SQLMesh (on the company blog), but reveals good issues with dbt.
• The data modeling divide — A discussion about different modeling techniques. OBT, star schema, activity schema, etc. and the divide within the community and tools companies for a consensus.

Gen AI 🤖

• Why AI will save the world — Marc Andreessen writes about the prevailing panic and 5 risks associated with AI, asserting that AI will probably do the world more good than harm. Still it has cold war vibes inside 🙃.

The single greatest risk of AI is that China wins global AI dominance and we – the United States and the West – do not.

I propose a simple strategy for what to do about this – in fact, the same strategy President Ronald Reagan used to win the first Cold War with the Soviet Union.

• The golden age of open source in AI is coming to an end — An article about changes in open-source code licenses creating less permissive models.
• Rush to use Generative AI pushes companies to get data in order — Garbage in garbage out. An article from the Wall Street Journal, obviously if you want to fine tune generative models you will have to be sure to have correct training datasets.
• Use ControlNet to generate QR Codes — A Chinese engineer used ControlNet to generate visually appealing and hidden QR Codes. The result is quite impressive and works most of the time.

Fast News ⚡️

• Which team should own data quality? — Wether it's data engineering, analytics engineering or more specialised functions supervised by a central governance this is a good question to have.
• The next chapter for CastorDoc — CastorDoc, previously Castor, is a a data catalog. They recently did a rebrand and Tristan shared the new associated vision. They unveiled 5 pillars to achieve the new vision in which AI-powered insights is the second one.
• Graph components with DuckDB — Max always amaze me with his experiments. This time he writes a graph algorithms in SQL to identify connections.
• Gotchas of streaming pipelines: profiling & performance — Feedbacks on how Lyft team increased performance on their streaming pipelines.
• The growing pains of database architecture — Figma team shared learnings about scaling Postgres instances.
• Backfills in data & machine learning — Backfilling is when you write or overwrite the historical data. Backfilling is one of the most complicated task in data engineering because it often requires design way ahead of problems. Dagster wrote a small guide about considerations you might have when doing backfills.
• Daft: a high-performance distributed dataframe library — Recently Polars took all the attention regarding dataframes manipulation. But this new library called Daft could also be a game changer. Daft is written in Rust, uses Arrow, can be distributed and can use complex types.
• SELECT Insights — A fresh new newsletter by Simon Späti. He shared a long list of links and genially structured the newsletter like a SQL query.

Data Economy 💰

• Cohere announces $270M Series C. Cohere is an OpenAI alternative, they propose an API and a Python, Go or Node SDK to add "language" to your traditional app.

See you next week ❤️.

Hey, I've been sick longer than I expected, but I'm finally well. I hope this email finds you all well, as well. I've had to catch up on almost 3 weeks of content. When I step back, the amount of articles shared each week is insane, there are countless articles about things that have already been written. Sometimes I feel like I'm trying to find a needle in a stack. Or several needles.

I wanted to write more about Microsoft Fabric and the states of data that were published last week but I'll do it another time.

Gen AI 🤖

As always the pace of innovation in this field is incredibly fast so here a few news I've seen I found worth it:

• Japan goes all in: copyright doesn’t apply to AI training — I'm far from being a law expert but it looks like something that will create precedence. The article is saying that it lays down with Japanese new strategy to become a leader in AI technologies, by removing barriers on training data they hope to open doors. Obviously artists (especially mangakas) were not happy about it.
• Sam Altman, OpenAI CEO did an Europe Tour — Sam went to Europe recently (Span, France, Poland, Germany and UL) in order to meet countries representatives. I guess that he did lobbying around the AI Act but also he was here to do real estate because OpenAI wants an European office.
• New Nvidia 144TB GPU — Nvidia is the clear winning of the AI race. They announced an insanely crazy new GPU and Google, Meta and Microsoft are already customers. Surprising.
• How DoorDash uses XcodeGen to eliminate project merge conflicts — Ok now I don't want to resolve a Git conflit anymore 😅 .
• US researchers developed a LLM-powered Minecraft agent: Voyager. Minecraft is a survival game and the agent has been designed to Minecraft learn life skills incrementally. In the end it generates a code that is used to send the agent in the cubic world.
• A new kind of camera— An artist developed an AI camera, the Paragraphica, that is a context-to-image camera. The camera is using location data to feed context to a generative algorithm.

Fast News ⚡️

• Meltano announced their Cloud — Meltano is an open-source data integration project that has been started at Gitlab. With a few configuration and a CLI you can write data pipelines using hundreds of connectors (using Singer spec). The pricing is based on the number of runs and not the volume of data. This is a major difference with the competition (Airbyte, Fivetran, Stitch).
• A ridesharing app simulation — Juraj developed over the last months a complete simulation of a ridesharing app (like Uber), he shared everything he did in blog posts and the results is kinda amazing. I recently spent hours on Mini Motorways so this is the kind of side projects I like.
• Breaking into data engineering as a self-taught developer — A few advice from a fellow data engineer who was data analyst before.
• What's the hype behind DuckDB? — This is a great post from Matt Palmer about DuckDB. If you want a quick intro about the tool this is the way to start. In the article Matt also showcases how you could use DuckDB to write a transfer pipeline like moving a Parquet file from a disk to S3.
• How Instacart Ads modularized data pipelines with Spark — A great deep dive on a Lakehouse architecture for streaming. The article describes a migration from "thousands of complex SQL lines" to composable Spark SQL.
• dbt at Zendesk ; setting foundations for scalability.

Data Economy 💰

• Databricks acquires bit.io — bit.io was "the fastest way to get a Postgres database". In order to start you just had to send data and your database was already setup. When looking at the press release Databricks acquisition is a team acquisition to improve their own developper experience.

Now I go back on Diablo — See you next week ❤️.

Hey, I've been sick in the last 3 days and it was impossible to write something. As I still want to send something, here a raw edition with no comments. See you on Friday.

Gen Ai 🤖

• QLoRA: Efficient Finetuning of Quantized LLMs — 65B parameter model on a single 48GB GPU reaching 99.3% of the performance level of ChatGPT on Vicuna.
• Modding Age of Empires II with a Sprite-Diffuser.
• Github Copilot Chat announcement.

Fast News ⚡️

• clickhouse-local vs DuckDB — DuckDB is not the only one to work great locally. In this test clickhouse works better.
• The Future of Data — Everyone wants a piece of the pie; no one wants to bake.
• Data Modeling, architecture Pattern, tools and the future — part 3 of Simon's guide.
• Microsoft Fabric — everyone was talking about it on LinkedIn. This is the Lakehouse integration for Analytics into Azure. Here are first impressions, how it includes with Power BI and a few remarks. Honestly this looks like a disguised Databricks.
• States of data season — Airbyte's state of data, Databricks's, lakeFS's.
• Engineering Levels: a simple framework for startups.
• Writing design docs for data pipelines.
• Databend and the rise of Data warehouse as a code.

Data Economy 💰

• dbt Labs reduced 15% of their staff. Tristan announced it on the blog and the company provided transition perks. It was a sad announcement.
• Snowflake acquired Neeva — A generative AI search company that was in difficulty got acquired by Snowflake.
• EU hits Meta with record €1.2B privacy fine — under GDPR.
• Elementl (Dagster) Raised $33m — to continue building the data orchestrator.

See you soon. ❤️

👋 Here is a special edition for me. Exactly 2 years ago, I sent out my first email newsletter. At the time, only 3 people received it. I already told the story in Robin's podcast, here is a written version. In 2021, I was doing Twitch lives twice a week, every Wednesday I was doing a data news round-up. One day, I decided to save the links on a blog created for the occasion, a few days later, 3 people subscribed. This is what made me decide to send emails containing my round-up. By chance.

So I want to thank Max, Théodore and Emiel, it is largely thanks to you that this newsletter exists. If you had not joined so early, I would never have realized that people would like to read my content. These bookmarks that I was saving mostly for myself.

Today, 104 editions later, I want to take a look back at my content creation journey, but also at my freelance journey that started one year earlier, in 2020.

The beginning

Before becoming a freelancer, I was working at Kapten, a French PHV company—an Uber competitor—where I was leading the data engineering team. We were a team of 6 people and our goal was to build the data platform for the company. During my time at Kapten, we built a data stack with Airflow, BigQuery and Metabase + Tableau. I was coming from the Hadoop world and BigQuery was a breath of fresh air. The component I'm most proud of is the ELT framework we built on top of Airflow to give total autonomy to analysts and scientists on the data loading and transformation processes.

In a nutshell it was an ETL-as-configuration on top of Airflow. You were able to define configs in Python to do full or incremental loading from different sources, processing in SQL or Python and exports. The framework and the processes were pretty strict, but it worked and gave full autonomy to analysts in building whatever they wanted to do. All the ownership was given back to others, we were just writing software and maintaining a platform.

I think it took almost a year to build the entire platform. We had set a goal: no broken Airflow pipelines in a 30-day sliding window. We achieved that. And we hit a plateau. We were doing less data engineering because everything was working well, less firefighting, looking for a new vision. As human beings, we wanted to fill the void, so we explored different things: real-time feature store, data lineage or data contracts—we call it this way today, but back in the days it was only schema management. But what was the next step for us?

I had done what I was hired to do: build a data platform for analytics and analysts. It was time for me to leave, at the same time the context changed: we got acquired and laid off. That's where my freelance journey started.

Going into freelance

I left the when the COVID was at a peak and a few people did not understand the move. To be honest I didn't even know where I was going but I was confident in my skillset and in my ability to sell my data engineering expertise. In retrospect I was just naive.

The Kapten experience brought me expertise on Airflow and GCP, a good knowledge about Kubernetes and a lead experience. In addition to my solid engineering and infra skills it creates a good resume.

By chance 2 of my former bosses heard about my freelancing and proposed me work. It led respectively to a 3-months and a 1-year mission with Equancy and Qonto. Then I did a mission with Yousign with Faouz that I met a few years earlier thanks to a LinkedIn message. The common point of the 3 missions was to build stuff around Airflow. In a blink my first company fiscal year was already done, with around €180k in revenue.

While I was at Qonto, we migrated to dbt, which was rapidly being adopted by French startups. This allowed me to add a new tool to my belt. Then it became a new expertise.

In my second fiscal year (2022), I had the privilege of working with the French tax authority to help them define the vision for the 2027 data platform and with the Ministry of Education to implement Superset and dashboards on that platform. In the blink of an eye, my second year was already over with less revenue (160k€) but in less time.

Along the way I also helped startups hiring—Folk, Modjo, Kard—and did mentoring—Blent.ai, Libeo, iBanFirst, nibble. I even hired 2 awesome interns who helped me on the blog for a few months. As 2023 is still running I'll keep it for another retrospective.

While my story is exciting, here are a few things to learn from it:

• Former co-workers are part of your network and are probably the ones who vouch for you the most.
• In terms of networking, participate in events, give to the community and you will receive something at some point. Don't be afraid to solicit people on LinkedIn, people respond more often than you'd think.
• Find the main reason why you want to freelance. It can be many things like money, freedom, issues with authority, digital nomadism, etc.
• If it's money I think you gonna miss the freedom part of being freelance. If you want to do a lot of ca$h you will work every day, in a long-term mission for a big company. Which is actually like a permanent position without the perks of it (at least in countries where we have a social system).
• Set your daily rate and (try to) stick to it. I started at €800, then went up to €1000 and now I'm at €1200. Don't forget that you are competing with agencies, often charging high prices.
• One of my strict conditions is to work only part-time. In fact, I work an average of 2.5 days a week. To be successful, you have to be organized and be aware of context switching. To be honest, this is very difficult and I am bad at it.
• In my opinion, to freelance in data engineering, you need at least two or three proven experiences in data engineering. Very often, as a freelancer, you are perceived as someone who knows things. To be assertive, you'll need to be confident in your recommendations.
• Identify your strengths and communicate clearly about it. Here how I say it: I'm a data engineer who built a lot of data platform for analytics, I have an expertise in Airflow, dbt, Superset and in infrastructure.

Juggling with content creation

Doing freelance data engineering is a great thing for me, I've been working with computers since I was young. It's always better when passion meets your work. Alongside this, I also started creating content in January 2021. This was one of my goals when I decided to go part-time so I could have time for content creation.

I did not set clear business objectives over my content creation. Finally I did an engineering school, not a business school. It's probably the reason why I often loose focus and do multiple things. Here is a small selection of what I tried:

• Twitch — I've done 4 months of Twitch at the beginning, but my 2-months holidays with no internet broke my routine. I don't think I'll go back to solo lives.
• I made YouTube videos — I have 7 videos, each video took me about 20 hours, hard to add it into my daily routine but it will come back.
• Twitter — even if went from 200 followers to 400 followers to Twitter I can't find my voice there. This is sad because Twitter is the social network I consume the most.
• LinkedIn — I tried multiple things on LinkedIn but I don't have the discipline to publish one post a day. In the end I went from 2000 followers to 6000+ in 2 years.
• Podcasts — the new thing I've recently started. Once again I loose focus, but, the podcast format is so statisfying to do.

And finally, the newsletter, which is my safe place. I've found discipline in writing my own content with my own tone. It takes me about a day of work per week. Basically, I spend 2 hours selecting content, 1 hour reading the content, 2 hours writing, and 1 hour post-processing. In the end, I'm proud of the quality of the newsletter, but one day is a lot and after 3 years, I have to wonder which direction to go in.

Actually, I don't care I'll continue like this. But why do I do content:

• I like to share / transmit to others, when I was a kid I wanted to be a maths teacher.
• It creates visibility for me and as a freelance I need to be visible.
• It helps me shape my ideas.
• I love the adrenaline rush I get when I do things publicly. Even though there are serious downsides to it, like FOMO or addiction, I love it.
• I hope that in the long run it will generate enough money for me to do less consulting. Blog subscriptions bring me 300 € / month. Which is less than 2% of my revenue 🫠.

Conclusion

This is a post that is more personnal than what I usually do. This time I did not promise anything like I already did in the past. Promises I didn't keep because I'm lazy. At least I learn from my mistakes.

Whether you are a customer, a friend or a subscriber thank you very much for your support over the past 3 years. Let's continue for another 3 years? ❤️

Data Council Austin is a yearly conference that features a great panel of speakers giving talks about the future of the data field. As I often do I've overlooked the 70 presentations and here a medley of what I've liked.

My personal selection

If you had only 3 videos to watch it should be the 3 following:

• Malloy an experimental language — This is my favourite talk. Llyod, founder of Looker, puts 30 years of data warehousing into perspective in 30 minutes, especially the fact that we see "data in rectangles." Since joining Google, he's been working on Malloy, a new way to query data. Malloy compiles in SQL and works on data semantics. The presentation gives another look at the semantic layer. During the demo, Llyod does some data analysis in the browser and it's just mind-blowing 🤯.
  
  At the same time someone Google also did a Calcite presentation.
• Data contracts, Accountable data quality — Data contracts is a trendy concepts that contains a lot of things. Chad Sanderson did the best recap about it. DE is often constant firefighting, a lot of (spaghetti) SQL to maintain. A lot of breaking changes are coming from upstream producers (form or content).
  
  At scale everything breaks without data quality, the modern data stack is good because self-service and easy to implement but lacks of everything to be mature in the future: ownership, data quality, context. It creates a non-consensual API, we pull data but never agreed on a contract (SLA, schema, etc.).
  
  The root cause is mainly because of miscommunication between producers and consumers. Data contracts aims to fix with API-based agreements between producers and consumers that capture the schema, semantics, distributions and enforcement policies of the data.
  
  You can also watch Whatnot data contracts implementation.
• Metric trees — It reminds my KPIs framework people were doing when I started to work in a consultancy firm. This is nice way to represent your company business. Still today 90% of the value a data team delivers is in the analytics. The analytics goal is to model correctly business. You should answer 4 questions: what happened, why did it happened, what's going to happen, what should we do next.
  
  Organisations are systems with inputs and outputs and a formula. Formulas have metrics, relationships and weights. In the end you can depicts all your KPIs with formulas.
  
  The data team strategy should be mainly to define and operationalise the company growth model. Using a metric tree as a logical representation of a growth model. You have 3 types of outputs: customer value, financial and strategic.

Other stuff I liked

• Snowflake optimisation guide — This is a pragmatic guide on how you can lower your Snowflake costs. In the current context we have to do more with less. The talk starts with a great introduction of Snowflake architecture. In a nutshell the speakers share tips about warehouses sizing and design, performance optimisation with pruning, clustering and query design.
• LLMs and Semantic layer —This is something I've in mind for a few time. This is a tool presentation but still it's relevant. On the same topic of self-service Whatnot shared how they turned data consumers in data constructors.
• Scaling Uber metrics systems (w/ Pinot) — uMetric migration from ES to Pinot. They created an unified layer where metrics uses the same logic for downstream consumers. uMetric manages definition, discovery, computation, verification and serving.
• Writing unit test for data science — Pragmatic guide about unit tests.
• Retro on data science by DJ Patil — DJ Patil has been US Chief Data Scientist. He coined the "data scientist" term back in 2008. He does a great retro.
• Dashboards as code — Using code to make BI dev better, this is DataOps, we have almost X as code in the whole data chain, only dashboards lacks of it.
• Growing the data Team and data Culture at GitLab — GitLab data playbook is well-known. The eng - director gap problem. This is when you have a director that manages an individual contributor.
• A deep-dive into the dbt manifest — How to do a dry-run in cloud data warehouse, load the manifest as dynamic dags, enforce polices or build monitoring.
• Augmenting the modern data stack — by merging batch and real-time technologies in one database.

See you soon ❤️.

Hey you, new Friday means Data News. This week is pretty stacked in term of content, especially video / audio content. I hope you will enjoy it as much as me.

Let's start with with my newly created podcast Minds of Data. In Minds of Data I'll met people from the data ecosystem in order to learn more about them. In the first episode I sad down with Joe Reis and we discussed about his professional journey before becoming the thought leader he is today, we also chatted about data engineering. You can listen the episode on Spotify, Apple Podcast and Deezer.

PS: this is my first episode ever so feedbacks are more than welcome.

As the same time in Paris we organised last Tuesday the May Airflow meetup. We had 3 talks, that you can find on YouTube. I really liked Benoit and Samy presentation about Cloud Composer—Managed Airflow on GCP. They shared good practices on how to manage Composer in the cloud, things like:

• Use the same configuration for staging and prod
• Use a secret manager to manage your Airflow connections
• Use IAM restrictions in the DAGs bucket
• Use operators and define the company policy around it
• Define clear policies to govern your Airflow

Also Airflow 2.6 went out this week with a new trigger DAG parameterizable UI, new alert notifications framework (callbacks) and a new graph interface in the grid view.

Gen AI 🤖

The pace of innovation and announcement in the (Gen) AI field doesn't deflate. I can't really cover the whole field because it moves so fast that I can't even keep up. This week the Google I/O Keynote was a major milestone.

Google I/O Keynote takeaways

What amazed me from the Google Keynote is the fact that Generative AI is treated like a product, like the 2007 iPhone—look at this ad. When you think about it AI has always been something hidden, like an API call, a score or a recommendation in a larger UI. In Google's Keynote AI gets a 26 minutes segment and then all the derivations lasting for 2h.

To me Google annual conference is a sign that the party is over, especially for OpenAI. Actually OpenAI deal with Microsoft was probably the best deal they could have go for. Even if as human we want to send models in the arena to get the most performant one, or masturbate ourselves comparing the size of parameters. In the end the best integrated models will win. And Google as a head start—as well as Microsoft, as they remind us in the Keynote they have 15 products used by billions of people: they have our e-mails, our photos, our maps and more. AI is a just a feature in their product, even if it needs an UI rethink, this is just a feature.

So in the end Google, an AI-first company from the beginning wants to put AI everywhere and wants to offer you an AI collaborator. Here are the major takeaways from the Keynote:

• They release PaLM 2, the last foundation model. It will exists in 4 sizes: Gecko, Otter, Bison and Unicorn each asking for different hardware resources to work.
• PaLM 2 will be natively integrated in Google products. Gmail will get enhance smart reply features, Maps will propose immersive view over a route and Photos will have a magic editor that will allow you in a single drag-n-drop to edit a picture.
• Google will create a sidekick that will be available in Workspace—Sheets, Docs and Slides—called Duet AI, you'll be able to ask the AI to create content for you unlocking productivity gains. Duet AI will also work in GCP (in the console and within the web IDE).
• According to the announcement PaLM 2 will particularly shine when fine-tuned (e.g. for IT security or medicine). You'll be able to do it by yourself within your own GCP instance in Vertex AI. They also released Imagen, Codey and Chirp resp. for image generation, code generation and speech-to-text.
• Bard, the conversational model—ChatGPT equivalent—is now opened to everyone (actually not in all countries). Bard works great for code generation, debugging and code explainability.
• Bard might also be the Zero-ETL solution we were all waiting for. In the demo the speaker asks Bard to find schools in an area, then asks for it to be saved in a Google Sheets, then asks to for a new column in the sheet if the school is public or private. To be honest, what prevents Bard in the future to do the same in a database?
• Finally Google tease their next-gen model Gemini which obviously will be awesome, to hear them and announce an evolution of the search interface will Gen AI as a new interactive way to search.

In the end I really like the keynote because it gives a new milestone about what we can expect as integration in the products we daily use.

Other stuff

• Hugging Face released an open model called StarCoder that has been trained on Github code that is meant to act as a Copilot. Still the model is not yet ready to be used as an instruction model—ChatGPT way.
• At the same time HF also introduced an open-source Chat UI.
• After Bill Gates, it Steve Wozniak—Apple co-founder—who gives his take on the AI breakthroughs in a BBC interview mainly we can't stop the march of progress, AI will be used to scam people and we have still to put guardrails, but human guardrails.
• Salesforce do not want to be leftover in the battle, they announced Slack GPT natively integrated in Slack to summarise or compose messages but also a way for partners to bring new kind of Gen AI apps.
• Also Salesforce did a makeup to Tableau with Tableau GPT, a way to provide AI-powered analytics. In Tableau Pulse you'll have access to auto-generated insights on your data. With a "For You" tab like you were in TikTok.

Fast News ⚡️

• Zero ELT could be the death of the modern data stack — Amazon launched this trend a few months ago. In the current situation we're far from killing any ELT processes, but it might come. For instance Zapier launched Zapier Tables some kind of data storage within your zaps.
• We need to talk about Excel — Let's be honest, as strong we try to kill Excel as strong he comes back. David shares interesting stories around Excel usage at companies that I can relate to. He finally mentions Count and Equals, two companies, that builds on top of tabular interfaces to do data.
• Determine BigQuery storage costs across an org — A SQL query that I did not tried. Please read it twice before running it blindly.
• Polars, laziness and SQL context — Daniel showcases the 2 features which should make you want to migrate to Polars.
• Building the seller analytics dashboard — An great example of what you should consider when building an analytics dashboard in the product and how you combine dbt and GraphQL APIs to build a pragmatic metrics store.
• OLTP vs. OLAP — One of the best explanation of the differences between both. Mainly it resides in the data storage. One being row-oriented while the other one is column-oriented, this is not the only difference.
• Correctly loading incremental data at scale & real-time denormalized data streaming platform.
• ExternalTaskSensor in Apache Airflow: how to calculate execution delta — I've seen multiple time that the delta computation was annoying for data engineering teams. This article deep-dives well on it.
• Upscaling LinkedIn's profile datastore while reducing costs — For optimisation geeks.

See you in a few days with Data Council takeaways ❤️.

Hey you, this is a Saturday edition of the Data News. I hope this email finds you well. This week you'll have less editorial content because I'm late. But still you'll find awesome articles that have been written recently.

As a reminder on Tuesday next week I'm organising the Apache Airflow Paris meetup that you should consider joining if in Paris. Also next week I'll publish my first podcast episode ever that I've recorded with Joe Reis—the co-author of the famous Fundamental of Data Engineering. I'm still looking for the name of the podcast, if you have ideas shoot.

Gen AI 🤖

• Google "We have no moat, and neither does OpenAI" — This is an internal note from a Google employee (which does not reflect Google views), that mainly says that open-source models will win over Google and OpenAI and closed-source policy for models might be a mistake especially in a world where some models leaks (e.g. Meta ones).
• If you already have access to OpenAI in Azure you can now use GPT-4—only in preview yet.

And more traditional AI:

• YOLO-NAS a new object detection model — you should have seen already this model that detects people in real-time in videos. This new one seems to be better than the previous one.
• Mojo, a new programming language ready for the AI — Mojo is a new programming language that looks like Python but at a lower level, this could unlock performance gains and new heights in AI models development.
• eBay’s blazingly fast billion-scale vector similarity engine.

Fast News ⚡️

• Paypal, template for data contract — PayPal is implementing a Data Mesh and they provided in the open all their thoughts with data contracts. In the Github repo they are sharing a YAML template describing what's in the contract. This is insanely exhaustive.
• Even Amazon can't make sense of serverless or microservices — PrimeVideo tech team wrote an article that could be summarised by: we migrated from functions based approach to a monolith in a VM. Internet found this ironical. By doing this they reduced cost by 90%.
• Lakehouse at Walmart — Samuel from Walmart describe the research they did and why they picked Hudi over Delta in order to implement a Lakehouse architecture. As a reminder the Lakehouse is the merge of the datalake and the data warehouse, which is mainly a way to add a SQL friendly processing engine on top of a datalake with ACID transactions.
• Safer deployment of streaming applications — This is how Grab deploy Flink applications.
• Why you should reconsider Debezium: challenges and alternatives — Warning: this article has been written by a CDC solution, but still this is relevant because it shows what is the reality of managing Debezium.
• Dataform: schedule daily updates using Cloud Functions — Dataform is a solution Google bought a few years ago that is a dbt alternative but for BigQuery. This article gives a great overview of the product. To be honest it looks like a bit hacky.
• 📺 Dev Deletes Entire Production Database, Chaos Ensues — If you want a greatly told story you should watch it, this is a YouTube video explaining how Gitlab remove the production database and how they fixed it. It reminds me my own horror story of deleting the whole /data folder in HDFS.
• Oracle is taking on Snowflake — I often say the Snowflake will become the new Oracle. This is fun to see that Oracle still try to catch up. They come up with a lot of news: they will implement Delta Sharing protocol, lower the storage for $118/TB to $25, partner with AWS and propose low-code data integration tool.
• Data modeling, again — Simon published the second part of his data modeling guide, this time he covered the different techniques you can use when modeling data: dimensional, vault, anchor and more. You might also want to see practical examples of data modeling, Sonny wrote a nice article using a hotel business as a support.
• Crafting your data team — Practical tips on how to get started with your data team in a new startup. In the post Marc gives you the qualities you should look for and what hiring you should prio first.
• 🎮 The CS:GO Liquid team announced a new data analyst, DeMars did work previously on a predictive analytics approach on Valorant trying to predict who whould win a round in different situations. This is fun to see our beloved data analyst position reaching other fields.
• Data projects on personal data — Petrica dive into her Medium data with DuckDB and Plotly and Stefen analysed his Uber spends with dbt and Postgres. As a reminder, doing personal data projects is still the best way to learn about technical stuff.

Data Economy 💰

• AuraML, an Indian-based company, raises $230k in pre-seed round. AuraML is a 3D synthetic data company, their engine is capable to generate 3D realistic-looking environnements your might want to use in other models.
• Mistral AI, a French Gen AI company, will probably raise €100m (link in French) in the following weeks. It looks like at the moment the company only hired a few French people that were working previously at Meta or Alphabet on LLaMa or DeepMind. The goal of the company is to provide the first French—hence European—alternative to OpenAI. Obviously this is heavily political and strategic for Europe so we will follow it in the next weeks.
• Anaconda is expanding and buying EduBlocks. EduBlocks is a scratch-like platform to write Python or HTML code. This is a cool thing in order to continue code democratisation.
• Open-source done differently. Sequoia—a VC—will support Sebastián Ramírez with an open-source fellowship. Sebastián is the creator of FastAPI, SQLModel and Typer. There isn't more detail in the press release but this is awesome to see.

See you next week ❤️.

Hey you, new edition of the newsletter. This week summer time arrived in Berlin and it was awesome. I managed to move forward with my client projects this week and it also feels relieving. So I'm pretty happy, sun and great projects 🙂.

Regarding the content, if you are in Paris on May 9th, we are organising the Paris Airflow Meetup in Algolia offices, it will be in English so you don't have any excuses not to come. Also I'll be a lot in Paris in May so if you want to have a 🍜 / 🍺 ping me.

What happened to the Semantic Layer? 🫠

This week dbt Labs disclose their vision about the semantic layer and especially what they want to do with the Transform acquisition. This is mainly a roadmap of the MeticFlow integration within dbt ecosystem. At the moment we have a dbt Semantic Layer that correspond to YAML definitions and MetricFlow—which was Transform open-source project—that is able to understand the semantics to generates SQL.

A lot of changes will happen to MetricFlow incl. breaking changes:

• the dbt metrics spec will change, in the current state actually not a lot of people were using it, dbt_metrics package will be deprecated, probably they will merge dbt and MetricFlow syntax to define semantics and metrics
• "The core MetricFlow package will become a stand-alone library for processing metric queries, generating a query plan, and rendering SQL against a target dialect." (cf. Github discussion)
• The license will change to BSL.
• The serving part of the system aka. the metrics store will be the paid service of dbt Labs and a part of the dbt Cloud offering. It means that you will define metrics and dimensions in YAML and then plug all you tools to dbt Cloud, it seems there isn't any open-source solution to do the serving—at least from dbt Labs side. And with the license change on MetricFlow dbt Labs are protecting themselves against someone using MetricFlow generation to propose such a paid service.
• There are more described in the Github discussion.

To add more spice to this Carlin wrote what happened to the Semantic Layer. Carlin works at Google in the Malloy team (Google semantic layer to say it fast—tbh it's probably more) and he gives his views and also a small retrospective on the semantic layers.

Gen AI 🤖

• DoorDash identifies Five big areas for using Generative AI — Doordash is a food delivery platform and they shared how they imagine Generative AI can help them in the future. Either by assisting humans, it can be customers (cart building, etc.) or employees (SQL writing or document drafting) ; either by improving actual AI stuff: search, discovery, information extraction.
• When it comes to SQL writing the field is on fire, a lot of companies are trying to rise from the dead the Slack chatbots answering to insights. I think of Shape (YCombinator, out of stealth this week) and Delphi Labs or Promptimize. Promptimize is a toolkit to evaluate and tests prompts, for instance you can "unit tests" you natural langage to SQL prompts with it—it has been open-source by Maxime Beauchemin (Airflow and Superset creator).
• Bard now helps you code — Google is finally going the Copilot way and proposes an alternative with Bard. Bard now can help you write code or Google Sheets functions, but it can do more by explaining or debugging code for you.
• 📺 The Inside Story of ChatGPT’s astonishing potential — A TED talk from OpenAI President and co-founder sharing his vision, the potentials and the limit of the technology. In the video you can feel Steve Jobs's 2007 iPhone keynote vibes. The video also greatly showcase ChatGPT plugins. I higly recommend you watching it.

Last but not least a more "traditional" AI category:

• End-to-end ML modeling in BigQuery — BigQuery added over the last years a lot of ML capabilities to the engine. This post showcases a lot of it (it uses a XGBoost model).
• Building a large scale unsupervised model anomaly detection system (part 2).
  

Fast News ⚡️

• From PostgreSQL to Snowflake: A data migration story — The migration lasted 9 months and included 8 steps. They went on this journey because in 2021 Postgres was already hitting read performance limits, degrading the downstream user experience in the BI tools. As Katia shares in the article a 9-months migration is a long tunnel where you encounter a lot of roadblocks and frustration but in the end everyone feels the difference: 10x performance gain—at least—on dashboard execution time.
• Building dbt CI/CD at scale — Every week a new great article about someone else dbt setup where you discover things. This time Damian shares how he designed checkout.com CI/CD pipelines—in Github. In a nutshell they get the actual production manifest, run a SQL Linter, validates models changes (by detecting what are the altered models and running them) and deploy to Airflow.
• Making the Most of Airflow — I already shared Matt's article last week and this week he continues with an awesome article about Airflow. In the article he gives a great overview of Airflow main concepts: DAGs and TaskFlow API (I've also wrote something about dynamic DAGs last year), DRY and what to do to not redevelop stuff and how to test.
• Building a Kimball dimensional model with dbt — Jonathan from Canva wrote a large article about dimensional modeling and how to do it with dbt. This is a 7-parts tutorial that shows you how to create fact and dimensions tables.
• Data engineering design principles you should follow — It treats mainly of software engineering principles like SOLID. Idempotence and determinacy are forgotten from the article and if you want to go deeper on the topic you can read the most important article on this topic: functional data engineering.
• Real-time denormalized data streaming platform part 1 and part 2 — Razorpay data team describes how and why they needed to move their ETL process from daily to near real-time. Technologically moving from Airflow batches to Spark running on top of Kafka.
• Toward declarative data orchestration with Kestra — A few weeks ago in the Airflow alternatives meetup we organised, we invited Kestra. A YAML-based orchestrator written on the JVM. Recently Benoit joined Kestra as their PO. In this article he shares his vision. It's mainly a question of vocabulary and reach, Kestra believes that with their own declarative YAML syntax they can offer data pipelines to the masses. YAML is enough simple for your analysts (they already do dbt) or business to write their own pipelines.
• Manage database schemas with Terraform in plain SQL — Atlas is an open-source schema management tool. The post showcases the atlas provider in Terraform that allows you to write SQL to manage your database in Terraform. I can't wait to see dbt reimplemented in Terraform.
• Automatically detecting breaking changes in SQL queries — When you alter a SQL query you can either do a breaking or a non-breaking change. What if with SQLglot you could detect a breaking change before it happens in production?

See you next week ❤️

Dear readers, I hope you're doing good. We are close to the second anniversary of the newsletter. Which is crazy. Retrospectively it means that I've written 900 words on average every week for the last 102 weeks. When you look at the first edition we came a long way—lmao.

We announced this week the May Paris Apache Airflow meetup. It will take place in Algolia offices, the 9th of May. We will have 3 speakers and for the first time all the presentations we will held in English. So if you're in Paris or in France do not hesitate to register.

Analytics engineering future

This week Tristan Handy—dbt Labs CEO—wrote a post about the future of analytics engineering: The next big step forwards for analytics engineering. As introduction Tristan gives the original vision of dbt that became mainstream, today. A lot of data teams embraced dbt, or at least the SQL with engineering practices to transform data in cloud data warehouses.

The content of the post is more about the future and the vision of the next big thing in analytics engineering: new models capabilities. In dbt Core 1.5 we will be able to define:

• Contracts — you will be able to define columns types and constraints and ask dbt to enforce it. If a model do not respect a contract it will not build. In dbt vocabulary build means run + other things.
• Access — you will be able to namespace models with groups and visibility. Models visibility will be either private, protected or public. This is a preambule to cross-project dependencies I guess.
• Versions — you will be able to define versions for models without breaking the downstream consumers. In order to do it you will have multiple SQL files suffixed with the version—_v<version> . To select a specific version you will have to do {{ ref('model_name', version=1) }} .

I think that these improvements are really important to bring analytics engineering to the next level, this is new capabilities that will bring the field new software engineering practices to data assets management. If we had to this the semantic layer new (through dbt Labs acquisition of Transform) we are going in the right direction.

Gen AI 🤖

• If you want to understand LLMs there is a note that has been written by an experts office of the French gov. You can read it in French or in English. To be honest this is a great quality note that you can share to people who wants to understand what are all the AI concepts. Might still be a bit too technical to share it to your parents.
• ChatClimate — This is a chat that has been trained with the last IPCC report (the GIEC for the French audience). He showcases well the search capabilities of ChatGPT-based system because every answer is completed with references to the report chapters.
• How to train your own Large Language Models — Now that you tried the previous chat, let's say that you want to run your own LLM. Replit team wrote a great overview of what you have to do.
• Building a ChatGPT Plugin for Medium.

Fast News ⚡️

• Building a Flink self-serve platform on Kubernetes at scale — Instacart engineering team migrated from Flink on EMR to Flink on Kubernetes. This article gives you an overview of the Kubernetes platform they implemented.
• fal-ai/isolate — Yet another package manager in Python. fal developed a new lightweight package manager to isolate environments for at function level. The project README is not yet really explicit.
• Data Engineering at Adyen — "Data engineers at Adyen are responsible for creating high-quality, scalable, reusable and insightful datasets out of large volumes of raw data". This is a good definition of one of the possible responsibilities of DE. This is a great article and they even included a flowchart to identify which role will suit you the most. It is interesting to read this post jointly with the future of data engineer at Meta. Which gives another perspective, which is very business oriented.
• Announcing dbterra: easily sync your jobs with dbt Cloud™️ — Eric developed a tool called dbterra that mixes dbt and Terraform in order to deploy open-source dbt project to dbt Cloud with configuration as code.
• Test Driven Development for SQL — A smal article that gives you a vanilla BigQuery framework with CTE to write unit tests. I think it has to be improve but it gives a greate boilerplate.
• Saving 💵 With BigQuery & dbt — A few tips to save money when using dbt and BigQuery. Mainly it says that you should consider switching your models to incremental.

Data Economy 💰

• Betterdata raises $1.65m seed round. A Singaporean company that provides a tool that generates synthetic data. Synthetic data are AI generated data. In Betterdata case you can use your own datasets and generate data that keep all the statistical metrics needed to do machine learning. This way you can work on data that is similar to yours but different. It's a technique to work with anonymised data.
• CoreDB raises $6.5m seed round. CoreDB is a managed Postgres service that put the emphase on the extensions in order to add more capabilities to your database cluster. CoreDB has been funded by the ex-CEO-CTO of Astronomer.
• A lot of companies announced recently layoffs, sadly. The biggest one being Meta with a new round of 4k people laying off 21000 people since last November. Astronomer as also let 100 people go recently, if you are heavily relying on Airflow it might be interesting to reach people out.
• Elon Musk, according to reports, founded a new AI company called X.AI Corp.

See you next week ❤️.

Hey you, the newsletter might be late today again, but this time this is not my fault. Ghost editor was down when I wanted to write. Anyway, here the weekly Data News, written faster than usual.

AI News 🤖

Yann le Cun did a 10 minutes interview at a major French radio. If you want to read the French transcript you can do it here. Mainly what he says:

• There is no doubt in the fact that one day there will be machines at least as intelligent as human. But ChatGPT isn't, it gives the impression but it is not.
• AI can amplify human intelligence like machines amplify human strength.
• Technology shift jobs. For instance before industrial revolution major part of the French population was working in fields, now it's less than 2%. It means we shouldn't be afraid of technology replacing jobs. He thinks that this will also allow more people to be creative.
• Regarding fake news and ethics he compares to e-mails. He thinks that like when we develop spamming filters to avoid fake mails we will develop the same to avoid fake news.
• For Yann ChatGPT has nothing revolutionary, but he admits it's good engineering. This is just a normal evolution of deep learning systems.
• (Last one because it's funny). He bets than in 10-15 years (or more) we will not have smartphones anymore but augmented reality glasses. We will also use voice to interact with machines, so we can interract with them hands in the pockets—I can't wait to use Siri and Alexa 2.0.

As a side project, if you want to practice machine learning this weekend you can replicate Rihab's project: detect wildfire smoke with YOLOv8 model.

Fast News ⚡️

• A tour of Airbyte’s Octavia CLI — Airbyte, an open-source extract-load platform, released a few months ago a CLI called Octavia that let’s you create integration pipelines. Jeremy wrote a post that showcases how to do it.
• Hot takes on the Modern Data Stack — Matt gives 5 hot takes about the MDS. I don’t totally agree with everything but this is a good read. He says that Redshift is not anymore competing in the warehousing space, which I agree with. He also says that Airflow is obsolete, I disagree, it became common recently to say bad things about Airflow. But as always the issue is between the chair and the keyboard. He is also hard with Airbyte and dbt.
• The new philosophers — It's been a long time since I've shared Benn's posts. Still my favorites. Saying smart things, weeks after weeks. This time he writes about the new marketing approach of the modern data stack ecosystem. Plenty of tools, so let's develop a new tools to avoid the other tools. And add his views about ChatGPT disruption: "We'll initially try to insert LLMs into the game we're currently playing [...]. Our data models won’t be augmented by LLMs; they’ll be built for LLMs". Probably no-one knows, yet, what it means.

• Castor announced Castor AI — Doing it the other way Castor released a feature that explains a SQL query in natural language. This is a good way to help business users understand what's happening in the transformation layer.
• How we made our reporting engine 17x faster — Teads engineering team explain how they significantly speed up their ads report generation. In a nutshell they replaced Spark (EMR) in-memory transformations by BigQuery.
• Large-Scale generation of ML podcast previews at Spotify with Google Dataflow — It became a common issue at vaste content platforms to generate previews to support the scale. This time Spotify explains how they did it with Apache Beam. As an input they take audio and transcript data and they generate podcast previews that will appear in your feed.
• Big savings on Big Data — This is the current trend, with the current economic situation we have to do more with less (or at least with what we have). At Lyft they optimised their ML platform to save time and money on workloads. Especially they lowered all the dev costs.
• LocalStack: Why local development for cloud workloads makes sense — It does the glue with the previous bullet point. This time Corey writes about LocalStack, a tool that emulates locally AWS APIs. The emulation could be the future mainly because it avoids increasing cloud costs for development.
• Using DuckDB with Polars — A nice showcase of the 2 new kids on the block working together. Mainly what you will do is querying in SQL with DuckDB Polars dataframes.
• Using Metrics Layer to standardize and scale experimentation at DoorDash — A very good exhaustive article about a metrics layer. At DoorDash a lot of teams are doing experimentation and they were in need of a common ground between metrics definition. That’s why they built this system. Mainly they define measures, dimensions and metrics in YAML that will be materialised and made accessible to Curie (their experimentation platform).

Data Economy 💰

• Cybersyn raises $62.9m Series A. Cybersyn is a data-as-a-service platform that provides public datasets for everyone. You can see it as a datasets marketplace of common public data. They are heavily supported by Snowflake so the dataset are accessible in Snowflake marketplace. For instance you can freely query the US Addresses dataset to get all the addresses in a zipcode.
• Rupert raises $8m in funding. Rupert wants to fill the gap between the data analyst and the business users by providing a no-code UI to create data alerts on top of your semantic layer.

Hey you, if I wasn't late in my newsletter writing it wouldn't be me. But here is your usual Data News. The main reason behind this delay is because I've played with LLMs yesterday. I've tried to run open-source models locally on my own laptop. There are still a few bugs and the results are not really at OpenAI level but this is fun to do.

This Tuesday we hosted the second part of the Airflow alternatives meetup with Prefect and Dagster. You can find the replay on YouTube.

Data modeling

Dear readers, I have to confess something. I did not care about data modeling for years. I mean, in the sense everyone understand it today, for 7 professional years I never did a star schema or something similar. I was in the Hadoop world and all I was doing was denormalisation. Denormalisation everywhere. The only normalisation I did was back at the engineering school while learning SQL with Normal Forms.

Actually what I cared was physical storage, data formats, logical partitioning or indexing.

But, actually, it's normal my role was not to translate business in tables. I still firmly believe that this is not the role of a data engineer. A data engineer should still be a software engineer working with data, empowering others with tooling and apps. Data modeling should not be a required data engineer skill. Enters the analytics engineer.

Still I feel that there is a hole in my skillset because I can't give relevant advices when it comes to model business with 3 facts tables instead of 5. And to be honest there isn't any good modern literature to answer this question. Simon started a multipart guide about data modeling. I hope he will fill the gaps. In the first part he treats about the history of modeling and the main concepts.

At the same time Maxime Beauchemin wrote a post about Entity-Centric data modeling. In comparison to the dimensional modeling it uses entities instead of facts. Which is easier to conceptually understand but also to use in machine learning.

When it comes to modeling it's hard not to mention dbt. In the recent years dbt simplified and revolutionised the tooling to create data models. dbt, as of today, is the leading framework. But alternatives are coming. This week I discovered SQLMesh, a all-in-one data pipelines tool. SQLMesh lets you define models like dbt but avoids you the burden of the Jinja ref/sources macros. Under the hood it uses sqlglot the SQL parser that has been developper by the same developper. It seems there is also a scheduler and a web UI included in the open-source version.

Gen AI 🤖

• It seems that Samsung employees leaked data to ChatGPT — Unsurprisingly OpenAI saves all the prompts we type (🫠) and can eventually improve models incrementally. It seems that Samsung employees gave confidential information to ChatGPT. Which means that OpenAI owns Samsung data. But is it really different than what we already have with Gmail or AWS? Or like when Tesla employees where watching consumers in-car footage for years.
• Italy decided to ban ChatGPT — In order to do it Italian Data Protection Watchdog ordered OpenAI to temporarily ceases processing Italian users' data. France and Germany might follow.
• OpenAI: Our approach to AI safety — 4 axes in which OpenAI wants to invest: improve safeguards, protect children, respect privacy and improve factual accuracy.
• Eight things to know about Large Language Models — A PDF that will give me a headache.
• On the practical side I've tried to run locally on my M1 Mac a LLM for the first time and it was a fun ride. In a nutshell I wanted to first run Vicuna an open-source chatbot that has great results when compared to GPT3.5. In order to run Vicuna (or other similar open-source models) you need to get the weights from the LLaMA Meta foundation 65B params model. You can get the model either by waiting after completing a Google form or by other channels remembering me the early days of internet 🧲. Except from the fact that the inference was super slow—while using douzains Go of RAM—the results were not as good as ChatGPT but still great. If you find it interesting tell me I'll write a post about what I launched and how.

Fast News ⚡️

• Twitter's recommendation algorithm — It was an Elon tweet. Twitter published on Github (here and here) their recommendation algorithm and they wrote a blogpost explaining how the recommendation is working. The machine learning is mainly in Python and uses PyTorch. But the algorithm as a whole contains a lot of features, filters and network algorithms.
• Microsoft data integration new capabilities — Few months ago I've entered the Azure world. Not really without pain. Today, Microsoft announces new low-code capabilities for Power Query in order to do "data preparation" from multiple sources. Disclaimer: I don't use Power Query and I don't plan to ever use it.
• One year as a dataviz journalist — Saturday is a good day to have a look at great data visualisations. Erin celebrates his 1-year anniversary as a viz journalist by putting light on the work he is proud of. I really like the "Farthest distance between World Cup stadiums" or the paths to become CCO.
• Life after orchestrators — Benjamin thinks that orchestrators are legacy systems and that we should all move in the real-time world where everything is simpler. No need to add trigger and to synchronise workflows together. Side node: Ben co-founded Popsink a real-time ETL company.
• Meta introduces Segment Anything — A new Foundation model enters the game. His name is SAM, and SAM wants to identify which image pixels belong to an object. Will traditional computer vision the next space to become has-been with the new AI innovations?
• ❤️ Reducing the lottery factor, for data teams — if you had to read only one article today you should read this one. The lottery factor, also named the bus factor is risk measurement about knowledge sharing. In data teams a lot of work have to be done in the early days to avoid knowledge to be lost later on. The article gives ~10 advices to apply to lower the risks. Among them I like the changelog, the pair-programming, the pre-recorded video and the stable credentials.
• The ultimate guide to hire your data team — An awesome canvas to conduct data interviews. This guide will help you before and during the interview. It includes a great list of example questions that you could ask in interviews.

PS: Data Council took place in Austin a few days ago. As soon as the videos will be out on YouTube I'll do a wrap-up of the sessions. Data Council is usually a moment of the year when the US data gratin gather to discuss.

Data Economy 💰

• Dozer raises $3m seed round. Dozer is a platform to develop real-time data apps, looking like a real-time ETL platform. With Dozer you can connect to multiple sources, do transformations (SQL, Python or JS) and then expose the output in APIs for frontend consumers (React, Vue or Python). YAML configured. It also looks like that Dozer is not really under a proper open-source license. If you want to go deeper to me Dozer looks like Materialize or Popsink but with a different vision, offering more an API as a serving layer than a database.
• Roboto AI raises $4.8m seed round. I hate this as much as I find it interesting. Roboto AI wants to create a AI-powered toolbox for people in robotics. In their demo you can use prompt to search over images or timeseries.

See you next week ❤️.

Dear readers, already 3 months done in 2023. We are slowly approaching the 2-years anniversary of the blog and the newsletter. We are almost 3000 and once again I want to thank you for the trust. To be honest time flies and I’d have preferred to do more for the blog in the start of the year but my freelancing activities and my laziness took me so much.

By the way, recently I’ve worked with Azure tooling and I changed a bit my mind. I had tried Azure years ago and the only memory I had of it was that it was not working. Like you ask for a VM and you don’t get a VM. But obviously it changed. Except the fact that they have a pretty bad vocabulary for things, it works, and the UI is surprisingly pleasant to use.

My personal preference hierarchy changed with this experience, which is subjective, is GCP > Azure > AWS. Still one complaint I might have is about documentation, sometimes docs page are not of great quality, the presentation is pretty bad and full of usage examples when we only need complete documentation of ressources.

If you did not register next week we’ll host an online meetup about Airflow alternatives and Prefect and Dagster teams will do a demo.

PS: brace yourself, April fool's is tomorrow. I've already seen a few "jokes" on LinkedIn.

Google Data Cloud & AI Summit

Two days ago Google announced new things at their Data Cloud & AI Summit. Here a small recap of what has been announced.

Pricing changes

First, a new BigQuery pricing model. Big changes—or should I say BigChanges—the flat-rate pricing will not be accessible starting on July 5 and will be replaced by a capacity pricing similar to what Snowflake is doing. It will start at $0.04 for a slot hour. It is hard to compare to the previous flat-rate pricing but the previous pricing was more around $0.028 for a slot hour (42% increase). Still Google says that it will lower your BigQuery costs because they have the smartest autoscaler on hearth 🫠 and will run only what's perfectly needed for your queries.

Let's take an example previously you could have run 100 BigQuery slots at every moment in time for $2000 a month. Tomorrow for instance you would be able to run 160 slot hour if you use BigQuery only 10h per day for the same amount ($2000). This changes means less computing power on average for the same price but with higher peaks.

Like good news never comes alone they will also increase the on-demand pricing by 25% starting July 5. It will costs $6,25 per TB compared to $5 before.

They also announced a "significant" increase in compression performance so that you should switch you storage pricing from logical (uncompressed) to physical (compressed—the actual bytes stored on disk). Compressed storage is at least twice more expensive than uncompressed but as they announce a 12:1 (previously 10:1) compression ratio your company wallet will be the winner. The icing on the cake.

This new pricing is sad to see, excluding the increase, I believe that for years, one of the strongest advantage of BigQuery was his apparent pricing transparency. Now you need to do multiplication and open and read 5 pages to understand the pricing.

5 paragraphs about the pricing, it was unexpected.

Looker Modeler

They also announced Looker Modeler, a single source of truth for BI metrics. This is finally over, we have Google take on the semantic layer and the evolution of the LookerML which was one of the first version. They created Looker Modeler as a metrics layer that will be accessible by all the application downstream.

In a nutshell it will mean—in Looker vocabulary:

• Data Engineer will create sources that will be available in Looker
• Analytics Engineer (or Data Analysts) will create Views from a table with Dimensions and Metrics thanks to the LookML
• Then AE will create Models on top of Explores—Explores are joined Views
• Then you will be able to access the Models through the Looker Modeler via a JDBC interface or a REST API.

Thanks to this we will be able to read LookML data from Tableau. Awesome!

If you teach someone SQL they can help themselves, if you teach them LookML they help everyone.

Gen App Builder

As an answer to OpenAI offering Google Cloud started to propose cloud offering around generative AI. They announced a web UI to create conversational AIs. In the demo you upload a FAQ (in CSV) and a "How to guide" (in PDF), you pick either "Chat" or "Search" mode and it will generate a app that you can give to your customers to user personalised with your data. You can also feed to the pre-trained models BigQuery tables, GCS buckets or websites URLs.

Gen AI 🤖

Google summit last section was a perfect transition to the GenAI category. Every week is richer than the previous one.

• Bill Gates published a note: The Age of AI has begun. He says that GPT being able to ace university Bio exam is the most important advance in technology since the graphical user interface in 1980. He thinks that AI will help reduce world's inequities like health inequities—I doubt that, like pills and medicine only rich and educated people will benefit AI in the end. Then he enlightens us with how AI can change health and education systems, what are the risks and the next frontiers.
• Then everyone started to freak out. An open letter has been written to ask a pause—at least 6 months—in giant AI experiments, it has been signed by almost 2000 people, with some notorious CEOs, researchers and Elon Musk. This pause should be use to develop with policymakers AI governance systems.
• Gary Marcus debated about the AI risk ≠ AGI risk. AGI means artificial general intelligence. He mainly thinks that LLMs are an “off-ramp” on the road to AGI.
• For B2B generative AI apps, is less more? — a16z, a huge US VC, predicts that we will enter in a second wave of AI called the SynthAI. Currently we generate information based on prompt, in the wave 2 we will generate insights based on information. This wave seems to be critical for B2B because AI should help decision making, but need to be concise for that.
• As a fun use-case, John wrote how we can rethink whisky search in the world of ChatGPT.

Fast News ⚡️

• The data engineer is dead, long live the data plaform engineer — This is a current trend of the marker that has been accelerate by the analytics engineer appearance. As AE are, theoretically, in between DE and DA it pushes the actual roles borders further. Meaning DE will have to do more infra stuff to support analytics initiatives. This is normal and Robert adds more to the table in his blogpost. You can also check this map that explains all data roles.
• Five charts that changed the world — A 5 minutes video by the BBC that shows 5 awesome charts that changed the course of the world.
• Databend, an open-source version of Snowflake — at least this is what they claim. This week I discovered this "open-source data warehouse written in Rust". I'll try it out when I have time. If you tried it you'd love to get your feedback.
• audit_helper in dbt — A blog post that showcase how you can use dbt audit_helper package to improve your models. As the competition gets harder every week Datafold also wrote a blogpost comparing audit_helper vs. data-diff.
• dbt-checkpoint, a list of pre-commit to ensure dbt quality — A list of 40 pre-commit written in Python that you can use to improve the quality of your dbt projects. It includes the useful check-script-has-no-table-name to check if a there is not table name leftovers.
• Strategies for effective data compaction — From Mixpanel and how they developed an event compactor system with PubSub.
• The 2 inventors of the Lempel-Ziv algorithm that is used in all ZIP files died recently. They wrote their proposal in 1977. RIP.

Sorry for this longer edition, and see you next week ❤️.

PS: if you follow me on LinkedIn you might see this content recycled there because.

Dear readers, I hope this new edition finds you well. It seems that you really liked the recent editions, which is perfect because it was fun to write. I feel that this week all the articles I found relevant for the newsletter are either AI related or technical. I really don't know how to deal with news overflow about the Gen AI landscape. Do you like all the GenAI hype? 👍 or 👎

Airflow alternatives meetup

This Tuesday also took place the first part of the Airflow alternatives Meetup with Mage and Kestra. It was an awesome online meetup. I really liked the presentations from Mage and Kestra and even if I was focus on hosting the event it was great to see 2 other visions about the future of orchestration. Which, to be honest, are not really far from Airflow.

Here are my takeaways about the event:

• Mage and Kestra have been both developed with Airflow flaws in mind, especially about deployment complexity, reusability and data sharing between tasks.
• The tagline "Modern replacement for Airflow" on Mage side makes sense. Out of the box Mage provide all-in-one web editor to write data pipelines with a great UX. In small browser text areas you will be able to write Python, SQL or R code and orchestrate theses transformations with drag-n-drop. I personally hate developing in the browser, but the promise looks good. But actually Mage and actual Airflow version are—almost—the same the only difference is the UX when developing pipelines.
• Tommy, Mage's CEO, said that for the moment they will focus on building the best open-source data pipelines tool. They got enough funding for the next 2 years.
• Facing the reality, even if worker management seems easier in Mage, the deployment is not yet ready to go. Either you go with Terraform script that will launch elastic containers either you go with helm, but requires Kubernetes.
• Now Kestra. One of last kid on the block. Ludovic, the CTO who presented Kestra at the event, said that he started the development while at a mission at Leroy Merlin where people were heavily unhappy about Airflow. Kestra is a YAML-based data pipeline tool mixed with string templating. The YAML approach allowed less-technical users to be able to write pipeline.
• Kestra vision is also very open, everything is accessible through APIs. Which leads to a variety of usage for a company. Under the hood Kestra is developed in Java which is totally different than other alternatives.
• Kestra in the future can easily looks like Mage, YAML being the mid-step before a "drag-n-drop" like UI.

It was so fun to organise this event and I'd love to do more live in the future with blef.fr. Still, in 2 weeks on April 4th the part 2 of the event with Prefect and Dagster will take place, I hope I'll see you there.

Gen AI 🤖

The newsletter is already to big for today so I'll try to keep it short especially on this Gen AI that is already spammed everywhere.

OpenAI is slowly starting to create a gigantic ecosystem and could become the next GAFA-like company. The non-profit research company manifesto is already far away. OpenAI released a study about Large Langage Models—LLMs—impact on the job market (sorry I wanted to read the pdf but my brain is already grilled) and announced ChatGPT plugins. In a nutshell OpenAI is has created a AI interface that everyone likes and will add on top of it a App Store experience with plugins. It reminds me of something.

Because OpenAI is not everything, some news of the alternative world. Mozilla announced Mozilla.ai a community-based open-source AI ecosystem, Stanford researches released Alpaca a model that behaves similarly than OpenAI text-davinci-003 but that costs a lot less ($600 to train it), there is also a list of open alternatives to ChatGPT.

I'd have love to speak about tools offering to translate human langage to SQL like sequel.ai or SQL translator but it would open the Pandora's box about self-service analytics and this is for next week.

Fast News ⚡️

• You broke Reddit: The Pi-day outage — a good retrospective post on Reddit outage for Pi-day—the 14th of March, 3/14 in US date format, unlucky we are all the rest of us we don't have a Pi-day. Jayme, a staff software engineer, shares that a Kubernetes version upgrade from 1.23 to 1.24 led to the outage. Actually Kubernetes introduces in 1.24 a terminology change from master to control-plane which was the trigger of the issue.
• Apache Arrow releases Arrow nanoarrow — Recently Arrow got a lot of light because of DuckDB or Pandas 2.0 and it's good. Arrow is a multi-langage interface to use in-memory data structures. Nanoarrow is a C library that acts as simplified interface for application developers in order to put Arrow everywhere.
• Avoiding data pipeline failures: the importance of backfilling capability — As I've already said in the past backfilling is one task that separates data engineers to great data engineers. Backfilling has to be thought at every step of a data pipeline design and development. This is a small LinkedIn article, but it's a good reminder.
• Datafold's data-diff now integrates dbt — You can now run data diff after a dbt run to compare your models to the production state and get a summarised view of the rows impacted.
• How LinkedIn reduced processing time with Apache Beam — Beam is a distributed processing framework that proposes a unified execution engine for batch and real-time. LinkedIn team decided to migrate to a lambda architecture and got 94% uplift in performance.
• How fast is DuckDB really? — Georges, Fivetran CEO, ran a performance test to have metrics on DuckDB performance. The article conclusion is that a Macbook M1 (and probably M2) can have better performance than a server. I can relate.
• If data lineage is the answer, what is the question? — A good list of use-cases where data lineage would be useful.
• Using CockroachDB to reduce feature store costs by 75% — More and more articles about costs optimisation, 2023 is the year were data engineers skills will be used to lower platform costs. And this is a good point.
• How shadow data teams are creating massive data debt.
• Distributed Machine Learning at Instacart.

Data Economy 💰

• Sifflet raises €12.8m Series A. Initially this is a data observability tool but it turned out they added features like lineage and cataloging. Often needed to better contextualise alerts but also to avoid tools multiplicity when working with big corporations.
• Hex raises $28m in a Venture Round[1]. Hex is a notebook-based analytics application. Cells are at the center of the analytics, they produce outputs than can be used later in other cells on in visualisation. The visualisation can be organised in a Notion-like document but with live data. I recently tried Hex, the UX is neat and I think the tool is worth it for production-ready explorations[2]. Here an example with the MAD data—I made no presentation effort.
• DragonflyDB raises $21m Series A. Dragonfly is a replacement for Redis claiming to outperform it in many way (throughput, snapshotting speed, scaling). I don't have a lot to say except the fact that we are going in a future with a lot of databases choices.

1. A venture round is when the series has not been specified.
2. I just coined the term, I mean, it's when you do a great exploration and you want to share a professional result to your stakeholders.

See you next week ❤️.

Hey you, I hope you had a great week. On my side I'm slowly starting to get on top of the things I had in queue. But, sadly, I work in LIFO so I feel that I'm never done. For people that are not use to it it means last in, first out. Which means that I get easily disturbed by a notification—or even a thought—and do something that I did not plan to do at first. It, probably, explains why you always get the newsletter late on Fridays—or Saturdays.

Thank you for the feedback about last week issue, it seems you liked it. I'll try to continue doing deep-dives on article from time to time.

Airflow alternatives meetup

We are organising next week with the Paris Apache Airflow Meetup group an online event to discuss about Airflow alternatives. At every Airflow meetup we often get questions about Airflow competition so we decided to give a voice to alternatives in order to understand how they compare with Airflow and more.

The first even will take place next week, on March 21st at 7PM CET (UTC+1) and we invited Mage and Kestra. We will host another event soon after with others. You can either register on LinkedIn either join the meetup event.

How lucky you are because I will host the event, so you'll hear my awesome French accent. It also means that if you have any questions that you want me to ask you can send them to me beforehand 🫠.

Gen AI 🤖

I will create a specific category for generative AI.

If you live in a cave or if you only read my newsletter to get news about the data world you might have missed that GPT-4 has been announced and released this week. I even had hard time navigating between data engineering memes and GPT4 tips on LinkedIn and my Twitter is divided between GPT-4 threads and protests in France. What a time to be alive. Politicians think we should work longer when we are slowly starting to discover new AI capabilities that will for sure impact workplaces.

I don't want to take the usual shortcut—but how could I not do that. Will AI replace jobs? I do think that AI should empower people, but will the capitalism think like this when an API call will be able to do the same job as a human? Does even capitalism think? Actually it's probably human decisions about AI that will lead to AI replacing people.

One field that has been totally impacted by the generative field is the Natural Langage Processing (NLP). On Reddit someone asked this if others were also witnessing panic in NLP orgs. The general feeling is that GPT made years of NLP research outdated.

A few other news:

• LinkedIn team also wrote a blog about AI principles stating that AI is like oxygen for the engineering team—I personally would have say that data was oxygen, but who cares—and that with great power comes great responsibility. The same week Microsoft (who owns LinkedIn) reportedly layed off the AI ethics and society teams. Great timing.
• Glaze, protecting artists from style mimicry — A tool developed by researchers at Chicago university will help digital artists by cloaking their art to avoid mimicry from deep learning trainings.
• Google and Microsoft will compete to include AI copilots in their offices suites — Microsoft announced 365 Copilot that will work in Word, Excel, Powerpoint and Outlook. On the other side Google announced the same for Google Docs and Gmail.

Fast News ⚡️

• Migrating from role to attribute-based access control — RBAC is probably one of the most use paradigm when it comes to autorisation especially because role based autorisations are faster to put in place. In the article Grab team explain how to migrated from roles to attributes autorisation on Kafka.
• Speeding up “Reverse ETL” — Ziqi works at Microsoft and details in this article what they had to consider to improve their Lakehouse exports to downstream databases. In short they switch SQL Server to columnar storage, disable indexes and locks when copying and played with parallelisation and batch size.
• Online gradient descent written in SQL — Max is one of the best when it comes to do great experiments. This time it shows that everything can be done in SQL. With recursive CTEs he implemented sklearn linear model and the code is not even that big.
• Data with Rust — This is a handbook that will showcase how to work in data engineering with Rust. At the moment only part 1 and 2 are written but it looks promising.
• Sharing data between dbt projects, dbt exposures to sources — When you have multiple dbt projects it can be a mess to reference a model from another project. This blog shows how you can automate it with a CI and definitions in exposures.
• Polars vs pandas : A new era for Python DataFrames — This comparison is also slowly starting to be a great debate in the data world. Will Polars overtake pandas in the coming years? Guillaume wrote yet another great comparison.
• Tracking the fake GitHub star black market with Dagster, dbt and BigQuery — Things are getting spicy here. Dagster team proposed a way to eventually identify Github projects buying stars.

Other few articles but with no comment:

• Introducing multi-modal index for the Lakehouse in Apache Hudi.
• How to be a good data analyst without good data.
• dbt Reimagined.
  

Data Economy 💰

• The Austrian data protection authority has decided that Meta tracking tools are in violation of the GDPR. It will create a precedent.
• Seldon raises $20m Series B. Seldon is a MLOps platform that helps you deploying models in production. At core Seldon provides a framework that you can configure to serve you models on top of Kubenertes.
• 👀 Adept raises $350m Series B. This is again a testimony of the frenzy about generative AI, and according to me the most impressive one. Adept want to create a general purpose AI teammate for everyone. At the moment it takes the form of a browser extension in which you can ask stuff when you navigate on Salesforce, Google Sheet or Craiglist.
• Cast AI raises $20m in funding. They propose an AI to cut your Kubernetes costs in half. Bold promise.

See you next week ❤️.

Dear readers, this week Data News lands on Saturday and will be a little bit different than usual because I found less relevant article and as promised last week I wanted to speak about the MAD Landscape.

I hope you will enjoy this topic focus edition where I speak about economics even if I'm a newbie about economy. In last minute I also added stuff about the Silicon Valley Bank that has been seized by the US FDIC, which will generate a crisis in scale-ups/startups world.

The MAD landscape

The Machine learning, Artificial intelligence & Data (MAD) Landscape is a company index that has been initiated in 2012 by Matt Turck a Managing Director at First Mark. First Mark is a NYC VC, in their portfolio they have Dataiku, ClickHouse and Astronomer among other tech or B2C companies.

Year after year the MAD Landscape has become an important tool to index the whole data landscape. The choice of categories is also a very clear way to categorise companies and to capture how the data field is changing. Obviously this kind of index is opinionated and they—Matt and his team—make editorial choices when they decide to include—or not—a company, but still, their selection depicts a reality.

Today I want to do a second lecture of the 4-parts article Matt wrote and to give my views on it. As Matt said in a LinkedIn live with Joe Reis the MAD landscape was not published last year (2022) because of time and the landscape has been totally shaken by 2 major events: the massive layoffs wave and the generative AI hype. As a reminder in 2021 edition money was flowing, Databricks did 2 huge rounds with $2.6b raised and Snowflake IPO was a success one year after.

In the MAD landscape we have 3 main parts that I will discuss about today:

• Infrastructure and open source infrastructure — all the data tools everyone wants to use (or not, Talend appears twice in the list 🙃) this part depicts well what a data engineering needs to create a data stack.
• Analytics — this is about the tools we will use to query the data lying down in the infrastructure.
• Machine learning & AI — this category has been totally shaken by the generative AI trend, enterprise machine learning in 2023 is not the same as previously.

Before going more into category changes and macro trends this MAD capture there are a few interesting facts highlighting some biais this index might have:

• 933 companies out of 1414 (65%) are US-based companies
• The continent repartition is 965 (68%) in North America, 182 (12%) in Europe , 74 (5%) in Asia and 192 companies are open-source, so they don't have a base country
• Median founding year is 2015, which means that half of the companies are less than 7 years old, and 20% are less than 3 years old
• GAFAM have logos everywhere. Amazon is the most represented one with 33, then Google with 30 logos, then Microsoft with 21. Apple and Meta are lower with both 2 logos. This is important to mention that IBM have 12 logos and IBM is the oldest company — 1911.

Mainly what these fact are saying is that the MAD landscape is dominated by US-based companies and US-based companies are nowadays thinking how the world should do data, trying to replicate their problem and their vision to everyone. Which is kinda broken. Obviously there are companies or VCs in Europe/Asia but rare are the one with the same impact. Diversity-wise this is a world dominated by the Northern Hemisphere (as always), there is no company in Africa or Southern America for instance.

Key insights

In a nutshell here are the key insights you need to know if you do want to read Matt's notes. First regarding data infrastructure:

• The consolidation will come in the next months/years — every sub-category has between 20 and 30 logos, even if every company think it's unique, they often do the same as other and the market might not be as large as thought. Also there are a lot of "single feature companies" which will compete with broader ones and more likely fails because of offering. Snowflake and Databricks are the adults who will whistle the end of recess.
• Quality and observability are the same — sorry but everyone want to be the "Datadog of data". When looking at the trend they all want to do the same.
• The future of data catalogs is unclear — I really like the definition of catalog Matt gives: "there is a need for an organised inventory of all data assets". Catalogs are still struggling being adopted even if they seem to be asked by a part of the industry. There are also too many alternative.
• With the recession, modern data stack is attacked — This is a big shortcut but true. Modern data stack is tightly coupled to ELT which means load first and think second. When you load first you have more data than you need which leads to avoidable costs. The actual MDS with unlimited computing power and storage might come to an end.
• If you want another perspective with a more exhaustive list of changes you can read Anna's takeaways about MAD 2023 infra category.

After infrastructure Matt also writes about all AI impacts:

• The index this year depicts the generative AI hype with a lot of early stage startup doing almost everything possible with generative algorithms.
• According to Matt we are now in the 3rd cycle AI hype. This is the largest one because it reached mainstream coverage. As a proof my father is using ChatGPT (in French "chat" means "cat" and he says CatGPT, which is a bit funny). But yeah AI became mainstream even if it was already everywhere before, but it was vertical models. But now everyone experiences the general purpose intelligence.
• Startups might have difficulties catching up tech giants on this because they need data and probably a lot of computing power they might not have.
• There are many backlashes AI companies will have to navigate through: impact on job market, algorithm bias, disinformation, hallucination—a word for AI is often wrong, and lastly AI is just boring.

In addition to this O'Reilly released their technology trends from the searches on their website, when we only focus on the data field what we see is:

• Overall Python is the most popular concept and the most growing one — I think it explains because Python is the best entry-level langage for the IT world.
• When it comes to data, data engineering is the most searched concept and growing
• Spark and Hadoop have been less searched than last year
• PowerBI is the 3rd most searched concept and I'm sad about it

Silicon Valley Bank—wat?

This is a bit last minute but this is freaking huge. Let me do a recap for you and why does it matter.

The Silicon Valley Bank (SVB) is a deposit bank based in California and has the biggest market share. SVB manages billions dollars of assets. Mainly the assets are coming from Silicon Valley startups, founders and employees. In a nutshell if you are a startup founder and you get millions from a seed round you put the money in the SVB.

2-3 years ago a lot of money was raised, the SVB got around $200 billion in deposit. The SVB wanted to put $80 billion of this money at work using Mortgage Backed Securities (MBS)—just as a reminder MBS were at the center of the 2008 financial crisis. MBS guarantees 1.5% return and because interests rates were low because of the pandemic it was ok.

In the last months the FED increased rates crossing recently the 4.5% mark which still was ok. But started triggering chain reaction of all actors. The SVB did a first mistake that I'm not able to explain.

Then VCs started panicking (e.g. Peter Thiel's) advising founders and startups to get the money out of the SVB. Which led to a bank run.

A bank run or run on the bank occurs when many clients withdraw their money from a bank, because they believe the bank may cease to function in the near future

Then the SVB did another mistake. One day later the stock was 60% down and later the same day the bank collapsed.

What happened here is huge and will have a big impact on every US-based scale-ups/startups—it's very well linked to the MAD landscape. Mainly the deposits were only insured until 250k and it means that a lot of companies will lack of cash and probably have difficulties to pay salaries and/or vendors soon.

As a reaction it'll, sadly, imply more layoffs in the coming days and weeks. Other are also afraid of a contagion to the whole banking system as the SVB collapse became the second-largest collapse of the US history.

Data Economy 💰

• Lonestar raises $5m in seed to put data centers on the Moon. Yep, you read it well. Apparently moon market projected to generate $105B in revenue over next decade. While in France we are fighting to retire earlier people wants to send my Twitter history backups to the moon.
• Employees are feeding sensitive business data to ChatGPT, raising security fears.

See you next week with an usual Data News ❤️.

Hello you, I hope this new Data News finds you well. After last week question about your consideration of a paying subscription I got a few feedbacks and it helped me a lot realise how you see the newsletter and what it means for a you. So thank you for that. I'll try to think about it in the following weeks to understand where I go for the third year of the newsletter and the blog.

Stay tuned and let's jump to the content.

This week I've published an compact article about how to get started with dbt. The idea behind this article is to define every dbt concept and objects from the CLI to the Jinja templating or models and sources. The article has been written as something you can add in your own internal dbt onboarding process for every newcomer.

Machine Learning Saturday 🤖

• How BlaBlaCar leverages machine learning to match passengers and drivers — BlaBlaCar is a carpooling company and in this article they detail what they did—in terms of machine learning—to improve trips listing with a Boost feature that proposes detours to drivers in order to be able to cover more countryside cities. It does not include any generative AI but greatly shows how machine learning can impact business problems.
• Sharing LinkedIn’s Responsible AI Principles — Very short article that lists the 5 principles LinkedIn aims to follow. In a nutshell AI should be use as a tool to empower members and augment their success, while prioritising trust, privacy, security, and fairness, providing transparency in AI usage, and the right governance should be put in place to maintain accountability over AI algorithms.
• Designing a regional experiment to measure incrementality — Monzo team did an geographical experiment in order to understand how their referral program works.

Fast News ⚡️

• Writing well: a data engineer’s advantage — This is probably a leftover part of the data engineer toolkit, but writing is an essential skill. Luuk gives a few advices on how to improve in your email communications with coworkers in order to announce new release or to seek for budget for a factorisation project.
• Here’s why your efforts to extract value from data are going nowhere — If data science is “making data useful,” then data engineering is “making data usable.”. This is a quote from Cassie article which I find awesome. But still, in order to make data works we still need to praise other data coworkers that have to do documentation and all the governance burden that no-one wants to do.
• Understanding slowly changing dimensions (SCD) in data warehousing — SCD modeling is an old technique but more and more relevant today as we need to keep track of transactional data. The article proposes 6 types of SCDs. I think the SCD type 2 is the most common and lossless one, but other are worth mentioning. As a side note, if you want to understand quickly what SCD are, dbt snapshots documentation page is the best path to go.
• How to run dbt with BigQuery in GitHub Actions — When you're starting with dbt you don't need any orchestrator or dbt Cloud, a CI/CD do it for sure. This article gives you the GitHub Action you need to setup.
• Snowflake: query acceleration service — Snowflake invented a boost, that you activate with a flag at warehouse creation (in Snowflake a warehouse is the compute isolation your queries run in, the bigger the warehouse is the more compute you use and pay). When you activate the query acceleration service when Snowflake thinks that a query can be accelerated it will launch more compute than actually specified by your warehouse. Not related, they also announced Snowpipe Streaming this week.
• Data ingestion pipeline with Operation Management — At Netflix they annotate video which can lead to thousand of annotation but they need to manage the annotation lifecycle each time the annotation algorithm runs. This article explains how they did it.
• Ensuring Data Consistency Across Replicas — Mixpanel details how they ensure that different zones Kafka consumers are writing the data in the same manner. This way, when a zone is unavailable they can use the other zone to still have the data without any duplication or lack of messages.
• Pandas 2.0.0 — A new major Pandas release is out. In the shadows of Polars that seems to revolutionise DataFrame computation Pandas came with a lot of optimisation and changes.
• AWS lambdas are still on Python 3.9 — Corey rant about AWS lambdas that are still using Python 3.9 while all the competition upgraded to at least Python 3.10.
• A small head's up, the Apache Airflow team has announced the Airflow Summit for 2023 which will be held in Toronto in September. They recently opened the call for presentations.

Data Economy 💰

• Qwak raises $12m Series A. Are the ducks the new elephants? Qwak proposes a all-in-one platform to manage all operations in a machine learning project. In the platform you do the feature engineering, the model creation, versionning, deployment and monitoring with all pipeline automated. I think a lot of platforms like this exists today.
• Announcing Tabular — Tabular has been released in public this week. Tabular is a cloud offer using Apache Iceberg. This is funny to see their offering because they offer a "managed data warehouse storage", which means without the compute. You bring your own compute. Some company also call it a lakehouse or a data lake, but the word shift is enough interesting to notice. At least for me.
• Insights from new data and AI Pegacorns — Ben from GradientFlow gave a few economic insights about the data Pegacorns (companies with more than $100m annual revenue). I don't have much to say on except that next year probably we'll see generative AI companies on the track to enter the selection.

I wanted to include a review of the 2023 MAD landscape in this newsletter but as I was late and it would have become a huge edition I'll try to write something on it specifically next week.

See you next week ❤️.

When I write dbt, I often mean dbt Core. dbt Core is an open-source framework that helps you organise data warehouse SQL transformation. dbt Core has been developed by dbt Labs, which was previously named Fishtown Analytics. The company has been founded in May 2016. dbt Labs also develop dbt Cloud which is a cloud product that hosts and runs dbt Core projects.

In this resource hub I'll mainly focus on dbt Core—i.e. dbt.

First let's understand why dbt exists. dbt was born out of the analysis that more and more companies were switching from on-premise Hadoop data infrastructure to cloud data warehouses. This switch has been lead by modern data stack vision. In terms of paradigms before 2012 we were doing ETL because storage was expensive, so it became a requirement to transform data before the data storage—mainly a data warehouse, to have the most optimised data for querying.

With the public clouds—e.g. AWS, GCP, Azure—the storage price dropped and we became data insatiable, we were in need of all the company data, in one place, in order to join and compare everything. Enter the ELT. In the ELT, the load is done before the transform part without any alteration of the data leaving the raw data ready to be transformed in the data warehouse.

In a simple words dbt sits on top of your raw data to organise all your SQL queries that are defining your data assets. And dbt only does the T of the ELT which is really clear in term of responsibilities.

dbt is a development framework that combines modular SQL with software engineering best practices to make data transformation reliable, fast, and fun.

It was the previous tag line dbt Labs had on their website. This is important to understand that dbt is a framework. Like every framework there are multiple hidden pieces to know before becoming proficient with it. Still it very easy to get started.

dbt concepts

There are a few concepts that are super important and we need to define them before going further:

• dbt CLI — CLI stands for Command Line Interface. When you have installed dbt you have available in your terminal the dbt command. Thanks to this you can run a lot of different commands.
• a dbt project — a dbt project is a folder that contains all the dbt objects needed to work. You can initialise a project with the CLI command: dbt init.
• YAML — in the modern data era YAML files are everywhere. In dbt you define a lot of configurations in YAML files. In a dbt project you can define YAML file everywhere. You have to imagine that in the end dbt will concatenate all the files to create a big configuration out of it. In dbt we use the .yml extension.
• profiles.yml — This file contains the credentials to connect your dbt project to your data warehouse. By default this file is located in your $HOME/.dbt/ folder. I recommend you to create your own profiles file and to specify the --profiles-dir option to the dbt CLI. A connection to a warehouse requires a dbt adapter to be installed.
• a model — a model is a select statement that can be materialised as a table or as a view. The models are most the important dbt object because they are your data assets. All your business logic will be in the model select statements. You should also know that model are defined in .sql files and that the filename is the name of the model by default. You can also add metadata on models (in YAML).
• a source — a source refers to a table that has been extracted and load—EL—by something outside of dbt. You have to define sources in YAML files.
• Jinja templating — Jinja is a templating engine that seems to exist forever in Python. A templating engine is a mechanism that takes a template with "stuff" that will be replaced when the template will be rendered by the engine. Contextualised to dbt it means that a SQL query is a template that will be rendered—or compiled—to SQL query ready to be executed against your data warehouse. By default you can recognise a Jinja syntax with the double curly brackets—e.g. {{ something }} .
• a macro — a macro is a Jinja function that either do something or return SQL or partial SQL code. Macro can be imported from other dbt packages or defined within a dbt project.
• ref / source macros — ref and source macros are the most important macros you'll use. When writing a model you'll use these macros to define the relationships between models. Thanks to that dbt will be able to create a dependency tree of all the relation between the models. We call this a DAG. Obviously source define a relation to source and ref to another model—it can also be other kind of dbt resources.

In a nutshell the dbt journey starts with sources definition on which you will define models that will transform these sources to something else you'll need in your downstream usage of the data.

dbt entities

I don't want to copy paste the dbt documentation here because I think they did it great, there are multiple dbt entities—or objects, I don't know how to name it, they name it resources, but I don't want to clash with the resource as a link. So there are multiple dbt entities you should be aware of before starting any project, the list below is exhaustive (I hope) but more, the list is sorted by priority:

• sources / models — you already know it, this is the key part of your data modelisation.
• tests — a way to define SQL tests either at column-level, either with a query. The trick is if the query returns results it means the test has failed.
• seeds — a way to quickly ingest static or reference files defined in CSV.
• incremental models — a syntax to define incrementally models with a if/else Jinja syntax. Here the reference. You can choose the strategy you want depending on your adapter (cf. examples on BigQuery).
• snapshots — this is how you do slowly changing dimension. This is a methodology that has been designed more than 20 years ago that optimise the storage used. The dbt snapshot page is the best illustration I know of the SCD.
• macros — a way to create re-usable functions.
• docs — in dbt you can add metadata on everything, some of the metadata is already expected by the framework and thank to it you can generate a small web page with your light catalog inside: you only need to do dbt docs generate and dbt docs serve .
• exposures — a way to define downstream data usage.
• metrics — in your modelisation you create dimensions and measures mainly, in dbt you can next define metrics that are measures group by dimensions. The idea is to use metrics downstream to avoid materialising everything. You can read my What is a metrics store to help you understand.
• analyses — a place to store queries that are either not finished either queries that you don't want to add in the main modelisation.

You can read dbt's official definitions.

Analytics engineering

dbt is becoming a popular framework while being extremely usable. A lot of companies have already picked dbt or aim to. There are multiple technological reasons for this, but technology is rarely the real reason. I think the reasons dbt is becoming the go-to are mainly organisational:

• dbt is a complete tool that you can give to analytics teams, it can become their unique playground. Within it they can do almost everything.
• The network effect. Because more and more companies are betting on it, more and more trained people there will be in the market. It's also a strategical choice in order to be able to hire people.
• The documentation, as I said earlier, is top of the notch.

dbt Labs also popularised the analytics engineer role. We can quickly summarise the role as in-between the data engineer and the data analyst. But because companies can have very versatile definition of role, I'd say that the analytics engineering is the practice to create a data model that represents accurately the business and that is optimised for a variety of downstream consumers. So the analytics engineers are the one doing this.

By the position of this role and the freshness of it, people are coming into analytics engineering from data analytics. Usually they don't have a lot of software engineering good practices and knowledge, which is obvious, but the dbt framework is also meant to bring this to the table.

This is also fair to say that dbt as a tool is very easy to use and very often the complexity of the dbt usage will lie in the SQL writing rather than the tool usage by himself. There are also a few questions in term of project structuration that needs to be done.

If you like this article you should subscribe to my weekly newsletter to not miss any other article of this kind.

Resources

As I only want to help you get started with concepts I know want to redirect you to other articles that I find relevant to go deeper:

• dbt annual conferences — Every year dbt do their annual conference called Coalesce which features a lot of dbt user and usage. I've covered with takeways the 2 last one: Coalesce 2021 and Coalesce 2022. In these articles there are a lot of cool presentations you should watch to understand deeper how dbt works.
• Introduction slides about dbt — This is a presentation I often give, you can also watch a talk I gave in French, there is also a great introduction by Seattle Data Guy that I recommend.
• You can do tests in dbt — like: environment-dependent unit testing in dbt, 7 dbt testing best practices or a guide to building reliable data with dbt tests.
• You have to get inspiration from others dbt projects — dbt @Beat, dbt @Vimeo, dbt @ShopBack.
• Optimisation — An issue with dbt is that everything will run in SQL, which means you'll have to optimise a lot of thing. dbt Labs team wrote about an optimisation of a long running model.
• A rant against dbt ref — A great article to make you think about dbt principles.
• How to monitor dbt models.
• Generate databases constraints with dbt.
• 🧑‍🏫 Online courses — I've tried any of the courses I'll recommend, but from the background of the mentors I think they are very relevant. There is first a Corise "Data modeling for the modern data warehouse" that lightly covers dbt and mainly how to do data modeling and the analytics engineers club that sells a training program to go "from analysts to engineer" in 10 weeks taught by an ex-dbt Labs employee. You can also contact me if you want something more personalised.

Dear readers, I hope you had a great week. Each time I look back and I see the amount of Fridays I've spent reading and writing I'm still surprised. For the last 2 newsletters I've tried to ask your for paying support. From number of people who really paid I can see that I failed to either word it correctly, either to propose a newsletter where you see the value of paying for it.

In any case, I need your honest feedback, what would make you consider paying for the content I create?

This is something I struggle with, I really like writing, I really like this newsletter, I really like the blog, but it takes me one day per week to be done. If I want to continue for years I have to find a way to make it sustainable for me, and also if I want to continue more in this direction I have to find a model that works. I'm open to all honest feedbacks.

A bit of infrastructure

This week I've seen a lot of articles that I can put under the infrastructure category, so here we are. The current data state is heavily dependent on infrastructure, wether it's cloud, on-premise or semi-related we need to understand where the data lands and where the code runs.

First Bucky gave his thoughts about the state of infra in 2023. In a nutshell, Javascript is the future of everything, we say it for years, you write once and you run it everywhere—in the browser, on servers—then workflows systems are a key piece of every software architecture, we have a fragmentation of tooling and we want to run tasks one after the other which means we need something to orchestrate them, finally the OLAP databases are evolving in something different with many more features.

In order to improve your data infra you should sometimes try to occasionally kill your data stack, chaos engineering is something that helps discover issues. Monte Carlo also wrote this week about chaos engineering, with a manifesto.

When it comes to data storage, the real-time ecosystem has also changed a lot in the last few years and a lot of tooling went out to simplify the burden of managing Kafka clusters, Materialize—a real-time platform—detailed their architecture. But if you want to continue using the underlying tools here an overlook of Flink architecture or a few techniques you should know as a Kafka streams developer.

Finally Whatnot shared how the migrated their CD processes to ArgoCD and Pinterest now uses HTTP/3, which I didn't even know it was existing.

Fast News ⚡️

• The future analytics developer experience — It's been a few months since we have articles complaining about the actual of analytics development experience. Often they are right. At the moment the best way to develop in your data warehouse is still in the query editor of BigQuery / Snowflake / etc., even if we have tools that are trying to provide a great experience as Petr is saying in the article we still lack something. I hope it will change.
• Measuring B2B customer satisfaction — Dashlane team shares how they are measuring customer satisfaction. I really like the KPI framework they put in place and how it translates in charts.
• Measuring everything — This blog is a proposition and a signal for you that you should measure absolutely everything to understand what is happening in your product. This goes further than being a data-driven enterprise, you have to put in place a framework the puts data measurement at every product choice, resulting in maturity increase.
• Stream processing vs real-time OLAP vs streaming database — Data storage  + compute field is slowly becoming a mess a lot of technologies that are so close but so far away at the same time. Hubert tries to explain stuff in the real-time category.
• Data Engineers and Kubernetes — A 101 guide about Kubernetes concepts and why you should as a data engineers understand kube basic entities.
• Coding patterns in Python — Startdatengineering is one of the best data engineering related blog and this time he propose a few patterns you might need to implement in Python when doing data pipelines.
• Etsy payments data model — Articles are often about technologies and rarely about the actual data modeling and this time Etsy team shared what was their reflexion while re-modeling payments. Sadly this is more about transactional improvements and choices rather than analytics.
• Shark attacks visualisation — This is a great example of embedded analytics. Louis deployed a version of ToucanToco—a BI tool—using Redshift to visualise data about shark attacks. In a surprising way 3 shark attacks were deadly in Italy, I'll more careful next time when I swim in the Mediterranean.

Data Economy 💰

• Qbeast raises €2.5m seed. This is interesting to see that data lakes platforms can still raise money in 2023. Qbeast propose a different way to organise data to optimise queries performance, still it seems they use Spark.
• OpenAI new strategy — Someone on Twitter reported that OpenAI privately announced a product called Foundry that would enable customers to run OpenAI on dedicated capacity with full control on the model configuration and profile.

See you next week ❤️ — small edition today, blank page issues 🫠

In last week newsletter I've also share what is a metrics store, which led to a longer edition than usual and I saw that a few people did not like it this way. It was a try I'll see in the future how I can do it better. Still, what is a metrics store? You can check out the post extracted from the newsletter.

On the same topic this week Pierre shared how to create a semantic layer in Preset—i.e. managed Apache Superset—to do so, it first defines metrics within dbt and then thanks to the CI/CD it pushes to Preset the metrics definition. This is a great example of a simple way to push down metrics to visualisation tools.

Is DataOps really a thing?

Last year DataOps has been used in many different ways to describe so many data-related different tasks. When you look deeply at it some companies put behind DataOps word just data stuff. Which is a bit misleading when you read that DataOps is "DevOps for data". Because all things wrapped DevOps is something different than software engineering.

I personally do share this perspective. Data engineering is mainly software engineering applied to data, or at least we try. If we see it this way, this is logical to say that DataOps is the movement to smoother the operation side, which technically means the infrastructure side—the IT as previous generations were saying, I don't like IT, it makes me feel old. Data engineering is also an infrastructure heavy field with a lot of technologies to put together to create something that works. This is why DataOps is important. This is why Infrastructure as Code is mandatory.

To me it stops here, all the marketing derivation of it saying we do data products using DataOps methodology is just marketing. Actually you are just writing code applied to data and using Docker containers to deploy it in the cloud. I think we should stick to software engineering vocabulary.

It also means that the data engineer role is constantly evolving. Especially with the new appearance of the analytics engineer role. Analytics engineers are taking tasks out of data engineers—which is for the better tbh. Data engineers will have to focus more on software and on infrastructure. Shifting the expertises. Analytics engineers will become the data modeling experts. Data engineers will own the infrastructure side and software related to data team—which is already a too broad field with different ownerships (DS, MLE, etc.).

In the end when I deploy data apps I end up doing Dockerfile with CI/CD processes and I look for cloud services to hosts my containers. If this is not DevOps what is it?

Fast News ⚡️

• Unveiling the three faces of documentation — Practical advices about data documentations and how you can leverage through 3 main axes: assets knowledge, business knowledge and team onboarding.
• Databricks announced a VS Code extension — This is a small news, but still interesting to see all-in-one platform like Databricks going this direction to provide end-users extension to support their way to write code rather than the vendor one.
• 📺 Understanding the business as a data analyst — A podcast about the business privilege position data analysts have, but also the responsibilities to understand and modelise it correctly in order to provide the best value to data users.
• Decrease ETL costs with Apache Arrow — I've often written data extraction with pandas by doing pd.read_sql because it's super handy and you can have something that works quickly, but the cost in memory can be high. This article shows how you can do it with Polars that leverage Arrow using less memory.
• Deploying data pipelines using the Saga pattern — When you enter the real time journey your way of thinking data pipeline is a bit different and it can be overwhelming when you come from the batch world. The Saga pattern is a pattern meant to ensure consistency first in the system. Here Picnic showcases the usage of dead letter queues.
• The case for being biased — It's been a long time since I've not featured Benn's posts, still awesomely written. It answers well to "Analytics is not about data. It's about truth" I've shared last week. Benn thinks about the role of a data team in the business decisional journey.
• Balancing quality and coverage with our data validation framework — Dropbox tech team developed a data validation framework in SQL. The validation runs as an Airflow operator every time a new data has been ingested. In terms of design only one query runs—performance reasons—and if the query returns something different than zeros, it means something is going wrong. This validation process is also a staging step before sending a table to production.
• I built a game for data visualization with streaming data — Fun project. How to use streaming data to create a real-time javascript visualisation as a video game.
• Pedram developed a NeoVim extension for dbt users. If you're not familiar with Vim or NeoVim, Simon explained what is Vim, and why this is more than an editor.

Data Economy 💰

• Europe data salary benchmark 2023 — Mikkel has become one of the best in Europe to picture correctly the data field by doing benchmark and studies across the whole market. This time he is looking at salaries. To me, as French, the most crazy number is to see that senior positions—5+ years—in Europe are compensated six figures.
• Side note, this week I realised that DuckDB Labs was the team behind DuckDB and not MotherDuck who did a partnership with them to propose the duck technology to everyone.

See you next week.

At the moment Transform is a small company compared to dbt Labs, only 40 employees according to LinkedIn and they raised around $25m. Which is only 10% of dbt Labs actual workforce. But I think this acquisition matters and will shape our data stacks.

In the past I've made jokes about the naming confusion the data field was into, especially with the following terms: semantic layer, metrics layer, metrics store, headless BI, features store. This is want I want to demystify today. I've spent the whole day reading and watching content in this category and I want to help you understand what it means for us. As a side note, it's fair to say that I also wasn't a believer in the actual necessity of this infrastructure piece. After a full day of research I'm more into it, but we have to be careful.

First, definitions

Before going further I have to write down some definitions. These definitions are mine and if you think I'm wrong you'd be more than happy to get your feedback on it. This is also super hard to have a universal definition across all vendors—as can be seen by this discussion.

• Measure — a measure is a value on which we can do all sort of computations (addition, multiplication, etc.), in a warehouse context we do aggregations on measures (sum, count, avg). A measure is often numerical but not necessarily. As an example the order price is a measure.
• Dimension — a dimension is something that categorises a measure, it adds context to a measure. You can use a dimension to filter or group the data. For instance the order date is a dimension.

Dear Data News friend, every week there is a bit of randomness when this email will truly land in your mailbox—which, btw, breaks all the rules of newsletter writing. Yeah, you know, you have to get your readers used to a fixed schedule, which they can trust and bla, bla, bla. The good news is that at least with me you can trust that I have no schedule except that you should have the newsletter on Friday or Saturday.

While I feel privileged to be able every week to send my thoughts to so many people, it takes me a significant amount of time to craft and write the newsletter. I ask you to consider supporting me by becoming a paying subscriber. Especially if you think like me that the newsletter is great.

Fast News ⚡️

• News from the generative AI universe — Google announced Bard a competitor to ChatGPT, but with better ethics, etc. In the same time Microsoft opened in beta the ChatGPT integration with Bing. Closer to us on the data space Hex proposed a prompt that can do magic for you.
• Big Data is Dead — A retrospective on why we don't need any more as much as computing power as before. Obviously the article is biased because it's from DuckDB mother company. As a reminder DuckDB runs on a single node fitting all computes in memory. But the article is relevant nonetheless.
• Migrating from Airflow to Dagster is now a breeze — In the orchestration competition Dagster made a step forward, they develop tooling to ease migration from one to the other and one side-effect is that you can orchestrate Dagster DAGs from Airflow. In order to understand Dagster philosophy you should now think with assets.
• Data Analytics framework in Python: from scientific approach to actionable implementation — A framework to conduct data analysis in Python.
• Should you measure the value of a data team? — Considerations about measuring the job a data team is doing and which metrics you should go for.
• Analytics is not about data. It's about truth. — This is an hot take this one because what's the truth?
• Rebuilding a Cassandra cluster using Yelp’s Data Pipeline — This is awesome when we can use our data engineering skills not only to do analytics but also to help fellow tech teams in tasks that are hard to do.
• How to fix your ETL to lower Snowflake Costs — Mark shares a 3 Snowflake queries that you can run to get table usage in order to identify what costs a lot.
• Reflecting on the past 6 years ff data engineering — This is a podcast episode (which I did not listen because of time).
• The complete guide to building reliable data with dbt tests — 10 practical points to improve your dbt tests.

Data Economy 💰

• Acceldata raises $50m in Series C. Acceldata looks like an enterprise data observability tool that does everything other data observability tools are doing. Like drawing charts that shows that you probably have issues 🫠.
• Recently the Kafka company (Confluent) acquired the Flink company (Immerok), economically it means a lot and reshuffle companies strategies. In addition RisingWave also shared views on why you probably need a stream processing system.
• Why big tech companies need so many people — this is a good economical question. For instance, Twitter, should be easy to copy. Why do they need thousands of engineers to develop a website that I can re-develop over a weekend?
• dbt Labs intends to acquire Transform. I just put this here for people who do not read the first part of the newsletter 🫠.

See you next week ❤️

Hey you, it's already February. Every week same analysis for me. I plan too many tasks but I slowly deliver. I guess that's how it is. Still I love this Friday rendezvous that we have together. I'm still amazed by how I changed my old habits to add the writing in my workflow. And it brings me a lot of joy.

This is also funny because I don't consider newsletter writing as work. Which is maybe a bit stupid but when I work on the newsletter I upskill myself, I read, I discover stuff, I meet with people. But the newsletter takes 1 day per week to be done, which is significant to say it's work. I wish everyone to find this little thing that is actually work but that makes work less work.

I'd like to write more about my time organisation and especially about my freelancing activities but today is a day where I have less time for the newsletter, so it's more an appetizer for later. Let's jump directly to the news.

ML Friday 🤖

• Netflix, discovering creative insights in promotional artwork — That probably the reason behind Netflix being now very conventional in term of artwork. The article shows our Netflix art creators are using past data to create new artworks. In the end this is a loophole, where everything looks like similar.
• ebay, Variable Hub a data access layer for risk decisioning — Looks like a feature store but for risk topics. The idea is to create a unified layer that stores all the data needed to take decisions.
• Lyft, powering millions of real-time decisions with LyftLearn Serving — Architecture of the decentralized system Lyft use to deploy and serve ml models.
• Spotify, Unleashing ML Innovation at Spotify with Ray — I've never used Ray in the past, but looks promising as a unified way to describe machine learning pipelines no matter the framework you want to use.

This is refreshing to see big tech machine learning articles that are still looking like machine learning we were doing 2 years ago.

Fast News ⚡️

• What's the Modern Data Stack? — Another post about what's the modern data stack. The article is a good summary of the required blocks composing a modern data stack. You can also get inspired by Stuart's modern data stack.
• Analytics Engineer- A Glorified BI Engineer? — I feel guilty, I still think that Analytics Engineers are BI Engineers. But BI Engineer for the modern data stack times. In this post Madison tried to compare the two roles. In the end actually, the answer depends. Analytics Engineer role is still unclear and depends company to company. What's often stays is that AE is between DE and DA, so the definition is often done complementarily to other positions.
• Microsoft Azure announced managed Airflow — Starting this week you'll be able to launch Apache Airflow within Azure Data Factory. The feature is in public preview. The way they integrated it within Azure looks a bit weird, but it at least exists now.
• Change data capture with DuckDB — Pedram had a sneak peek of the future, he tried a CDC setup (with Striim) that writes to GCS and then DuckDB compute metrics downstream.
• Data team as % of workforce — Mikkel is a reference when speaking about data team size. This week he categorised companies by data team size as % of workforce. For instance he found that Marketplace companies have bigger data teams than B2B ones. It makes sense.
• 2023 state of databases for Serverless & Edge — I did not know that serverless databases field was so innovative right now. All things considered this is a normal evolution, databases connections are from an old time and web developers wants direct access to databases. This is interesting to see how serverless Postgres is going.
• Think is SQL, avoid writing SQL in a top to bottom approach — A nice post about the mismatch between the logical query processing order and the syntaxic order of SQL queries.
• Parquet best practices: the art of filtering — How to leverage Parquet filtering to save processing time.
• Optimizing dbt development with Snowflake clones — dbt development in large data warehouse can become expensive if you ask every dbt developer to dbt run the whole SQL tree. Montreal analytics propose a solution with Snowflake db clones. You can also use the dbt --defer option which does something similar.
• What if we use CHANGELOG in our data projects? — This is important to have a consistent nomenclature when naming commits and changes, sadly the same should apply to dashboards, but hard to do.
• How we deployed a simple wildlife monitoring system on Google Cloud — Artefact engineering a serverless platform on GCP to do wildlife monitoring.
• 📺 Seinfeld-like sitcom generated by AI 24/7 live on Twitch — This is amazing how far we are able to go today in terms of content generation.

Data Economy 💰

• Select Star raises $15m in Series A. Select Star is another data catalog that automatically connects to your tools and provides the usual data catalog UI based on a search bar with metadata management inside. Nothing new under the sun.

See you next week ❤️.

PS: and sorry it was a fast data news today. I have a big presentation to prepare for Monday. I wish you a great weekend.

Dear Data News readers it's a joy every week to write this newsletter, we are slowly approaching the second birthday of this newsletter. In order to celebrate this together I'd love to receive your stories about data—can be short or long, anonymous or not. This is an open box, just write me with what you have on the mind and I'll bundle an edition with it.

This is fun because I'm usually not someone who's good at having habits. Every week to be honest I get hit by Friday. I don't write in advance. Every week you get a taste of my current mood. I often try to sync my travels on Fridays, even if internet is terrible in the train, this is still a good way to fill the +8 hours travel time I'm used to.

Today I take the following commitment: I will never use any generative algorithm to write something in the newsletter. Fun story because one year ago I had an intern working with me on the blog to whom I had given the task to write code that was able to learn from my writings to generate a Data News edition. One year later, different views. In ChatGPT times, my idea is just boring.

On the other side, at the moment I'm not really organised to check if articles that I share have been totally written by humans, but same shit, I'll do as much as I can to avoid sharing empty articles like I've always did. It might be a good use-case for GPTZero.

As a data professional this is probably the height to not want to use AI. But right now the field feels like when cryptocurrencies arrived. Awesome raw ideas with sharks circling around waiting for a new productivity highness.

PS: last week I did a—bad—joke about Apache naming and a reader pointed me an article about the ASF and non-Indigenous appropriation.

This is enough about my life, let's jump to the news.

Back to the roots, a few engineering articles

I did not know how to put together these articles, so here a few loose articles. In my manage and schedule dbt guide in a nutshell I say that in dbt projects you have 2 lifecycles. The first one is the developing experience and the second is the dbt runtime. It means you have to run dbt somewhere:

• Jonathan proposed a creative way to do it in Dagster — every dbt model is a software defined asset, which means that the whole data chain is reactive and every model are refreshed on a trigger rather than on a cron-based schedule.
• Astronomer team developed an awesome library that is meant to translate dbt DAG to Airflow DAG: astronomer-cosmos. You either have a DbtDag object or a DbtTaskGroup, that dynamically creates an Airflow DAG from your dbt project. It looks very promising. Cosmos reads dbt models files and do not use the manifest.

In term of data modeling ThoughtSpot wrote about the best data modeling methods and Chad—the pope of Data Contracts—wrote about data contracts for the warehouse, mainly it shift the responsibilities to data producers in order to enforce schema and semantic, but in the data world it is sometimes rather an utopia. Producers are often software teams that, sadly, does not care about data teams.

Finally Noah shared how he improved data quality by removing 80% of the tests and Ronald proposed a framework to create data products in Airflow.

Data people are creatives 🪄

This is a new category that will appear in the next Data News edition. In this category I'll share stuff that we can do with data. The idea is to inspire others by promoting the end use-case rather than just the technology. I'll be more than happy to share what you do.

• Are Airbnb guests less energy efficient than their host? — Max tries to find if Airbnb guests energy consumption is higher than the hosts' one. I'm always amazed by straight to the point analyses like this.
• Automated object detection in CSGO — PandaScore, a French company that generates data from public—and probably private—e-sports data, showcases how they used OCR to get data in CSGO live streams. I did something similar last year on Teamfight Tactics.
• Football data pipeline project — This is more a technical walk-through to build a Streamlit dashboard on the Premier League. Still this is interesting.

Fast News ⚡️

• Airbyte announced a free sync plan. Starting today the connectors that are in alpha and beta will be free to use in Airbyte Cloud. It needs only one side of the sync to be in alpha/beta to have it for free. Once GA you'll have 2 weeks before being charged.
• Earlier in January Fivetran also announced a free plan. Starting February you will be able to use it to sync up to 500k distinct rows for free plus other perks.
• SQLAlchemy 2.0 released — This is a major release with a lot of breaking changes. As I'm far from being an expert in SQLAlchemy  I can't say more than it seems to be shiny new better ORM.
• Metaplane announced data tests preview in pull requests — This is a way to compare the SQL code in a PR to the live production data to see directly in Github what have changed. It gives ideas.
• Snowflake released min_by and max_by functions — With these new min/max functions you can in a select statement get the first/last status for an id. This is a great shortcut.
• How to compare two tables for quality in BigQuery — Giorgios propose a simple query to compare 2 tables in BigQuery. If you are a Snowflake user there is a minus operation to do it even easier and if you use dbt you can avoid this boilerplate by use dbt_utils.equality function.
• How misused terminology is damaging the data field — The title is a bit exaggerated and terminology gatekeeping damage even more the field. Actually in the end we all do stuff with data, right?
• How you can have impact as an Engineering Manager — Good question and good article. In a nutshell it's about your team and other teams and how you interact with other people in terms of behaviour, processes and practices.

Data Economy 💰

• Microsoft finally announced their "multi-billion dollar" investment—probably $10b—in OpenAI. Nothing more to say, you might have guessed my opinion in the introduction.
• whalesync raises $1.8m pre-seed to create another data movement SaaS that is connectors based. With bidirectional connectors. The difference with similar product is the possibility to also sync to Postgres. Usually tools like this only do it between SaaS. The enable also web page creation automation for SEO, which is unrelated to the data movement business.
• Komprise raises $37m Series D to build yet another all-in-one data platform to do everything about data.

See you next week ❤️.

Hey, new Friday, new Data News edition. I'm so happy to see new people coming every week. Thank you for every recommendation you do about the blog or the Data News. This kindness for my content gives me wings.

This week I don't want to be late, so let's start the weekly wrap-up. I got less inspired this week, it means shorter edition.

As a side note we are looking for speakers for a late February Airflow Meetup. Still open topics, so whatever you want to share—have to be related to Airflow at some point—we'll be happy to welcome you as speaker.

The current state of data

This week Benjamin Rogojan livestreamed an online conference featuring awesome data voices: state of data infra. Matt wrote his takeaways on Medium about the conference. In parallel Ben released the results of a survey about data infras he run among his followers. The main thing to notice is that the average company is a Finance company using Airflow with BigQuery and they struggle—like you probably—to hire people.

This is also time for my views about the state of data. After 2 years of running the newsletter writing every week about trends and following "influencers" for you I'm bored. If I'm being honest I'm French and probably I was born bored, but still. When I was a young professional I was so hype by new technologies, right now it's harder for me. I personally feel that data ecosystem is in a in-between state. In between the Hadoop era, the modern data stack and the machine learning revolution everyone—but me—waits for. But, funny, in the end we are still copying data from database to database by using CSVs, like 40 years ago.

If we go back to this week articles:

• Matt Hawkins tried to find the origins of the term "modern data stack".
• Pedram wrote about the state of data testing — in the end of the article obviously because it's on Datafold blog they share data-diff, still the article is relevant near the four facets of data quality: accuracy, completeness, consistency and integrity.
• Apache Doris, to me it looks like a character from Nemo, actually it's the new real-time warehouse of the Apache Foundation.
• There is an introduction post about DataHub — when you look at what you have to run to launch a data catalog: 4 components and 4 different data storage. Don't be surprised if no ones uses data catalogs. When I think that some people are saying Airflow is complex to launch.

In a nutshell I just want to solve problems and empower people with what I build and I don't care if my stack is a post-modern aquarium, I just want it to be blazingly boring.

Data modeling techniques

Data modeling as of today is probably the most important skills of every data practitioner. We don't really care about your role or your tools. This is about optimisation. Optimisation at different levels, it can be performance optimisation, costs optimisation, business understanding optimisation. Yeah, in fine, optimisation.

There are many techniques out there to do it, I don't want to enumerate them because that's not really the intention. Still, aim for simplicity, keep it simple stupid and think about your consumers.

PS: this feedback about the Medallion architecture—bronze, silver, gold—might be interesting for you.

Fast News ⚡️

• Why I moved my dbt workloads to GitHub and saved over $65,000 — With the dbt Cloud price increase I already shared companies started to look for innovative way to run dbt. This time this is an example demonstrating that you can do it in Github Actions.
• 10 Common Misconceptions about Airflow — Airflow grown a lot and probably users that lost faith in Airflow a while back while never come back. Still this post tries to revalidate Airflow. Shortly, in recent Airflow versions it's easy for instance to get started, the UI is great—and tbh always has been, the scheduler is stable.
• Lights on Versatile Data Kit — A YouTube video about a tool developed by vmware that is an alternative to dbt—yeah, sorry this is the best way to define it.
• Data Engineering job market in Stockholm — Alexander shared on a personal blog his job research in Sweden. Spoiler: out of 43 application he got 6 offers. This is a short post but describes well his experience.
• Why the super rich are inevitable — Except the fact that we should eat the rich. I just want to talk about the way the information is displayed. Alvin—the author—explained economical concept with a scrollable visualisation and with some simulation to help people understand concepts. I found it very pleasant and it looks like something data teams could do to package data analyses.
• All you need to know to get started with Vertex AI Pipelines — Will people continue to do Data Science by themselves in 2023? Probably not like before and with more APIs in it. For that you can follow this overview about Vertex AI—the Google Cloud Platform manage machine learning product.
• BigQuery Ingestion-Time Partitioning and Partition Copy With dbt — Christophe from Teads wrapped-up how they contributed to dbt 1.4 by adding ingestion-time partitioned table support for BigQuery.
• Don't target 100% coverage — Yes. This is about JavaScript, but you can still send it to your boss that is asking for 100% coverage for data tests.
• Choose your adventure: How changing how you spend your free time can genuinely make you feel like you have more of it and take care of your well-being.

Data Economy 💰

• Cumul.io raises €10m Series A. Embedded analytics is the capabilities to introduce Business Intelligence apps within "traditional" software platforms like SaaS application or public website. Cumul.io provides a complete SDK to integrates Analytics in your app. Either by doing it yourself either by letting your customer do it.
• Lay-offs are continuing at big tech. Google and Microsoft announced respectively 6% and ~5% jobs cuts. According to layoffs.fyi in January this year around 40k people got laid off in tech, it represents 25% of last year total lay-offs—150k. If it happened to you recently, you can reach me, I'll do whatever I can do to help you.

See you next week ❤️.

Hey. I have busy weeks, I'm sorry Data News are coming on Saturday again. This is a bit hard to travel by train, work and write at the same time. Plus I'm a fast context switcher, so it piles up. Also a few of you have sent me messages recently and I've not yet answered, I see you and I did not forget you. Now that I'm back in Berlin it'll be easy.

Last week we organised the first Paris Airflow meetup of the year. It was a round table that I've moderated with Benoit Pimpaud, Furcy Pin and Marc Lamberti. We talked about the place of Airflow in 2023, the unbundling of Airflow and the best way to run your Airflow DAGs today.

The discussion was in French and the recording will be released next week. In the meantime you can still check my article Using Airflow the wrong way that summarize a bit the operators vs. containers debate. During the meetup we did not talk about Airflow alternatives, currently Mage is the rising tool that everyone tries out as a replacement for Airflow?

Enjoy the Data News.

Polars—Pandas are freezing

Recently influencers are betting that Rust will be the de-facto language in data engineering. The history repeat, we've seen it with Scala, Go or even Julia at some scale. In the end Python and SQL are still here for good. But with Rust the approach is different. The idea is not to replace Python but to replace the underlying bindings that are used by Python libraries.

And it makes sense, for instance ruff a Python linter that is build in Rust that claims to be extremely faster that the usual stuff.

On the data processing side there is Polars, a DataFrame library that could replace pandas. Let's have a quick look at it. In this overview I'll not talk about performance because I don't have the time to do a proper benchmark—and I've never done this. Just the experience of a beginner that knows pandas very well.

The installation is pretty straight forward, you can do it with pip. When compared to pandas this is awesome because it seems polars as no dependencies so it does not need to build wheels like pandas.

pip install polars

Regarding the imports the documentation continues to treat me well. It looks like stuff I know with pandas.

import polars as pl

Then I can do my first CSV import, in the example I load a French railway open dataset about lost and found objects in stations.

df = pl.read_csv("lost-objects-stations.csv", sep=";")

Then you can use the same code as pandas to select the data (head, ["col"], etc.). I want now to try a group by.

df.groupby("Station").agg([pl.count()]).sort("count", reverse=True)

# Same code but it pandas
df.groupby("Station")["Date"].count().sort_values(ascending=False)

And lastly (because if I continue the newsletter gonna be too long for you to read), I just try to convert a str Series to datetime.

df = df.with_columns(
	df["Date"].str.strptime(pl.Datetime, fmt="%Y-%m-%dT%H:%M:%S%Z").alias("Date")
)

# Same code in Pandas
pd_df["Date"] = pd.to_datetime(pd_df["Date"], format="%Y-%m-%dT%H:%M:%S%Z", utc=True)

To be honest I try polars for 15 minutes and I can already see how I could switch to it if I have the guaranty it is way faster. APIs are quite similar so I'm far from being lost.

🫠 If after this small introduction you want a deeper comparison of Polars you can check Modern Polars by Kevin Heavey or a 40 minutes YouTube video that explains Polars internals.

Hiring processes

The current state of the data market is weird. At the same time we have a lot of lay-offs and a lot of companies that are still looking for data folks. Which is often a critical hiring for them, but they struggle. There is a huge gap between jobs, what folks are looking for and what companies are looking for.

This week Teads shared their engineering hiring process. The process is not focused entirely on data, but still this is relevant because it can give ideas to hiring companies or junior looking for advices. They have a short 4 touchpoint interview which looks like a good compromise.

When focusing on data more, Galen wrote about what he looks for in data analyst candidates. One of the most interesting advice he gives that I can press is: you should spend time mastering the technologies you've chosen. With the current state of data this is easy to loose focus, so listen to this intervention. Stop chasing the last data trends and master what you daily use. I think that mastery in one domain can be easily transferable in other domain.

AI Saturday

• How DoorDash upgraded a heuristic with ML to save thousands of cancelled orders — When running a marketplace this is an usual problem to deal with. DoorDash shares the models they used to replace their intuition.
• 👀 Building a defensible Machine Learning company in the age of foundation models — This article is very complete, this is probably the best written article about the actual trends in the machine learning. A whole ecosystem is shifting from build it yourself to consume foundation models and APIs built by others.

Fast News ⚡️

• Announcing dbt-fal adapter — I shared fal months ago when they launched. I'm still on their Discord and when dbt finally announced Python models support I was a bit sceptical about fal offering the same thing. But because of the small scoped dbt solution—Python code only runs in the warehouse. With this release you can really mix Python and SQL code.
• How we cut our Databricks costs by 50% — We can always find optimization in our cloud setup to save costs.
• How to land a job in progressive data — If you want to use your skills to Do Good you have to look at Brittany's post about progressive data.
• Statement of objections issued against Google’s data processing terms — The German office competition regulation said that Google should do more in being explicit about how the data in processed to help Google business.
• How query engines work — This is a web book that explains how query engines work. I did not read it yet but it looks great.
• Analysis of Confluent buying Immerok — Jesse Anderson analyses last week news of Confluent (Kafka) buying Immerok (Flink) and what it implies in the real-time low-level technologies competition between Kafka / Flink / Spark.
• On Data Contracts, Data Products and Muesli — Another post on data contracts, a bit to long for me to read it. Sorry.
• Extracting, converting, and querying data in local files using clickhouse-local — This is awesome how fat Clickhouse can go. Looks like a wider alternative to DuckDB but also a good trend for other warehouse: provide a local experience that lives out of the cloud.
• ByteGraph: A Graph Database for TikTok — ByteGraph is the open-source graph database developed by the company behind TikTok. This article shows you what are the key concepts to understand it. To be honest I'm quite impressed with the first line stating that it has been designed to support OLAP, OLSP and OLTP workloads.

Data Economy 💰

• Metaplane raises $8.4m seed funding. This is a bold claim, Metaplane wants to be the Datadog for data. Operating in the data observability space the usual set of features: tests, data quality monitoring based on historical data, lineage and alerts.
• XetHub raises $7.5m seed round. XetHub brings git to data files management. They support up to 1TB repositories with git-like commands (checkout, push, commit, pull, etc.). I think that XetHub is super useful when in data science we need to keep the data alongside the models. When commit a change on a big file their repo hub summarise data diffs.
• Generative AIs are booming, following all the stories with possible Microsoft $10b investment in OpenAI, Seek AI raises $7.5m seed round. Seek AI promise is a prompt where you ask your data anything and the AI responds on top of the raw data directly.

See you next week, maybe on Friday ❤️.

Happy new year 🎆. For those who were already subscribed at the start of last year I tried to put resolutions and objectives for the year that I did not succeed to follow. The year was so different to what I was expected. Maybe this is an excuse. Anyway I did not reach my goals. What about if we don't care for this year?

Still, what happened was awesome and here a small personal / professional throwback:

• I worked for the French public sector as a freelancer: tax administration and education ministry. It makes sense for me and this is something I also really care about.
• Bootstrapped a coaching activity with companies and individuals—this is a new exercise but I feel it's close to management that I can't do in freelance.
• I moved to Berlin, talk to my first meetup ever in English, met awesome people there but I'd like to met more.
• We restarted the Paris Airflow Meetup and people liked it. There are still a few seats left for next Tuesday meetup.
• I started to pay myself after 1 year and half of unemployment pay. This is maybe my main source of stress. Will I be next year able to find missions to pay me for the whole year? My business plan asks for 100k€ in revenue.
• This year I deeply learned Superset—adding the tools to my tools expertise list.
• My written content got around 100k views last year. The blog crossed the 2000 members mark (❤️) and I won the best data science newsletter award. On LinkedIn and Twitter I multiplied 2 my followers. Everywhere I was starting from the bottom and now we're here.
• I talked in Robin's podcast about the newsletter and my data engineering journey.

I'm also sorry to start the year late with my newsletter sending. On the last 3 days I was teaching DataOps at a French school and I did not manage to find the time to write to you. And you know what, this is the first time in 7 years of teaching that more than 80% of the class wants to become data engineer.

As a conclusion of this introduction, I want to thank everyone reading this newsletter and sharing feedback or good words about it. It means so much to me and it fuels me. For sure the Data News will be here for a new year and new stuff is coming.

Time for the news—I have around 30 links to share today so it might be less opinionated than usual. Happy reading.

Data team role

I really like all the thoughts around data team role, missions, vision and strategy. I still think that we did not reach any form of consensus about data teams. In term of tooling the modern data stack proposed something that works but the modern data team is still behind. Here the latest ideas I've seen this week:

• Should software teams start learning from analytics engineers? — Petr reverse the common idea where analytics teams should learn from software. Why actually everyone is just a part of engineering that helps all of us getting better at data and software.
• Data Teams as support teams — Chad from Zendesk thinks that data teams are often misaligned with customers and because of the supportive nature of the relationship between something does not work. He then digs in modeling and analytics value to understand what are the impact on the relationship—this is fun to read that someone from Zendesk does not want to be a support team!
• ❤️ Elbows of data — This is a good follow-up to Chad's post. Katie toss the term elbow of data who are "folks who have insisted on being involved in driving the company forward, whether they were invited to or not". When we do data we have skills and understanding to help our company. Once again our main role should be to empower stakeholders.

Data Science Saturday 🤖

• How to invest better in acquisition channels? — Marianne detailed how data science helped Qonto understanding their acquisition channels investment.
• Data science has a tool obsession.
• Selecting the best image for each merchant using exploration and ml.
• Introduction to Graph Machine Learning (related Grab Graph service platform).

We are in a middle of ChatGPT frenzy. A new day means a new interrogation about our future. Our future as developers but our future as humans. ChatGPT is seeking for money at high valuation amount. Still, should we trust OpenAI to be open as the name is saying 🫠?

If you want to understand better what's behind ChatGPT you can have a look at minGPT a minimal re-implementation via PyTorch.

At the same time it seems that following initial Microsoft investment in OpenAI, Bing will use GPT models to improve their text and images search. Who would have say that Bing would kill Google?

Final note: How China is building a parallel generative AI universe.

Fast News ⚡️

• Why I'm using (Neo)vim as a Data Engineer and Writer in 2023 — If you want to take 2023 beginning as a sign to move to vim Simon wrote a great post for you.
• CircleCI’s unnoticed holiday security breach — CircleCI had a security breach a few days ago.
• What if we rewrite everything? — Navigating through the technical debt and spending our entire career doing the same stuff over again. What is the right strategy to have? Probably Keep It Simple Stupid.
• Why It’s So Hard to Become a Staff Engineer — A feedback to help people bringing the gap between senior and staff. I think this is even relevant to data world.
• Introducing ADBC: Database Access for Apache Arrow — When I see "minimal-overhead alternative to JDBC/ODBC for analytical applications" I'm instantly in. My all professional life I've heard architect saying JDBC is bad so if something better can come so we don't talk about it. You can also listen a related podcast about Arrow vision.
• Recap: a data catalog for people who hate data catalogs — This one hurts. You may have noticed if you read me that I'm not very tender with current state of data catalogs. This week Chris started a small footprint data catalog written in Python called Recap. I'll have a look at it soon.
• Observability, Tick — Nigel wrote a small post detailing how a smal startup can do observability without spending a lot of money.

Data Economy 💰

The economic situation is obviously not at his best. Previously data was not always impacted about the difficulties but it's also coming to the data world. That's why data fundraising will become a data economy wrap-up.

• Astronomer laid off  20% of their staff—which represents 76 folks—and moved from a co-CEO structure to only one CEO. I appreciate the transparency effort that has been done to make this note public. But I still struggle to see Astronomer value and strategy, but this is hard because Astro hires a lot of core Airflow contributors and have important contributions to the data community.
• Salesforce is laying off 10% of their staff—roughly 8000 people—including folks at Tableau. They acquired Tableau in 2019 and analysts are saying that Tableau ex-employees are more often impacted by the lay-off.

In search of consolidation and levers to do companies are also merging:

• Qlik wants to acquire Talend. Qlik and Talend are two old BI giants. The first one has been founded in 1993 and the second one in 2005. They had obviously been challenged by the cloud vendors and the modern data stack vision that does not include them.
• Confluent signed a deal to acquired Immerok. They are respectively the home companies of Kafka and Flink. This is to be honest a natural move because the two technologies works at the best together and ksqlDB never took the place it should have been in the market. Sadly also right now they are challenged by real-time tooling that is way easier to setup.

Finally a fundraise:

• Chaos Genius is raising $3.3m Seed Round. They propose an optimisation platform for Snowflake to help you save up to 30% of your warehouse costs.

See you next week ❤️.

You can listen the podcast on all the platforms:

• Apple Podcasts/Itunes: bit.ly/3X3qlOQ
• Spotify: bit.ly/3GnfWXb
• Google Podcast: bit.ly/3VPSAPR
• Deezer: bit.ly/3ZjWsM6

Hey you, this is the last article of the year and it's gonna be about the articles and trends that made 2022 according to me. You'll see articles that I've already share during the year.

Once again thank you everyone for your support this year and see you next week for the first Data News of 2023. Sorry for the delay, I had a blank page syndrome today. Now let's jump to my selection.

ANALYTICS ENGINEERING

We have to be honest in 2022 Analytics Engineering shaped up the data field and concentrated a lot of data discussions. Analytics Engineering can be seen as a renaming of the BI Engineering, if we look at it more precisely it mainly comes out of the data roles specialisation. Analytics Engineers is a specialized role between the Data Engineer and the Data Analyst. Madison had a look a job posting to see what are the skills companies really want in Analytics Engineers.

Analytics engineers provide clean data sets to end users, modeling data in a way that empowers end users to answer their own questions. [...], an analytics engineer spends their time transforming, testing, deploying, and documenting data. Analytics engineers apply software engineering best practices like version control and continuous integration to the analytics code base.¹

Analytics Engineering brought back light on data modeling. Preset wrote a gentle introduction to data modeling. In a nutshell data modeling is the techniques we can use to structure the data in data warehouses. Nowadays we have:

• Dimensional modeling — Introduced in 1996 by Ralph Kimball. We often use the Snowflake Schema or the Star Schema (that is a special case of the previous one, here Snowflake is not the data warehouse technology but more the shape of the table relationships—drawing a snowflake).
• Entity modeling — Introduced by Bill Inmon. In this methodology you use the 3NF (third normal form) to model your business entities to avoid redundancy. This approach is less flexible than the previous one.
• OBT—One big table ; I don't really know who introduced OBT except the fact that Fivetran mentioned it in 2020. This is often the easiest approach to start. Everything in one table, denormalised.

As a final note, a Reddit thread discussing is Kimball's Dimensional Modelling dead in 2022?

In order to complete the AE articles list here a few I recommend as the best 2022 analytics engineering articles:

• Factless Fact table — not so absurd it may sound at first
• Testing: Our assertions vs. reality — Probably the best talk of 2022 about testing. This is a YouTube video.
• Super Tables: The road to building reliable and discoverable data products — LinkedIn data modeling choices explained and the introduction of Super Tables concept.
• Understanding the Snowflake Query Optimizer — In order to become better at data modeling you'll need to understand how the underlying warehouse engine is working. This article is a good way to go to understand how Snowflake works.
• Stop using so many CTEs — This is a vendor article that showcases "Chained CTEs" in Hex. Still relevant because in today's data world CTE are everywhere and a lot of data transformations are just a bunch of SQL queries around a few hundreds of lines with a lot of CTEs. But CTEs are untestable blocs of code.
• 7 Antifragile Principles for a Successful Data Warehouse — Something to look at to create a healthy data warehouse.
• My guide about managing and scheduling dbt from dev to production.

DATA TEAMS

3 piece of content that I feel are relevant and not really trendy. This is more something long term that we have to have in mind:

• Building more effective data teams using the JTBD framework — Data teams are still in between with no really good practices when it comes to routines or organisation. The Job To Be Done framework can be something to look at.
• Building Modern Data Teams — The most complete resource hub with around 40 articles on how to build data teams, data strategies or think about data work / hiring.
• Emerging Architectures for Modern Data Infrastructure — The updated version of the a16z vision about modern data infrastructure.

ENGINEERING

In loose, a few of the best 2022 data engineering articles:

• Introducing Software-Defined Assets — Best article to rethink the data pipelines and to consider datasets like assets.
• The rise of the data reliability engineer — Data Engineers have a large part of their daily job that is close to SREs while not being SREs.
• The best website to understand visually machine learning models.
• Joe Reis blog. He started blogging recently after writing the excellent Fundamental of Data Engineering book. I often surprise myself agreeing to everything he says, if you have to follow someone except me I think it should be him.
• The many layers of data lineage — The best metaphor to understand what you can do with data lineage.
• Design Patterns in Machine Learning Code and Systems — Because we need design pattern even if I disliked the design pattern classes I had back in engineering school.
• 3 tips to take back control of your time.

A GLIMPSE INTO THE FUTURE

This year people talked about a lot of things, with no research here what I can remember:

• Data Mesh — The Mesh has been assimilated and tried by multiples organisations, what we've seen is that it requires a minimal size to be started, we have yet to figure out if the organisational changes are worth it.
• Data contracts — An interface between the data producers and the data consumers. The interface can take multiple form, we often summarize it as a schema registry. Very useful in a mesh organisation.
• Semantic Layer / Metric Layer / Headless BI — "Something“² between the data warehouse and the BI tool that will probably shape trends next year.
• Unbundling of Airflow — This is year many Airflow alternatives went public, all with their own vision and great promises, in addition the one-dag-to-rule-them-all strategy has been challenged and execution has also been delocated to other system leaving Airflow like an empty shell. But in the end he'll be back.
• GPT-3 applications — It has the potential to revolutionize industries through automation and augmenting human intelligence, but has also raised concerns about its potential negative impact on employment (this bullet has been generated by ChatGPT).

Now that I've said this, I think that 3 technologies will shape data engineering next year:

• Wasm — WebAssembly is a portable compilation target in the browser. In human words it means you can run your favourite language code in a Firefox tab. One example is PyScript, that allows us to run Python in HTML. Thanks to Wasm we can use a decentralised power: your stakeholders laptops.
• DuckDB — A single node in-memory OLAP database. We did not see yet the full potential. What I think about DuckDB.
• Dagger — A programmable CI/CD engine that you can run everywhere.

1. What is an analytics engineer? (Claire Carroll)
2. Semantic Layer is more than just something. To be honest for the moment I take it sarcastically, because I'm not sure this is something really important—at least when I see my own French market.

Hey you, if you just subscribed yesterday to the Data News I wish you a warm welcome ❤️‍🔥. The Data News is your Friday weekly data curation in which I select for you the most interesting—according to me—data articles of the last week. I hope you'll enjoy it ✨.

Christmas is coming, so whether you celebrate it or not, I wish you a great end of the year and good time with family and/or friends. There will be a last Data News next week that will be my 10 2022' must-read articles. In the meantime you can read Prukalpa's 5 must-read data blogs from 2022.

The Advent of Data is also coming to an end tomorrow, it has been an awesome ride, I'm so happy we put together such an awesome list of content and I'm so grateful to the 24 creators who accepted the rules and wrote something for this first year. I'll do a wrap-up of the Advent in January to celebrate what we achieved together.

Remember: the Advent of Data was your daily spark of data joy in December. Every day a new data article has been published by a data creator.

Guide—manage and schedule dbt

I published 2 days ago the most complete guide about dbt management and scheduling, in case you missed it you have to check it out! Original deep post that are exclusive to Data News members are something I'm willing to do more next year, to bring you additional value to this newsletter.

Next year I plan to talk about:

• Data engineering and analytics engineering career paths
• State of the data integration—related to another 2023 project 📚
• We have too many choices, my framework to take a decision
• Something you want me to write on?

Let's go back to dbt. So this guide in a nutshell will give you ideas on how you can manage dbt repository(ies)/project(s), what you have to think about to provide a top-notch developer experience, how to host and schedule your dbt code.

I'm really proud of the development experience part of the guide because I think that this is a still a unresolved part of every dbt project, something is still broken. From the first contact, the local installation, the (web?) IDE, the useless copy-pastes, the code reviews, the tooling to the development environments there is a lot to say.

As an extension this week two great articles have been written about custom dbt setups. Monzo team detailed how they created their own framework on top of dbt to follow their growth and Albert from Superside explained how they migrated from dbt Cloud to a custom setup with CI/CD, S3, Docker and Airflow.

PS: small question, I did not email you for the guide, would you have wanted to receive an email for it?

Give yourself a book Christmas 🎁

If you need gift ideas for yourself I have a few books to propose to you. The selection is a mix between 2 things I love—data engineering and visualisation.

Here the selection 📚:

• Fundamental of Data Engineering — It rapidly became a best seller, Joe and Matt wrote a greatly structured book that covers all the data engineering topics, I firmly recommend it from juniors to seniors.
• The Data Warehouse Toolkit, 3rd Edition — With the rapid rise of the Analytics Engineering role the data modelisation came back as number one priority for a lot of data teams. Dimensional modeling has been a reference for years which is the main purpose of the Kimball method.
• Effective Data Storytelling — Data storytelling have been something really trendy in the last years, but in a lot of data teams because of the dashboard constraint we often lack of creativity, context or storytelling. This book is a must-read if you want to drive actions with data.

Obviously there are more books that went release this year that are awesome, but I just mentioned what you should absolutely have.

Fast News ⚡️

• Functional Data Engineering - A Blueprint — Ananth, from Data Engineering Weekly, the best data engineering newsletter, wrote a great follow-up to Maxime's functional data engineering post. In the post he shows how we can apply entity and event schematisation to Lakehouse architecture.
• Tips for hiring junior Data Engineers — MOST. IMPORTANT. POST. OF. 2022. Every data engineer has been junior once, this is important not to gatekeep others by forgetting we once knew nothing about data engineering. This is our duty as senior to hire juniors and to help them. I guaranty you this is the most satisfying feeling, aside from my recommendation the article is awesome and speaks the truth. Last point I think the max ratio is 3 juniors for 1 senior.
• 💥 BigQuery data lineage — It looks like a something huge, but I'm note sure tbh. Soon we will have a data lineage tab in the BigQuery UI. In order to have it you'll have to activate Data Catalog/Dataplex. This is in public preview.
• Panel discussion about licenses in open-source, relation with VCs, etc. Between Doug Cutting (Hadoop co-founder), Maxime Beauchemin (Airflow & Superset creator) and David Nalley (Apache Foundation president). This is really geeky about licensing but few of you might find it interesting.
• Maybe Snowflake isn’t for you! — Thoughts around the expensive price of Snowflake that reminds Oracle. tl;dr: take control back of your tools to find the holy added value every data people has spoken about.
• Managed transcription with OpenAI whisper and Hugging Face inference endpoints — I don't even understand the first chart of the article but it looks cool.
• Personal Finances with Airflow, Docker, Great Expectations and Metabase — when you're a nerd and you like to extend the data pleasure on Saturday.
• StackOverflow 2022 developer survey — 15% of respondents developers are in data roles (but they can have multiple hats) and when it comes to technologies SQL and Python come just after JavaScript and HTML/CSS that are just everywhere. Last number is that Spark is the framework that pays the most nowadays—not a good sign for Spark future.
• Working with large CSV files in Python from Scratch — Good pattern to optimize your pandas computes by leveraging partionning.
• Data Reprocessing Pipeline in Asset Management Platform @Netflix (I did not read it, but I want to keep a track of it—looks interesting).

Data Fundraising 💰

• Qualytics raised $2.5m Seed round. The newcomers in the data quality space that is already quite crowded. Qualytics is a small team—8 employees on LinkedIn—based in the US proposing a "data firewall" that protects and compares your data to detect drifts, anomalies and history discrepancies.

See you next week for the last edition of 2022 ❤️. Enjoy holidays.

But this article is not about the pricing which can be very subjective depending on the context—what is 1200$ for dev tooling when you pay them more than $150k per year, yes it's US-centric but relevant.

Let's go deeper than this to list what are today the options out there to schedule dbt in production. We will also cover what it means to manage dbt¹. This article will be written like a guide that aim to be exhaustive by listing all the possible solutions but if you feel I missed something do not hesitate to ping me.

dbt, a small reminder

Everyone—incl. me—is speaking about dbt, but what the heck is dbt. In simple words dbt Core is a framework that helps you organise all your warehouse transformation. The framework usage grew a lot over the last years. It's important to say that a lot of the usages we have today have not been initially designed by Fishtown Analytics.

At first dbt transformations were only SQL queries, but in the recent version with supported warehouse it has been possible to add Python transformations. dbt responsibility is to transform the collection of queries into an usable DAG. The dependencies between the queries are humanly defined—which means prone to error—thanks to 2 handful function source and ref. These 2 functions are called macros because they use Jinja, a Python templating engine, in dbt macros transform Python+SQL code in SQL, we can say that we have templated queries.

Everything I just mention before we can consider it static. If we do a parallel with software development, this is your codebase. Python and SQL together in the dbt framework is your codebase. You can do development on your codebase. In order to go in production you'll have to manage and schedule dbt.

To manage dbt you will have to answer multiple questions, but mainly dbt management is how the data team develop on dbt, how the project is validated/deployed, how you get alerted when something goes wrong, how you monitor.

In addition the dbt management you will have to find the place where dbt will be scheduled. Where dbt will run. dbt scheduling is tricky but not really complicated. If you followed what we've just seen dbt is in a SQL queries orchestrator. dbt does not run the queries, all the queries are sent to the underlying warehouse, which means that theoretically dbt does not need a lot of computing power—CPU/RAM—because he is only sending SQL queries sequentially to your data warehouse which does the work.

Obviously every dbt project has been designed differently but if we simplify the workflow all dbt project will need at some point to run one or multiple dbt CLI commands.

In this guide we will first see how we can manage dbt, i.e. git structures, how to code, the CI/CD and the deployment then in the second part how we schedule dbt code,i.e. on which server and triggers.

How to manage dbt 🧑‍🔧

One of the dbt founding principle is to bring software engineering practices to the data development work, especially into SQL development world. In order to follow-up on this we will try to treat the workflow like an engineering project, even if sometimes it could feel over-engineered.

You have to consider development and deployment when managing dbt project(s):

• Like every engineering project the management will obviously start with a git repository—depending on your scale it can be multiple repositories, but if you're just starting I recommend you to go with a single one.
• The next step is the development experience. What we often call DevEx. Sometimes data teams forget it. In order to understand this point we have to ask ourselves who are the dbt developers and what do they need.
• After development often comes deployment. It can be deployment in all environment or as a lot of data only in production, because only production exists. But before sending your code to production you still want to validate some stuff, static or not, in the CI/CD pipelines.

Git repositories considerations

This is the everlasting debate of every software engineering team, monorepo or multirepo? This is tightly linked to another dbt related question which is mono-project or multi-project. By default and by design dbt is meant to work with mono-project, but when you're starting to grow or when you want to have clear domain borders the single project can quickly reach limits.

As I said previously if you're just starting with dbt and you're a small team I still recommend you to go with one repo one project. Try first to organise correctly the models folder before trying to structure at a higher level.

The first question you'll probably hit is: how do I put models in different schema/datasets? This is the first step of project organisation. The solution is to override the generate_schema_name macro.

Then if you want to go for multiple projects you'll maybe have to decide how you do the interface between multiple projects, within the dbt toolkit you have 2 solutions:

• Every project can define exposures². Exposures are then a way to define a downstream usage of models in the project. With the exposure nomenclature you can regroup multiple models in the depends_on for an type:application that is supposed to use them. ‌‌‌‌If we imagine some-company, with 2 projects—domains—Ops and Marketing we can have in the Ops exposures the models that we want the outside to be aware of define in it. Then with some kind of automation we can generate sources accordingly in the Marketing project.‌‌‌‌ summarized everything here, to go further GoDataDriven team did an awesome talk at Coalesce explaining how you can achieve this: dbt & data mesh: the perfect pair (?).

• The other solution is to go for a dbt packages structure. In this solution every project—domain—can be installed as a dep in other projects, but I think it will end up in a nightmare of dependencies management. In addition you'll have to be smart in the way you run the models in the end because packages installation could duplicate models execution.

Once project/repo structure has been defined there are still open questions, here are a few:

• How do I structure my dbt models folder? You can opt for the dbt recommended solution or for Brice's recommendations. Personally here my only advice is: don't be shy to create folders to separate concerns.
• One YAML to rule them all — Do you want to create only one big YAML file that describe all the sources and all the models or do you split it. In my opinion sources have to be at schema/database level and YAML models have to be at the model level. So it means one YAML per SQL file.
• Who is the real owner of the git repo? Data engineering or analytics team? — It depends but I'm in favour if possible to give the responsibilities and ownership to the analytics team, dbt is their playground, as a data—platform—engineer it's your responsibility to help them, but it's up to them to learn by doing. Under the hood it also means that dbt project(s) have to be independent(s) from other tools (e.g. dbt repo should not be in the Airflow repo).