Skip to content

Data News — Week 21

Christophe Blefari
Christophe Blefari
2 min read

This week small incremental news but a nice detailed ML article at the end for those that missed it.

Apache Airflow 2.1.0 and official Helm Chart 🎉

Airflow release the first official Helm Chart, with documentation page explaining how to use it for newcomers. If you are interested in a post explaining how to run Airflow on GKE with Helm please drop me a message!

On the other hand huge improvement with this new release, we got a cross-DAG dependency UI and a calendar view in the DAG UI. Here the full changelog.

Airflow Calendar View in 2.1.0

I write SQL everyday can you lint me?

SQLFluff is becoming the best SQL linter in class, here a large overview of all the superpowers that comes with it. I loves the fact the SQLFluff can lint dbt templates out of the box and the rules references that could be a model for all data documentation.

Microsoft announced Power BI in Jupyter notebooks

This week Microsoft announced an embed integration of Power BI inside Jupyter notebooks. If you want to see more announcements from Microsoft the Build 2021 is available online.

Embed PowerBI dashboard in Jupyter (gif from Microsoft article)

Wait, Python is getting faster

Two weeks ago Guido Van Rossum said that the Python Core developpers are aiming to 2x Python performance in the next major release 3.11. I think this is a good news for all data developpers out there!

If you interested in the plan to do it, the detailed roadmap is on Github. In the best scenario we could achieve a 5x uplift.

Secoda announced a dbt integration

Secoda (another Data Discovery tool) announced a dbt Cloud integration to their tool. In this news we can learn two things:

  • We have out there a lot of Data Discovery tools and we need maybe a consolidation
  • dbt is becoming the main transformation tools and everyone trying to integrate it

ML Friday, Machine Learning at Carrefour: tackling promotional product shortage

1 month ago Carrefour posted an article explaining how the tackle promotional product shortage thanks to ML. They tried to explain how they manage the shortage to avoid impacting the NPS.

datanews

Christophe Blefari

Data Engineering Coach that enjoys all kind of data platform.