New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Don't Update Serialized DAGs in DB if DAG didn't change #9850
Don't Update Serialized DAGs in DB if DAG didn't change #9850
Conversation
We should not update the "last_updated" column unncessarily. This is first of few optimizations to DAG Serialization that would also aid in DAG Versioning
| @@ -120,5 +150,5 @@ def test_bulk_sync_to_db(self): | |||
| dags = [ | |||
| DAG("dag_1"), DAG("dag_2"), DAG("dag_3"), | |||
| ] | |||
| with assert_queries_count(7): | |||
| with assert_queries_count(10): | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will optimize the above queries to lower this count in followup PRs
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh this is added an extra query for each (bulk) write?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, one to check if DAG is updated or not.
airflow/models/serialized_dag.py
Outdated
| serialized_dag_from_db: SerializedDagModel = ( | ||
| session | ||
| .query(cls) | ||
| .filter(cls.dag_id == dag.dag_id) | ||
| .one_or_none() | ||
| ) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| serialized_dag_from_db: SerializedDagModel = ( | |
| session | |
| .query(cls) | |
| .filter(cls.dag_id == dag.dag_id) | |
| .one_or_none() | |
| ) | |
| serialized_dag_from_db: SerializedDagModel = session.query(cls).get(dag.dag_id) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
get will only provide performance boost if the object is in memory isn't it?
Otherwise under the hood, get calls one which in turn calls one_or_none:
https://github.com/sqlalchemy/sqlalchemy/blob/rel_1_3_17/lib/sqlalchemy/orm/loading.py#L287
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Given we want to check that the SerializedDAG is changed or not, we want to access the DB rather than getting anything from the memory.
I feel .one_or_none is better here but happy to hear your thoughts.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In the AirBnB talk they mentioned about really big DAGs and the need to use compression in DAG serialization for DB representation of the DAG (they implemented it in their fork). I think adding one extra read here might add quite heavy DB throughput. It's an edge case, but possibly an important one and some companies already experiencing it.
I know it would be a bit bigger change (involving DB migration) but possibly we should calculate hash of the DAG and compare it and read it from the DB rather than then DAG itself. That seems like perfectly doable and not that complex.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah already going to do that one in other PR. Calculating hash of the serialized JSON. But want to do it in a separate PR
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cool!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In this case query is better as you will have to replace it with https://docs.sqlalchemy.org/en/13/orm/query.html#sqlalchemy.orm.query.Query. later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated in 453dee8
|
With this change does it still make sense to have the |
At least in this PR it makes sense to keep it to avoid pulling Serialized Dag from DB for comparing if DAG has changed or not |
|
Monday earliest. Trying to recover after the summit :) |
Sure, thanks :) |
Looks good as long as we plan to optimise it in follow-up PRs as discussed :)
We should not update the "last_updated" column unnecessarily. This is first of few optimizations to DAG Serialization that would also aid in DAG Versioning (cherry picked from commit 1a32c45)
We should not update the "last_updated" column unnecessarily. This is first of few optimizations to DAG Serialization that would also aid in DAG Versioning (cherry picked from commit 1a32c45)
We should not update the "last_updated" column unnecessarily. This is first of few optimizations to DAG Serialization that would also aid in DAG Versioning (cherry picked from commit 1a32c45)
We should not update the "last_updated" column unnecessarily. This is first of few optimizations to DAG Serialization that would also aid in DAG Versioning (cherry picked from commit 1a32c45)
We should not update the "last_updated" column unnecessarily. This is first of few optimizations to DAG Serialization that would also aid in DAG Versioning (cherry picked from commit 1a32c45)
We should not update the "last_updated" column unnecessarily. This is first of few optimizations to DAG Serialization that would also aid in DAG Versioning
Make sure to mark the boxes below before creating PR: [x]
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.