★ wanayoo — archive 1999 https://github.com/apache/airflow/pull/9850Nouvelle recherche | Portail wanayoo
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't Update Serialized DAGs in DB if DAG didn't change #9850

Merged
merged 2 commits into from Jul 20, 2020

Conversation

kaxil
Copy link
Member

@kaxil kaxil commented Jul 16, 2020

We should not update the "last_updated" column unnecessarily. This is first of few optimizations to DAG Serialization that would also aid in DAG Versioning


Make sure to mark the boxes below before creating PR: [x]

  • Description above provides context of the change
  • Unit tests coverage for changes (not needed for documentation changes)
  • Target Github ISSUE in description if exists
  • Commits follow "How to write a good git commit message"
  • Relevant documentation is updated including usage instructions.
  • I will engage committers as explained in Contribution Workflow Example.

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

We should not update the "last_updated" column unncessarily. This is first of  few optimizations to DAG Serialization that would also aid in DAG Versioning
@@ -120,5 +150,5 @@ def test_bulk_sync_to_db(self):
dags = [
DAG("dag_1"), DAG("dag_2"), DAG("dag_3"),
]
with assert_queries_count(7):
with assert_queries_count(10):
Copy link
Member Author

@kaxil kaxil Jul 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will optimize the above queries to lower this count in followup PRs

Copy link
Member

@ashb ashb Jul 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh this is added an extra query for each (bulk) write?

Copy link
Member Author

@kaxil kaxil Jul 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, one to check if DAG is updated or not.

serialized_dag_from_db: SerializedDagModel = (
session
.query(cls)
.filter(cls.dag_id == dag.dag_id)
.one_or_none()
)
Copy link
Member

@ashb ashb Jul 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
serialized_dag_from_db: SerializedDagModel = (
session
.query(cls)
.filter(cls.dag_id == dag.dag_id)
.one_or_none()
)
serialized_dag_from_db: SerializedDagModel = session.query(cls).get(dag.dag_id)

Copy link
Member

@ashb ashb Jul 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

@kaxil kaxil Jul 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

@kaxil kaxil Jul 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given we want to check that the SerializedDAG is changed or not, we want to access the DB rather than getting anything from the memory.

I feel .one_or_none is better here but happy to hear your thoughts.

Copy link
Member

@potiuk potiuk Jul 16, 2020 •

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the AirBnB talk they mentioned about really big DAGs and the need to use compression in DAG serialization for DB representation of the DAG (they implemented it in their fork). I think adding one extra read here might add quite heavy DB throughput. It's an edge case, but possibly an important one and some companies already experiencing it.

I know it would be a bit bigger change (involving DB migration) but possibly we should calculate hash of the DAG and compare it and read it from the DB rather than then DAG itself. That seems like perfectly doable and not that complex.

Copy link
Member Author

@kaxil kaxil Jul 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah already going to do that one in other PR. Calculating hash of the serialized JSON. But want to do it in a separate PR

Copy link
Member

@potiuk potiuk Jul 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool!

Copy link
Member

@potiuk potiuk Jul 16, 2020 •

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this case query is better as you will have to replace it with https://docs.sqlalchemy.org/en/13/orm/query.html#sqlalchemy.orm.query.Query. later.

Copy link
Member Author

@kaxil kaxil Jul 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in 453dee8

@kaxil kaxil requested a review from ashb Jul 16, 2020
@ashb
Copy link
Member

@ashb ashb commented Jul 17, 2020

With this change does it still make sense to have the min_serialized_dag_update_interval setting?

@kaxil
Copy link
Member Author

@kaxil kaxil commented Jul 17, 2020

With this change does it still make sense to have the min_serialized_dag_update_interval setting?

At least in this PR it makes sense to keep it to avoid pulling Serialized Dag from DB for comparing if DAG has changed or not

@kaxil
Copy link
Member Author

@kaxil kaxil commented Jul 18, 2020

@potiuk - Can you take a look at this please. @ashb is on 6 week's paternal leave. I have addressed all the comments on this and #9851

@potiuk
Copy link
Member

@potiuk potiuk commented Jul 18, 2020

Monday earliest. Trying to recover after the summit :)

@kaxil
Copy link
Member Author

@kaxil kaxil commented Jul 18, 2020

Monday earliest. Trying to recover after the summit :)

Sure, thanks :)

potiuk
potiuk approved these changes Jul 20, 2020
Copy link
Member

@potiuk potiuk left a comment

Looks good as long as we plan to optimise it in follow-up PRs as discussed :)

@kaxil kaxil merged commit 1a32c45 into apache:master Jul 20, 2020
38 checks passed
@kaxil kaxil deleted the dont-update-dags-if-not-needed branch Jul 20, 2020
@kaxil kaxil added this to the Airflow 1.10.12 milestone Jul 21, 2020
kaxil added a commit that referenced this issue Aug 13, 2020
We should not update the "last_updated" column unnecessarily. This is first of  few optimizations to DAG Serialization that would also aid in DAG Versioning

(cherry picked from commit 1a32c45)
kaxil added a commit that referenced this issue Aug 13, 2020
We should not update the "last_updated" column unnecessarily. This is first of  few optimizations to DAG Serialization that would also aid in DAG Versioning

(cherry picked from commit 1a32c45)
kaxil added a commit that referenced this issue Aug 15, 2020
We should not update the "last_updated" column unnecessarily. This is first of  few optimizations to DAG Serialization that would also aid in DAG Versioning

(cherry picked from commit 1a32c45)
pcandoalmeida added a commit to pcandoalmeida/airflow that referenced this issue Oct 1, 2020
We should not update the "last_updated" column unnecessarily. This is first of  few optimizations to DAG Serialization that would also aid in DAG Versioning

(cherry picked from commit 1a32c45)
cfei18 pushed a commit to cfei18/incubator-airflow that referenced this issue Mar 5, 2021
We should not update the "last_updated" column unnecessarily. This is first of  few optimizations to DAG Serialization that would also aid in DAG Versioning

(cherry picked from commit 1a32c45)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants