[WIP] Remove old lineage stuff

[jason810496]

closes: #44983

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[jason810496]

When attempting to remove the lineage logic from the core module, I noticed that it causes failures in tests related to OpenLineage listener capturing hook-level lineage (#41482).

For example, removing apply_lineage and prepare_lineage from BaseOperator:

airflow/airflow/models/baseoperator.py

Lines 705 to 739 in 0efd9e6

	@prepare_lineage
	def pre_execute(self, context: Any):
	"""Execute right before self.execute() is called."""
	if self._pre_execute_hook is None:
	return
	ExecutionCallableRunner(
	self._pre_execute_hook,
	context_get_outlet_events(context),
	logger=self.log,
	).run(context)
	def execute(self, context: Context) -> Any:
	"""
	Derive when creating an operator.
	Context is the same dictionary used as when rendering jinja templates.
	Refer to get_template_context for more context.
	"""
	raise NotImplementedError()
	@apply_lineage
	def post_execute(self, context: Any, result: Any = None):
	"""
	Execute right after self.execute() is called.
	It is passed the execution context and any results returned by the operator.
	"""
	if self._post_execute_hook is None:
	return
	ExecutionCallableRunner(
	self._post_execute_hook,
	context_get_outlet_events(context),
	logger=self.log,
	).run(context, result)

Results in the following test failure:

FAILED providers/tests/openlineage/extractors/test_manager.py::test_extractor_manager_gets_data_from_pythonoperator - assert 0 == 1
 +  where 0 = len([])
 +    where [] = HookLineage(inputs=[], outputs=[]).outputs

airflow/providers/tests/openlineage/extractors/test_manager.py

Line 301 in 0efd9e6

def test_extractor_manager_gets_data_from_pythonoperator(session, dag_maker, hook_lineage_collector):

It seems OpenLineage is still coupled with the lineage module and might need to be moved to compact.lineage for now (or to the OpenLineage module in a future PR ).

After some experimentation, implementing an on_load callback in the OpenLineageProviderPlugin to monkey-patch the core module at runtime prevents the OpenLineage test failures, even with the lineage module removed from the core.

Based on https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/plugins.html#interface

I’m not sure if this is a suitable long-term solution for maintaining OpenLineage compatibility while cleaning up the lineage module ?

cc @Lee-W @uranusjr