#122 spark4 upgrade

[MatloaItumeleng]

Adds support for Spark 4.

Release Notes

• Adds support for Spark 4 / Scala 2.13 cross-build with Spark3/ Scala 2.12 and Scala 2.13
• Adds Ansi-handling for Spark4
• Bumps spark-commons from 0.6.3 to 1.0.0

Related

Closes #122
related to #123

[github-actions]

JaCoCo code coverage report - scala 2.12.20

Metric (instruction)	Coverage	Threshold	Status
Overall	83.14%	80.0%	✅
Changed Files	93.46%	80.0%	✅

Report	Coverage (O/Ch)	Threshold (O/Ch)	Status (O/Ch)
spark-data-standardization Jacoco Report - scala:2.12.20	83.14% / 93.46%	80.0% / 80.0%	✅/✅

File Path	Coverage	Threshold	Status
Standardization.scala	84.4%	60.0%	✅
TypeParser.scala	94.18%	60.0%	✅

---

Run 35618146402 · Event: pull_request

[yruslan]

Looks good to me. I just have a commend and a question.

It would be ideal if it can be tested with one of Aqueduct pipelines that has casting errors. But I understand that it is not easy to setup such a test. From my side, I'd be happy to merge this and release a new version. And when Aqueduct starts using the library, we can come back with bug reports in worst case.

[lsulak]

I think that this file must also be updated:

https://github.com/AbsaOSS/spark-data-standardization/blob/master/.github/workflows/release.yml

because there is nothing saying that Spark 4 + Java 17 artifact will be released - this is very important

[lsulak]

Also, JaCoCo only runs on 2.12 / Spark 3.5, so the new Spark 4 code path has no coverage check.

[github-actions]

JaCoCo code coverage report - scala 2.13.17, spark 4.1.2

Metric (instruction)	Coverage	Threshold	Status
Overall	78.2%	75.0%	✅
Changed Files	86.31%	80.0%	✅

Report	Coverage (O/Ch)	Threshold (O/Ch)	Status (O/Ch)
spark-data-standardization Jacoco Report - spark:4.1.2 scala:2.13.17	78.2% / 86.31%	75.0% / 80.0%	✅/✅

File Path	Coverage	Threshold	Status
Standardization.scala	91.82%	60.0%	✅
TypeParser.scala	85.76%	60.0%	✅

---

Run 38405390425 · Event: pull_request

[github-actions]

JaCoCo code coverage report - scala 2.12.20, spark 3.5.7

Metric (instruction)	Coverage	Threshold	Status
Overall	83.32%	80.0%	✅
Changed Files	93.96%	80.0%	✅

Report	Coverage (O/Ch)	Threshold (O/Ch)	Status (O/Ch)
spark-data-standardization Jacoco Report - spark:3.5.7 scala:2.12.20	83.32% / 93.96%	80.0% / 80.0%	✅/✅

File Path	Coverage	Threshold	Status
Standardization.scala	91.96%	60.0%	✅
TypeParser.scala	94.18%	60.0%	✅

---

Run 38405390425 · Event: pull_request

[MatloaItumeleng]

Have done few updates workflows now run cross build for JaCoCo , build test and the release , README indicates how to build per Scala Spark minor version and the change in the artifact which will not conflict per version

[yruslan]

Looks good to me. I've only checked the logic changes and how they are testes. I didn't look at CI/CD bit.

It is very hard to predict all edge cases with Aqueduct compatibility. It might work best if these changes are released, maybe as an RC, and then Aqueduct can be built for Spark 4 and test it with Glue 6 in UAT. And if there is still a bug or a incompatible behavior, it can be reported and hotfixed.

[yruslan]

Btw Alex and Kevin are planning to turn ANSI off for Spark 4 jobs as part of built-in configuration, so I am optimistic that there shouldn't be many issues with Aqueduct.

[ABLL526]

LGTM, discussed on call.