📝 Update the section on data cleaning

* Add validation categories and procedures
* Add text encoding with charset_normalizer and ftfy
* Add DataProfiler and cerberus

Author: veit

CHANGELOG.rst

@@ -23,6 +23,12 @@ emergencies when we need to start branches for older versions.
 Added
 ~~~~~
 
+* 📝 Update the section on data cleaning
+
+  * Add validation categories and procedures
+  * Add text encoding with charset_normalizer and ftfy
+  * Add DataProfiler and cerberus
+
 * 📝 Add Narwhals
 * 📝 Add protobuf-py
 

docs/clean-prep/categories.rst

@@ -0,0 +1,62 @@
+.. SPDX-FileCopyrightText: 2026 cusy GmbH
+..
+.. SPDX-License-Identifier: BSD-3-Clause
+
+Categories of data quality
+==========================
+
+There are various ways to categorise data quality issues – by the cause of the
+problem, by its likely or potential impact, by the type of data element, or by
+the level of a data hierarchy at which the error or inconsistency becomes
+apparent, to name but a few. In this tutorial, categorisation is generally
+implicit, and we focus on the methods for detecting problems and the tools
+suitable for this purpose. However, it is useful to at least briefly touch upon
+other ways of categorising data problems before we turn our attention to
+detection.
+
+Impossible data
+    such as temperatures below absolute zero (0 Kelvin or −273.15 °C) or
+    nonsensical date and time values – whilst these cannot occur in the
+    :class:`datetime` data type in Python, they can occur in
+    :doc:`../data-processing/serialisation-formats/json/index` and :doc:`SQLite
+    <python-basics:save-data/sqlite/index>`, where text fields are usually used
+    to store date information.
+Special values
+    The `IEEE-754 <https://en.wikipedia.org/wiki/IEEE_754>`_ standard for
+    representing floating-point numbers also includes special values for
+    positive and negative infinity (+∞, −∞) as well as *Not a Number* (``NaN``)
+    values.
+Unspecified Values
+    Values that conform to a specific format may nevertheless be invalid, such
+    as email addresses.
+Inconsistencies
+    Groups of related field values for each record may be mutually exclusive. To
+    avoid such inconsistencies, the :abbr:`DRY (Don’t Repeat Yourself)`
+    principle recommends disregarding calculable values. Checksums, however, are
+    the exact opposite of this.
+
+    *‘Single Source of Truth’* is an example of database normalisation and
+    simplifies updates; with denormalised data stores, on the other hand, joins
+    and calculations can be avoided during data analysis.
+
+Anomalies and data drift
+    The checks carried out to date have made it possible to declare data
+    invalid. However, anomalies and data drift do not necessarily support this
+    conclusion and therefore rarely allow for automated data cleansing.
+Malware
+    Not all input or output data should be processed in analysis systems.
+
+    .. figure:: exploits_of_a_mom_2x.png
+       :alt: Hi, this is your son’s school. We’re having some  computer trouble.
+             Oh, dear - did he break something? In a way –
+             Did you really name your son Robert'); DROP TABLE Students; -- ?
+             Oh, yes. Little bobby tables, we call him.
+             Well, weve lost this year’s student records. I hope you’re happy.
+             And I hope, youe’ve learned to sanitize your database inpots.
+       :target: https://xkcd.com/327/
+
+       Exploits of a Mom
+
+    Ein offensichtliches Beispiel für versehentlich offengelegte Ausgabedaten
+    sind :abbr:`z. B. (zum Beispiel)` aus fortlaufenden Personenkennziffern
+    generierte URLs mit personenbezogenen Daten.

docs/clean-prep/exploits_of_a_mom_2x.png

@@ -0,0 +1,3 @@
+SPDX-FileCopyrightText: 2026 cusy GmbH
+
+SPDX-License-Identifier: BSD-3-Clause

docs/clean-prep/exploits_of_a_mom_2x.png.license

@@ -2,67 +2,60 @@
 ..
 .. SPDX-License-Identifier: BSD-3-Clause
 
-Data cleansing and validation
-=============================
+Cleaning and validating data
+============================
 
-In the following, we want to give you a practical overview of various libraries
-and methods for `data cleansing <https://en.wikipedia.org/wiki/Data_cleansing>`_
-and validation with Python. Besides well-known libraries like
-:doc:`/workspace/numpy/index` and ,:doc:`/workspace/pandas/index`
-we also use several small, specialised libraries like :doc:`dedupe
-<deduplicate>`, :doc:`fuzzywuzzy <string-matching>`, :doc:`voluptuous
-<voluptuous>`, :doc:`tdda <tdda>` and :doc:`hypothesis <hypothesis>`. We prefer
-these more lightweight solutions to large, universal systems like `Great
-Expectations <https://greatexpectations.io/>`_ or `MobyDQ
-<https://ubisoft.github.io/mobydq/>`_.
+*‘Garbage in, garbage out’* is a stark reminder that it is almost impossible to
+draw meaningful insights from poor-quality data. Whilst there are highly
+regulated, safety-critical sectors in which data is routinely checked at every
+stage, the general standard of data validation in most analytical projects tends
+to be rather low.
 
-.. tip::
-   `cusy seminar: Cleanse and validate data with Python
-   <https://cusy.io/en/our-training-courses/cleanse-and-validate-data-with-python.html>`_
+However, if we write code to analyse data without first checking it, some
+results will be misleading, incorrect or invalid. The same applies if we fail to
+validate our output. In doing so, we end up contributing to the very *‘bad
+data’* that we all rail against. *Postel’s Law* – the principle of robustness
+proposed in the TCP standard – is helpful here:
+
+    *“be conservative in what you do, be liberal in what you accept from
+    others.”* [#]_
+
+This stands in stark contrast to one of Python’s principles:
 
-Overview
---------
+    *“Errors should never pass silently.”* [#]_
 
-.. csv-table:: GitHub-Insights
-    :header: "Name", "Stars", "Mitwirkende", "Commit-Aktivität", "Lizenz"
+The :doc:`XML <../data-processing/serialisation-formats/xml-html/index>`
+specification also requires that malformed XML documents be rejected:
 
-    "`scikit-learn <https://github.com/scikit-learn/scikit-learn>`_",".. image:: https://raster.shields.io/github/stars/scikit-learn/scikit-learn",".. image:: https://raster.shields.io/github/contributors/scikit-learn/scikit-learn",".. image:: https://raster.shields.io/github/commit-activity/y/scikit-learn/scikit-learn",".. image:: https://raster.shields.io/github/license/scikit-learn/scikit-learn"
-    "`fg-data-profiling <https://github.com/Data-Centric-AI-Community/fg-data-profiling>`_",".. image:: https://raster.shields.io/github/stars/Data-Centric-AI-Community/fg-data-profiling",".. image:: https://raster.shields.io/github/contributors/Data-Centric-AI-Community/fg-data-profiling",".. image:: https://raster.shields.io/github/commit-activity/y/Data-Centric-AI-Community/fg-data-profiling",".. image:: https://raster.shields.io/github/license/Data-Centric-AI-Community/fg-data-profiling"
-    "`Hypothesis <https://github.com/HypothesisWorks/hypothesis>`_",".. image:: https://raster.shields.io/github/stars/HypothesisWorks/hypothesis",".. image:: https://raster.shields.io/github/contributors/HypothesisWorks/hypothesis",".. image:: https://raster.shields.io/github/commit-activity/y/HypothesisWorks/hypothesis",".. image:: https://raster.shields.io/github/license/HypothesisWorks/hypothesis"
-    "`marshmallow <https://github.com/marshmallow-code/marshmallow>`_",".. image:: https://raster.shields.io/github/stars/marshmallow-code/marshmallow",".. image:: https://raster.shields.io/github/contributors/marshmallow-code/marshmallow",".. image:: https://raster.shields.io/github/commit-activity/y/marshmallow-code/marshmallow",".. image:: https://raster.shields.io/github/license/marshmallow-code/marshmallow"
-    "`dedupe <https://github.com/dedupeio/dedupe>`_",".. image:: https://raster.shields.io/github/stars/dedupeio/dedupe",".. image:: https://raster.shields.io/github/contributors/dedupeio/dedupe",".. image:: https://raster.shields.io/github/commit-activity/y/dedupeio/dedupe",".. image:: https://raster.shields.io/github/license/dedupeio/dedupe"
-    "`pandera <https://github.com/unionai-oss/pandera>`_",".. image:: https://raster.shields.io/github/stars/unionai-oss/pandera",".. image:: https://raster.shields.io/github/contributors/unionai-oss/pandera",".. image:: https://raster.shields.io/github/commit-activity/y/unionai-oss/pandera",".. image:: https://raster.shields.io/github/license/unionai-oss/pandera"
-    "`thefuzz <https://github.com/seatgeek/thefuzz>`_",".. image:: https://raster.shields.io/github/stars/seatgeek/thefuzz",".. image:: https://raster.shields.io/github/contributors/seatgeek/thefuzz",".. image:: https://raster.shields.io/github/commit-activity/y/seatgeek/thefuzz",".. image:: https://raster.shields.io/github/license/seatgeek/thefuzz"
-    "`Voluptuous <https://github.com/alecthomas/voluptuous>`_",".. image:: https://raster.shields.io/github/stars/alecthomas/voluptuous",".. image:: https://raster.shields.io/github/contributors/alecthomas/voluptuous",".. image:: https://raster.shields.io/github/commit-activity/y/alecthomas/voluptuous",".. image:: https://raster.shields.io/github/license/alecthomas/voluptuous"
-    "`datacleaner <https://github.com/rhiever/datacleaner>`_",".. image:: https://raster.shields.io/github/stars/rhiever/datacleaner",".. image:: https://raster.shields.io/github/contributors/rhiever/datacleaner",".. image:: https://raster.shields.io/github/commit-activity/y/rhiever/datacleaner",".. image:: https://raster.shields.io/github/license/rhiever/datacleaner"
-    "`popmon <https://github.com/ing-bank/popmon>`_",".. image:: https://raster.shields.io/github/stars/ing-bank/popmon",".. image:: https://raster.shields.io/github/contributors/ing-bank/popmon",".. image:: https://raster.shields.io/github/commit-activity/y/ing-bank/popmon",".. image:: https://raster.shields.io/github/license/ing-bank/popmon"
-    "`TDDA <https://github.com/tdda/tdda>`_",".. image:: https://raster.shields.io/github/stars/tdda/tdda",".. image:: https://raster.shields.io/github/contributors/tdda/tdda",".. image:: https://raster.shields.io/github/commit-activity/y/tdda/tdda",".. image:: https://raster.shields.io/github/license/tdda/tdda"
-    "`Validr <https://github.com/guyskk/validr>`_",".. image:: https://raster.shields.io/github/stars/guyskk/validr",".. image:: https://raster.shields.io/github/contributors/guyskk/validr",".. image:: https://raster.shields.io/github/commit-activity/y/guyskk/validr",".. image:: https://raster.shields.io/github/license/guyskk/validr"
-    "`Probatus <https://github.com/ing-bank/probatus>`_",".. image:: https://raster.shields.io/github/stars/ing-bank/probatus",".. image:: https://raster.shields.io/github/contributors/ing-bank/probatus",".. image:: https://raster.shields.io/github/commit-activity/y/ing-bank/probatus",".. image:: https://raster.shields.io/github/license/ing-bank/probatus"
+    “„Validating and non-validating processors alike MUST report violations of
+    this specification's well-formedness constraints … that they read.”* [#]_
 
-Dormant projects
-----------------
+The sections on :doc:`procedures/index`, :doc:`procedures/ranges` and
+:doc:`procedures/regression_tests` do not require any prior knowledge of Python
+and are therefore generally suitable for professionals in the fields of data
+management, business management and quality assurance.
+
+We will then provide you with a practical overview of various :doc:`libraries
+and methods <libs-methods/index>` for `data cleaning
+<https://en.wikipedia.org/wiki/Data_cleansing>`_ and validation using Python.
+
+.. tip::
+   `cusy seminar: Cleanse and validate data with Python
+   <https://cusy.io/en/our-training-courses/cleanse-and-validate-data-with-python.html>`_
 
-.. csv-table:: GitHub-Insights
-    :header: "Name", "Stars", "Mitwirkende", "Commit-Aktivität", "Lizenz"
+----
 
-    "`Bulwark <https://github.com/ZaxR/bulwark>`_",".. image:: https://raster.shields.io/github/stars/ZaxR/bulwark",".. image:: https://raster.shields.io/github/contributors/ZaxR/bulwark",".. image:: https://raster.shields.io/github/commit-activity/y/ZaxR/bulwark",".. image:: https://raster.shields.io/github/license/ZaxR/bulwark"
-    "`PandasSchema <https://github.com/multimeric/PandasSchema>`_",".. image:: https://raster.shields.io/github/stars/multimeric/PandasSchema",".. image:: https://raster.shields.io/github/contributors/multimeric/PandasSchema",".. image:: https://raster.shields.io/github/commit-activity/y/multimeric/PandasSchema",".. image:: https://raster.shields.io/github/license/multimeric/PandasSchema"
-    "`pandas-validation <https://github.com/jmenglund/pandas-validation>`_",".. image:: https://raster.shields.io/github/stars/jmenglund/pandas-validation",".. image:: https://raster.shields.io/github/contributors/jmenglund/pandas-validation",".. image:: https://raster.shields.io/github/commit-activity/y/jmenglund/pandas-validation",".. image:: https://raster.shields.io/github/license/jmenglund/pandas-validation"
-    "`Opulent-Pandas <https://github.com/danielvdende/opulent-pandas>`_",".. image:: https://raster.shields.io/github/stars/danielvdende/opulent-pandas",".. image:: https://raster.shields.io/github/contributors/danielvdende/opulent-pandas",".. image:: https://raster.shields.io/github/commit-activity/y/danielvdende/opulent-pandas",".. image:: https://raster.shields.io/github/license/danielvdende/opulent-pandas"
-    "`signpost <https://github.com/ilsedippenaar/signpost>`_",".. image:: https://raster.shields.io/github/stars/ilsedippenaar/signpost",".. image:: https://raster.shields.io/github/contributors/ilsedippenaar/signpost",".. image:: https://raster.shields.io/github/commit-activity/y/ilsedippenaar/signpost",".. image:: https://raster.shields.io/github/license/ilsedippenaar/signpost"
+.. [#] Jon Postel: `Transmission Control Protocol
+       <https://www.rfc-editor.org/info/rfc761/#section-2.10>`_, 1980
+.. [#] Tim Peters: :pep:`The Zen of Python <20>`, 1999
+.. [#] `XML Specification 1.0, Section 5.1
+       <https://www.w3.org/TR/2008/REC-xml-20081126/#proc-types>`_, 1998
 
 .. toctree::
     :hidden:
     :titlesonly:
     :maxdepth: 0
 
-    nulls.ipynb
-    outliers.ipynb
-    string-matching.ipynb
-    deduplicate.ipynb
-    hypothesis.ipynb
-    tdda.ipynb
-    voluptuous.ipynb
-    scikit-learn-reprocessing.ipynb
-    dask-pipeline.ipynb
+    categories
+    procedures/index
+    libs-methods/index

docs/clean-prep/index.rst

@@ -18,7 +18,7 @@
     "1. [Google Maps Geocoder](https://developers.google.com/maps/documentation/geocoding/guides-v3/overview)\n",
     "2. [Open Notify API for ISS location](http://api.open-notify.org)\n",
     "\n",
-    "We will use them to track ISS location and next transit time with respect to a list of cities. To create our charts and parallelise data intelligently, we will use Dask, specifically [Dask Delayed](../performance/dask.ipynb)."
+    "We will use them to track ISS location and next transit time with respect to a list of cities. To create our charts and parallelise data intelligently, we will use Dask, specifically [Dask Delayed](../../performance/dask.ipynb)."
    ]
   },
   {