diff --git a/docs/notebooks/Gallery.ipynb b/docs/notebooks/Gallery.ipynb index 60d2a46e..4a0036a9 100644 --- a/docs/notebooks/Gallery.ipynb +++ b/docs/notebooks/Gallery.ipynb @@ -7,11 +7,34 @@ "source": [ "# Tutorial Gallery\n", "\n", + "PyEarthTools aims to cater for the following use cases and user groups:\n", + "\n", + " - Scientists learning machine learning, and data scientists learning Earth system science\n", + "\n", + " - People working on their own workstations and people working in supercomputing facilities (such as NCI and Jasmine)\n", + "\n", + " - Those trying to develop a new model given data inputs and science goals - i.e. tabula rasa approaches\n", + "\n", + " - Those wanting to quickly access and evaluate pre-defined architectures, simple or advanced.\n", + "\n", + " - Those seeking to perform model intercomparison, such as between different experiments, or between model architectures\n", + "\n", + " - Frontier model development of world-class Earth system science models, for scientific purposes\n", + "\n", + " - Running other people's pretrained models (in any class of model), for research purposes\n", + "\n", + "As a result, users going through these tutorials will need to be aware of the difference between individual workstation setups and supercomputing facility setups, and may need to spend some time identifying the tutorials of greatest relevance to them. The gallery page starts with introductory examples and proceeds to more complex examples to match a common journey faced by adopters.\n", + "\n", + "\n", "This gallery is organised into:\n", "\n", "- **[Quick Start (Low Hardware Requirements)](#Quick-Start-(Low-Hardware-Requirements))**. \n", - " *Good for newcomers. Includes tips and tricks, and dealing with common questions when first adopting PyEarthTools.* \n", - "- **[Working with Station Data (Medium Hardware Requirements)](#Working-with-Station-Data-(Medium-Hardware-Requirements))** \n", + " *Good for newcomers wanting a first working example. Includes tips and tricks, and dealing with common questions when first adopting PyEarthTools.*\n", + "- **[Introduction To Machine Learning](#Introduction-To-Machine-Learning)** \n", + " *Structured learning journey for those using machine learning for the first time. Introduced AutoEncoders and core\n", + " PyEarthTools concepts for data pipelines. Then proceeds to outline PyEarthTools classes for model definitions and links\n", + " into a structure experiment workflow which may be of interest. Suitable for delivery as a hands-on training workshop or self-directed learning.* \n", + "- **[Working with Weather Station Data (Medium Hardware Requirements)](#Working-with-Station-Data-(Medium-Hardware-Requirements))** \n", " *Examples of how to connect to a new data source and handle new data structures.* \n", "- **[Tutorials on Specific Modelling Objectives](#Tutorials-on-Specific-Modelling-Objectives)**. \n", " *Good starting points for people interested in specific modelling opportunities, or for those looking to see an end-to-end approach.* \n", @@ -45,6 +68,26 @@ "| **LUCIE Climate Model** | Make predictions from a climate model | ![Image showing output from the LUCIE model](https://pyearthtools.readthedocs.io/en/latest/_images/notebooks_tutorial_LUCIE_LUCIE-Inference_6_1.png) | [LUCIE Inference](./tutorial/LUCIE/LUCIE-Inference.ipynb) | 13 Nov 2025 |\n" ] }, + { + "cell_type": "markdown", + "id": "181fe484-f2bb-412f-ba6e-f0c3a9520bd1", + "metadata": {}, + "source": [ + "## Introduction to Scientific Machine Learning\n", + "\n", + "These tutorials are optimised for the NCI supercomputing facility, mainly due to ease of data access. Subsequent work will generalise these for use on workstations and laptops. The examples are not likely to require large GPUs, but may require a data replication step into the workstation which is not trivial.\n", + "\n", + "| Topic | Description | Image | Notebooks | Last Tested |\n", + "|-------|--------------|-------|-------------|-------------|\n", + "| **Introduction For Workshops** | Reading material introducing AutoEncoders and time-series modelling concepts | (no image) | [IntroductionForWorkshops](./tutorial/IntroductionForWorkshops.ipynb) | 29 July 2026 |\n", + "| **Simple AutoEncoder** | Training a simple AutoEncoder using PyEarthTools for data access and PyTorch for the modelling | (no image) | [AutoEncoder Example](./tutorial/AutoEncoder_Example.ipynb) | 29 July 2026 |\n", + "| **Improved AutoEncoder** | Improving accuracy (performance) of the AutoEncoder Model | (no image) | [AutoEncoder - Improving Results](AutoEncoder_ImprovingResults.ipynb) | 29 July 2026 |\n", + "| **PyEarthTools Simple AutoEncoder** | Training a simple pre-defined wrapped autoencoder | (no image) | TBD | n/a |\n", + "| **Other PyEarthTools Simple Models** | Training other simple wrapped models | (no image) | TBD | n/a |\n", + "| **Suggested Experimental Workflow** | Overview of a model development lifecycle | (no image) | TBD | n/a |\n", + "| **Advanced Architecture Examples** | Training more advanced architectures | (no image) | TBD | n/a |" + ] + }, { "cell_type": "markdown", "id": "a6f26875-9a0c-40b2-87ad-39cb1f8037e9", @@ -171,7 +214,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.14.5" + "version": "3.12.13" } }, "nbformat": 4, diff --git a/docs/notebooks/tutorial/IntroductionForWorkshops.ipynb b/docs/notebooks/tutorial/IntroductionForWorkshops.ipynb index 826faf08..a152619b 100644 --- a/docs/notebooks/tutorial/IntroductionForWorkshops.ipynb +++ b/docs/notebooks/tutorial/IntroductionForWorkshops.ipynb @@ -30,13 +30,13 @@ "source": [ "## Introduction to Time Series Forecasting\n", "\n", - "This content may seem obvious, but in this tutorial we are trying to make a prediction of future conditions, based on previously obtained observational data. There are a lot of model types which do not fall in this category, including re-analysis, downscaling, feature detection (e.g. detecting cold fronts, rain bands or cyclones) within data. \n", + "This content may seem obvious, but in this tutorial eventually want to make a prediction of future conditions, based on previously obtained observational data. There are a lot of model types which do not fall in this category, including re-analysis, downscaling and feature detection (e.g. detecting cold fronts, rain bands or cyclones) within data. We need to introduce specific language to describe the components of this modelling challenge.\n", "\n", - "A lot of time series forecasting outside of science is done on one-dimensional data (e.g. a single weather station), or on non-physical data (e.g. a spreadsheet of train passenger numbers by time of day).\n", + "A lot of time series forecasting outside of science is done on one-dimensional data (e.g. a single weather station), or on non-physical data (e.g. a spreadsheet of train passenger numbers by time of day). Time-series data in Earth System data may be done on one dimensional data, a graph of data (e.g. a collection of weather stations, with known locations), a grid of data (like a temperature map) or a volume of data (like a stack of maps, or a cube, based on latitude, longitude and height). Gridded data may refer to 2d data or a volume of data so long as it has clear spatial coordinates. \n", "\n", - "A lot of the time, the observations are taken under significant uncertainty. As such, there are statistical models, machine learning models, and physical models whose focus are on establishing initial conditions in some fashion. 3d VAR and 4d VAR are two methods grounded in physical science for performing this function. A lot of the time in physical modelling, we will be dealing with an coordinate system of some kind, such as lat, lon, height and time. \n", + "A lot of the time, the observations are taken under significant uncertainty. This could be due to physical sensor characteristics or positional uncertainty. As such, there are statistical models, machine learning models and physical models, whose focus is on establishing initial conditions in some fashion. 3d VAR and 4d VAR are two methods grounded in physical science for performing this function. A lot of the time in physical modelling, we will be dealing with a coordinate system of some kind, such as lat, lon, height and time. \n", "\n", - "Training such a time-series models involves taking a large archive of historical data, and repeatedly presenting sequences to a machine learning model to form the training process. It is common, but by no means the only (or even best) practise to do that by walking through the data from the oldest to the most recent data. There is a lot more that could be said around the biases and issues that could be present in taking a simple approach to model training data, but it is out of scope for this introduction.\n", + "Training such a time-series model (regardless of coordinate structure) involves taking a large archive of historical data, and repeatedly presenting sequences of data to a machine learning model to form the training process. It is common, but by no means the only (or even best) practise to do that by walking through the data from the oldest to the most recent data. There is a lot more that could be said around the biases and issues that could be present in taking a simple approach to model training data, but it is out of scope for this introduction. That said, a failure to understand cross-validation and data leakage between test/train/validate splits is a common issue when first training an ML model.\n", "\n", "Two essential questions for any time-series models are \"how do you gather observations\" and \"how far in the future do you predict\". There is usually some kind of window of previous observations used. They are typically grouped for convenience into \"time steps\". These time steps could be for example, hourly, daily, monthly, per-minute, or even more frequently. The model's view of the world is a view of these time steps, and all observations at a given time step are taken as simulteneous. \n", "\n", @@ -128,7 +128,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.13.9" + "version": "3.12.13" } }, "nbformat": 4,