diff --git a/src/concepts/pipeline/artifact_deployment.md b/src/concepts/pipeline/artifact_deployment.md index 6a4555280..85309f03d 100644 --- a/src/concepts/pipeline/artifact_deployment.md +++ b/src/concepts/pipeline/artifact_deployment.md @@ -18,11 +18,13 @@ There are two categories of user-defined functions: - UDFs for internal use only: These UDFs are published to the `udf` and `udf_js` datasets in the `moz-fx-data-shared-prod` project and managed via [bigquery-etl](https://github.com/mozilla/bigquery-etl/tree/main/sql/moz-fx-data-shared-prod/udf) - public UDFs that can be used even outside of Mozilla projects: These UDFs are published to the `mozfun` project and managed via [bigquery-etl](https://github.com/mozilla/bigquery-etl/tree/main/sql/mozfun) -The UDF deploy is triggered through the `publish_public_udfs` and `publish_persistent_udfs` Airflow task in the [`bqetl_artifact_deployment`](https://workflow.telemetry.mozilla.org/tree?dag_id=bqetl_artifact_deployment) Airflow DAG. The DAG no longer runs on a nightly schedule; it is triggered by CircleCI on merges to `main` and by Jenkins after schema deploys, and tasks can still be triggered manually by clearing them. +The UDF deploy is triggered through the `publish_public_udfs` and `publish_persistent_udfs` Airflow task in the [`bqetl_artifact_deployment`](https://workflow.telemetry.mozilla.org/tree?dag_id=bqetl_artifact_deployment) Airflow DAG. The DAG no longer runs on a nightly schedule; it is triggered by CircleCI on merges to `main` and by Jenkins after schema deploys, and tasks can still be triggered manually by clearing them. To avoid redundant overlapping runs, the DAG skips a run if another run is already queued, unless the `generate_sql` parameter is set. ## Table Deployment -Tables get deployed through the [`publish_tables_and_views` Airflow task](https://workflow.telemetry.mozilla.org/tree?dag_id=bqetl_artifact_deployment), which replaced the former separate `publish_new_tables` task. This task will run all SQL generators, generate schemas for each query and will deploy schemas for tables and apply any schema-compatible changes for existing tables. The task will fail if changes that are incompatible with existing schemas (such as removing fields or changing field types) are to be applied. +Tables get deployed through the [`publish_tables_and_views` Airflow task](https://workflow.telemetry.mozilla.org/tree?dag_id=bqetl_artifact_deployment), which replaced the former separate `publish_new_tables` task. This task will run all SQL generators, generate schemas for each query and will deploy schemas for tables and apply any schema-compatible changes for existing tables across `moz-fx-data-shared-prod`, `moz-fx-data-experiments`, `moz-fx-data-marketing-prod`, `moz-fx-glam-prod`, and `moz-fx-data-bq-people`. The task will fail if changes that are incompatible with existing schemas (such as removing fields or changing field types) are to be applied. New tables are first initialized via a separate `bqetl_artifact_initialize` Airflow DAG, which runs `bqetl query initialize` across `moz-fx-data-shared-prod`, `moz-fx-data-experiments`, `moz-fx-data-marketing-prod`, and `moz-fx-data-bq-people`. + +Static tables, which are deployed from CSV files rather than generated SQL, are published separately via the `publish_static_tables` task for `moz-fx-data-shared-prod` and `mozdata`. ## View Deployment diff --git a/src/concepts/pipeline/event_pipeline.md b/src/concepts/pipeline/event_pipeline.md index ba2a1cd5c..a0982f905 100644 --- a/src/concepts/pipeline/event_pipeline.md +++ b/src/concepts/pipeline/event_pipeline.md @@ -125,7 +125,7 @@ On the pipeline side, the event data is made available in different datasets: its event payload for Firefox versions before 62. - [`events`](../../datasets/batch_view/events/reference.md) contains a row for each event received from main pings and event pings. See [`STMO#52582`](https://sql.telemetry.mozilla.org/queries/52582/source). - For applications that collect events through Glean, each application has a separate `events` dataset. -- [`events_daily`](../../datasets/bigquery/events_daily/reference.md) encodes each client-day's events as a compressed string of Unicode characters, for use with the `mozfun.event_analysis` functions for funnel and event-count analysis. +- [`events_daily`](../../datasets/bigquery/events_daily/reference.md) (generated for Firefox Desktop, Fenix, and Mozilla VPN; not available for other Glean applications) encodes each client-day's events as a compressed string of Unicode characters, for use with the `mozfun.event_analysis` functions for funnel and event-count analysis. The Firefox Desktop version of this table (`telemetry_derived.events_daily_v1`) is deprecated and no longer actively scheduled (see [bug 1805722](https://bugzilla.mozilla.org/show_bug.cgi?id=1805722)). # Data tooling diff --git a/src/cookbooks/publishing_datasets.md b/src/cookbooks/publishing_datasets.md index cf3ae60af..c1cba0f6c 100644 --- a/src/cookbooks/publishing_datasets.md +++ b/src/cookbooks/publishing_datasets.md @@ -19,12 +19,13 @@ owners: labels: application: firefox incremental: true # incremental queries add data to existing tables - schedule: daily # scheduled in Airflow to run daily public_json: true public_bigquery: true review_bugs: # Bugzilla bug IDs of data review - 1414839 incremental_export: false # non-incremental JSON export writes all data to a single location +scheduling: + dag_name: bqetl_ssl_ratios ``` The following options define how data is published: @@ -41,11 +42,12 @@ The following options define how data is published: - `incremental`: indicates how data gets updated based on the query and Airflow configuration - `true`: data gets incrementally updated - `false`: the entire table data gets updated -- `review_bugs`: list of Bugzilla bug numbers of the data review +- `review_bugs`: list of Bugzilla bug numbers of the data review; required, and enforced by CI validation, whenever `public_bigquery` or `public_json` is set - `legacy`: set to `true` to indicate a dataset that selects only from legacy Telemetry -Data will get published when the query is executed in Airflow. Metadata of available public -data on Cloud Storage is updated daily through a separate Airflow task. +Data for `public_json` gets published when the query runs in Airflow, and metadata of available public +data on Cloud Storage is updated daily through a separate Airflow task. Tables flagged `public_bigquery` +are deployed to the `mozilla-public-data` project nightly via the `bqetl_artifact_deployment` Airflow DAG. More information about accessing public data can be found in [Accessing Public Data](../cookbooks/public_data.md). diff --git a/src/datasets/active_profiles.md b/src/datasets/active_profiles.md index 663cf5df7..97998cd1e 100644 --- a/src/datasets/active_profiles.md +++ b/src/datasets/active_profiles.md @@ -1,5 +1,7 @@ # Intro +**Note:** This dataset is retired/archived. The pipeline no longer exists in Mozilla's current `bigquery-etl` or `telemetry-airflow` repositories, and the table is no longer generated; this page is kept for historical reference. See [Scheduling](#scheduling) below. + The `active_profiles` dataset gives client-level estimates of whether a profile is still an active user of the browser at a given point in time, as well as probabilistic forecasts of the client's future activity. These quantities are estimated by a model that attempts to infer diff --git a/src/datasets/batch_view/clients_daily/reference.md b/src/datasets/batch_view/clients_daily/reference.md index 64ae45f34..57cbb843c 100644 --- a/src/datasets/batch_view/clients_daily/reference.md +++ b/src/datasets/batch_view/clients_daily/reference.md @@ -55,6 +55,8 @@ This dataset is updated daily via the [telemetry-airflow](https://github.com/mozilla/telemetry-airflow) infrastructure. The job runs as part of the [`main_summary` DAG](https://github.com/mozilla/bigquery-etl/blob/master/dags/bqetl_main_summary.py). +Downstream tasks that depend on this dataset include `jetstream`, `operational_monitoring`, `parquet_export`, and `catalyst`. + ## Schema The data is partitioned by `submission_date`. diff --git a/src/datasets/bigquery/clients_last_seen/reference.md b/src/datasets/bigquery/clients_last_seen/reference.md index 266952ea0..3c9fa8085 100644 --- a/src/datasets/bigquery/clients_last_seen/reference.md +++ b/src/datasets/bigquery/clients_last_seen/reference.md @@ -71,6 +71,7 @@ Please see this [section](../../../concepts/segments.md) for descriptions regard - `is_core_active_v1`: Boolean indicating if the client satisfies conditions of being core active on that day. - `activity_segments_v1`: The activity segment applicable to the client that day. +- `segment`: Duplicate of `activity_segments_v1`, using the same logic. - `is_regular_user_v3`: Boolean indicating if the client satisfies conditions of being a regular user on that day. - `is_new_or_resurrected_v3`: Boolean indicating if the client satisfies conditions of being a regular user on that day. - `is_weekday_regular_v1`: Boolean indicating if the client satisfies conditions of being a weekday regular user on that day. @@ -91,7 +92,10 @@ Please see this [section](../../../concepts/segments.md) for descriptions regard - `days_had_8_active_ticks_bits`: Each bit field represents if a client had at least 8 active ticks on that day. This can be used to approximate the threshold of 1 URI, and is useful for determining activity for clients using Private Browsing Mode where URI counts are not recorded. -- `days_since_visited_8_active_ticks`: Number of days since the client had at least 8 active ticks. +- `days_since_had_8_active_ticks`: Number of days since the client had at least 8 active ticks. + +- `days_active_bits`: Each bit field represents if a client had at least 1 active tick and visited at least 1 URI on that day. +- `days_since_active`: Number of days since the client was active (had at least 1 active tick and visited at least 1 URI). ### New Profile Specific