Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
aa6cf64
concepts/pipeline/artifact_deployment.md: Mention bqetl_artifact_init…
doc-correctness[bot] Nov 20, 2026
a9c6b32
concepts/pipeline/artifact_deployment.md: Add note on static table de…
doc-correctness[bot] Nov 20, 2026
9eec87d
concepts/pipeline/artifact_deployment.md: List projects covered by ta…
doc-correctness[bot] Nov 20, 2026
5ff4bc4
concepts/pipeline/artifact_deployment.md: Document concurrency guard …
doc-correctness[bot] Nov 20, 2026
e3343e1
concepts/pipeline/event_pipeline.md: Note Firefox Desktop events_dail…
doc-correctness[bot] Nov 20, 2026
c086df8
concepts/pipeline/event_pipeline.md: Clarify events_daily is only gen…
doc-correctness[bot] Nov 20, 2026
8055e29
cookbooks/publishing_datasets.md: Move scheduling out of labels into …
doc-correctness[bot] Nov 20, 2026
68a0629
cookbooks/publishing_datasets.md: Note review_bugs is CI-enforced whe…
doc-correctness[bot] Nov 20, 2026
cc6aca0
cookbooks/publishing_datasets.md: Clarify separate deployment paths f…
doc-correctness[bot] Nov 20, 2026
efca1f4
datasets/active_profiles.md: Add retired/archived notice to Intro sec…
doc-correctness[bot] Nov 20, 2026
d705633
datasets/batch_view/clients_daily/reference.md: Note downstream tasks…
doc-correctness[bot] Nov 20, 2026
24ffc30
datasets/bigquery/clients_last_seen/reference.md: Fix field name to m…
doc-correctness[bot] Nov 20, 2026
67fff73
datasets/bigquery/clients_last_seen/reference.md: Document new days_a…
doc-correctness[bot] Nov 20, 2026
7c7b508
datasets/bigquery/clients_last_seen/reference.md: Document new segmen…
doc-correctness[bot] Nov 20, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions src/concepts/pipeline/artifact_deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,13 @@ There are two categories of user-defined functions:
- UDFs for internal use only: These UDFs are published to the `udf` and `udf_js` datasets in the `moz-fx-data-shared-prod` project and managed via [bigquery-etl](https://github.com/mozilla/bigquery-etl/tree/main/sql/moz-fx-data-shared-prod/udf)
- public UDFs that can be used even outside of Mozilla projects: These UDFs are published to the `mozfun` project and managed via [bigquery-etl](https://github.com/mozilla/bigquery-etl/tree/main/sql/mozfun)

The UDF deploy is triggered through the `publish_public_udfs` and `publish_persistent_udfs` Airflow task in the [`bqetl_artifact_deployment`](https://workflow.telemetry.mozilla.org/tree?dag_id=bqetl_artifact_deployment) Airflow DAG. The DAG no longer runs on a nightly schedule; it is triggered by CircleCI on merges to `main` and by Jenkins after schema deploys, and tasks can still be triggered manually by clearing them.
The UDF deploy is triggered through the `publish_public_udfs` and `publish_persistent_udfs` Airflow task in the [`bqetl_artifact_deployment`](https://workflow.telemetry.mozilla.org/tree?dag_id=bqetl_artifact_deployment) Airflow DAG. The DAG no longer runs on a nightly schedule; it is triggered by CircleCI on merges to `main` and by Jenkins after schema deploys, and tasks can still be triggered manually by clearing them. To avoid redundant overlapping runs, the DAG skips a run if another run is already queued, unless the `generate_sql` parameter is set.

## Table Deployment

Tables get deployed through the [`publish_tables_and_views` Airflow task](https://workflow.telemetry.mozilla.org/tree?dag_id=bqetl_artifact_deployment), which replaced the former separate `publish_new_tables` task. This task will run all SQL generators, generate schemas for each query and will deploy schemas for tables and apply any schema-compatible changes for existing tables. The task will fail if changes that are incompatible with existing schemas (such as removing fields or changing field types) are to be applied.
Tables get deployed through the [`publish_tables_and_views` Airflow task](https://workflow.telemetry.mozilla.org/tree?dag_id=bqetl_artifact_deployment), which replaced the former separate `publish_new_tables` task. This task will run all SQL generators, generate schemas for each query and will deploy schemas for tables and apply any schema-compatible changes for existing tables across `moz-fx-data-shared-prod`, `moz-fx-data-experiments`, `moz-fx-data-marketing-prod`, `moz-fx-glam-prod`, and `moz-fx-data-bq-people`. The task will fail if changes that are incompatible with existing schemas (such as removing fields or changing field types) are to be applied. New tables are first initialized via a separate `bqetl_artifact_initialize` Airflow DAG, which runs `bqetl query initialize` across `moz-fx-data-shared-prod`, `moz-fx-data-experiments`, `moz-fx-data-marketing-prod`, and `moz-fx-data-bq-people`.

Static tables, which are deployed from CSV files rather than generated SQL, are published separately via the `publish_static_tables` task for `moz-fx-data-shared-prod` and `mozdata`.

## View Deployment

Expand Down
2 changes: 1 addition & 1 deletion src/concepts/pipeline/event_pipeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ On the pipeline side, the event data is made available in different datasets:
its event payload for Firefox versions before 62.
- [`events`](../../datasets/batch_view/events/reference.md) contains a row for each event received from main pings and event pings. See [`STMO#52582`](https://sql.telemetry.mozilla.org/queries/52582/source).
- For applications that collect events through Glean, each application has a separate `events` dataset.
- [`events_daily`](../../datasets/bigquery/events_daily/reference.md) encodes each client-day's events as a compressed string of Unicode characters, for use with the `mozfun.event_analysis` functions for funnel and event-count analysis.
- [`events_daily`](../../datasets/bigquery/events_daily/reference.md) (generated for Firefox Desktop, Fenix, and Mozilla VPN; not available for other Glean applications) encodes each client-day's events as a compressed string of Unicode characters, for use with the `mozfun.event_analysis` functions for funnel and event-count analysis. The Firefox Desktop version of this table (`telemetry_derived.events_daily_v1`) is deprecated and no longer actively scheduled (see [bug 1805722](https://bugzilla.mozilla.org/show_bug.cgi?id=1805722)).

# Data tooling

Expand Down
10 changes: 6 additions & 4 deletions src/cookbooks/publishing_datasets.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,13 @@ owners:
labels:
application: firefox
incremental: true # incremental queries add data to existing tables
schedule: daily # scheduled in Airflow to run daily
public_json: true
public_bigquery: true
review_bugs: # Bugzilla bug IDs of data review
- 1414839
incremental_export: false # non-incremental JSON export writes all data to a single location
scheduling:
dag_name: bqetl_ssl_ratios
```

The following options define how data is published:
Expand All @@ -41,11 +42,12 @@ The following options define how data is published:
- `incremental`: indicates how data gets updated based on the query and Airflow configuration
- `true`: data gets incrementally updated
- `false`: the entire table data gets updated
- `review_bugs`: list of Bugzilla bug numbers of the data review
- `review_bugs`: list of Bugzilla bug numbers of the data review; required, and enforced by CI validation, whenever `public_bigquery` or `public_json` is set
- `legacy`: set to `true` to indicate a dataset that selects only from legacy Telemetry

Data will get published when the query is executed in Airflow. Metadata of available public
data on Cloud Storage is updated daily through a separate Airflow task.
Data for `public_json` gets published when the query runs in Airflow, and metadata of available public
data on Cloud Storage is updated daily through a separate Airflow task. Tables flagged `public_bigquery`
are deployed to the `mozilla-public-data` project nightly via the `bqetl_artifact_deployment` Airflow DAG.

More information about accessing public data can be found in
[Accessing Public Data](../cookbooks/public_data.md).
Expand Down
2 changes: 2 additions & 0 deletions src/datasets/active_profiles.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Intro

**Note:** This dataset is retired/archived. The pipeline no longer exists in Mozilla's current `bigquery-etl` or `telemetry-airflow` repositories, and the table is no longer generated; this page is kept for historical reference. See [Scheduling](#scheduling) below.

The `active_profiles` dataset gives client-level estimates of whether a profile
is still an active user of the browser at a given point in time, as well as probabilistic forecasts
of the client's future activity. These quantities are estimated by a model that attempts to infer
Expand Down
2 changes: 2 additions & 0 deletions src/datasets/batch_view/clients_daily/reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,8 @@ This dataset is updated daily via the
[telemetry-airflow](https://github.com/mozilla/telemetry-airflow) infrastructure.
The job runs as part of the [`main_summary` DAG](https://github.com/mozilla/bigquery-etl/blob/master/dags/bqetl_main_summary.py).

Downstream tasks that depend on this dataset include `jetstream`, `operational_monitoring`, `parquet_export`, and `catalyst`.

## Schema

The data is partitioned by `submission_date`.
Expand Down
6 changes: 5 additions & 1 deletion src/datasets/bigquery/clients_last_seen/reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ Please see this [section](../../../concepts/segments.md) for descriptions regard

- `is_core_active_v1`: Boolean indicating if the client satisfies conditions of being core active on that day.
- `activity_segments_v1`: The activity segment applicable to the client that day.
- `segment`: Duplicate of `activity_segments_v1`, using the same logic.
- `is_regular_user_v3`: Boolean indicating if the client satisfies conditions of being a regular user on that day.
- `is_new_or_resurrected_v3`: Boolean indicating if the client satisfies conditions of being a regular user on that day.
- `is_weekday_regular_v1`: Boolean indicating if the client satisfies conditions of being a weekday regular user on that day.
Expand All @@ -91,7 +92,10 @@ Please see this [section](../../../concepts/segments.md) for descriptions regard
- `days_had_8_active_ticks_bits`: Each bit field represents if a client had at least 8 active ticks on that day.
This can be used to approximate the threshold of 1 URI, and is useful for determining activity for clients using
Private Browsing Mode where URI counts are not recorded.
- `days_since_visited_8_active_ticks`: Number of days since the client had at least 8 active ticks.
- `days_since_had_8_active_ticks`: Number of days since the client had at least 8 active ticks.

- `days_active_bits`: Each bit field represents if a client had at least 1 active tick and visited at least 1 URI on that day.
- `days_since_active`: Number of days since the client was active (had at least 1 active tick and visited at least 1 URI).

### New Profile Specific

Expand Down