fix - sso login issue for new users and stats missing issue

[LalitDeore]

Issues:

1. Currently, we are not storing onprem stats in a GCP bucket when the Datastore entity size limit is exceeded.
2. Fix the way daily_statistics and onprem_stats are loaded from Datastore. When a new field is added to the daily_statistics struct and the main backend is deployed before another backend, the older backend fail to load daily_statistics. On the next stats update, this reset daily_statistics and cause the organization to lose its stats.
3. When SSO is required for a tenant, new users are unable to log in because their account must first be connected to SSO. Since this is their first login, they cannot complete the SSO connection.

Fixes:

1. Store the on-prem stats in a GCP bucket when the Datastore entity size limit is reached.
2. Store daily_statistics as a JSON string in Datastore instead of mapping it directly to the DailyStatistics struct. This prevents issues when different backend versions have different fields in the struct and ensures stats are preserved during deployments.
3. When SSO is required but the user's account has not yet been connected to SSO, use the login authentication mode. This allows the user to log in and connect their account to SSO.

[frikky]

"Store daily_statistics as a JSON string in Datastore instead of mapping it directly to the DailyStatistics struct. This prevents issues when different backend versions have different fields in the struct and ensures stats are preserved during deployments."

This specifically is really bad, especially since we actually use these stats quite frequently. About 40% of our current CPU usage is literally JSON unmarshalling right now. And that's kind of something I don't want to exacerbate. Field not mapping is not a "lets change the type" problem. It's a problem of keeping the same between backend problem. And it only occurs if you REMOVE a field in a struct array. Adding fields doesnt matter.

[LalitDeore]

"Store daily_statistics as a JSON string in Datastore instead of mapping it directly to the DailyStatistics struct. This prevents issues when different backend versions have different fields in the struct and ensures stats are preserved during deployments."

This specifically is really bad, especially since we actually use these stats quite frequently. About 40% of our current CPU usage is literally JSON unmarshalling right now. And that's kind of something I don't want to exacerbate. Field not mapping is not a "lets change the type" problem. It's a problem of keeping the same between backend problem. And it only occurs if you REMOVE a field in a struct array. Adding fields doesnt matter.

I need to look into the JSON unmarshalling issue. We lose daily_statistics not only when we remove a field from it, but also when we add a new field to daily_statistics. I tested this by adding a field directly to daily_statistics in the DB for an org, and after running a workflow, it reset the daily_statistics for that org.

PS: I am making this PR a draft to try another fix instead of JSON unmarshalling. I have raised a separate PR for the SSO issue so we can push it early :)

[frikky]

"Store daily_statistics as a JSON string in Datastore instead of mapping it directly to the DailyStatistics struct. This prevents issues when different backend versions have different fields in the struct and ensures stats are preserved during deployments."
This specifically is really bad, especially since we actually use these stats quite frequently. About 40% of our current CPU usage is literally JSON unmarshalling right now. And that's kind of something I don't want to exacerbate. Field not mapping is not a "lets change the type" problem. It's a problem of keeping the same between backend problem. And it only occurs if you REMOVE a field in a struct array. Adding fields doesnt matter.

I need to look into the JSON unmarshalling issue. We lose daily_statistics not only when we remove a field from it, but also when we add a new field to daily_statistics. I tested this by adding a field directly to daily_statistics in the DB for an org, and after running a workflow, it reset the daily_statistics for that org.

PS: I am making this PR a draft to try another fix instead of JSON unmarshalling. I have raised a separate PR for the SSO issue so we can push it early :)

No. Struct JSON mapping does not fail when you have a new field. If so, please create a test program proving it, because that entirely goes against how structs work.

PS: If you want it to be a string, how do you intend to make it indexable? You need to think about this architecturally, for long-term use. And known data types should never go into strings/bytes. Unknown data is the only option, and even then, we "want" to use maps or interfaces (where possible).

[frikky]

bump