Specified why we need to test on staging #23325
Conversation
In an engineering meeting on 4th December 2024 we discussed that performance on our local machines is not comparable to a production environment, the outcome of this meeting was an agreement that we would test migrations on staging (or production-like environment) in order to understand the performance qualities.
WalkthroughThe migration review GitHub workflow file was updated to revise the wording of a checklist item. The item previously stated "Tested on the staging database servers" and was changed to "Tested performance on the staging database servers," clarifying that performance testing is required and that performance on local machines is not comparable to a production environment. No other modifications were made to the checklist or the workflow's logic. No changes were made to any exported or public entities. Tip ⚡️ Faster reviews with caching
Enjoy the performance boost—your workflow just got faster. 📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
⏰ Context from checks skipped due to timeout of 90000ms (5)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #23325 +/- ##
==========================================
- Coverage 72.36% 72.35% -0.01%
==========================================
Files 1513 1513
Lines 107338 107338
Branches 13230 13230
==========================================
- Hits 77671 77666 -5
- Misses 28652 28670 +18
+ Partials 1015 1002 -13
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
| @@ -30,7 +30,7 @@ jobs: | |||
|
|
|||
| ### General requirements | |||
|
|
|||
| - [ ] :warning: Tested on the staging database servers | |||
| - [ ] :warning: Tested performance on the staging database servers | |||
There was a problem hiding this comment.
Maybe adding the why directly in the checklist helps to provide context in the future?
| - [ ] :warning: Tested performance on the staging database servers | |
| - [ ] :warning: Tested performance on staging database servers, as performance on local machines is not comparable to a production environment |
There was a problem hiding this comment.
@allouis I do wonder whether / where we have documented the how here, and if there's maybe a link or reference we could provide to get to that documentation easily from the checklist.
There was a problem hiding this comment.
From our discussion on a call, I took these notes -- maybe some of this is already documented, and some of it still needs to be documented. Either way it would be nice to have a pointer in the checklist item(s) to where to look this up.
When should we test on staging servers?
=> When the migration does something potentially costly in a potentially large table.
What are potentially large or difficult tables?
email_recipientsposts
What are potentially costly operations?
DDL / changing data structures:
- adding an index
- dropping columns
- maybe: adding a default value to a previously nullable column
DML / changing data:
- calculating data
- loops (and potentially nested loops)
The distinction between MySQL vs. SQLite might be necessary to test when ...
- renaming columns
- changing data involving timestamps or booleans
- adding lots of data (SQLite might have low limits)
What we didn't have a good insight on yet was how to test on staging servers, which I think we should write up somewhere.
Co-authored-by: Leif Singer <[email protected]>
In an engineering meeting on 4th December 2024 we discussed that performance on our local machines is not comparable to a production environment, the outcome of this meeting was an agreement that we would test migrations on staging (or production-like environment) in order to understand the performance qualities.