FAL-2260 Prepare opencraft release branch for lilac.2 by samuelallan72 · Pull Request #175 · open-craft/configuration

samuelallan72 · 2021-10-05T05:49:03Z

Port the following PRs / commits from koa to lilac:

These are skipped because they have already been upstreamed in lilac 🎉 :

[SE-4304] [SE-4495] feat: add celerybeat configuration #170

And this isn't required any more (see comments below):

Remove AppleId third party backend from defaults #140

Test instructions:

For each patch from edx/configuration@open-release/koa.master...open-craft:opencraft-release/koa.3 , verify that it has either been upstreamed or ported cleanly.

Launch a lilac instance with ocim, using this configuration branch, and verify everything is deployed successfully. (wip - deploying on https://manage.opencraft.com/instance/29580/ )

Notes / Concerns:

Not sure about 93ba25e - we don't need to use our fork of tubular any more, since upstream has merged https://github.com/edx/tubular/pull/532 . The upstream fix is in lilac.master though, not lilac.2. I'd recommend modifying that commit to pin to edx/tubular@open-release/lilac.master , but not sure if that's what we want to do here. I'm not sure why we're pinning it here, and not in private extra-configuration. CC @lgp171188
Also not sure about Remove AppleId third party backend from defaults #140 - upstream had changed the import paths CC @kaizoku
I couldn't find an associated PR for edx/configuration@4df1323 CC @pkulkark could you check this?

michaelbukachi · 2021-10-12T01:24:50Z

I don't see the changes from [SE-4304] [SE-4495] feat: add celerybeat configuration.

samuelallan72 · 2021-10-12T03:33:31Z

@michaelbukachi

I don't see the changes from [SE-4304] [SE-4495] feat: add celerybeat configuration.

This is this commit: dd2f2ac The diff shows less changes than the original commit, because most of it has been upstreamed.

CC @gabor-boros do we need the 'install simple-beat to control celerybeat processes' task? Will it be upstreamed?

michaelbukachi · 2021-10-12T18:56:19Z

@swalladge looks good.

gabor-boros · 2021-10-13T17:05:53Z

@swalladge I'm a bit confused by your question. The changes are upstreamed as part of

Am I missing something here?

samuelallan72 · 2021-10-13T22:17:58Z

@gabor-boros

I'm a bit confused by your question. The changes are upstreamed as part of ...

Am I missing something here?

aaah sorry, I was the one missing something. :P Yep it's all been upstreamed; git was playing tricks on me when I checked the diff and applied the commit though. 🤦‍♂️

gabor-boros · 2021-10-13T22:34:06Z

@swalladge I believe some PRs/commits are missing. Correct me if I overlooked it, but OCIM now can provision instances that are using multi tenant redis. Now, the celery upgrade is not upstreaming yet as I'm still fighting with the ticket in the current sprint.

Taking that into account, shouldn't we port that to opencraft lilac as well? Since it is something that will never end up on edx/lilac but we want to use it, I guess we should cherry-pick that too.

Cc: @pomegranited

samuelallan72 · 2021-10-13T23:10:50Z

@gabor-boros

I believe some PRs/commits are missing. Correct me if I overlooked it, but OCIM now can provision instances that are using multi tenant redis. Now, the celery upgrade is not upstreaming yet as I'm still fighting with the ticket in the current sprint.

Taking that into account, shouldn't we port that to opencraft lilac as well?

Sure yes definitely. Are these commits/PRs you're talking about on edx/configuration and in koa? I didn't see any other commits related to redis/celery. If they're in edx/configuration master, then we can backport to our lilac branch after this commit is merged (ie. after we actually cut our lilac release).

gabor-boros · 2021-10-17T07:57:35Z

@gabor-boros

I believe some PRs/commits are missing. Correct me if I overlooked it, but OCIM now can provision instances that are using multi tenant redis. Now, the celery upgrade is not upstreaming yet as I'm still fighting with the ticket in the current sprint.

Taking that into account, shouldn't we port that to opencraft lilac as well?

Sure yes definitely. Are these commits/PRs you're talking about on edx/configuration and in koa? I didn't see any other commits related to redis/celery. If they're in edx/configuration master, then we can backport to our lilac branch after this commit is merged (ie. after we actually cut our lilac release).

Well, it seems I remembered wrongly. Sorry about the confusion!

pkulkark · 2021-10-18T08:06:34Z

@swalladge

I couldn't find an associated PR for edx@4df1323 CC @pkulkark could you check this?

I believe that was a backport of this upstream PR: https://github.com/edx/configuration/pull/6465.

samuelallan72 · 2021-10-18T21:55:57Z

@pkulkark

I believe that was a backport of this upstream PR: edx#6465.

Ok. Was there no corresponding PR for the backport?

kaizoku · 2021-10-18T22:44:49Z

@swalladge, for #140 , the AppleIdAuth backend is in lilac now, so that particular PR shouldn't be necessary.
I do see they noted that the path changed, which is reflected in lms/envs/production.py, so we'll need to updated those paths for EDXAPP_THIRD_PARTY_AUTH_BACKENDS in playbooks/roles/edxapp/defaults/main.yml too.

…ements #162 (cherry picked from commit 9daa748)

#164 (cherry picked from commit 590f352)

… it into forum role (#159) #159 (cherry picked from commit 066fbf5)

This PR add the flags to sync the edx services logs to s3 bucket. (cherry picked from commit eb402aa)

…ls_extra_requirements_koa feature: Allow installing extra requirements when using the `credentials` role (cherry picked from commit 7ce36c7)

(cherry picked from commit 4df1323)

…ygithub-fix [FAL-2409] Use the tubular pygithub fix in our common branch (cherry picked from commit a957ce3)

samuelallan72 · 2021-10-19T00:22:11Z

@kaizoku

the AppleIdAuth backend is in lilac now, so that particular PR shouldn't be necessary.

Awesome! :)

I do see they noted that the path changed, which is reflected in lms/envs/production.py, so we'll need to updated those paths for EDXAPP_THIRD_PARTY_AUTH_BACKENDS in playbooks/roles/edxapp/defaults/main.yml too.

Looks like that's already been done, so that's all good now.

samuelallan72 · 2021-10-19T00:36:13Z

+        DOMAIN: "{{ COMMON_GIT_MIRROR }}"
+        PATH: "open-craft"
+        REPO: "tubular.git"
+        VERSION: "opencraft-release/koa.3"


@pomegranited what are your thoughts here? We can't use open-release/lilac.2 for tubular because https://github.com/edx/tubular/pull/531 didn't make it to lilac.2, but we should default to a lilac release here to avoid issues. Maybe we can default to open-release/lilac.master here?

@swalladge That sounds safe -- tubular isn't changed very often, so we're not risking much by using a potentially-shifting lilac.master target.

👍 I updated in 06fb8f4 - would you mind sanity checking this?

I'm also deploying a new sandbox instance for testing. CC @michaelbukachi if you'd like to make another review pass now that there has been some changes.

@swalladge 06fb8f4 looks good 👍 Thanks for simplifying that drift too.

@michaelbukachi To test this, you can check the sandbox server for retirement-related errors in /edx/var/log/**/*.log, and verify that you're able to create and delete a user successfully.

Previously, we pinned to an opencraft release as a quick fix, but that quick fix is now merged to lilac upstream. (The fix being https://github.com/edx/tubular/pull/531 ) It didn't make it to lilac.2, so we need to pin to lilac.master.

michaelbukachi

LG

pomegranited · 2021-10-29T00:53:35Z

@swalladge I don't see your change from https://github.com/edx/configuration/pull/6462 in this branch? It was partially backported to our koa.3 release branch by #176 but that port unfortunately doesn't include the EDXAPP_ENABLE_MONGODB_INDEXES and FORUM_ENABLE_MONGODB_INDEXES gates that your upstream PR included, creating some code drift.

I think https://github.com/edx/configuration/pull/6462 should be ported to this branch, and the *_ENABLE_MONGODB_INDEXES flags enabled in Ocim. Any client sites that want to use them will have to enable them too, CC @Agrendalath

I will include this in the scope of FAL-2543, which is semi-related.

samuelallan72 · 2021-10-29T01:00:51Z

@pomegranited

I don't see your change from edx#6462 in this branch?

Yeah they wouldn't be unless we already had backported it to our koa branch, or upstream had included it in lilac.2...

I think edx#6462 should be ported to this branch, and the *_ENABLE_MONGODB_INDEXES flags enabled in Ocim

Sounds like a good plan 👍

I will include this in the scope of FAL-2543, which is semi-related.

Thanks! :D

Agrendalath · 2021-10-29T01:24:08Z

@pomegranited, @swalladge, huh, I had no idea we've created two tickets for this 🤦

samuelallan72 · 2021-10-29T01:33:28Z

@Agrendalath oh haha I didn't even notice they were different tickets. Looks like we forgot to backport to the client branch in the first ticket, and the second ticket didn't notice that we'd already implemented it upstream.

samuelallan72 requested a review from michaelbukachi October 5, 2021 05:49

samuelallan72 self-assigned this Oct 5, 2021

samuelallan72 force-pushed the samuel/prepare-lilac.2-release branch from 8e54316 to 082fa19 Compare October 5, 2021 05:58

samuelallan72 requested review from kaizoku and lgp171188 October 11, 2021 04:51

samuelallan72 changed the title ~~WIP: FAL-2260 Prepare opencraft release branch for lilac.2~~ FAL-2260 Prepare opencraft release branch for lilac.2 Oct 11, 2021

samuelallan72 requested a review from pkulkark October 11, 2021 04:53

michaelbukachi reviewed Oct 12, 2021

View reviewed changes

Comment thread playbooks/retire_host.yml

samuelallan72 force-pushed the samuel/prepare-lilac.2-release branch from 93ba25e to 188133c Compare October 13, 2021 22:18

0x29a and others added 7 commits October 19, 2021 10:50

BB-3669 feat: ansible task to install extra edx django service requir…

694e6ce

…ements #162 (cherry picked from commit 9daa748)

[SE-4419] Add script to generate the daily uptime reports (#164)

a07798c

#164 (cherry picked from commit 590f352)

[SE-4364] Update reindex_courses command to use --all option and move…

0573de1

… it into forum role (#159) #159 (cherry picked from commit 066fbf5)

[SE-4481]: Upload edx-services logs to s3 bucket (#167)

1116e75

This PR add the flags to sync the edx services logs to s3 bucket. (cherry picked from commit eb402aa)

Merge pull request #169 from open-craft/joaocabrita/bb_4342-credentia…

e0219d1

…ls_extra_requirements_koa feature: Allow installing extra requirements when using the `credentials` role (cherry picked from commit 7ce36c7)

fix: put edx service logs sync behind flag

b2e8606

(cherry picked from commit 4df1323)

Merge pull request #173 from open-craft/guruprasad/FAL-2409-tubular-p…

b5eeb9b

…ygithub-fix [FAL-2409] Use the tubular pygithub fix in our common branch (cherry picked from commit a957ce3)

samuelallan72 force-pushed the samuel/prepare-lilac.2-release branch from 188133c to b5eeb9b Compare October 19, 2021 00:20

samuelallan72 commented Oct 19, 2021

View reviewed changes

michaelbukachi approved these changes Oct 19, 2021

View reviewed changes

samuelallan72 merged commit 0252892 into opencraft-release/lilac.2 Oct 25, 2021

samuelallan72 deleted the samuel/prepare-lilac.2-release branch October 25, 2021 00:56

Uh oh!

Conversation

samuelallan72 commented Oct 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

michaelbukachi commented Oct 12, 2021

Uh oh!

samuelallan72 commented Oct 12, 2021

Uh oh!

michaelbukachi commented Oct 12, 2021

Uh oh!

gabor-boros commented Oct 13, 2021

Uh oh!

samuelallan72 commented Oct 13, 2021

Uh oh!

gabor-boros commented Oct 13, 2021

Uh oh!

samuelallan72 commented Oct 13, 2021

Uh oh!

gabor-boros commented Oct 17, 2021

Uh oh!

pkulkark commented Oct 18, 2021

Uh oh!

samuelallan72 commented Oct 18, 2021

Uh oh!

kaizoku commented Oct 18, 2021

Uh oh!

samuelallan72 commented Oct 19, 2021

Uh oh!

samuelallan72 Oct 19, 2021

Choose a reason for hiding this comment

Uh oh!

pomegranited Oct 19, 2021

Choose a reason for hiding this comment

Uh oh!

samuelallan72 Oct 19, 2021

Choose a reason for hiding this comment

Uh oh!

michaelbukachi Oct 19, 2021

Choose a reason for hiding this comment

Uh oh!

pomegranited Oct 19, 2021

Choose a reason for hiding this comment

Uh oh!

michaelbukachi left a comment

Choose a reason for hiding this comment

Uh oh!

pomegranited commented Oct 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

samuelallan72 commented Oct 29, 2021

Uh oh!

Agrendalath commented Oct 29, 2021

Uh oh!

samuelallan72 commented Oct 29, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

samuelallan72 commented Oct 5, 2021 •

edited

Loading

pomegranited commented Oct 29, 2021 •

edited

Loading