Skip to content

Github archive revert #2757

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
garrrikkotua merged 33 commits into from
Jan 22, 2025
Merged

Github archive revert #2757

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
0218ca7
Github archive shutdown
gaspergrom Jan 10, 2025
4a942f1
fix for consumer timeouts
themarolt Jan 10, 2025
08acd39
fix for consumer timeouts
themarolt Jan 10, 2025
f111a3c
changed questdb activities query log to be debug on info level
themarolt Jan 10, 2025
c238b3e
handle kafka heartbeat errors
themarolt Jan 10, 2025
7fd2a05
handle kafka heartbeat errors
themarolt Jan 10, 2025
88f46de
handle kafka heartbeat errors
themarolt Jan 10, 2025
39b768f
handle kafka messages individually
themarolt Jan 10, 2025
43062c2
Test queue perf (#2758)
themarolt Jan 11, 2025
9a17ec5
Remove GDPR checks and related logic from contributor components (#2762)
Sameh16 Jan 13, 2025
e515ed0
Improvement/lfx 1957 unused code (#2759)
emlimlf Jan 13, 2025
7f1b676
Remove unused features, configs, and sample data (#2740)
skwowet Jan 13, 2025
0d65240
Fix dropdown issues
gaspergrom Jan 13, 2025
9c982d1
Remove backend tests and related code (#2748)
skwowet Jan 13, 2025
2589705
Test queue perf (#2763)
themarolt Jan 13, 2025
bd1d1a9
start
garrrikkotua Jan 14, 2025
0c771dc
fix as it was
garrrikkotua Jan 14, 2025
fde9b61
introduce env and switch libs
garrrikkotua Jan 16, 2025
33e3ca8
kinda workds
garrrikkotua Jan 16, 2025
a7cf90f
Merge branch 'main' into improvement/github-archive-shutdown
garrrikkotua Jan 20, 2025
a0473d3
update pnpm lock
garrrikkotua Jan 20, 2025
7d92756
Merge branch 'main' into improvement/github-archive-shutdown
garrrikkotua Jan 20, 2025
4565618
add missing file
garrrikkotua Jan 20, 2025
88199e0
works
garrrikkotua Jan 21, 2025
5434e97
don't push attrs in non sf mode
garrrikkotua Jan 21, 2025
1ea90e6
format back
garrrikkotua Jan 21, 2025
facbac2
fix issues
garrrikkotua Jan 21, 2025
f11b804
Merge branch 'main' into improvement/github-archive-shutdown
garrrikkotua Jan 21, 2025
cacb58d
fix refresh repo settings
garrrikkotua Jan 21, 2025
d5c390c
fix lint in run worker
garrrikkotua Jan 21, 2025
1676e8b
Enable github archive by env
gaspergrom Jan 21, 2025
71cb3ee
fix linting and types
garrrikkotua Jan 22, 2025
e5e4233
remove any
garrrikkotua Jan 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion backend/config/custom-environment-variables.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,8 @@
"callbackUrl": "CROWD_GITHUB_CALLBACK_URL",
"privateKey": "CROWD_GITHUB_PRIVATE_KEY",
"webhookSecret": "CROWD_GITHUB_WEBHOOK_SECRET",
"isCommitDataEnabled": "CROWD_GITHUB_IS_COMMIT_DATA_ENABLED"
"isCommitDataEnabled": "CROWD_GITHUB_IS_COMMIT_DATA_ENABLED",
"isSnowflakeEnabled": "CROWD_GITHUB_IS_SNOWFLAKE_ENABLED"
},
"githubIssueReporter": {
"appId": "CROWD_GITHUB_ISSUE_REPORTER_APP_ID",
Expand Down
3 changes: 2 additions & 1 deletion backend/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,8 @@
"script:purge-tenants-and-data": "SERVICE=script TS_NODE_TRANSPILE_ONLY=true tsx src/bin/scripts/purge-tenants-and-data.ts",
"script:import-lfx-memberships": "SERVICE=script TS_NODE_TRANSPILE_ONLY=true tsx src/bin/scripts/import-lfx-memberships.ts",
"script:fix-missing-org-displayName": "SERVICE=script TS_NODE_TRANSPILE_ONLY=true tsx src/bin/scripts/fix-missing-org-displayName.ts",
"script:syncActivities": "SERVICE=script TS_NODE_TRANSPILE_ONLY=true tsx src/bin/scripts/syncActivities.ts"
"script:syncActivities": "SERVICE=script TS_NODE_TRANSPILE_ONLY=true tsx src/bin/scripts/syncActivities.ts",
"script:refreshGithubRepoSettings": "SERVICE=script TS_NODE_TRANSPILE_ONLY=true tsx src/bin/scripts/refresh-github-repo-settings.ts"
},
"dependencies": {
"@aws-sdk/client-comprehend": "^3.159.0",
Expand Down
13 changes: 13 additions & 0 deletions backend/src/api/integration/helpers/githubAuthenticate.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
import Permissions from '../../../security/permissions'
import IntegrationService from '../../../services/integrationService'
import PermissionChecker from '../../../services/user/permissionChecker'

export default async (req, res) => {
new PermissionChecker(req).validateHas(Permissions.values.tenantEdit)
const payload = await new IntegrationService(req).connectGithub(
req.params.code,
req.body.installId,
req.body.setupAction,
)
await req.responseHandler.success(req, res, payload)
}
9 changes: 9 additions & 0 deletions backend/src/api/integration/helpers/githubConnectInstallation.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
import Permissions from '../../../security/permissions'
import IntegrationService from '../../../services/integrationService'
import PermissionChecker from '../../../services/user/permissionChecker'

export default async (req, res) => {
new PermissionChecker(req).validateHas(Permissions.values.tenantEdit)
const payload = await new IntegrationService(req).connectGithubInstallation(req.body.installId)
await req.responseHandler.success(req, res, payload)
}
9 changes: 9 additions & 0 deletions backend/src/api/integration/helpers/githubGetInstallations.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
import Permissions from '../../../security/permissions'
import IntegrationService from '../../../services/integrationService'
import PermissionChecker from '../../../services/user/permissionChecker'

export default async (req, res) => {
new PermissionChecker(req).validateHas(Permissions.values.tenantEdit)
const payload = await new IntegrationService(req).getGithubInstallations()
await req.responseHandler.success(req, res, payload)
}
25 changes: 19 additions & 6 deletions backend/src/api/integration/helpers/githubMapRepos.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,27 @@
import { GITHUB_CONFIG } from '@/conf'

import Permissions from '../../../security/permissions'
import IntegrationService from '../../../services/integrationService'
import PermissionChecker from '../../../services/user/permissionChecker'

const isSnowflakeEnabled = GITHUB_CONFIG.isSnowflakeEnabled === 'true'

export default async (req, res) => {
new PermissionChecker(req).validateHas(Permissions.values.tenantEdit)
const payload = await new IntegrationService(req).mapGithubRepos(
req.params.id,
req.body.mapping,
true,
req.body?.isUpdateTransaction ?? false,
)
let payload
if (isSnowflakeEnabled) {
payload = await new IntegrationService(req).mapGithubReposSnowflake(
req.params.id,
req.body.mapping,
true,
req.body?.isUpdateTransaction ?? false,
)
} else {
payload = await new IntegrationService(req).mapGithubRepos(
req.params.id,
req.body.mapping,
true,
)
}
await req.responseHandler.success(req, res, payload)
}
13 changes: 13 additions & 0 deletions backend/src/api/integration/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,22 @@ export default (app) => {
app.get(`/integration/autocomplete`, safeWrap(require('./integrationAutocomplete').default))
app.get(`/integration/global`, safeWrap(require('./integrationGlobal').default))
app.get(`/integration/global/status`, safeWrap(require('./integrationGlobalStatus').default))

app.get(
'/integration/github-installations',
safeWrap(require('./helpers/githubGetInstallations').default),
)

app.post(
'/integration/github-connect-installation',
safeWrap(require('./helpers/githubConnectInstallation').default),
)

app.get(`/integration`, safeWrap(require('./integrationList').default))
app.get(`/integration/:id`, safeWrap(require('./integrationFind').default))

app.put(`/authenticate/:code`, safeWrap(require('./helpers/githubAuthenticate').default))

app.put(`/integration/:id/github/repos`, safeWrap(require('./helpers/githubMapRepos').default))
app.get(`/integration/:id/github/repos`, safeWrap(require('./helpers/githubMapReposGet').default))
app.get(
Expand Down
58 changes: 55 additions & 3 deletions backend/src/bin/jobs/refreshGithubRepoSettings.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,15 @@ import cronGenerator from 'cron-time-generator'
import { timeout } from '@crowd/common'
import { getServiceChildLogger } from '@crowd/logging'

import { GITHUB_CONFIG } from '../../conf'
import GithubReposRepository from '../../database/repositories/githubReposRepository'
import SequelizeRepository from '../../database/repositories/sequelizeRepository'
import GithubIntegrationService from '../../services/githubIntegrationService'
import IntegrationService from '../../services/integrationService'
import { CrowdJob } from '../../types/jobTypes'

const IS_GITHUB_SNOWFLAKE_ENABLED = GITHUB_CONFIG.isSnowflakeEnabled === 'true'

const log = getServiceChildLogger('refreshGithubRepoSettings')

interface Integration {
Expand All @@ -33,8 +36,7 @@ interface Integration {
}
}

export const refreshGithubRepoSettings = async () => {
log.info('Updating Github repo settings.')
const refreshForSnowflake = async () => {
const dbOptions = await SequelizeRepository.getDefaultIRepositoryOptions()

const githubIntegrations = await dbOptions.database.sequelize.query(
Expand Down Expand Up @@ -124,7 +126,7 @@ export const refreshGithubRepoSettings = async () => {
}

// Map new repos
await integrationService.mapGithubRepos(
await integrationService.mapGithubReposSnowflake(
integration.id,
newFullMapping,
true,
Expand All @@ -150,6 +152,56 @@ export const refreshGithubRepoSettings = async () => {
log.info('Finished updating Github repo settings.')
}

const refreshForGitHub = async () => {
log.info('Updating Github repo settings.')
const dbOptions = await SequelizeRepository.getDefaultIRepositoryOptions()

interface Integration {
id: string
tenantId: string
integrationIdentifier: string
}

const githubIntegrations = await dbOptions.database.sequelize.query(
`SELECT id, "tenantId", "integrationIdentifier" FROM integrations
WHERE platform = 'github' AND "deletedAt" IS NULL
AND "createdAt" < NOW() - INTERVAL '1 minute' AND "integrationIdentifier" IS NOT NULL`,
)

for (const integration of githubIntegrations[0] as Integration[]) {
log.info(`Updating repo settings for Github integration: ${integration.id}`)

try {
const options = await SequelizeRepository.getDefaultIRepositoryOptions()
options.currentTenant = { id: integration.tenantId }

const integrationService = new IntegrationService(options)
// newly discovered repos will be mapped to default segment of the integration
await integrationService.updateGithubIntegrationSettings(integration.integrationIdentifier)

log.info(`Successfully updated repo settings for Github integration: ${integration.id}`)
} catch (err) {
log.error(
`Error updating repo settings for Github integration ${integration.id}: ${err.message}`,
)
} finally {
await timeout(1000)
}
}

log.info('Finished updating Github repo settings.')
}

export const refreshGithubRepoSettings = async () => {
log.info('Updating Github repo settings.')

if (IS_GITHUB_SNOWFLAKE_ENABLED) {
await refreshForSnowflake()
} else {
await refreshForGitHub()
}
}

const job: CrowdJob = {
name: 'Refresh Github repo settings',
// every day
Expand Down
1 change: 1 addition & 0 deletions backend/src/conf/configTypes.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,7 @@ export interface GithubConfiguration {
privateKey: string
webhookSecret: string
isCommitDataEnabled: string
isSnowflakeEnabled: string
globalLimit?: number
callbackUrl: string
}
Expand Down
19 changes: 18 additions & 1 deletion backend/src/database/repositories/githubReposRepository.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,24 @@ export default class GithubReposRepository {
)
}

static async updateMapping(
static async updateMapping(integrationId, mapping, options: IRepositoryOptions) {
const tenantId = options.currentTenant.id

await GithubReposRepository.bulkInsert(
'githubRepos',
['tenantId', 'integrationId', 'segmentId', 'url'],
(idx) => `(:tenantId_${idx}, :integrationId_${idx}, :segmentId_${idx}, :url_${idx})`,
Object.entries(mapping).map(([url, segmentId], idx) => ({
[`tenantId_${idx}`]: tenantId,
[`integrationId_${idx}`]: integrationId,
[`segmentId_${idx}`]: segmentId,
[`url_${idx}`]: url,
})),
options,
)
}

static async updateMappingSnowflake(
integrationId,
newMapping: Record<string, string>,
oldMapping: {
Expand Down
8 changes: 7 additions & 1 deletion backend/src/serverless/integrations/types/regularTypes.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,13 @@ import { PlatformType } from '@crowd/types'
export type Repo = {
url: string
name: string
updatedAt: string
updatedAt?: string
createdAt?: string
owner?: string
available?: boolean
fork?: boolean
private?: boolean
cloneUrl?: string
}

export type Repos = Array<Repo>
Expand Down
64 changes: 64 additions & 0 deletions backend/src/serverless/integrations/usecases/github/rest/getInstalledRepositories.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
import axios, { AxiosRequestConfig } from 'axios'

import { getServiceChildLogger } from '@crowd/logging'

import { Repos } from '../../../types/regularTypes'

const log = getServiceChildLogger('getInstalledRepositories')

const getRepositoriesFromGH = async (page: number, installToken: string): Promise<any> => {
const REPOS_PER_PAGE = 100

const requestConfig = {
method: 'get',
url: `https://api.github.com/installation/repositories?page=${page}&per_page=${REPOS_PER_PAGE}`,
headers: {
Authorization: `Bearer ${installToken}`,
},
} as AxiosRequestConfig

const response = await axios(requestConfig)
return response.data
}

const parseRepos = (repositories: any): Repos => {
const repos: Repos = []

for (const repo of repositories) {
repos.push({
url: repo.html_url,
owner: repo.owner.login,
createdAt: repo.created_at,
name: repo.name,
fork: repo.fork,
private: repo.private,
cloneUrl: repo.clone_url,
})
}

return repos
}

export const getInstalledRepositories = async (installToken: string): Promise<Repos> => {
try {
let page = 1
let hasMorePages = true

const repos: Repos = []

while (hasMorePages) {
const data = await getRepositoriesFromGH(page, installToken)

if (data.repositories) {
repos.push(...parseRepos(data.repositories))
}

hasMorePages = data.total_count && data.total_count > 0 && data.total_count > repos.length
page += 1
}
return repos.filter((repo) => !repo.private && !repo.fork)
} catch (err: any) {
log.error(err, 'Error fetching installed repositories!')
throw err
}
}
Loading
Loading