Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Search #7237

Merged
merged 57 commits into from
Oct 3, 2024
Merged

Search #7237

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
dd5055d
fix test
ijreilly Aug 23, 2024
0904917
Merge branch 'main' of github.com:twentyhq/twenty into search
ijreilly Aug 28, 2024
81d6005
Comments with twenty orm hints
ijreilly Aug 28, 2024
c343155
Merge branch 'main' of github.com:twentyhq/twenty into search
ijreilly Aug 29, 2024
8d76f96
WIP - search with query builder and trgm works (prereq: add trgm exte…
ijreilly Sep 2, 2024
ed90ee9
WIP
ijreilly Sep 3, 2024
66f8747
wip - Add search parameter
ijreilly Sep 3, 2024
77f4063
wip - search on trigram works in front
ijreilly Sep 3, 2024
0989b70
ts_vector works
ijreilly Sep 3, 2024
b758fc8
wip
ijreilly Sep 5, 2024
570b9d3
wip - spilo dockerfile works
ijreilly Sep 6, 2024
c612fc0
Merge branch 'main' of github.com:twentyhq/twenty into search
ijreilly Sep 9, 2024
324ac9b
Merge branch 'main' of github.com:twentyhq/twenty into search
ijreilly Sep 11, 2024
3cbccd3
implementation of search endpoint
ijreilly Sep 11, 2024
6d1092f
Add limit arg to search endpoint
ijreilly Sep 12, 2024
552b4e0
Add feature flag
ijreilly Sep 12, 2024
b258648
wip
ijreilly Sep 16, 2024
cc4764a
Merge branch 'main' of github.com:twentyhq/twenty into search
ijreilly Sep 17, 2024
3401747
wip - searchvector created and works but typeorm issue + no indexes yet
ijreilly Sep 18, 2024
a1213c5
split emails in two words for tsvector
ijreilly Sep 18, 2024
51dcb01
Add searchVector to custom object entity
ijreilly Sep 18, 2024
0716119
create searchVector field at custom object creation
ijreilly Sep 19, 2024
fc053e3
Handle gin index creation with decorator
ijreilly Sep 20, 2024
7e50d5a
Merge branch 'main' of github.com:twentyhq/twenty into search
ijreilly Sep 20, 2024
3929800
remove merge conflict trace
ijreilly Sep 20, 2024
1d67d22
wip
ijreilly Sep 20, 2024
2a1103c
Merge branch 'main' of github.com:twentyhq/twenty into search
ijreilly Sep 20, 2024
24c7d6e
generate graphql for front
ijreilly Sep 20, 2024
6b1d822
wip
ijreilly Sep 20, 2024
4059b61
Add typeOrmMetadata column
ijreilly Sep 23, 2024
3eec4ef
Merge branch 'main' of github.com:twentyhq/twenty into search
ijreilly Sep 23, 2024
1cdbe30
wip
ijreilly Sep 24, 2024
2c70842
Merge branch 'main' of github.com:twentyhq/twenty into search
ijreilly Sep 24, 2024
b08b5aa
wip
ijreilly Sep 24, 2024
5a23053
Fix infinite loop
ijreilly Sep 24, 2024
dff5266
Use .getMany()
ijreilly Sep 26, 2024
b1001c7
Merge branch 'main' of github.com:twentyhq/twenty into search
ijreilly Sep 26, 2024
85a00c9
Concatenate phone fields for ts vector column expressoin
ijreilly Sep 26, 2024
3fa7252
Remove phones from person label identifier
ijreilly Sep 26, 2024
9a06c25
Remove code from paradedb tests
ijreilly Sep 27, 2024
581045b
clean code
ijreilly Sep 27, 2024
eb39b2d
Merge branch 'main' of github.com:twentyhq/twenty into search
ijreilly Sep 30, 2024
88a4d19
minor improvements
ijreilly Sep 30, 2024
1c52263
Extend sync-index to standard indexes on custom objects
ijreilly Sep 30, 2024
3d7bbac
Add feature flag in FE
ijreilly Sep 30, 2024
5605645
Fix standard index factory issue
ijreilly Oct 1, 2024
ef9ab33
Merge branch 'main' of github.com:twentyhq/twenty into search
ijreilly Oct 1, 2024
b05c027
Code improvements
ijreilly Oct 1, 2024
54c17d1
Fix index creation
ijreilly Oct 2, 2024
4652fc7
Fix SQL injection breach
ijreilly Oct 2, 2024
cae5738
Refactor generateFields
ijreilly Oct 2, 2024
5771352
Nullify searchVector when deletedAt is not null
ijreilly Oct 2, 2024
d404c22
improve code quality
ijreilly Oct 2, 2024
2e95942
Fix escapedWord
ijreilly Oct 2, 2024
c6fe5f9
improve code quality
ijreilly Oct 3, 2024
adfc8ee
Fix BTREE indexes inclusion of deletedAt + add tests
ijreilly Oct 3, 2024
1bed77f
Follow conventions for addIndexType
ijreilly Oct 3, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions packages/twenty-front/src/generated-metadata/graphql.ts
Original file line number Diff line number Diff line change
Expand Up @@ -375,6 +375,7 @@ export enum FieldMetadataType {
RichText = 'RICH_TEXT',
Select = 'SELECT',
Text = 'TEXT',
TsVector = 'TS_VECTOR',
Uuid = 'UUID'
}

Expand Down
1 change: 1 addition & 0 deletions packages/twenty-front/src/generated/graphql.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -280,6 +280,7 @@ export enum FieldMetadataType {
RichText = 'RICH_TEXT',
Select = 'SELECT',
Text = 'TEXT',
TsVector = 'TS_VECTOR',
Uuid = 'UUID'
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,9 @@ import { useKeyboardShortcutMenu } from '@/keyboard-shortcut-menu/hooks/useKeybo
import { CoreObjectNameSingular } from '@/object-metadata/types/CoreObjectNameSingular';
import { getCompanyDomainName } from '@/object-metadata/utils/getCompanyDomainName';
import { useFindManyRecords } from '@/object-record/hooks/useFindManyRecords';
import { useSearchRecords } from '@/object-record/hooks/useSearchRecords';
import { makeOrFilterVariables } from '@/object-record/utils/makeOrFilterVariables';
import { Opportunity } from '@/opportunities/Opportunity';
import { Person } from '@/people/types/Person';
import { LightIconButton } from '@/ui/input/button/components/LightIconButton';
import { SelectableItem } from '@/ui/layout/selectable-list/components/SelectableItem';
Expand Down Expand Up @@ -165,8 +167,21 @@ export const CommandMenu = () => {
[closeCommandMenu],
);

const { records: people } = useFindManyRecords<Person>({
skip: !isCommandMenuOpened,
const isTwentyOrmEnabled = useIsFeatureEnabled(
'IS_QUERY_RUNNER_TWENTY_ORM_ENABLED',
);

const isWorkspaceMigratedForSearch = useIsFeatureEnabled(
'IS_WORKSPACE_MIGRATED_FOR_SEARCH',
);

const isSearchEnabled =
useIsFeatureEnabled('IS_SEARCH_ENABLED') &&
isTwentyOrmEnabled &&
isWorkspaceMigratedForSearch;

const { records: peopleFromFindMany } = useFindManyRecords<Person>({
skip: !isCommandMenuOpened || isSearchEnabled,
objectNameSingular: CoreObjectNameSingular.Person,
filter: commandMenuSearch
? makeOrFilterVariables([
Expand All @@ -183,9 +198,24 @@ export const CommandMenu = () => {
: undefined,
limit: 3,
});
const { records: peopleFromSearch } = useSearchRecords<Person>({
skip: !isCommandMenuOpened || !isSearchEnabled,
objectNameSingular: CoreObjectNameSingular.Person,
limit: 3,
searchInput: commandMenuSearch ?? undefined,
});

const { records: companies } = useFindManyRecords<Company>({
skip: !isCommandMenuOpened,
const people = isSearchEnabled ? peopleFromSearch : peopleFromFindMany;

const { records: companiesFromSearch } = useSearchRecords<Company>({
skip: !isCommandMenuOpened || !isSearchEnabled,
objectNameSingular: CoreObjectNameSingular.Company,
limit: 3,
searchInput: commandMenuSearch ?? undefined,
});

const { records: companiesFromFindMany } = useFindManyRecords<Company>({
skip: !isCommandMenuOpened || isSearchEnabled,
objectNameSingular: CoreObjectNameSingular.Company,
filter: commandMenuSearch
? {
Expand All @@ -195,6 +225,10 @@ export const CommandMenu = () => {
limit: 3,
});

const companies = isSearchEnabled
? companiesFromSearch
: companiesFromFindMany;

const { records: notes } = useFindManyRecords<Note>({
skip: !isCommandMenuOpened,
objectNameSingular: CoreObjectNameSingular.Note,
Expand All @@ -207,8 +241,8 @@ export const CommandMenu = () => {
limit: 3,
});

const { records: opportunities } = useFindManyRecords({
skip: !isCommandMenuOpened,
const { records: opportunitiesFromFindMany } = useFindManyRecords({
skip: !isCommandMenuOpened || isSearchEnabled,
objectNameSingular: CoreObjectNameSingular.Opportunity,
filter: commandMenuSearch
? {
Expand All @@ -218,6 +252,17 @@ export const CommandMenu = () => {
limit: 3,
});

const { records: opportunitiesFromSearch } = useSearchRecords<Opportunity>({
skip: !isCommandMenuOpened || !isSearchEnabled,
objectNameSingular: CoreObjectNameSingular.Opportunity,
limit: 3,
searchInput: commandMenuSearch ?? undefined,
});

const opportunities = isSearchEnabled
? opportunitiesFromSearch
: opportunitiesFromFindMany;

const peopleCommands = useMemo(
() =>
people.map(({ id, name: { firstName, lastName } }) => ({
Expand Down
Original file line number Diff line number Diff line change
@@ -1,16 +1,29 @@
import { useEffect } from 'react';
import { useRecoilState, useRecoilValue } from 'recoil';
import { useRecoilCallback, useRecoilValue } from 'recoil';

import { useIsLogged } from '@/auth/hooks/useIsLogged';
import { currentUserState } from '@/auth/states/currentUserState';
import { currentWorkspaceState } from '@/auth/states/currentWorkspaceState';
import { useFindManyObjectMetadataItems } from '@/object-metadata/hooks/useFindManyObjectMetadataItems';
import { objectMetadataItemsState } from '@/object-metadata/states/objectMetadataItemsState';
import { ObjectMetadataItem } from '@/object-metadata/types/ObjectMetadataItem';
import { FieldMetadataType } from '~/generated-metadata/graphql';
import { WorkspaceActivationStatus } from '~/generated/graphql';
import { generatedMockObjectMetadataItems } from '~/testing/mock-data/objectMetadataItems';
import { isDeeplyEqual } from '~/utils/isDeeplyEqual';
import { isUndefinedOrNull } from '~/utils/isUndefinedOrNull';

const filterTsVectorFields = (
objectMetadataItems: ObjectMetadataItem[],
): ObjectMetadataItem[] => {
return objectMetadataItems.map((item) => ({
...item,
fields: item.fields.filter(
(field) => field.type !== FieldMetadataType.TsVector,
),
}));
};

export const ObjectMetadataItemsLoadEffect = () => {
const currentUser = useRecoilValue(currentUserState);
const currentWorkspace = useRecoilValue(currentWorkspaceState);
Expand All @@ -21,26 +34,32 @@ export const ObjectMetadataItemsLoadEffect = () => {
skip: !isLoggedIn,
});

const [objectMetadataItems, setObjectMetadataItems] = useRecoilState(
objectMetadataItemsState,
const updateObjectMetadataItems = useRecoilCallback(
({ set, snapshot }) =>
() => {
const filteredFields = filterTsVectorFields(newObjectMetadataItems);
const toSetObjectMetadataItems =
isUndefinedOrNull(currentUser) ||
currentWorkspace?.activationStatus !==
WorkspaceActivationStatus.Active
? generatedMockObjectMetadataItems
: filteredFields;

if (
!isDeeplyEqual(
snapshot.getLoadable(objectMetadataItemsState).getValue(),
toSetObjectMetadataItems,
)
) {
set(objectMetadataItemsState, toSetObjectMetadataItems);
}
},
[currentUser, currentWorkspace?.activationStatus, newObjectMetadataItems],
);

useEffect(() => {
const toSetObjectMetadataItems =
isUndefinedOrNull(currentUser) ||
currentWorkspace?.activationStatus !== WorkspaceActivationStatus.Active
? generatedMockObjectMetadataItems
: newObjectMetadataItems;
if (!isDeeplyEqual(objectMetadataItems, toSetObjectMetadataItems)) {
setObjectMetadataItems(toSetObjectMetadataItems);
}
}, [
currentUser,
currentWorkspace?.activationStatus,
newObjectMetadataItems,
objectMetadataItems,
setObjectMetadataItems,
]);
updateObjectMetadataItems();
}, [updateObjectMetadataItems]);

return <></>;
};
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
import { RecordGqlConnection } from '@/object-record/graphql/types/RecordGqlConnection';

export type RecordGqlOperationSearchResult = {
[objectNamePlural: string]: RecordGqlConnection;
};
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
import { useQuery, WatchQueryFetchPolicy } from '@apollo/client';
import { useRecoilValue } from 'recoil';

import { currentWorkspaceMemberState } from '@/auth/states/currentWorkspaceMemberState';
import { useObjectMetadataItem } from '@/object-metadata/hooks/useObjectMetadataItem';
import { ObjectMetadataItemIdentifier } from '@/object-metadata/types/ObjectMetadataItemIdentifier';
import { getRecordsFromRecordConnection } from '@/object-record/cache/utils/getRecordsFromRecordConnection';
import { RecordGqlOperationGqlRecordFields } from '@/object-record/graphql/types/RecordGqlOperationGqlRecordFields';
import { RecordGqlOperationSearchResult } from '@/object-record/graphql/types/RecordGqlOperationSearchResult';
import { RecordGqlOperationVariables } from '@/object-record/graphql/types/RecordGqlOperationVariables';
import { useSearchRecordsQuery } from '@/object-record/hooks/useSearchRecordsQuery';
import { ObjectRecord } from '@/object-record/types/ObjectRecord';
import { getSearchRecordsQueryResponseField } from '@/object-record/utils/getSearchRecordsQueryResponseField';
import { SnackBarVariant } from '@/ui/feedback/snack-bar-manager/components/SnackBar';
import { useSnackBar } from '@/ui/feedback/snack-bar-manager/hooks/useSnackBar';
import { useMemo } from 'react';
import { logError } from '~/utils/logError';

export type UseSearchRecordsParams = ObjectMetadataItemIdentifier &
RecordGqlOperationVariables & {
onError?: (error?: Error) => void;
skip?: boolean;
recordGqlFields?: RecordGqlOperationGqlRecordFields;
fetchPolicy?: WatchQueryFetchPolicy;
searchInput?: string;
};

export const useSearchRecords = <T extends ObjectRecord = ObjectRecord>({
objectNameSingular,
searchInput,
limit,
skip,
recordGqlFields,
fetchPolicy,
}: UseSearchRecordsParams) => {
const currentWorkspaceMember = useRecoilValue(currentWorkspaceMemberState);
const { objectMetadataItem } = useObjectMetadataItem({
objectNameSingular,
});
const { searchRecordsQuery } = useSearchRecordsQuery({
objectNameSingular,
recordGqlFields,
});

const { enqueueSnackBar } = useSnackBar();

const { data, loading, error } = useQuery<RecordGqlOperationSearchResult>(
searchRecordsQuery,
{
skip:
skip || !objectMetadataItem || !currentWorkspaceMember || !searchInput,
variables: {
search: searchInput,
limit: limit,
},
fetchPolicy: fetchPolicy,
onError: (error) => {
logError(
`useSearchRecords for "${objectMetadataItem.namePlural}" error : ` +
error,
);
enqueueSnackBar(
`Error during useSearchRecords for "${objectMetadataItem.namePlural}", ${error.message}`,
{
variant: SnackBarVariant.Error,
},
);
},
},
);

const queryResponseField = getSearchRecordsQueryResponseField(
objectMetadataItem.namePlural,
);

const result = data?.[queryResponseField];

const records = useMemo(
() =>
result
? (getRecordsFromRecordConnection({
recordConnection: result,
}) as T[])
: [],
[result],
);

return {
objectMetadataItem,
records: records,
loading,
error,
};
};
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
import { useRecoilValue } from 'recoil';

import { useObjectMetadataItem } from '@/object-metadata/hooks/useObjectMetadataItem';
import { objectMetadataItemsState } from '@/object-metadata/states/objectMetadataItemsState';
import { RecordGqlOperationGqlRecordFields } from '@/object-record/graphql/types/RecordGqlOperationGqlRecordFields';
import { generateSearchRecordsQuery } from '@/object-record/utils/generateSearchRecordsQuery';

export const useSearchRecordsQuery = ({
objectNameSingular,
recordGqlFields,
computeReferences,
}: {
objectNameSingular: string;
recordGqlFields?: RecordGqlOperationGqlRecordFields;
computeReferences?: boolean;
}) => {
const { objectMetadataItem } = useObjectMetadataItem({
objectNameSingular,
});

const objectMetadataItems = useRecoilValue(objectMetadataItemsState);

const searchRecordsQuery = generateSearchRecordsQuery({
objectMetadataItem,
objectMetadataItems,
recordGqlFields,
computeReferences,
});

return {
searchRecordsQuery,
};
};
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
import gql from 'graphql-tag';

import { ObjectMetadataItem } from '@/object-metadata/types/ObjectMetadataItem';
import { mapObjectMetadataToGraphQLQuery } from '@/object-metadata/utils/mapObjectMetadataToGraphQLQuery';
import { RecordGqlOperationGqlRecordFields } from '@/object-record/graphql/types/RecordGqlOperationGqlRecordFields';
import { getSearchRecordsQueryResponseField } from '@/object-record/utils/getSearchRecordsQueryResponseField';
import { capitalize } from '~/utils/string/capitalize';

export type QueryCursorDirection = 'before' | 'after';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

style: QueryCursorDirection is defined but not used in this file. Consider removing if unnecessary.


export const generateSearchRecordsQuery = ({
objectMetadataItem,
objectMetadataItems,
recordGqlFields,
computeReferences,
}: {
objectMetadataItem: ObjectMetadataItem;
objectMetadataItems: ObjectMetadataItem[]; // TODO - what is this used for?
recordGqlFields?: RecordGqlOperationGqlRecordFields;
computeReferences?: boolean;
}) => gql`
query Search${capitalize(objectMetadataItem.namePlural)}($search: String, $limit: Int) {
${getSearchRecordsQueryResponseField(objectMetadataItem.namePlural)}(searchInput: $search, limit: $limit){
edges {
node ${mapObjectMetadataToGraphQLQuery({
objectMetadataItems,
objectMetadataItem,
recordGqlFields,
computeReferences,
})}
}
pageInfo {
hasNextPage
hasPreviousPage
startCursor
endCursor
}
}
}
`;
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
import { capitalize } from '~/utils/string/capitalize';

export const getSearchRecordsQueryResponseField = (objectNamePlural: string) =>
`search${capitalize(objectNamePlural)}`;
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
export type Opportunity = {
__typename: 'Opportunity';
id: string;
createdAt: string;
updatedAt?: string;
deletedAt?: string | null;
name: string | null;
};
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,5 @@ import { FieldMetadataType } from '~/generated-metadata/graphql';

export type SettingsSupportedFieldType = Exclude<
FieldMetadataType,
FieldMetadataType.Position
FieldMetadataType.Position | FieldMetadataType.TsVector
>;
Loading
Loading