Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add frontend and backend part for job to materialize a volume annotation #6086

Merged
merged 39 commits into from
May 9, 2022
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
435f2f1
add frontend and backend part for job to apply merger mode tracing
MichaelBuessemeyer Feb 28, 2022
ee1f591
renaming variable
MichaelBuessemeyer Feb 28, 2022
4f7dcb0
Merge branch 'master' of github.com:scalableminds/webknossos into add…
MichaelBuessemeyer Mar 21, 2022
db2998a
add annotation type as job parameter and make layer names more readable
MichaelBuessemeyer Mar 21, 2022
719a98a
fix layer selection for merger mode job modal
MichaelBuessemeyer Mar 22, 2022
c2190cd
add job output dataset name to start job modals
MichaelBuessemeyer Mar 31, 2022
a9fb85e
add output dataset name to nuclei & neuron & merger mode job params
MichaelBuessemeyer Mar 31, 2022
7f045f5
Merge branch 'master' of github.com:scalableminds/webknossos into add…
MichaelBuessemeyer Mar 31, 2022
d0be7b9
refactor starting modal for flood fills to use the same root componen…
MichaelBuessemeyer Mar 31, 2022
60a5acf
Merge branch 'master' into add-apply-merger-mode-job
MichaelBuessemeyer Apr 4, 2022
c252d40
Apply suggestions from code review
MichaelBuessemeyer Apr 5, 2022
7512f3a
Merge branch 'master' of github.com:scalableminds/webknossos into add…
MichaelBuessemeyer Apr 5, 2022
0aa7e31
apply pr feedback
MichaelBuessemeyer Apr 5, 2022
9d16876
Merge branch 'add-apply-merger-mode-job' of github.com:scalableminds/…
MichaelBuessemeyer Apr 5, 2022
7688824
disable jobs
MichaelBuessemeyer Apr 5, 2022
28a66f9
update docs
MichaelBuessemeyer Apr 25, 2022
b81be12
apply pr feedback
MichaelBuessemeyer Apr 25, 2022
77d6e32
Merge branch 'master' of github.com:scalableminds/webknossos into add…
MichaelBuessemeyer Apr 25, 2022
cd5d318
add changelog entry
MichaelBuessemeyer Apr 25, 2022
485c2d3
Update docs/volume_annotation.md
MichaelBuessemeyer Apr 27, 2022
7601d6f
only show merger mode job button whe jobs are enabled
MichaelBuessemeyer Apr 27, 2022
ef24b0a
Merge branch 'master' into add-apply-merger-mode-job
MichaelBuessemeyer Apr 27, 2022
5d3a7e9
add output segmentation layer name as user input field to apply merge…
MichaelBuessemeyer Apr 28, 2022
04e4f96
add segmentation output layer name to apply merger mode job api call
MichaelBuessemeyer May 2, 2022
2d740f7
Merge branch 'master' of github.com:scalableminds/webknossos into add…
MichaelBuessemeyer May 2, 2022
740efd6
make merger job available for volume annotation without skeletons
MichaelBuessemeyer May 4, 2022
4ec2dc9
Merge branch 'master' of github.com:scalableminds/webknossos into add…
MichaelBuessemeyer May 4, 2022
a810b43
fix returnLink of materialize job
MichaelBuessemeyer May 5, 2022
0bbbe0d
make materialize volume anotation job better compatible with volume …
MichaelBuessemeyer May 5, 2022
310375f
update doc images and add merge with fallback layer section to docs
MichaelBuessemeyer May 5, 2022
9f6a3e7
Apply suggestions from code review
MichaelBuessemeyer May 5, 2022
c02083c
apply review feedback
MichaelBuessemeyer May 5, 2022
edff465
Merge branch 'master' into add-apply-merger-mode-job
fm3 May 9, 2022
b43250b
Update docs/volume_annotation.md
MichaelBuessemeyer May 9, 2022
4d9ca99
Merge branch 'master' into add-apply-merger-mode-job
MichaelBuessemeyer May 9, 2022
4af34cf
apply pr feedback
MichaelBuessemeyer May 9, 2022
dad50a8
Merge branch 'master' of github.com:scalableminds/webknossos into add…
MichaelBuessemeyer May 9, 2022
b581c4b
Merge branch 'add-apply-merger-mode-job' of github.com:scalableminds/…
MichaelBuessemeyer May 9, 2022
89bb93f
rephrase commennt to make it clearer
MichaelBuessemeyer May 9, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions app/controllers/JobsController.scala
Original file line number Diff line number Diff line change
Expand Up @@ -258,4 +258,34 @@ class JobsController @Inject()(jobDAO: JobDAO,
}
}

def runApplyMergerModeJob(organizationName: String,
dataSetName: String,
layerName: String,
annotationId: String): Action[AnyContent] =
sil.SecuredAction.async { implicit request =>
log(Some(slackNotificationService.noticeFailedJobRequest)) {
for {
organization <- organizationDAO.findOneByName(organizationName) ?~> Messages("organization.notFound",
organizationName)
_ <- bool2Fox(request.identity._organization == organization._id) ?~> "job.applyMergerMode.notAllowed.organization" ~> FORBIDDEN
dataSet <- dataSetDAO.findOneByNameAndOrganization(dataSetName, organization._id) ?~> Messages(
"dataSet.notFound",
dataSetName) ~> NOT_FOUND
userAuthToken <- wkSilhouetteEnvironment.combinedAuthenticatorService.findOrCreateToken(
request.identity.loginInfo)
command = "apply_merger_mode"
commandArgs = Json.obj(
"organization_name" -> organizationName,
"dataset_name" -> dataSetName,
"layer_name" -> layerName,
"webknossos_token" -> RpcTokenHolder.webKnossosToken,
"user_auth_token" -> userAuthToken.id,
"annotation_id" -> annotationId,
)
job <- jobService.submitJob(command, commandArgs, request.identity, dataSet._dataStore) ?~> "job.couldNotRunApplyMergerMode"
js <- jobService.publicWrites(job)
} yield Ok(js)
}
}

}
2 changes: 1 addition & 1 deletion app/models/job/Job.scala
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ case class Job(
}
case "export_tiff" =>
Some(s"$dataStorePublicUrl/data/exports/${_id.id}/download")
case "infer_nuclei" | "infer_neurons" =>
case "infer_nuclei" | "infer_neurons" | "apply_merger_mode" =>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure, whether this is everything I need to do here for the new job. I did not test the jobs list view and therefore this is likely untested.

returnValue.map { resultDatasetName =>
s"/datasets/$organizationName/$resultDatasetName/view"
}
Expand Down
2 changes: 2 additions & 0 deletions conf/messages
Original file line number Diff line number Diff line change
Expand Up @@ -342,6 +342,7 @@ job.couldNotRunComputeMeshFile = Failed to start mesh file computation job.
job.couldNotRunNucleiInferral = Failed to start nuclei inferral job.
job.couldNotRunNeuronInferral = Failed to start neuron inferral job.
job.couldNotRunGlobalizeFloodfills = Failed to start job for globalizing floodfills.
job.couldNotRunApplyMergerMode = Failed to start job to apply merger mode tracing.
job.disabled = Long-running jobs are not enabled for this webKnossos instance.
jobs.worker.notFound = Could not find this worker in the database.
job.export.fileNotFound = Exported file not found. The link may be expired.
Expand All @@ -353,6 +354,7 @@ job.inferNuclei.notAllowed.organization = Currently nuclei inferral is only allo
job.inferNeurons.notAllowed.organization = Currently neuron inferral is only allowed for datasets of your own organization.
job.meshFile.notAllowed.organization = Calculating mesh files is only allowed for datasets of your own organization.
job.globalizeFloodfill.notAllowed.organization = Globalizing floodfills is only allowed for datasets of your own organization.
job.applyMergerMode.notAllowed.organization = Applying merger mode tracings is only allowed for datasets of your own organization.

agglomerateSkeleton.failed=Could not generate agglomerate skeleton.

Expand Down
1 change: 1 addition & 0 deletions conf/webknossos.latest.routes
Original file line number Diff line number Diff line change
Expand Up @@ -209,6 +209,7 @@ POST /jobs/run/exportTiff/:organizationName/:dataSetName c
POST /jobs/run/inferNuclei/:organizationName/:dataSetName controllers.JobsController.runInferNucleiJob(organizationName: String, dataSetName: String, layerName: Option[String])
POST /jobs/run/inferNeurons/:organizationName/:dataSetName controllers.JobsController.runInferNeuronsJob(organizationName: String, dataSetName: String, layerName: String, bbox: String)
POST /jobs/run/globalizeFloodfills/:organizationName/:dataSetName controllers.JobsController.runGlobalizeFloodfills(organizationName: String, dataSetName: String, newDataSetName: Option[String], layerName: Option[String], annotationId: Option[String], annotationType: Option[String])
POST /jobs/run/applyMergerMode/:organizationName/:dataSetName controllers.JobsController.runApplyMergerModeJob(organizationName: String, dataSetName: String, layerName: String, annotationId: String)
GET /jobs/:id controllers.JobsController.get(id: String)
PATCH /jobs/:id/cancel controllers.JobsController.cancel(id: String)
POST /jobs/:id/status controllers.WKRemoteWorkerController.updateJobStatus(key: String, id: String)
14 changes: 14 additions & 0 deletions frontend/javascripts/admin/admin_rest_api.js
Original file line number Diff line number Diff line change
Expand Up @@ -1084,6 +1084,20 @@ export function startGlobalizeFloodfillsJob(
);
}

export function startApplyMergerModeJob(
organizationName: string,
datasetName: string,
layerName: string,
annotationId: string,
): Promise<APIJob> {
return Request.receiveJSON(
`/api/jobs/run/applyMergerMode/${organizationName}/${datasetName}?layerName=${layerName}&annotationId=${annotationId}`,
{
method: "POST",
},
);
}

export function getDatasetDatasource(
dataset: APIMaybeUnimportedDataset,
): Promise<APIDataSourceWithMessages> {
Expand Down
16 changes: 15 additions & 1 deletion frontend/javascripts/oxalis/view/action-bar/toolbar_view.js
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
// @flow
import { Radio, Tooltip, Badge, Space, Popover } from "antd";
import { useSelector, useDispatch } from "react-redux";
import React, { useEffect } from "react";
import React, { useEffect, useState } from "react";

import { LogSliderSetting } from "oxalis/view/components/setting_input_views";
import { addUserBoundingBoxAction } from "oxalis/model/actions/annotation_actions";
Expand Down Expand Up @@ -29,6 +29,7 @@ import { updateUserSettingAction } from "oxalis/model/actions/settings_actions";
import { usePrevious, useKeyPress } from "libs/react_hooks";
import { userSettings } from "types/schemas/user_settings.schema";
import ButtonComponent from "oxalis/view/components/button_component";
import { ApplyMergerModeModal } from "oxalis/view/right-border-tabs/starting_job_modals";
import Constants, {
ToolsWithOverwriteCapabilities,
type AnnotationTool,
Expand Down Expand Up @@ -187,6 +188,7 @@ function AdditionalSkeletonModesButtons() {
const isMergerModeEnabled = useSelector(
state => state.temporaryConfiguration.isMergerModeEnabled,
);
const [showApplyMergerModeModal, setShowApplyMergerModeModal] = useState<boolean>(false);
const isNewNodeNewTreeModeOn = useSelector(state => state.userConfiguration.newNodeNewTree);
const toggleNewNodeNewTreeMode = () =>
dispatch(updateUserSettingAction("newNodeNewTree", !isNewNodeNewTreeModeOn));
Expand Down Expand Up @@ -222,6 +224,18 @@ function AdditionalSkeletonModesButtons() {
/>
</ButtonComponent>
</Tooltip>
{isMergerModeEnabled && (
<Tooltip title="Apply this merger mode tracing.">
<ButtonComponent
style={narrowButtonStyle}
onClick={() => setShowApplyMergerModeModal(true)}
faIcon="fa fa-cogs"
/>
</Tooltip>
)}
{showApplyMergerModeModal && (
<ApplyMergerModeModal handleClose={() => setShowApplyMergerModeModal(false)} />
)}
MichaelBuessemeyer marked this conversation as resolved.
Show resolved Hide resolved
</React.Fragment>
);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,16 @@
import React, { useEffect, useState, type Node } from "react";
import { type APIDataset, type APIJob } from "types/api_flow_types";
import { Modal, Select, Button } from "antd";
import { startNucleiInferralJob, startNeuronInferralJob } from "admin/admin_rest_api";
import {
startNucleiInferralJob,
startNeuronInferralJob,
startApplyMergerModeJob,
} from "admin/admin_rest_api";
import { useSelector } from "react-redux";
import { getColorLayers } from "oxalis/model/accessors/dataset_accessor";
import { getColorLayers, getSegmentationLayers } from "oxalis/model/accessors/dataset_accessor";
import { getUserBoundingBoxesFromState } from "oxalis/model/accessors/tracing_accessor";
import Toast from "libs/toast";
import { type OxalisState, type UserBoundingBox } from "oxalis/store";
import { type UserBoundingBox } from "oxalis/store";
import { Unicode, type Vector3 } from "oxalis/constants";
import { capitalizeWords, computeArrayFromBoundingBox, rgbToHex } from "libs/utils";

Expand All @@ -28,23 +32,26 @@ type StartingJobModalProps = {
jobName: string,
description: Node,
isBoundingBoxConfigurable?: boolean,
chooseSegmentationLayer?: boolean,
};

function StartingJobModal(props: StartingJobModalProps) {
const isBoundingBoxConfigurable = props.isBoundingBoxConfigurable || false;
const chooseSegmentationLayer = props.chooseSegmentationLayer || false;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main changes here and below are that I enabled StartingJobModal to not only select color layers but also segmentation layers when chooseSegmentationLayer=true is passed to the component. The changes are mainly removing "color" from that variable names as the layers the component now works with might also be of kind segmentation.

const { dataset, handleClose, jobName, description, jobApiCall } = props;
const userBoundingBoxes = useSelector((state: OxalisState) =>
getUserBoundingBoxesFromState(state),
);
const [selectedColorLayerName, setSelectedColorLayerName] = useState<?string>(null);
const userBoundingBoxes = useSelector(state => getUserBoundingBoxesFromState(state));
const [selectedLayerName, setSelectedLayerName] = useState<?string>(null);
const [selectedBoundingBox, setSelectedBoundingBox] = useState<?UserBoundingBox>(null);
const colorLayerNames = getColorLayers(dataset).map(layer => layer.name);
const layerNames = (chooseSegmentationLayer
? getSegmentationLayers(dataset)
: getColorLayers(dataset)
).map(layer => layer.name);
useEffect(() => {
if (colorLayerNames.length === 1) {
setSelectedColorLayerName(colorLayerNames[0]);
if (layerNames.length === 1) {
setSelectedLayerName(layerNames[0]);
}
});
if (colorLayerNames.length < 1) {
if (layerNames.length < 1) {
return null;
}
const onChangeBoundingBox = (selectedBBoxId: number) => {
Expand All @@ -54,15 +61,15 @@ function StartingJobModal(props: StartingJobModalProps) {
}
};
const startJob = async () => {
if (selectedColorLayerName == null) {
if (selectedLayerName == null) {
return;
}
try {
let apiJob;
if (isBoundingBoxConfigurable) {
apiJob = await jobApiCall(selectedColorLayerName, selectedBoundingBox);
apiJob = await jobApiCall(selectedLayerName, selectedBoundingBox);
} else {
apiJob = await jobApiCall(selectedColorLayerName);
apiJob = await jobApiCall(selectedLayerName);
}
if (!apiJob) {
return;
Expand All @@ -87,24 +94,24 @@ function StartingJobModal(props: StartingJobModalProps) {
};

const ColorLayerSelection = (): Node =>
colorLayerNames.length > 1 ? (
layerNames.length > 1 ? (
<React.Fragment>
<p>Please select the layer that should be used for the inferral.</p>
<p>Please select the layer that should be used for this job.</p>
<div style={{ textAlign: "center" }}>
<Select
showSearch
style={{ width: 300 }}
placeholder="Select a color layer"
optionFilterProp="children"
value={selectedColorLayerName}
onChange={setSelectedColorLayerName}
value={selectedLayerName}
onChange={setSelectedLayerName}
filterOption={(input, option) =>
option.children.toLowerCase().indexOf(input.toLowerCase()) >= 0
}
>
{colorLayerNames.map(colorLayerName => (
<Select.Option key={colorLayerName} value={colorLayerName}>
{colorLayerName}
{layerNames.map(layerName => (
<Select.Option key={layerName} value={layerName}>
{layerName}
</Select.Option>
))}
</Select>
Expand Down Expand Up @@ -166,7 +173,7 @@ function StartingJobModal(props: StartingJobModalProps) {
) : null;

const hasUnselectedOptions =
selectedColorLayerName == null || (isBoundingBoxConfigurable && selectedBoundingBox == null);
selectedLayerName == null || (isBoundingBoxConfigurable && selectedBoundingBox == null);

return (
<Modal
Expand Down Expand Up @@ -198,7 +205,7 @@ function StartingJobModal(props: StartingJobModalProps) {
}

export function NucleiInferralModal({ handleClose }: Props) {
const dataset = useSelector((state: OxalisState) => state.dataset);
const dataset = useSelector(state => state.dataset);
philippotto marked this conversation as resolved.
Show resolved Hide resolved
return (
<StartingJobModal
dataset={dataset}
Expand Down Expand Up @@ -229,7 +236,7 @@ export function NucleiInferralModal({ handleClose }: Props) {
}

export function NeuronInferralModal({ handleClose }: Props) {
const dataset = useSelector((state: OxalisState) => state.dataset);
const dataset = useSelector(state => state.dataset);
return (
<StartingJobModal
dataset={dataset}
Expand Down Expand Up @@ -268,3 +275,34 @@ export function NeuronInferralModal({ handleClose }: Props) {
/>
);
}

export function ApplyMergerModeModal({ handleClose }: Props) {
const dataset = useSelector(state => state.dataset);
const annotationId = useSelector(store => store.tracing.annotationId);
return (
<StartingJobModal
dataset={dataset}
handleClose={handleClose}
jobName="apply merger mode"
chooseSegmentationLayer
jobApiCall={async colorLayerName =>
startApplyMergerModeJob(
dataset.owningOrganization,
dataset.name,
colorLayerName,
annotationId,
)
}
description={
<>
<p>
Start a job that take the current state of this merger mode tracing and apply it to the
segmentation layer. This will create a new dataset which contains the merged
segmentation layer. If this dataset has more than one segmentation layer, please select
the segmentation layer to the merging should be applied to.
</p>
</>
}
/>
);
}