New motion correction methods, CIAtah QoL updates, etc.

Main features and changes:

• Increment to CIAtah version 4.11.0.
• Addition of additional spinal cord motion correction functions and demos. These allow for correction of large motion (LD-MCM), non-rigid motion (displacement fields), and cross-session registration (CS-MCM). See below for details of function or https://bahanonu.github.io/ciatah/pipeline_detailed_spinal.
• Improved DeepLabCut importer function to also allow output of tensors with all tracking data.
• QoL improvements for a number of CIAtah functions and class methods.
• Updated README to include additional information on spinal cord imaging, Fiji + N5 viewing of files, and other changes.

New functions

• ciapkg.demo.cross_session_motion_correction - Script to allow multi-step cross-session registration directly on movies. Includes automated registration as well as the ability to perform manual correction to help with large or unusual shifts using a GUI.
• ciapkg.demo.displacementFields - Displacement field motion correction example across several images. This is to help validate that motion correction will work on user systems.
• ciapkg.demo.ldmcm -
• ciapkg.motion_correction.ldmcm - LD-MCM motion correction using deep learning and control point registration. This demo script includes the necessary steps that need to be run after conducting DeepLabCut-based tracking of features in the imaging movie.
• ciapkg.video.scaleMatrix8bit - Scales an input matrix to 8-bit (256) range. Can also save video out to an AVI file.
• ciapkg.view.displacementFieldCompare -Displays different frames along with the displacement fields.

Updated @ciatah

• ciatah.computeManualSortSignals - Update display of information.
• ciatah.computeMatchObjBtwnTrials - Added the option to only keep cells within a specific region of the FOV, useful for when want to register cells in a single region and don't want other cells affecting alignment. Add support for normcorre and imregdemons for registration.
• ciatah.getRegistrationSettings - Add support for gradient-based filtering for motion correction input. Add displacement field-based motion correction. (Updated 2024.02.29 [18:17:06])
• ciatah.modelBatchCopyFiles - Easier user switching between copying or moving files.
• ciatah.modelExtractSignalsFromMovie - Allow input of images from one cell extraction method to CELLMax and EXTRACT, e.g. run PCA-ICA quickly then use as input. Update ROI method of calling other algorithm images.
• ciatah.modelGetSignalsImage - Added support for registering images (e.g. after cross-day registration) using multiple methods instead of just turboreg.
• ciatah.modelPreprocessMovieFunction - Updated NormCorre parameter window to estimate window size from the first frame. Make correlation calculation parallel. Allow detrend movie to use movie subsets to reduce memory overhead. Reduce double calling of mean() for calculation and plotting when conducting detrending. Further eliminate nanmean/nanmin/nanmax usage, use dims instead of (:) [waste memory], and general refactoring. Fix thisMovieMinMask(row,:) = logical(max(isnan(squeeze(thisMovie(row,:,:))),[],2,'omitnan')); as was 3 instead of row. Add support for input of prior motion correction coordinates to be user facing. Empty imagesc() call leading to face appearing. For the curious, see https://blogs.mathworks.com/steve/2006/10/17/the-story-behind-the-matlab-default-image/.
• ciatah.modelVarsFromFiles - CELLMax by default uses the estimated dF/F.
• ciatah.modelVerifyDataIntegrity - Updated movieInformation to have additional movie information.
• ciatah.runPipeline - Use guiFoldersSelected to remember prior user selected folders on next call of CIAtah GUI.
• ciatah.viewMatchObjBtwnSessions - Additional options for choosing which cells to display and additional cellmaps saved out as TIFF files.
• ciatah.viewMovie - Fix readHDF5Subset function unable to be found in certain cases.
• ciatah.viewObjmaps - Update display of cell overlap.
• `` -

Updated +ciapkg

• ciapkg.behavior.importDeepLabCutData - Added conversion to tensor and cell to the import function from other functions.
• ciapkg.classification.matchObjBtwnTrials - Made ability to switch between different motion correction and image alignment methods user visible.
• ciapkg.download.downloadGithubRepositories - Automatically convert string inputs to character arrays.
• ciapkg.download.example_downloadTestData - Added additional link.
• ciapkg.hdf5.appendDataToHdf5.m - Flag to silence command line output.
• ciapkg.hdf5.downsampleHdf5Movie - Integrated TIFF support from other functions. options.newFilenameTwo is sufficient to initiate secondary movie downsampling. Option to convert data type during downsampling.
• ciapkg.image.cropMatrix - Update to callback. Added support for automatic detection of crop borders from main pre-processing functions. Remove workspace unpacking of options.
• ciapkg.image.thresholdImages - fastThresholding respects normalization=0, so threshold functions as absolute value. Also make sure remove unconnected works with fastThresholding at all times.
• ciapkg.io.getMovieFileType - Added mp4 support to output.
• ciapkg.io.loadDependencies - Update Bio-Formats to 7.3.0. Upgraded ImageJ download site.
• ciapkg.motion_correction.computeManualMotionCorrection - Better support for adding custom fill values for both translation and rotation.
• ciapkg.motion_correction.getNoRMCorreParams - Change default parameters and run final output through NoRMCorre parameter function to reduce chance of errors.
• ciapkg.motion_correction.turboregMovie - Changes to GUI display. Correct user options input to normcorre.apply_shifts. Add displacement field-based motion correction. Add back in ability to remove NaNs from each frame before calculating shifts or displacements. Add support for gradient-based filtering for motion correction input. Support loading of prior coordinates from MAT-file in standard of modelPreprocessMovieFunction. Added additional options for displacement field-based motion correction. Update to row assignment for removeInputMovieEdges.
• ciapkg.movie_processing.detrendMovie - Fix detrendDegree passing.
• ciapkg.movie_processing.dfofMovie - Updated to allow users to adjust movies with negative values.
• ciapkg.movie_processing.normalizeMovie - matlabdisk no longer converts to cell array, bad memory usage. Improve other memory usage in function. Change how NaN check before filtering is run. Remove on a per-frame basis so that the NaNs can be added back in at the correct pixel locations. Further eliminate nanmean/nanmin/nanmax usage, use dims instead of (:) [waste memory], and general refactoring. Added movie subset support to reduce memory usage with parfor. Fix detrendDegree passing. Added subfxnFft movie subset support to reduce memory usage with parfor.
• ciapkg.overloaded.suptitle - Added font name option.
• ciapkg.settings.cnmfeSettings - Updated merge settings to reduce duplicates in output.
• ciapkg.view.playMovie - Users can now input a cell array of movies to have the extra movie play instead of the 'extraMovie' option. Added support for additional NWB series.

Updated docs

• help_spatial_filtering
• pipeline_detailed_signal_extraction_validation
• pipeline_detailed_signal_sorting_manual

Main features and changes:

• Increment to CIAtah version 4.6.1.

New functions

• ciapkg.behavior.importDeepLabCutData (see prior pull request).
• ciapkg.tracking.annotateOpenFieldParameters - Requests information on arena size and center area for open field analysis from circular or square arenas.

Updated @ciatah

• ciatah - Add guiFoldersSelected, prior user selected folders kept on next call.
• ciatahMainGui - List of files in selected folder now contain file dates. Folder are now sorted to appear first in the list. Get around error with incorrect filter for input images applied. Allow user to reset folders along with prior select folders stay selected on next reload of object.

Updated +ciapkg

• 'ciapkg.io.loadMovieList' - Added mp4 support. Update AVI support to do additional cdata checks. To reduce automatic conversion to double when reading HDF5 with largeMovieLoad=1 and a specific set of frames, force outputMovie default value to be single. This is made consistent across all image types, checks if float or not then re-initializes default outputMovie.

• ciapkg.io.readFrame - Update AVI support to do additional cdata checks. Add support for reading multiple files if a character cell array of char vectors is given. Add bioformats passing arguments for loadMovieList as options.

• ciapkg.io.loadDependencies - Ensure display location of SlideBook6Reader.jar saving. Add success output.

• ciapkg.io.saveMatrixToFile - Comment updates.

• ciapkg.view.playMovie - Add support for mp4 files. Add support for overlaying DeepLabCut markers on the movie (including with confidence measure). Users can change size of overlay markers. Change default colormap to gray. Added downsample (both type) support for extra movie, mirrors what happens in the 1st movie. Also improved downsampling when just subsampling by removing unnecessary calls to imresize. Improved tracking point handling to ensure in the correct axes and can turn off prior frame tracking. Better handling of quiver plots with tracking. Convert all nanmin/nanmax to 'omitnan' and (:) to [1 2 3 4] to ensure compatible with 3- and 4-d tensors. Additional Bio-Formats support.

• ciapkg.view.changeFont - Update to handle legend, since use TextColor property instead of Color for text. Users can also put in a blank fontSize to avoid changing that property, can skip UI that would normally appear.

• ciapkg.view.setFigureDefaults - Disable Matlab UI font size change for code and text, that is left to the user.

• `ciapkg.beha...

Code and GUI improvements, NoRMCorre and additional file formats support, etc.

Main features and changes:

• Increment to CIAtah version 4.5.7.
• NoRMCorre is now supported within main CIAtah GUI and as a switch within turboregMovie (to keep support for surrounding motion correction processing code). This gives users another option for motion correction with non-rigid registration is needed.
  • Note this runs a custom forked version of NoRMCorre in which I moved all code within a normcorre package and re-wrote to make compatible to improve namespace and integration: https://github.com/bahanonu/NoRMCorre.
• Improved support for Bio-Formats files (better file handling, ability to grab data via internal series name, etc.). Additional integration into main CIAtah GUI in next release.
• The main module for pre-processing (modelPreprocessMovieFunction) has several new options, speed/code improvements, NoRMCorre integration, etc. Further support for read-from-disk pre-processing in next release.
• Added support for SlideBook (.sld) files.
• Support for MATLAB's new Name=value syntax for passing Name-Value arguments to functions.
• General switch in many functions to fast thresholding.
• Switching away from using nanmean to mean('omitnan'), to be completed in next release.
• Improved loading of CIAtah directory (with ciapkg.loadBatchFxns).

New functions

• ciapkg.bf.getSeriesNoFromName - Returns series number for Bio-Formats series with name given by user.
• ciapkg.bf.dispFileSeries - Returns series number for Bio-Formats series with name given by user.
• writeMatfileToHDF5 - Writes a MAT-file dataset to HDF5 by reading the matrix in parts to avoid large overhead.
• mergeStructs - Copies fields in fromStruct into toStruct, if there is an overlap in field names, fromStruct overwrites toStruct unless specified otherwise.
• writeDataToMatfile - Converts 3D movie matrix stored in a file (HDF5, TIF, AVI, NWB) into a MAT-file for use with larger-than-memory processing.
• getNoRMCorreParams - Automatically returns default NoRMCorre parameters or ask user to input parameters.

Updated @ciatah

• ciatahMainGui - Improve speed of movie display by better passing of movie information to ciapkg.io.readFrame. If processed and raw movie are the same length, sliders move in sync. By default use 'painters' renderer as that can produce smoother rendering than opengl. Added file sizes to the folder files preview. Reset movie dimension on folder select change so movie previews run without outputting initial error (movies would still preview, users just might be confused by the warning). Feature to allow preview of HDF5 datasets. Expanding to Bio-Formats and other types of data with internal structures for data. Added ability to open folder in OS GUI, e.g. Windows Explorer. If manual input for filtering folders set back to no filter to avoid GUI loops. Make sure preview data and open folder buttons are associated with correct previewDataHandle and openFoldersHandle handles.
• getRegistrationSettings - Add additional black background support. Added largeMovieLoad setting.
• modelAddNewFolders - Add additional check for when user adds blank folder paths (NOT empty folders) and remove them. Empty folder check update.
• modelExtractSignalsFromMovie - CELLMax support for loading prior settings. Update to EXTRACT support: added additional options, do not automatically eliminate summary section, and more. Switch order of mergeStructs and supplying cell radius to EXTRACT, else empty vector can be passed depending on user input.
• modelGetSignalsImages - Ensure when loading cell extraction CIAtah-style that only MAT files are loaded.
• modelModifyRegionAnalysis - Misc code standard updates.
• modelPreprocessMovieFunction - Change all nested and local functions to start with "nestedfxn_" or "localfxn_" prefixes for clarity. Added option to fix "broken" frames, e.g. frames in which the camera gives out garbled or improperly formatted data. Fix with mean of the movie to reduce effect on downstream analysis. Disable local function "getMovieFileType" in favor or CIAtah package function. Added option to place prefix on all output files. Detrending now does everything within nested function instead of calling normalizeMovie() to reduce memory overhead. Added support for MAT-file based processing of movies. Add additional black background support. Added largeMovieLoad setting support, improve memory allocation for select movies. Make sure readFrame is compatible with all HDF5 dataset names. NormCorre integrated into the pipeline for end users.
• modelSaveImgToFile - Update PaperPosition to auto to avoid error "Positioning Figure for ResizeFcn. Set PaperPositionMode to 'auto' (match figure screen size) to avoid resizing and this warning." Further change to PaperPositionMode to avoid errors.
• modelVarsFromFiles - Enforce that NWB loading only looks for .nwb files and ensure that manual or automatic classifications are loaded when loading NWB files. Ensure when loading cell extraction CIAtah-style that only MAT files are loaded.
• viewMovieRegistrationTest - Updated so more settings are pulled from CIAtah object settings.
• viewSubjectMovieFrames - Update code standards.

Updated +ciapkg

• signalSorter - Misc updates.
• downloadCnmfGithubRepositories - Update URLs to include a backup URL for cvx since would often timeout.
• appendDataToHdf5 - comments updated.
• downsampleHdf5Movie - Added Bio-Formats support. Switch chunking to automatic by default, generally improved reading from data later than small chunks. Improved Name-Value comments.
• readHDF5Subset - minor updates.
• computeImageFeatures - Make default thresholding fast thresholding. Improved annotation of code and added options.fastThresholding. Also by default plots are not made. Refactored extra features code and finding centroid to avoid unnecessary function calls, speeding up code.
• filterImages - Make default thresholding fast thresholding.
• findCentroid - Make default thresholding fast thresholding. Update to ensure threshold based on max value if user inputs non-normalized (e.g. distribution forced to zero to one) thresholded images.
• groupImagesByColor - Added fast thresholding option and other threshold image options.
• thresholdImages - manageParallelWorkers now passed options.waitbarOn value to reduce command line clutter if user request in thresholdImages.
• getFileInfo - Add trial # to output.
• getFileList - Improved support multiple directory input. Added folder filter.
• getMovieFileType - Added MAT-file support. Add SlideBook Bio-Formats support.
• getOptions - In 2021a MATLAB introduced Name=value syntax for passing name-value arguments. These are passed within varargin as a string array instead of a char array as was the case with comma-separated syntax. getOptions now checks for isstring in addition to ischar to add support for this syntax to CIAtah functions. See https://www.mathworks.com/help/matlab/release-notes.html?rntext=&startrelease=R2021a&endrelease=R2021a&groupby=release&sortby=descending&searchHighlight=#mw_77c0932f-4a31-44e6-b550-db5a736c2de3.
• load_tif_movie - Speed improvements related to imfinfo calls. Added tiffread support and ScanImageTiffReader support (not user accessible currently).
• loadDependencies - Added PatchWarp (https://github.com/ryhattori/PatchWarp). Code clean up. Updated NoRMCorre to use https://github.com/bahanonu/NoRMCorre as that is a package version, cleaner namespace. Added CIAtah utilities repository (https://github.com/bahanonu/ciatah_utils) to separate outside code from main repository. Add SlideBook .jar reader to Bio-Formats download. Ensure NoRMCorre is in default options. Ensure bfmatlab_readers directory exists, else websave errors occur.
• loadMovieList - Speed improvements related to imfinfo calls. Add option to load movie into a MAT-file. Improve Bio-Formats support, including using bfGetPlane to partially load data along with adding ndpi support. Add SlideBook Bio-Formats support. Remove local function getMovieFileType to force use of CIAtah getMovieFileType function. Updated getIndex call to avoid issues with certain Bio-Formats. Ensure getting correct Bio-Formats frames and loadMovieList output. Hide TIF warnings.
• loadSignalExtraction - Added option to load manual sorting as well and output as standard vector.
• manageParallelWorkers - Added displayInfo option.
• readFrame - Ensure subfxn_loadFrame nested function outputs frame data (even if file does not contain the requested frame, e.g. empty matrix).
• saveMatrixToFile - Use Fast_Tiff_Write to write out TIF files instead of saveastiff by default. Give option with options.tifWriter.
• computeManualMotionCorrection - Check if inputs are sparse, convert to single. By default use 'painters' renderer as that can produce smoother rendering than opengl. Update to make sure titles update correctly for test images. Add renderer option to allow users to choose painters or opengl.
• turboregMovie - Refactor code to remove need to transform movie into cell array, performance and memory improvements. Added NoRMCorre support, function will eventually be merged with "registerMovie" or renamed to indicate support for multiple algorithms. Reason to integrate NoRMCorre into this function is takes advantage of existing pre-processing pipeline and integration with other algorithms (e.g. cross-session). Check if inputs are sparse, convert to single. Use custom bahanonu NoRMCorre that is within a package and update to use reference picture instead of template. Updated to which('normcorre.normcorre') from which('normcorre') since normcorre now inside a package. Additional NoRMCorre support and getNoRMCorreParams checking before running NoRMCorre.
• downsampleMovie - Update information for users/comments.
• normalizeMovie - Switch away from using nanmean to mean('omitnan'). Detrend flight code ref...

Improved initialization, movie processing selection, GUI speed-ups, and misc. improvements

Main features and changes:

• Increment to CIAtah version to 4.2.0.
• GUI improvements to ciapkg.motion_correction.computeManualMotionCorrection that significantly speed up GUI responsiveness, e.g. when doing hundreds of sessions.
• computeMatchObjBtwnTrials now supports manual alignment of sessions before automated alignment and matching of cells across sessions.
• Replace calls to suptitle with ciapkg.overloaded.suptitle due to function being dropped from newer Matlab releases.
• Several fixes to improve usability and initializing CIAtah.
• Addition of detrending function to API, alias for existing functionality.
• Misc fixes for edge cases and other improvements.

New functions

• ciapkg.api.getMovieFileType - API access for getMovieFileType.
• ciapkg.api.getMovieInfo - API access for getMovieInfo.
• ciapkg.api.readFrame - API access for readFrame.
• ciapkg.movie_processing.detrendMovie - Detrend a movie to account for photobleaching, etc. Alias for call to normalizeMovie.

Updated @ciatah

• ciatah - Misc code fixes to conform to better Matlab language standards.
• ciatahMainGui - Additional check only for movie previews of supported movie files. "Folder files" list selection additional regexp support. Changed so that 'Start selected method' button selects the correct folders like occurs when pressing enter. The folder list will no longer reset when selecting certain GUI elements that should not impact folder list.
• computeMatchObjBtwnTrials - Added manual alignment option before automated for cases in which there are large shifts or rotations along with changes in bulk cells identified that might be hard for automated to align.
• modelDownsampleRawMovies - Change TIF finding to split apart the information file before using for regular expression. Additional updates to handle CIAtah v4.0 API switch.
• modelModifyRegionAnalysis - Fix modelVarsFromFilesCheck and modelVarsFromFiles recursion edge case.
• modelPreprocessMovieFunction - Updated suptitle to ciapkg.overloaded.suptitle. Fix "No movies" dialog box displaying when movies successfully run. Add modelPreprocessMovieFunction settings saved in CIAtah class to saved output file for later retrieval. For selecting turboreg crop coordinates, switched to ciapkg.io.readFrame when single frame is requested as this is faster and avoids long read times of all frame information as occurs in certain types of TIFF files or hard drives. Misc code fixes to conform to better Matlab language standards.
• modelVerifyDataIntegrity - Updated suptitle to ciapkg.overloaded.suptitle.
• runPipeline - Eliminate the close all figure when initializing the main GUI. Unexpected for some users and undesirable in certain cases.
• viewMovie - Updated suptitle to ciapkg.overloaded.suptitle.
• getRegistrationSettings - Handle "User selects specific value" going into infinite loop. Allow cancel option to prevent loop.
• initializeObj - Wrap calls to diverging_map to avoid potential ciatah initialization errors on macOS.
• modelPreprocessMovieFunction - Add verification that turboreg MEX function is in the path.
• viewMovieRegistrationTest - Additional updates to handle CIAtah v4.0 API switch.

Updated +ciapkg

• signalSorter - Updated suptitle to ciapkg.overloaded.suptitle. Also limit number of GUI re-runs to prevent infinite looping. Fixed issue with gca.XRuler.Axle.LineStyle being an empty object when axes created, leading to failure to assign and error loops. Make all openFigure to ciapkg.api.openFigure.
  downloadGithubRepositories - Update _external_programs to call ciapkg.getDirExternalPrograms() to standardize call across all functions. If multiple repos requested but one already downloaded, previously exited before downloading the other repos.
• readHDF5Subset - Fix display of information on error.
• createPeakTriggeredImages - Ensure only unique signal peaks used.
• fftImage - Misc code fixes to conform to better Matlab language standards.
• getMovieFileType - Update all strcmp to endsWith to ensure finding file extension as there are cases where software will export metadata into files with naming schema like NAME.tif.xml for NAME.tif and this can cause issues when using strcmp without endsWith-like checks. Update docs. Added varargin support.
• getMovieInfo - Directly call loadMovieList and bypass ciapkg.api.
• loadMovieList - AVI now read(...,'native') is faster.
• jsonRead - Fix edge case where the end of the imported string is a space, causing jsondecode to fail.
• loadMovieList - AVI now read(...,'native') is faster.
• manageMiji - Added forceLoad option.
• manageParallelWorkers - Added nCoresFree option for users to set the number of cores to remain free. Added ability to input just the number of workers to open as 1st single input argument that aliases for the "setNumCores" Name-Value input, still support other input arguments as well.
• modelAddOutsideDependencies - No longer use Fiji-based Miji.
• readFrame - AVI now read(...,'native') is faster.
• resetMiji - Update to use manageMiji to setup Miji, avoid certain issues. Also search for imagej instead of Fiji path.
• computeManualMotionCorrection - Change number of pixels to translate per click and UI improvements. Improved pre-allocation, minimize GUI slowdown for large inputs over time. Improve speed of GUI and responsiveness. Force altInputImages to initialize as input class type to avoid automatic conversion to double (save space/ram). Significant speed improvements by removing certain calls to figure, axis, imagesc, etc. that slow down UI over time when many sessions aligned.
• downsampleMovie - Update for Matlab standards.
• viewAddTextToMovie - Added varargin for future additions.
• changeFont - Updated so doesn't change Axes backgroundcolor when changing font color, only Axes text.
• displayMatchingObjs - Updated suptitle to ciapkg.overloaded.suptitle
• openFigure - Option to add a title to the figure, e.g. "Figure 1" becomes "Fig. 1 - Lorem ipsum". Also make user accessible change figure background color.
• playMovie - AVI now read(...,'native') is faster. Also add primaryTrackingPointNback option to allow a trailing number of points from prior frames to appear.
• exampleFxn - updated for better help.
• getMovieFileType - Added support for checking if a cell was accidentally input instead of a string path along with verifying that input was a string. Added support for oir and czi/lsm Olympus and Zeiss file formats that already was in loadMovieList.
• computeManualMotionCorrection - Improved pre-allocation, minimize GUI slowdown for large inputs over time. Improve speed of GUI and responsiveness.
• turboregMovie - Additional display of information. Added verification that turboreg MEX function is in the path.
• changeFont - Updated so can update font size, name, color more independent of one another.
• downloadCnmfGithubRepositories - Update display of information if CVX is not automatically found.

Updated +ciapkg.api

• convertInscopixIsxdToHdf5 - Update +image package to +inscopix.

CIAtah 4.0

Main features and changes

• All CIAtah functions have been moved to +ciapkg folder so they are part of the ciapkg package or sub-packages. The ciapkg folder has been removed.
  • This improves namespace handling (e.g. eliminates naming conflicts) to make it easier to integrate CIAtah with other packages or when using the GUI or API with user scripts.
• Major version incremented due to large change in organization of many CIAtah/ciapkg functions and thus the underlying API.
  • All relevant CIAtah functions can now be called via ciapkg.api.FUNCTION_NAME, which allows for a central location to call all functions as well as allowing importing of relevant CIAtah functions into a function if want to reduce visual clutter. Full function can be found in respective +ciapkg sub-packages.
• Updated to handle CIAtah v4.0 switch to all functions inside ciapkg package and ciatah class.
• Current support is MATLAB 2019b or later due to function of nested packages.
  • See https://www.mathworks.com/help/matlab/matlab_prog/upgrade-code-for-r2019b-changes-to-function-precedence-order.html#mw_8a31cdce-6dc1-4ae0-ae4a-ca0fbae7d1ff.
• Updates to certain functions for speed, UI, and other improvements.
• Move all NWB functions to ciapkg.nwb sub-package.
• Re-organized several File Exchange functions and removed unused ones.

Example of movie slider and cell-extraction in the main GUI.

Updated @ciatah functions

• ciatahMainGui - Added support for file list when selecting a specific folder. Font scaling support.
• modelPreprocessMovieFunction - Fix to handle folders with no files.
• modelExtractSignalsFromMovie - Added nwbpkg support. Updated EXTRACT support.

Updated +ciapkg functions

• Move Turboreg to _external_programs
• loadBatchFxns - Improved handling of external programs both in adding and removing from path. Additional support for removing specific packages that are not always needed.
• cropMatrix - Updated to allow specifying the size of the rectangle.
• getFileList - Added support to exclude adding the input directory to each file path.
• getMovieFileType - Added HDF5 capitalized file extension.
• getOptions - Ensure no warnings are shown. Added fix to handle users calling API version of getOptions with getOptions variable input arguments.
• loadDependencies - Added Turboreg (moved from within ciapkg) to make explicit that this is an external program. Update Schnitzer NWB package to new URL (https://github.com/schnitzer-lab/nwbpkg). Default to download GRAMmar.
• loadMovieList - Added HDF5 capitalized file extension. Ensure that loadMovieList has all output arguments set no matter return conditions.
• computeManualMotionCorrection - Added acceleration based on rapid user clicks or holding down direction keys. Add support for 90 degree quick rotation, fix passing of flip dims. Change number of pixels to translate per click and UI improvements. Improved pre-allocation.
• turboregMovie - Additional matlab disk normalizeType options. Additional display of information.
• normalizeVector - Added percentile name-value options for soft zero to one.
• playMovie - Update to avoid caxis with rgb movies, making them hard to view.
• setupNwb - Update to include check for each of the NWB dependencies, moved from saveNeurodataWithoutBorders. Added nwbpkg support.
• cmdLinePipeline - Updated to handle CIAtah v4.0 switch to all functions inside ciapkg package.
• saveNeurodataWithoutBorders - Added nwbpkg support.

Updated to license, move functions into ciapkg package, gramm graphics, detrend support, speed improvements, movie controls to CIAtah main GUI, etc.

Main features and changes:

• Add sliders to allow users to quickly scroll through both movies in main ciatah GUI.
• Added detrend support for ciatah class modelPreprocessMovieFunction.
• Moved ciapkg overloaded functions into sub-package ciapkg.overloaded to clean up namespace and not interfere with users namespace, updated dependent functions accordingly.
• Moved startup into ciapkg to prevent changing user settings at startup.
• Moved loadBatchFxns into ciapkg package and updated all functions accordingly.
• Speed improvements when setting up ciatah class.
• Added downloading of Gramm (https://github.com/piermorel/gramm) for graphics as external program.
• Updated to MIT license from GPL-3.0 License.

Example of movie slider and cell-extraction in the main GUI.

New functions

• None.

Updated @ciatah

• ciatahMainGui - Update to improve handling of output variables when figure closes to avoid calling invalid handles.
• computeCellDistances - Added gramm plots that show the distribution of cell-cell distances.
• getRegistrationSettings - Added support for certain options (e.g. motion correction) to give a vector and also added movie detrend options.
• modelPreprocessMovieFunction - Add support for multiple reference frames, the mean is taken before input to motion correction. Added support for detrending movies.
• ciatahMainGui - Add sliders to allow users to quickly scroll through both movies. Updated to ensure movie callback playback loop operations only occur if handles to movies are still valid, e.g. if main GUI figure is still open.

Updated ciapkg

• signalSorter - Changed call to overloaded functions to ciapkg.overloaded to avoid altering users Matlab workspace native functions. Change selection of cell on the maps to be instantaneous for where the mouse location is, much faster, don't have to wait for ginputCustom cross-hairs to show. Users can still show crosshairs with 'V'.
• thresholdImages - Fast border calculation using convn for options.fastThresholding==1, less precise than bwboundaries or bwareafilt but works for fast display purposes.
• manageMiji - Check that ImageJ already in java path to prevent duplicate loading and wasting time.

Updated +ciapkg

• `io.loadDependencies' - Added backup URL for downloading mij.jar.
• loadBatchFxns - Remove redundant loading and unloading of external programs via additional checks. Moved loadBatchFxns into ciapkg package. Use ciapkg.getDir to get directory as standard IO.
• startup - moved into ciapkg package.
• loadDependencies - Added gramm (https://github.com/piermorel/gramm) support/downloading for graphics plotting.

GUI updates, fast thresholding, and misc updates.

Main features and changes:

• CIAtah main GUI now displays cell extraction outputs.
• Threshold images has a fast thresholding option.
• Improved callback support in various methods.
• viewCellExtractionOnMovie now supports read from disk.

Updated @ciatah

• ciatahMainGui - Update to ensure NWB switch is made after user selects it. Updated cell extraction loading and allow a cache for faster loading if user switches between methods in the same GUI load. Added support for folder loading button and some other additional improvements. Cell extraction now thresholded for cleaner visuals.
• getRegistrationSettings - Switch to callbacks to close instead of using pause, allows for more flexibility going forward and use with MATLAB Online.
• modelAddNewFolders - Add 'next' button to initial menu, in part to add MATLAB Online support.
• modelExtractSignalsFromMovie - Update to add fix for CELLMax with ROI.
• modelGetSignalsImages - Updated to allow loading raw files without calling modelVarsFromFiles.
• viewCellExtractionOnMovie - Add support for view from disk and cell overlays.

Updated ciapkg

• thresholdImages - Added support for fast thresholding using vectorized form, faster than parfor loop.
• loadMovieList - dims.three fix for reading tifs, esp. ImageJ >4GB.
• playMovie - Added feature to input line overlays on input movie (e.g. to overlay cell extraction outputs).

Updated +ciapkg

• readFrame - Updated to have backup read method for different tiff styles.
• updatePkg - Updated comparison to ensure ignoring of lower level versions after higher level versions are found to differ.
• version - Updated to trim version string to make sure new line is removed.

Improved pre-processing UI, Bio-Formats support, easier ImageJ/Miji loading, and more.

Main features and changes:

• modelPreprocessMovieFunction - CIAtah now remembers the user's analysis order and there are options to reset it. Further, the pre-processing settings are now all displayed on a single page.
• Bio-Formats (https://www.openmicroscopy.org/bio-formats/) support added. Currently added support for Zeiss (CZI, LSM) and Olympus (OIR) formats with more to follow.
• Switch support from Miji/Fiji combination to ImageJ/Miji combination using .jar files that is easier to install and should have easier support across Windows/Linux/OSX.
• Update README to include webinars and workshop recordings on calcium imaging analysis and CIAtah.

New functions

• ciapkg.io.getOptions - clone of getOptions in anticipation of moving all functions to ciapkg sub-package.
• @ciatah\modelVarsFromFilesCheck - Checks whether a specific folder has cell extraction variables loaded, if not loads them.

Updated @ciatah

• computeManualSortSignals - Added modelVarsFromFilesCheck() to check and load signals if user hasn't already. Fix check to make sure variables are loaded.
• computeMatchObjBtwnTrials - Added modelVarsFromFilesCheck() to check and load signals if user hasn't already.
• getRegistrationSettings - Switch to single page support. Multi-column layout since most computers are widescreen now.
• modelExtractSignalsFromMovie - Use filesep in getAlgorithmRootPath to avoid issues in Unix-based systems. Fix issue of passing multiple movies to PCA-ICA.
• modelModifyMovies - manageMiji('startStop','closeAllWindows'); added to improve support for ImageJ over Fiji implementation of Miji.
• modelModifyRegionAnalysis - Added modelVarsFromFilesCheck() to check and load signals if user hasn't already.
• modelPreprocessMovieFunction
  • Fixed thisFrame issue when displaying area to use for motion correction if treatMoviesAsContinuous=0 and processMoviesSeparately=0, e.g. would load reference frame from each and if there were 3 movies, would assume RGB, causing display to be white.
  • Also update so the display frame takes into account custom frame list range. Updated checking of options. Also save ordering of options selected.
  • Added manageMiji('startStop','closeAllWindows'); support.
• modelVarsFromFilesCheck - Add check for objLocations.
• viewCellExtractionOnMovie - Added modelVarsFromFilesCheck() to check and load signals if user hasn't already.
• viewCreateObjmaps - Added modelVarsFromFilesCheck() to check and load signals if user hasn't already.
• viewMovie - Added support for simple and advanced settings. Added modelVarsFromFilesCheck() to check and load signals if user hasn't already.
• viewMovieRegistrationTest - Added modelVarsFromFilesCheck() to check and load signals if user hasn't already.
• viewObjmaps - Added modelVarsFromFilesCheck() to check and load signals if user hasn't already.
• viewSubjectMovieFrames - Added modelVarsFromFilesCheck() to check and load signals if user hasn't already.

Updated ciapkg

• signalSorter - Since ciapkg.io.loadSignalExtraction already supports loading NWB or CIAtah MAT-file cell extraction files, remove remove redundancy and only call ciapkg.io.loadSignalExtraction rather than loadNeurodataWithoutBorders. Added support for progress bar when movie not input.
• cropMatrix - Support switched from [nSignals x y] to [x y nSignals]. Updated to make imrect the default method of selecting the coordinates.
• loadMovieList - Added support for Bio-Formats compatible files, specifically Olympus (OIR) and Zeiss (CZI, LSM).
• manageMiji - Add support for setting up ImageJ along with closing all windows to future proof any changes to those calls.
• modelAddOutsideDependencies - Updated to add support for ImageJ call instead of Fiji.
• saveMatrixToFile - updated TIFF saving to add support for export of multi-channel color timeseries TIFF stack if in format [x y C t] where x,y = width/height, C = RGB channels, t = frames.
• computeManualMotionCorrection - Add default gamma option. Update to fix issue of prior figure (even after clf) maintaining previous key press and exiting, causing uiwait to fail.
• normalizeMovie - Detrend movie now uses nanmean to get around issues in motion corrected videos.
• createMovieFromVector - Users can now manually change value assigned to center line or signal.
• playMovie - Add support for playing RGB movie of dimension [x y C t] if input directly as matrix.
• loadBatchFxns - Added manageMiji('startStop','closeAllWindows'); support.

Updated +ciapkg

• cmdLinePipeline - Added CNMF/CNMF-e and EXTRACT to cell extraction examples.
• loadDependencies - Switched to support for original MIJ calling of ImageJ using just jar files, easier compatibility across MATLAB versions and OSes.
• updatePkg - Added additional URLs for compatibility. Updated to ciatah from calciumImagingAnalysis URLs.

Misc

• ciapkg.io.updatePkg - option on whether to update package (only 0 for now). Also alert user if behind a version.
• ciapkgRoot moved into +ciapkg folder.
• ciapkg.version - Update VERSION path.
• ciapkg.versionOnline - Update VERSION path.
• plotSignalsGraph - Fix minIncrementAmount issue.
• • ciapkg.versionOnline - Now checks multiple location of VERSION file to future proof in case it is moved.
• ciatah - checks online for version at startup.
• ciapkg.nwb.setupNwb - Fix for options.defaultObjDir leading to incorrect NWB folder and cores not being generated.

New load/save cell-extraction outputs, EXTRACT support, and misc. updates

New

• Added support for the new cell-extraction method EXTRACT (https://github.com/schnitzer-lab/EXTRACT-public) to CIAtah.
• ciapkg.io.saveSignalExtraction - universal function to save NWB or CIAtah-style cell-extraction outputs
• ciapkg.io.loadSignalExtraction - function to load either NWB or CIAtah-style cell-extraction outputs.
• ciapkg.pkgName - Outputs the package name, e.g. CIAtah.

Updated:

• calciumImagingAnalysisMainGui changed to ciatahMainGui.
• computeManualSortSignals - If user hasn't called modelVarsFromFiles, computeManualSortSignals called the function. However, this lead to a mismatch between computeManualSortSignals fileNum and obj.fileNum, leading to mismatch between xcoords, etc. and input signals/images.
• loadDependencies - Added EXTRACT support to list of functions to download.
• ciapkg.pkgName - Option on whether to update package (only 0 for now). Also alert user if behind a version.
• modelExtractSignalsFromMovie - Update saveRunTimes to handle case in which user selects multiple movies for cell extraction. Convert ndSparse outputs to single from cell-extraction algorithms (e.g. for CELLMax/EXTRACT) when saving as NWB. Updated EXTRACT support to include additional options.
• signalSorter - User can now input just NWB path without a blank variable for inputSignals. Also added support for CIAtah mat files.
• saveNeurodataWithoutBorders - Update to checking if only a single signal input.
• computeCnmfeSignalExtraction_batch - extractedSignalsType, extractedSignalsEstType struct update.
• changeFont - Add font name support.
• Updated docs.

Adding ciapkg API sub-package, improved directory management, etc.

New

ciapkg.io.loadDependencies - Download and load CIAtah dependencies. Branch from class function.

ciapkg.getDirExternalPrograms - Returns the directory where external programs are stored. All functions should call this to find external program directory.

ciapkg.getDirPkg - Standardized location to obtain relevant CIAtah directories, e.g. location of default data folder.

ciapkg.api - This package will eventually contain pass-through functions for all CIAtah functions, e.g. this will allow CIAtah to keep organization clear by putting functions in related subpackages but also allow users to quickly access all functions using ciapkg.api.

Updated

ciapkg.nwb.setupNwb - Update _external_programs to call ciapkg.getDirExternalPrograms() to standardize call across all functions.

ciapkg.versionOnline - Updated to handle new VERSION file that includes datestamp on 2nd line.

example_downloadTestData - Function now calls data directory via standardized ciapkg.getDirPkg('data') to avoid placing data in incorrect folder.

createHdf5File - Close space_id, dset_id, and fid with low-level HDF5 functions before appending data with hdf5write to avoid read/write issues.

readHDF5Subset - Updated support for files with datasets that contain 2D matrices.

loadMovieList - Fixed loading HDF5 datasetname that has only a single frame, loadMovieList would ask for 3rd dimension information that did not exist.

loadNeurodataWithoutBorders - Parse the algorithm associated with the NWB signal extraction data.

saveNeurodataWithoutBorders - Function checks that yaml, matnwb, and nwb_schnitzer_lab loaded, else tries to load to make sure all dependencies are present and active. Update _external_programs to call ciapkg.getDirExternalPrograms() to standardize call across all functions. Added a check for inputs with a single signal and function returns as it is not supported.

computeCnmfeSignalExtraction_batch - Added trace origin type to output structure.

playMovie - Added feature to sub-sample movie to make display run faster for larger movies.

plotSignalsGraph - Function outputs the modified traces for parent functions to use for additional plotting behavior.

CIAtah class

A number of functions change calling to external programs from hardcoded value to calling ciapkg.getDirExternalPrograms().

loadDependencies - Calls non-class function for use in more functions without needing to load CIAtah class.

modelAddNewFolders - Added support for direct input of method type, useful for command-line or unit testing. 'Add CIAtah example folders.' now adds the absolute path to avoid path errors if user changes Matlab current directory.

modelExtractSignalsFromMovie - adding CELLMax and EXTRACT to public repo in anticipation of their repos being made public. Update _external_programs to call ciapkg.getDirExternalPrograms() to standardize call across all functions.

modelPreprocessMovieFunction - _inputMovieF0 now saved to processing subfolder.

README update, image/signal loading streamline, and misc. other updates

• ciapkg.io.matlabToolboxCheck - Checks whether a MATLAB toolbox/feature has a license available and is installed.
• ciapkg.io.dependencyCheck now re-directs to ciapkg.io.matlabToolboxCheck.
• ciapkg.io.updatePkg - notify users if behind on a version.
• ciapkg.versionOnline - check the online version of CIAtah.
• README update.
• modelGetSignalsImages - Reorganized returnType to be outside raw signals flag, so common filtering mechanism regardless of variables loaded into RAM or not. This fixes if a user loads variables into RAM then uses cross-session alignment, the viewMatchObjBtwnSessions method may not display registered images (cross-session alignment is still fine and valid).
• viewMatchObjBtwnSessions - Take out parfor in one loop that calls obj functions to eliminate serialization issues. Also change color mapping of cross session maps so that it covers the full range using the actual global IDs matched across sessions rather than diluting the range by including global IDs that don't meet the criteria. Makes the maps more colorful.
• applyImagesToMovie - Remove parallelization across inputImages to reduce memory overhead and serialization memory issues (e.g. transferring a duplicate of the movie to all workers).
• Misc other changes and update to README.