Skip to content

Releases: dadoonet/fscrawler

v2.9 🌈

08 Mar 10:34
@dadoonet dadoonet
fscrawler-2.9
740f0d6
Compare
Choose a tag to compare
v2.9 🌈 Latest
Latest

What's Changed

New Contributors

Full Changelog: fscrawler-2.8...fscrawler-2.9

Contributors

dadoonet, dependabot, and cbb-colab
Assets 2
Loading

v2.8 🌈

08 Mar 10:32
@release-drafter release-drafter
fscrawler-2.8
38bdefb
Compare
Choose a tag to compare
v2.8 🌈

What's Changed

  • #1356: ci(Mergify): configuration update (thanks to @dadoonet)
  • #1322: Update Log4J 2.15.0 and Elasticsearch 7.16.1 (thanks to @dadoonet)
  • #1276: Revert "Remove a52f2ab6-086b-4285-a7a1-78ecdc6404ba vulnerability id" (thanks to @dadoonet)
  • #1275: Remove a52f2ab6-086b-4285-a7a1-78ecdc6404ba vulnerability id (thanks to @dadoonet)
  • #1228: latest docker tag should be only the latest stable version (thanks to @dadoonet)

🚀 New features

🚨 Bug Fixes

  • #1358: fs.ocr.enabled is always false (thanks to @ywjung)
  • #1286: Fix starting fscrawler with Docker (thanks to @dadoonet)
  • #1271: fix: not working optional libraries (e.g. jpeg2000) (thanks to @NickUfer)

💉 Updated features

📝 Documentation updates

  • #1345: Improve documentation for settings (thanks to @cbb-colab)
  • #1310: Update ocr.rst, the path was wrong and not working (thanks to @sahin52)
  • #1256: Add section Workaround for huge temporary files (thanks to @dfbm)

🚦 Tests

Thanks to

@NickUfer, @cbb-colab, @cwperry, @dadoonet, @dependabot, @dependabot[bot], @dfbm, @mergify[bot], @sahin52 and @ywjung

Contributors

dadoonet, cwperry, and 7 other contributors
Assets 2
Loading

FSCrawler 2.7 🌈

05 Aug 11:10
@release-drafter release-drafter
fscrawler-2.7
2598ce2
Compare
Choose a tag to compare
FSCrawler 2.7 🌈

The FSCrawler team is pleased to announce the FSCrawler 2.7 release!

FSCrawler

FS Crawler offers a simple way to index binary files into elasticsearch.

Usage

Download FSCrawler 2.7:

wget https://repo1.maven.org/maven2/fr/pilato/elasticsearch/crawler/fscrawler-es7/2.7/fscrawler-es7-2.7.zip

Start FS crawler with:

bin/fscrawler job_name

FS crawler will read a local file (default to ~/.fscrawler/{job_name}/_settings.json).
If the file does not exist, FS crawler will propose to create your first job.

$ bin/fscrawler job_name
18:28:58,174 WARN  [f.p.e.c.f.FsCrawler] job [job_name] does not exist
18:28:58,177 INFO  [f.p.e.c.f.FsCrawler] Do you want to create it (Y/N)?
y
18:29:05,711 INFO  [f.p.e.c.f.FsCrawler] Settings have been created in [~/.fscrawler/job_name/_settings.json]. Please review and edit before relaunch

Create a directory named /tmp/es or c:\tmp\es, add some files you want to index in it and start again:

$ bin/fscrawler job_name
18:30:34,330 INFO  [f.p.e.c.f.FsCrawlerImpl] Starting FS crawler
18:30:34,332 INFO  [f.p.e.c.f.FsCrawlerImpl] FS crawler started in watch mode. It will run unless you stop it with CTRL+C.
18:30:34,682 INFO  [f.p.e.c.f.FsCrawlerImpl] FS crawler started for [job_name] for [/tmp/es] every [15m]

More details in the documentation.

New features

  • #991: Add Workplace Search connector.
  • #1203: Add FTP crawler. By helsonxiao.
  • #1211: Add file.content_type field on folders.
  • #1210: Add file.filename field on folders.
  • #1179: Automatically create Custom Sources.
  • #1037: Split console logs and actual logs and add a banner :).
  • #1036: Support ssl verification configurable. By TommyLike.
  • #1035: Log index errors in documents.log.
  • #1031: Add an external Log4J2 configuration file.
  • #907: Add path_prefix option.
  • #820: Generate FSCrawler docker images. By toto1310.
  • #776: Report HEAP size at startup.
  • #752: Add option to ignore symlinks. By budachst.
  • #715: Allow custom index name in the REST API. By kikkauz.
  • #698: Add Cross-Origin Resource Sharing (CORS) headers to RestServer. By isaac-ipl.
  • #692: Allow running OCR but not on PDF files.
  • #673: Add support for YAML configuration.
  • #663: Add Patterns table to includes and excludes. By wrathagom.

Fixed Bugs

  • #1224: Fix NPE in Console when running with Docker.
  • #1217: Check if date is null when formatting it to RFC3339.
  • #1204: Split build and deploy phases for Docker images.
  • #1201: 2.7 - Docker image broken. By agrantdeakin.
  • #1194: Elasticsearch node settings should not be null by default.
  • #1193: Corrupt PDF can lead to a StackOverflow.
  • #1137: Ignore errors when parsing a 0 byte file.
  • #1085: fscrawler.bat added a CD to move to the appropriate directory. By CircuitGuy.
  • #1084: InputStream must have > 0 bytes. By yuanzhian.
  • #1066: Start fscrawler instead of internal services.
  • #1041: Fixed an issue that caused an error when running in a windows environment. By muraken720.
  • #1006: Running fscrawler with no argument now lists existing jobs. By janhoy.
  • #1005: Fix ENTRYPOINT in Dockerfile to allow variable substitution. By Maijin.
  • #994: Using cloud id gives "invalid IPv6 Address". By tdaroly.
  • #973: Fix SSH crawling from Windows machine.
  • #899: FSCrawler can't index .doc or .docx elements. By LaaKii.
  • #895: java.lang.NoSuchMethodError: parsing some Word files. By mwaltersbmc.
  • #860: Bug Syntax error in fscrawler file, to init fscrawler. By CarlosRCDev.
  • #847: sun.jnu.encoding=UTF-8 added in .bat and .sh both. By shahariaazam.
  • #834: FS Crawler freezes when crawling a 0 byte TXT file. By dansfelix.
  • #819: Fix Percentage computation.
  • #760: Allow passing test parameters to Maven CLI.
  • #714: fix release-drafter. By jetersen.
  • #701: Change log level and display logs only if filters on content.
  • #691: OCR without pdf_ocr. By Newmski.
  • #686: Wait for healthy index when creating the index.
  • #681: SSH dirs should be seen as dirs and not files.
  • #680: trying to index remote files with ssh - files seen as folder. By sblanc0054.
  • #660: Fix authentication when sending announcement email.

Main changes

  • #1218: Isolate WorkplaceSearchClient and ElasticsearchClient.
  • #1213: Switch back to Java 11.
  • #1049: Update Dockerfile to use JDK14. By mario-89.
  • #1212: Let's use JsonPath.
  • #1207: Generate only 2 docker images.
  • #1206: Detect when fscrawler runs in foreground and adapt logs.
  • #1205: Add logs to the console when running a Docker instance.
  • #1172: Move CI from Travis to GitHub actions.
  • #872: Add more information to the _simulate API.
  • #700: Add dependency convergence checks.
  • #695: Exclude the PDFParser from the DefaultParser.
  • #694: Display full names when catching parsing errors.
  • #693: Move fs.pdf_ocr setting to fs.ocr.pdf_strategy.
  • #675: Warn in case of Tika error.
  • #1219: Update to Elasticsearch 7.14.0 and 6.8.18.
  • #1180: Bump tika.version from 1.26 to 1.27.

Removed

  • #978: files lost. By bluebell1990.

Have fun!
-FSCrawler team

Assets 4
Loading

FSCrawler 2.6

09 Jan 18:34
@release-drafter release-drafter
fscrawler-2.6
e710d00
Compare
Choose a tag to compare
FSCrawler 2.6

What's Changed

Thanks to

@FredDut, @Jdecaudin, @Quix0r, @babadofar, @barts2108, @coder-sa, @ctamisier, @dadoonet, @edjeavons, @fgaujous, @gpcmol, @it20one, @kneubi, @shadiakiki1986, @soruly, @vakopian, @xcorail, Ajitpal Singh and Julien Decaudin

Assets 2
Loading