Skip to content

fix: align arrow version with spark and add docker integration tests #205

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jackye1995 merged 6 commits into from
Feb 6, 2026
Merged

fix: align arrow version with spark and add docker integration tests #205

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
8e9bc20
initial commit
hamersaw Feb 5, 2026
26c73cc
added lightweight testing and fixed env vars in Makefile
hamersaw Feb 6, 2026
929c40c
working e2e
hamersaw Feb 6, 2026
3e89a1f
Merge remote-tracking branch 'upstream/main' into feature/docker-tests
hamersaw Feb 6, 2026
b76a0bf
keeping default Spark 3.5 and updating Spark 4.0 to Arrow 18.1
hamersaw Feb 6, 2026
469933a
removing maven profiles and hardcoding versions in each module
hamersaw Feb 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,10 @@ docker-shell:
docker-down: check-docker-compose
cd docker && ${DOCKER_COMPOSE} down

.PHONY: docker-test
docker-test: check-docker-compose
cd docker && docker exec spark-lance pytest /home/lance/tests/ -v --timeout=120

# =============================================================================
# Documentation
# =============================================================================
Expand Down Expand Up @@ -151,6 +155,7 @@ help:
@echo " docker-up - Start docker containers"
@echo " docker-shell - Open shell in spark-lance container"
@echo " docker-down - Stop docker containers"
@echo " docker-test - Run integration tests in spark-lance container"
@echo ""
@echo "Documentation:"
@echo " serve-docs - Serve documentation locally"
5 changes: 4 additions & 1 deletion docker/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ WORKDIR ${SPARK_HOME}

ENV SPARK_VERSION=3.5.8
ENV SPARK_MAJOR_VERSION=3.5
ENV LANCE_SPARK_VERSION=0.1.3-beta.7
ENV LANCE_SPARK_VERSION=0.1.3-beta.8
Comment thread
hamersaw marked this conversation as resolved.
ENV LANCE_NS_VERSION=0.4.5

# Download spark
Expand Down Expand Up @@ -82,6 +82,9 @@ RUN mkdir -p /home/lance/warehouse /home/lance/notebooks /home/lance/spark-event
# Copy notebooks if available
COPY notebooks/ /home/lance/notebooks/

# Copy tests
COPY tests/ /home/lance/tests/

# Add a notebook command
RUN echo '#! /bin/sh' >> /bin/notebook \
&& echo 'export PYSPARK_DRIVER_PYTHON=jupyter-lab' >> /bin/notebook \
Expand Down
2 changes: 2 additions & 0 deletions docker/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
pytest==8.3.5
pytest-timeout==2.3.1
jupyterlab==3.6.7
jupyter-server==1.24.0
pandas==2.3.1
Expand Down
Loading