Skip to content

Commit e482568

Browse files
authored
Add some TLP doco about shared local repo access (#185)
Rework local repo page available from TLP, move "uninteresting" simple LRM to bottom, add intro about locking, to not have it buried in some module...
1 parent f251fdc commit e482568

File tree

2 files changed

+67
-32
lines changed

2 files changed

+67
-32
lines changed

maven-resolver-named-locks/src/site/markdown/index.md.vm

+22-7
Original file line numberDiff line numberDiff line change
@@ -23,28 +23,41 @@ Named locks are essentially locks that are assigned to some given (opaque) ID. I
2323
resources that each can have unique ID assigned (i.e., file with an absolute path, some entities with unique ID),
2424
then you can use named locks to make sure they are being protected from concurrent read and write actions.
2525

26-
Named locks provide support classes for implementations, and provide out of the box seven named lock implementations and three name mappers.
26+
Named locks provide support classes for implementations, and provide out of the box several lock and name mapper implementations.
2727

28-
Out of the box, "local" (local to JVM) named lock implementations are the following:
28+
Following implementations are "local" (local to JVM) named lock implementations:
2929

3030
- `rwlock-local` implemented in `org.eclipse.aether.named.providers.LocalReadWriteLockNamedLockFactory` that uses
3131
JVM `java.util.concurrent.locks.ReentrantReadWriteLock`.
3232
- `semaphore-local` implemented in `org.eclipse.aether.named.providers.LocalSemaphoreNamedLockFactory` that uses
3333
JVM `java.util.concurrent.Semaphore`.
34+
- `noop` implemented in `org.eclipse.aether.named.providers.NoopNamedLockFactory` that uses no locking.
35+
36+
Note about "local" locks: they are in-JVM, in a way, they properly coordinate in case of multithreaded access from
37+
same JVM, but do not cover accesses across multiple processes and/or multiple hosts access.
38+
In other words, local named locks are only suited within one JVM with a multithreaded access.
39+
40+
Following named lock implementations use underlying file system advisory file locking:
41+
3442
- `file-lock` implemented in `org.eclipse.aether.named.providers.FileLockNamedLockFactory` that uses
3543
JVM `java.nio.channels.FileLock`.
36-
- `noop` implemented in `org.eclipse.aether.named.providers.NoopNamedLockFactory` that uses no locking.
3744

38-
Out of the box, "distributed" named lock implementations are the following (separate modules which require additional dependencies):
45+
The `file-lock` implementation uses file system advisory file locking, hence, concurrently running Maven processes
46+
set up to use `file-lock` implementation can safely share one local repository. This is almost certain on
47+
local file systems across all operating systems. In case of NFS mounts, file advisory locking MAY work if NFSv4+
48+
used with complete setup (with all the necessary services like `RPC` and `portmapper` needed to implement NFS advisory
49+
file locking, check your NFS and/or OS manuals for details). In short: if your (local or remote) FS correctly support
50+
and implements advisory locking, it should work. Local FS usually does, while with NFS your mileage may vary.
51+
52+
Finally, "distributed" named lock implementations are the following (separate modules which require additional dependencies and remote services):
3953

4054
- `rwlock-redisson` implemented in `org.eclipse.aether.named.redisson.RedissonReadWriteLockNamedLockFactory`.
4155
- `semaphore-redisson` implemented in `org.eclipse.aether.named.redisson.RedissonSemaphoreNamedLockFactory`.
4256
- `semaphore-hazelcast-client` implemented in `org.eclipse.aether.named.hazelcast.HazelcastClientCPSemaphoreNamedLockFactory`.
4357
- `semaphore-hazelcast` implemented in `org.eclipse.aether.named.hazelcast.HazelcastCPSemaphoreNamedLockFactory`.
4458

45-
Local named locks are only suited within one JVM with a multithreaded build.
46-
Sharing a local repository between multiple Maven processes (i.e., on a busy CI server) requires a distributed named lock!
47-
59+
Sharing a local repository between multiple hosts (i.e., on a busy CI server) may be best done with one of distributed named lock,
60+
if NFS locking is not working for you.
4861

4962
The aforementioned (opaque) IDs need to be mapped from artifacts and metadata.
5063

@@ -54,3 +67,5 @@ Out of the box, name mapper implementations are the following:
5467
- `gav` implemented in `org.eclipse.aether.internal.impl.synccontext.named.GAVNameMapper`.
5568
- `discriminating` implemented in `org.eclipse.aether.internal.impl.synccontext.named.DiscriminatingNameMapper`.
5669
- `file-gav` implemented in `org.eclipse.aether.internal.impl.synccontext.named.FileGAVNameMapper`.
70+
71+
Note: the `file-gav` name mapper MUST be used with `file-lock` named locking, no other mapper will work with it.

src/site/markdown/local-repository.md

+45-25
Original file line numberDiff line numberDiff line change
@@ -25,30 +25,16 @@ remote, but also to store the artifacts locally installed (locally built and
2525
installed, to be more precise). Both of these artifacts were stored in bulk
2626
in the local repository.
2727

28+
## Implementations
29+
2830
Local repository implementations implement the `LocalRepositoryManager` (LRM)
2931
interface, and Resolver out of the box provides two implementations for it:
30-
"simple" and "enhanced".
31-
32-
## Simple LRM
32+
"enhanced" used at runtime and "simple" meant to be used in tests and alike
33+
scenarios (is not meant for production use).
3334

34-
Simple is a fully functional LRM implementation, but is used
35-
mainly in tests, it is not recommended in production environments.
36-
37-
To manually instantiate a simple LRM, one needs to invoke following code:
35+
### Enhanced LRM
3836

39-
```java
40-
LocalRepositoryManager simple = new SimpleLocalRepositoryManagerFactory()
41-
.newInstance( session, new LocalRepository( baseDir ) );
42-
```
43-
44-
Note: This code snippet above instantiates a component, that is not
45-
recommended way to use it, as it should be rather injected whenever possible.
46-
This example above is merely a showcase how to obtain LRM implementation
47-
in unit tests.
48-
49-
## Enhanced LRM
50-
51-
Enhanced LRM on the other hand is enhanced with several extra
37+
Enhanced LRM is enhanced with several extra
5238
features, one most notable is scoping cached content by its origin and context:
5339
if you downloaded an artifact A1 from repository R1
5440
and later initiate build that requires same artifact A1, but repository R1
@@ -58,7 +44,7 @@ Those two, originating from two different repositories may not be the same thing
5844
This is meant to protect users from "bad practice" (artifact coordinates are
5945
unique in ideal world).
6046

61-
### Split Local Repository
47+
#### Split Local Repository
6248

6349
Latest addition to the enhanced LRM is *split* feature. By default, split
6450
feature is **not enabled**, enhanced LRM behaves as it behaved in all
@@ -75,7 +61,7 @@ The split feature is implemented by the `LocalPathPrefixComposer` interface,
7561
that adds different "prefixes" for the locally stored artifacts, based on
7662
their context.
7763

78-
#### Note About Release And Snapshot Differentiation
64+
##### Note About Release And Snapshot Differentiation
7965

8066
The prefix composer is able to differentiate between release and snapshot
8167
versioned artifacts, and this is clear-cut: Maven Artifacts are either
@@ -99,7 +85,7 @@ The GAV level metadata gets differentiated based on version it carries, so
9985
they may end up in releases or snapshots, depending on their value of
10086
`metadata/version` field.
10187

102-
#### Use Cases
88+
##### Use Cases
10389

10490
Most direct use case is simpler local repository eviction. One can delete all
10591
locally built artifacts without deleting the cached ones, hence, no
@@ -139,7 +125,7 @@ $ mvn ... -Daether.enhancedLocalRepository.split \
139125
For complete reference of enhanced LRM configuration possibilities, refer to
140126
[configuration page](configuration.html).
141127

142-
#### Split Repository Considerations
128+
##### Split Repository Considerations
143129

144130
**Word of warning**: on every change of "split" parameters, user must be aware
145131
of the consequences. For example, if one change all aspects of split
@@ -149,7 +135,7 @@ is unchanged! Simply put, as all prefixes will be "new", the composed paths will
149135
point to potentially non-existing locations, hence, resolver will consider
150136
it as a "new" local repository in every aspect.
151137

152-
#### Implementing Custom Split Strategy
138+
##### Implementing Custom Split Strategy
153139

154140
To implement custom split strategy, one needs to create a component of
155141
type `LocalPathPrefixComposerFactory` and override the default component
@@ -160,3 +146,37 @@ class that provides all the defaults.
160146
The factory should create a stateless instance of a composer
161147
configured from passed in session, that will be used with the enhanced LRM
162148
throughout the session.
149+
150+
### Simple LRM
151+
152+
Simple is a fully functional LRM implementation, but is used
153+
mainly in tests, it is not recommended in production environments.
154+
155+
To manually instantiate a simple LRM, one needs to invoke following code:
156+
157+
```java
158+
LocalRepositoryManager simple = new SimpleLocalRepositoryManagerFactory()
159+
.newInstance( session, new LocalRepository( baseDir ) );
160+
```
161+
162+
Note: This code snippet above instantiates a component, that is not
163+
recommended way to use it, as it should be rather injected whenever possible.
164+
This example above is merely a showcase how to obtain LRM implementation
165+
in unit tests.
166+
167+
## Shared Access to Local Repository
168+
169+
In case of shared (multi-threaded, multi-process or even multi host) access
170+
to local repository, coordination is a must, as local repository is hosted
171+
on file system, and each thread may read and write concurrently into it,
172+
causing other threads or processes to get incomplete or partially written data.
173+
174+
Hence, since Resolver 1.7.x version, there is a pluggable API called "Named Locks"
175+
available, providing out of the box lock implementations for cases like:
176+
177+
* multi-threaded, in JVM locking (the default)
178+
* multi-process locking using file system advisory locking
179+
* multi-host locking using Hazelcast or Redisson (needs Hazelcast or Redisson cluster)
180+
181+
For details see [Named Locks module](maven-resolver-named-locks/).
182+

0 commit comments

Comments
 (0)