Skip to content

Commit

Permalink
hadoop: user root in sdk as normal user (#3213)
Browse files Browse the repository at this point in the history 
* hadoop: user root in sdk as normal user

* add doc for juicefs.supergroup
  • Loading branch information
@tangyoupeng
tangyoupeng authored Feb 6, 2023
1 parent 7db5227 commit b7788e1
Show file tree
Hide file tree
Showing 3 changed files with 39 additions and 29 deletions.
1 change: 1 addition & 0 deletions docs/en/deployment/hadoop_java_sdk.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -181,6 +181,7 @@ Please refer to the following table to set the relevant parameters of the JuiceF
| `juicefs.debug` | `false` | Whether enable debug log |
| `juicefs.access-log` | | Access log path. Ensure Hadoop application has write permission, e.g. `/tmp/juicefs.access.log`. The log file will rotate automatically to keep at most 7 files. |
| `juicefs.superuser` | `hdfs` | The super user |
| `juicefs.supergroup` | `supergroup` | The super user group |
| `juicefs.users` | `null` | The path of username and UID list file, e.g. `jfs://name/etc/users`. The file format is `<username>:<UID>`, one user per line. |
| `juicefs.groups` | `null` | The path of group name, GID and group members list file, e.g. `jfs://name/etc/groups`. The file format is `<group-name>:<GID>:<username1>,<username2>`, one group per line. |
| `juicefs.umask` | `null` | The umask used when creating files and directories (e.g. `0022`), default value is `fs.permissions.umask-mode`. |
Expand Down
41 changes: 21 additions & 20 deletions docs/zh_cn/deployment/hadoop_java_sdk.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -177,26 +177,27 @@ make win

#### 其它配置

| 配置项 | 默认值 | 描述 |
|---------------------------|-------------|-----------------------------------------------------------------------------------------------------------------------------------------------|
| `juicefs.bucket` | | 为对象存储指定跟格式化时不同的访问地址 |
| `juicefs.debug` | `false` | 是否开启 debug 日志 |
| `juicefs.access-log` | | 访问日志的路径。需要所有应用都有写权限，可以配置为 `/tmp/juicefs.access.log`。该文件会自动轮转，保留最近 7 个文件。 |
| `juicefs.superuser` | `hdfs` | 超级用户 |
| `juicefs.users` | `null` | 用户名以及 UID 列表文件的地址，比如 `jfs://name/etc/users`。文件格式为 `<username>:<UID>`，一行一个用户。 |
| `juicefs.groups` | `null` | 用户组、GID 以及组成员列表文件的地址，比如 `jfs://name/etc/groups`。文件格式为 `<group-name>:<GID>:<username1>,<username2>`，一行一个用户组。 |
| `juicefs.umask` | `null` | 创建文件和目录的 umask 值（如 `0022`），如果没有此配置，默认值是 `fs.permissions.umask-mode`。 |
| `juicefs.push-gateway` | | [Prometheus Pushgateway](https://github.com/prometheus/pushgateway) 地址，格式为 `<host>:<port>`。 |
| `juicefs.push-auth` | | [Prometheus 基本认证](https://prometheus.io/docs/guides/basic-auth)信息，格式为 `<username>:<password>`。 |
| `juicefs.push-graphite` | | [Graphite](https://graphiteapp.org) 地址，格式为 `<host>:<port>`。 |
| `juicefs.push-interval` | 10 | 指标推送的时间间隔，单位为秒。 |
| `juicefs.fast-resolve` | `true` | 是否开启快速元数据查找（通过 Redis Lua 脚本实现） |
| `juicefs.no-usage-report` | `false` | 是否上报数据。仅上版本号等使用量数据，不包含任何用户信息。 |
| `juicefs.block.size` | `134217728` | 单位为字节，同 HDFS 的 `dfs.blocksize`，默认 128 MB |
| `juicefs.file.checksum` | `false` | DistCp 使用 `-update` 参数时，是否计算文件 Checksum |
| `juicefs.no-bgjob` | `false` | 是否关闭后台任务（清理、备份等） |
| `juicefs.backup-meta` | 3600 | 自动将 JuiceFS 元数据备份到对象存储间隔（单位：秒），设置为 0 关闭自动备份 |
| `juicefs.heartbeat` | 12 | 客户端和元数据引擎之间的心跳间隔（单位：秒），建议所有客户端都设置一样 |
| 配置项 | 默认值 | 描述 |
|---------------------------|--------------|-------------------------------------------------------------------------------------------------------------|
| `juicefs.bucket` | | 为对象存储指定跟格式化时不同的访问地址 |
| `juicefs.debug` | `false` | 是否开启 debug 日志 |
| `juicefs.access-log` | | 访问日志的路径。需要所有应用都有写权限，可以配置为 `/tmp/juicefs.access.log`。该文件会自动轮转，保留最近 7 个文件。 |
| `juicefs.superuser` | `hdfs` | 超级用户 |
| `juicefs.supergroup` | `supergroup` | 超级用户组 |
| `juicefs.users` | `null` | 用户名以及 UID 列表文件的地址，比如 `jfs://name/etc/users`。文件格式为 `<username>:<UID>`，一行一个用户。 |
| `juicefs.groups` | `null` | 用户组、GID 以及组成员列表文件的地址，比如 `jfs://name/etc/groups`。文件格式为 `<group-name>:<GID>:<username1>,<username2>`，一行一个用户组。 |
| `juicefs.umask` | `null` | 创建文件和目录的 umask 值（如 `0022`），如果没有此配置，默认值是 `fs.permissions.umask-mode`。 |
| `juicefs.push-gateway` | | [Prometheus Pushgateway](https://github.com/prometheus/pushgateway) 地址，格式为 `<host>:<port>`。 |
| `juicefs.push-auth` | | [Prometheus 基本认证](https://prometheus.io/docs/guides/basic-auth)信息，格式为 `<username>:<password>`。 |
| `juicefs.push-graphite` | | [Graphite](https://graphiteapp.org) 地址，格式为 `<host>:<port>`。 |
| `juicefs.push-interval` | 10 | 指标推送的时间间隔，单位为秒。 |
| `juicefs.fast-resolve` | `true` | 是否开启快速元数据查找（通过 Redis Lua 脚本实现） |
| `juicefs.no-usage-report` | `false` | 是否上报数据。仅上版本号等使用量数据，不包含任何用户信息。 |
| `juicefs.block.size` | `134217728` | 单位为字节，同 HDFS 的 `dfs.blocksize`，默认 128 MB |
| `juicefs.file.checksum` | `false` | DistCp 使用 `-update` 参数时，是否计算文件 Checksum |
| `juicefs.no-bgjob` | `false` | 是否关闭后台任务（清理、备份等） |
| `juicefs.backup-meta` | 3600 | 自动将 JuiceFS 元数据备份到对象存储间隔（单位：秒），设置为 0 关闭自动备份 |
| `juicefs.heartbeat` | 12 | 客户端和元数据引擎之间的心跳间隔（单位：秒），建议所有客户端都设置一样 |

#### 多文件系统配置

Expand Down
26 changes: 17 additions & 9 deletions sdk/java/libjfs/guid.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -68,12 +68,16 @@ func (m *mapping) lookupUser(name string) uint32 {
if !m.local {
return m.genGuid(name)
}
u, _ := user.Lookup(name)
if u != nil {
id_, _ := strconv.ParseUint(u.Uid, 10, 32)
id = uint32(id_)
} else {
if name == "root" { // root in hdfs sdk is a normal user
id = m.genGuid(name)
} else {
u, _ := user.Lookup(name)
if u != nil {
id_, _ := strconv.ParseUint(u.Uid, 10, 32)
id = uint32(id_)
} else {
id = m.genGuid(name)
}
}
m.usernames[name] = id
m.userIDs[id] = name
Expand All @@ -90,12 +94,16 @@ func (m *mapping) lookupGroup(name string) uint32 {
if !m.local {
return m.genGuid(name)
}
g, _ := user.LookupGroup(name)
if g == nil {
if name == "root" {
id = m.genGuid(name)
} else {
id_, _ := strconv.ParseUint(g.Gid, 10, 32)
id = uint32(id_)
g, _ := user.LookupGroup(name)
if g == nil {
id = m.genGuid(name)
} else {
id_, _ := strconv.ParseUint(g.Gid, 10, 32)
id = uint32(id_)
}
}
m.groups[name] = id
m.groupIDs[id] = name
Expand Down

0 comments on commit b7788e1

Please sign in to comment.