Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hadoop: user root in sdk as normal user #3213

Merged
merged 2 commits into from
Feb 6, 2023
Merged

hadoop: user root in sdk as normal user #3213

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f703609
hadoop: user root in sdk as normal user
tangyoupeng Feb 6, 2023
05f1733
add doc for juicefs.supergroup
tangyoupeng Feb 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/en/deployment/hadoop_java_sdk.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,6 +181,7 @@ Please refer to the following table to set the relevant parameters of the JuiceF
| `juicefs.debug` | `false` | Whether enable debug log |
| `juicefs.access-log` | | Access log path. Ensure Hadoop application has write permission, e.g. `/tmp/juicefs.access.log`. The log file will rotate automatically to keep at most 7 files. |
| `juicefs.superuser` | `hdfs` | The super user |
| `juicefs.supergroup` | `supergroup` | The super user group |
| `juicefs.users` | `null` | The path of username and UID list file, e.g. `jfs://name/etc/users`. The file format is `<username>:<UID>`, one user per line. |
| `juicefs.groups` | `null` | The path of group name, GID and group members list file, e.g. `jfs://name/etc/groups`. The file format is `<group-name>:<GID>:<username1>,<username2>`, one group per line. |
| `juicefs.umask` | `null` | The umask used when creating files and directories (e.g. `0022`), default value is `fs.permissions.umask-mode`. |
Expand Down
41 changes: 21 additions & 20 deletions docs/zh_cn/deployment/hadoop_java_sdk.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,26 +177,27 @@ make win

#### 其它配置

| 配置项 | 默认值 | 描述 |
|---------------------------|-------------|-----------------------------------------------------------------------------------------------------------------------------------------------|
| `juicefs.bucket` | | 为对象存储指定跟格式化时不同的访问地址 |
| `juicefs.debug` | `false` | 是否开启 debug 日志 |
| `juicefs.access-log` | | 访问日志的路径。需要所有应用都有写权限，可以配置为 `/tmp/juicefs.access.log`。该文件会自动轮转，保留最近 7 个文件。 |
| `juicefs.superuser` | `hdfs` | 超级用户 |
| `juicefs.users` | `null` | 用户名以及 UID 列表文件的地址，比如 `jfs://name/etc/users`。文件格式为 `<username>:<UID>`，一行一个用户。 |
| `juicefs.groups` | `null` | 用户组、GID 以及组成员列表文件的地址，比如 `jfs://name/etc/groups`。文件格式为 `<group-name>:<GID>:<username1>,<username2>`，一行一个用户组。 |
| `juicefs.umask` | `null` | 创建文件和目录的 umask 值（如 `0022`），如果没有此配置，默认值是 `fs.permissions.umask-mode`。 |
| `juicefs.push-gateway` | | [Prometheus Pushgateway](https://github.com/prometheus/pushgateway) 地址，格式为 `<host>:<port>`。 |
| `juicefs.push-auth` | | [Prometheus 基本认证](https://prometheus.io/docs/guides/basic-auth)信息，格式为 `<username>:<password>`。 |
| `juicefs.push-graphite` | | [Graphite](https://graphiteapp.org) 地址，格式为 `<host>:<port>`。 |
| `juicefs.push-interval` | 10 | 指标推送的时间间隔，单位为秒。 |
| `juicefs.fast-resolve` | `true` | 是否开启快速元数据查找（通过 Redis Lua 脚本实现） |
| `juicefs.no-usage-report` | `false` | 是否上报数据。仅上版本号等使用量数据，不包含任何用户信息。 |
| `juicefs.block.size` | `134217728` | 单位为字节，同 HDFS 的 `dfs.blocksize`，默认 128 MB |
| `juicefs.file.checksum` | `false` | DistCp 使用 `-update` 参数时，是否计算文件 Checksum |
| `juicefs.no-bgjob` | `false` | 是否关闭后台任务（清理、备份等） |
| `juicefs.backup-meta` | 3600 | 自动将 JuiceFS 元数据备份到对象存储间隔（单位：秒），设置为 0 关闭自动备份 |
| `juicefs.heartbeat` | 12 | 客户端和元数据引擎之间的心跳间隔（单位：秒），建议所有客户端都设置一样 |
| 配置项 | 默认值 | 描述 |
|---------------------------|--------------|-------------------------------------------------------------------------------------------------------------|
| `juicefs.bucket` | | 为对象存储指定跟格式化时不同的访问地址 |
| `juicefs.debug` | `false` | 是否开启 debug 日志 |
| `juicefs.access-log` | | 访问日志的路径。需要所有应用都有写权限，可以配置为 `/tmp/juicefs.access.log`。该文件会自动轮转，保留最近 7 个文件。 |
| `juicefs.superuser` | `hdfs` | 超级用户 |
| `juicefs.supergroup` | `supergroup` | 超级用户组 |
| `juicefs.users` | `null` | 用户名以及 UID 列表文件的地址，比如 `jfs://name/etc/users`。文件格式为 `<username>:<UID>`，一行一个用户。 |
| `juicefs.groups` | `null` | 用户组、GID 以及组成员列表文件的地址，比如 `jfs://name/etc/groups`。文件格式为 `<group-name>:<GID>:<username1>,<username2>`，一行一个用户组。 |
| `juicefs.umask` | `null` | 创建文件和目录的 umask 值（如 `0022`），如果没有此配置，默认值是 `fs.permissions.umask-mode`。 |
| `juicefs.push-gateway` | | [Prometheus Pushgateway](https://github.com/prometheus/pushgateway) 地址，格式为 `<host>:<port>`。 |
| `juicefs.push-auth` | | [Prometheus 基本认证](https://prometheus.io/docs/guides/basic-auth)信息，格式为 `<username>:<password>`。 |
| `juicefs.push-graphite` | | [Graphite](https://graphiteapp.org) 地址，格式为 `<host>:<port>`。 |
| `juicefs.push-interval` | 10 | 指标推送的时间间隔，单位为秒。 |
| `juicefs.fast-resolve` | `true` | 是否开启快速元数据查找（通过 Redis Lua 脚本实现） |
| `juicefs.no-usage-report` | `false` | 是否上报数据。仅上版本号等使用量数据，不包含任何用户信息。 |
| `juicefs.block.size` | `134217728` | 单位为字节，同 HDFS 的 `dfs.blocksize`，默认 128 MB |
| `juicefs.file.checksum` | `false` | DistCp 使用 `-update` 参数时，是否计算文件 Checksum |
| `juicefs.no-bgjob` | `false` | 是否关闭后台任务（清理、备份等） |
| `juicefs.backup-meta` | 3600 | 自动将 JuiceFS 元数据备份到对象存储间隔（单位：秒），设置为 0 关闭自动备份 |
| `juicefs.heartbeat` | 12 | 客户端和元数据引擎之间的心跳间隔（单位：秒），建议所有客户端都设置一样 |

#### 多文件系统配置

Expand Down
26 changes: 17 additions & 9 deletions sdk/java/libjfs/guid.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,12 +68,16 @@ func (m *mapping) lookupUser(name string) uint32 {
if !m.local {
return m.genGuid(name)
}
u, _ := user.Lookup(name)
if u != nil {
id_, _ := strconv.ParseUint(u.Uid, 10, 32)
id = uint32(id_)
} else {
if name == "root" { // root in hdfs sdk is a normal user
id = m.genGuid(name)
} else {
u, _ := user.Lookup(name)
if u != nil {
id_, _ := strconv.ParseUint(u.Uid, 10, 32)
id = uint32(id_)
} else {
id = m.genGuid(name)
}
}
m.usernames[name] = id
m.userIDs[id] = name
Expand All @@ -90,12 +94,16 @@ func (m *mapping) lookupGroup(name string) uint32 {
if !m.local {
return m.genGuid(name)
}
g, _ := user.LookupGroup(name)
if g == nil {
if name == "root" {
id = m.genGuid(name)
} else {
id_, _ := strconv.ParseUint(g.Gid, 10, 32)
id = uint32(id_)
g, _ := user.LookupGroup(name)
if g == nil {
id = m.genGuid(name)
} else {
id_, _ := strconv.ParseUint(g.Gid, 10, 32)
id = uint32(id_)
}
}
m.groups[name] = id
m.groupIDs[id] = name
Expand Down