Compare commits

..

34 Commits

Author SHA1 Message Date
Massaki Archambault d309d20dd8 fix typo 2024-08-14 22:02:54 -04:00
Massaki Archambault 014d682d69 create changelog 2024-08-14 21:58:12 -04:00
Massaki Archambault 331bdcd083 remove media folder 2024-08-14 12:05:35 -04:00
Massaki Archambault e4b0496116 change user configuration of gitlab to user name
This should be easier to configure than having to hunt down the user id
2024-08-14 12:05:03 -04:00
Massaki Archambault d1c82e1329 rewrite readme 2024-08-13 23:45:09 -04:00
Massaki Archambault 46217c1a9d rename project to gitforgefs 2024-08-13 23:32:29 -04:00
Massaki Archambault d3e211e1cb default to config.yaml 2024-08-13 22:22:50 -04:00
Massaki Archambault 53450730c9 tweak host match regex 2024-08-13 22:18:20 -04:00
Massaki Archambault 3180af5bd4 update dependencies 2024-08-09 17:12:48 -04:00
Massaki Archambault 70f269e25e fix inode collision 2024-08-09 16:43:30 -04:00
Massaki Archambault caa030b03b remove staticInoChan, as it's redundant 2024-08-09 16:17:47 -04:00
Massaki Archambault ed1dc2517a add gitea forge support 2024-08-09 16:13:57 -04:00
Massaki Archambault bc8ae0a3c8 refactor to respect naming convention 2024-08-08 23:44:12 -04:00
Massaki Archambault 1a01c9ecea refactor platform -> forge 2024-08-04 21:45:51 -04:00
Massaki Archambault 0e5fed0bbd refactor gitlab current user 2024-08-04 21:39:15 -04:00
Massaki Archambault 382a0f6d8d add support for github current user 2024-08-04 21:16:03 -04:00
Massaki Archambault 937e5c341d add support for github users 2024-08-04 21:00:34 -04:00
Massaki Archambault a0aaa4491b add support for github orgs 2024-08-04 18:59:57 -04:00
Massaki Archambault 4e2b631a0c refactor config loader and add github config 2024-08-04 15:57:39 -04:00
Massaki Archambault 36b3963ac3 fix missing mutex lock 2024-07-19 00:12:39 -04:00
Massaki Archambault 2d0a62dc45 hide archived project by default
prefix archived project name with a "." by default
so they appear hidden on the filesystem.
At the same time, added the configuration gitlab.archived_project_handling
to set how archived projects are handled
2024-07-18 00:36:57 -04:00
Massaki Archambault 6d0d3fdfc0 guard gitlab cache Map with RWMutex to prevent concurrent read/write
fixes #11
2024-07-17 23:35:18 -04:00
Massaki Archambault 0a50158239 add config tests 2024-06-08 01:02:35 -04:00
Massaki Archambault a4ed751abd move config loader to its own package 2024-06-07 23:42:13 -04:00
Massaki Archambault e26f0ae865 fix cache invalidation 2024-06-07 12:17:40 -04:00
Massaki Archambault 4667c12e47 fix regression that caused more queries then necessary to be made to gitlab api 2024-06-06 21:46:29 -04:00
Massaki Archambault 8e350a7dac switch to slog for logging 2024-06-06 01:51:34 -04:00
Massaki Archambault b0b7f7b36d improve hostname match regex 2024-05-08 19:34:29 -04:00
Massaki Archambault 0ece3f05a3 Check if git support --initial-branch before attempting to use it on init
`git init --initial-branch` was only added in git
version 0.28.0. We parse the git version and
check if the argument is supported before using it
Fixes #8
2024-05-07 23:58:14 -04:00
Massaki Archambault 324be5f1f6 entirely skip passing the --depth argument to git if the depth is configured to 0
fixes #7
2024-05-07 23:33:02 -04:00
Massaki Archambault de68c4952e reorganize gitlab package 2024-05-05 23:59:01 -04:00
Massaki Archambault cf77b16b23 refactor to decouple fstree package from git package 2024-05-05 19:53:04 -04:00
Massaki Archambault 652b7b43a6 rename fs package to fstree to avoid collision with go-fuse 2024-05-05 16:23:07 -04:00
Massaki Archambault b7683d4f24 refactor to decouple fs package from gitlab package 2024-05-05 16:09:03 -04:00
12 changed files with 26 additions and 124 deletions

3
.gitignore vendored
View File

@ -53,5 +53,4 @@ tags
# End of https://www.toptal.com/developers/gitignore/api/go,vim,code # End of https://www.toptal.com/developers/gitignore/api/go,vim,code
config.yaml config.yaml
__debug_*

20
.vscode/launch.json vendored
View File

@ -1,20 +0,0 @@
{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0",
"configurations": [
{
"name": "Run",
"type": "go",
"request": "launch",
"mode": "debug",
"program": "${workspaceRoot}",
"args": [
"-debug",
"-config",
"${workspaceRoot}/config.yaml"
]
}
]
}

View File

@ -52,7 +52,7 @@ Merge requests to add support to other forges are welcome.
Install [go](https://golang.org/) and run Install [go](https://golang.org/) and run
``` sh ``` sh
go install github.com/badjware/gitforgefs@latest go install github.com/badjware/gitforgefs
``` ```
The executable will be in `$GOPATH/bin/gitforgefs` or `~/go/bin/gitforgefs` by default. For convenience, add `~/go/bin` in your `$PATH` if not done already. The executable will be in `$GOPATH/bin/gitforgefs` or `~/go/bin/gitforgefs` by default. For convenience, add `~/go/bin` in your `$PATH` if not done already.
@ -92,24 +92,4 @@ While the filesystem lives in memory, the git repositories that are cloned are s
Simply use `make` to create the executable. The executable will be in `bin/`. Simply use `make` to create the executable. The executable will be in `bin/`.
See `make help` for all available targets. See `make help` for all available targets.
## Troubleshooting
### My application claims that a file or a folder doesn't exists.
Some applications doesn't resolve symlinks properly. Try setting the `fs.use_symlinks` configuration to `false`.
### `docker` fails to run with the message _error while creating mount source path_
This happens because `docker` is running as a different user than the one who created the mount. Follow these steps to allow docker access to the mount:
1. Open the file `/etc/fuse.conf` as root.
2. Add `user_allow_other` to the file, then close and save your modifications.
3. Open your `gitforgefs` configuration.
4. Add the `allow_other` to your mountoptions. The mount option are configured in `fs.mountoptions`.
``` yaml
fs:
mountoptions: allow_other,nodev,nosuid
```
5. Restart your mount.

View File

@ -3,15 +3,8 @@ fs:
#mountpoint: /mnt #mountpoint: /mnt
# Mount options to pass to `fusermount` as its `-o` argument. Can be overwritten via the command line. # Mount options to pass to `fusermount` as its `-o` argument. Can be overwritten via the command line.
# Some applications need the `allow_other` option to function properly (eg: docker). If you need to use `allow_other`,
# you must also add `user_allow_other` in /etc/fuse.conf.
# See mount.fuse(8) for the full list of options. # See mount.fuse(8) for the full list of options.
#mountoptions: nodev,nosuid #mountoptions: nodev,nosuid
#mountoptions: allow_other,nodev,nosuid
# Use a symlink to point to the real location of the repository instead of doing a loopback
# Using symlinks is more performant and allow cloning to be asynchronous, but may cause compatibility issues with some applications
# use_symlinks: false
# The git forge to use as the backend. # The git forge to use as the backend.
# Must be one of "gitlab", "github", or "gitea" # Must be one of "gitlab", "github", or "gitea"

View File

@ -32,7 +32,6 @@ type (
FSConfig struct { FSConfig struct {
Mountpoint string `yaml:"mountpoint,omitempty"` Mountpoint string `yaml:"mountpoint,omitempty"`
MountOptions string `yaml:"mountoptions,omitempty"` MountOptions string `yaml:"mountoptions,omitempty"`
UseSymlinks bool `yaml:"use_symlinks,omitempty"`
Forge string `yaml:"forge,omitempty"` Forge string `yaml:"forge,omitempty"`
} }
GitlabClientConfig struct { GitlabClientConfig struct {
@ -90,7 +89,6 @@ func LoadConfig(configPath string) (*Config, error) {
FS: FSConfig{ FS: FSConfig{
Mountpoint: "", Mountpoint: "",
MountOptions: "nodev,nosuid", MountOptions: "nodev,nosuid",
UseSymlinks: false,
Forge: "", Forge: "",
}, },
Gitlab: GitlabClientConfig{ Gitlab: GitlabClientConfig{

View File

@ -31,7 +31,7 @@ var _ = (fs.NodeReaddirer)((*groupNode)(nil))
// Ensure we are implementing the NodeLookuper interface // Ensure we are implementing the NodeLookuper interface
var _ = (fs.NodeLookuper)((*groupNode)(nil)) var _ = (fs.NodeLookuper)((*groupNode)(nil))
func newGroupNodeFromSource(ctx context.Context, source GroupSource, param *FSParam) (fs.InodeEmbedder, error) { func newGroupNodeFromSource(source GroupSource, param *FSParam) (*groupNode, error) {
node := &groupNode{ node := &groupNode{
param: param, param: param,
source: source, source: source,
@ -85,7 +85,7 @@ func (n *groupNode) Lookup(ctx context.Context, name string, out *fuse.EntryOut)
Ino: group.GetGroupID() + groupBaseInode, Ino: group.GetGroupID() + groupBaseInode,
Mode: fuse.S_IFDIR, Mode: fuse.S_IFDIR,
} }
groupNode, _ := newGroupNodeFromSource(ctx, group, n.param) groupNode, _ := newGroupNodeFromSource(group, n.param)
return n.NewInode(ctx, groupNode, attrs), 0 return n.NewInode(ctx, groupNode, attrs), 0
} }
@ -93,17 +93,10 @@ func (n *groupNode) Lookup(ctx context.Context, name string, out *fuse.EntryOut)
repository, found := repositories[name] repository, found := repositories[name]
if found { if found {
attrs := fs.StableAttr{ attrs := fs.StableAttr{
Ino: repository.GetRepositoryID() + repositoryBaseInode, Ino: repository.GetRepositoryID() + repositoryBaseInode,
} Mode: fuse.S_IFLNK,
if n.param.UseSymlinks {
attrs.Mode = fuse.S_IFLNK
} else {
attrs.Mode = fuse.S_IFDIR
}
repositoryNode, err := newRepositoryNodeFromSource(ctx, repository, n.param)
if err != nil {
panic(err)
} }
repositoryNode, _ := newRepositoryNodeFromSource(repository, n.param)
return n.NewInode(ctx, repositoryNode, attrs), 0 return n.NewInode(ctx, repositoryNode, attrs), 0
} }

View File

@ -2,11 +2,7 @@ package fstree
import ( import (
"context" "context"
"errors"
"fmt"
"os"
"syscall" "syscall"
"time"
"github.com/hanwen/go-fuse/v2/fs" "github.com/hanwen/go-fuse/v2/fs"
) )
@ -15,7 +11,7 @@ const (
repositoryBaseInode = 2_000_000_000 repositoryBaseInode = 2_000_000_000
) )
type repositorySymlinkNode struct { type repositoryNode struct {
fs.Inode fs.Inode
param *FSParam param *FSParam
@ -30,53 +26,24 @@ type RepositorySource interface {
} }
// Ensure we are implementing the NodeReaddirer interface // Ensure we are implementing the NodeReaddirer interface
var _ = (fs.NodeReadlinker)((*repositorySymlinkNode)(nil)) var _ = (fs.NodeReadlinker)((*repositoryNode)(nil))
func newRepositoryNodeFromSource(ctx context.Context, source RepositorySource, param *FSParam) (fs.InodeEmbedder, error) { func newRepositoryNodeFromSource(source RepositorySource, param *FSParam) (*repositoryNode, error) {
if param.UseSymlinks { node := &repositoryNode{
return &repositorySymlinkNode{ param: param,
param: param, source: source,
source: source,
}, nil
} else {
localRepositoryPath, err := param.GitClient.FetchLocalRepositoryPath(ctx, source)
if err != nil {
return nil, fmt.Errorf("failed to fetch local repository path: %w", err)
}
// The path must exist to successfully create a loopback. We poll the filesystem until its created.
// This of course add latency, maybe we should think of a way of mitigating it in the future.
// We do not care in the case of a symlink. A symlink pointing on nothing is still a valid symlink.
for ctx.Err() == nil {
_, err := os.Stat(localRepositoryPath)
if err == nil {
return fs.NewLoopbackRoot(localRepositoryPath)
} else if errors.Is(err, os.ErrNotExist) {
// wait for the file to be created
// TODO: think of a more efficient way of archiving this
time.Sleep(100 * time.Millisecond)
} else {
// error, filesystem
return nil, fmt.Errorf("error while waiting for the local repository to be created: %w", err)
}
}
// error, context cancelled
return nil, fmt.Errorf("context cancelled while waiting for the local repository to be created: %w", ctx.Err())
} }
// Passthrough the error if there is one, nothing to add here
// Errors on clone/pull are non-fatal
return node, nil
} }
func (n *repositorySymlinkNode) Readlink(ctx context.Context) ([]byte, syscall.Errno) { func (n *repositoryNode) Readlink(ctx context.Context) ([]byte, syscall.Errno) {
// Create the local copy of the repo // Create the local copy of the repo
// TODO: cleanup // TODO: cleanup
localRepositoryPath, err := n.param.GitClient.FetchLocalRepositoryPath(ctx, n.source) localRepositoryPath, err := n.param.GitClient.FetchLocalRepositoryPath(n.source)
if err != nil { if err != nil {
n.param.logger.Error(err.Error()) n.param.logger.Error(err.Error())
} }
return []byte(localRepositoryPath), 0 return []byte(localRepositoryPath), 0
} }
type repositoryLoopbackNode struct {
fs.LoopbackNode
param *FSParam
source RepositorySource
}

View File

@ -19,7 +19,7 @@ type staticNode interface {
} }
type GitClient interface { type GitClient interface {
FetchLocalRepositoryPath(ctx context.Context, source RepositorySource) (string, error) FetchLocalRepositoryPath(source RepositorySource) (string, error)
} }
type GitForge interface { type GitForge interface {
@ -28,8 +28,6 @@ type GitForge interface {
} }
type FSParam struct { type FSParam struct {
UseSymlinks bool
GitClient GitClient GitClient GitClient
GitForge GitForge GitForge GitForge
@ -77,7 +75,7 @@ func (n *rootNode) OnAdd(ctx context.Context) {
} }
for groupName, group := range rootGroups { for groupName, group := range rootGroups {
groupNode, _ := newGroupNodeFromSource(ctx, group, n.param) groupNode, _ := newGroupNodeFromSource(group, n.param)
persistentInode := n.NewPersistentInode( persistentInode := n.NewPersistentInode(
ctx, ctx,
groupNode, groupNode,

View File

@ -84,7 +84,7 @@ func NewClient(logger *slog.Logger, p config.GitClientConfig) (*gitClient, error
return c, nil return c, nil
} }
func (c *gitClient) FetchLocalRepositoryPath(ctx context.Context, source fstree.RepositorySource) (localRepoLoc string, err error) { func (c *gitClient) FetchLocalRepositoryPath(source fstree.RepositorySource) (localRepoLoc string, err error) {
rid := source.GetRepositoryID() rid := source.GetRepositoryID()
cloneUrl := source.GetCloneURL() cloneUrl := source.GetCloneURL()
defaultBranch := source.GetDefaultBranch() defaultBranch := source.GetDefaultBranch()
@ -98,12 +98,12 @@ func (c *gitClient) FetchLocalRepositoryPath(ctx context.Context, source fstree.
localRepoLoc = filepath.Join(c.CloneLocation, hostname, strconv.Itoa(int(rid))) localRepoLoc = filepath.Join(c.CloneLocation, hostname, strconv.Itoa(int(rid)))
if _, err := os.Stat(localRepoLoc); os.IsNotExist(err) { if _, err := os.Stat(localRepoLoc); os.IsNotExist(err) {
// Dispatch clone msg // Dispatch clone msg
msg := c.cloneTask.WithArgs(ctx, cloneUrl, defaultBranch, localRepoLoc) msg := c.cloneTask.WithArgs(context.Background(), cloneUrl, defaultBranch, localRepoLoc)
msg.OnceInPeriod(time.Second, rid) msg.OnceInPeriod(time.Second, rid)
c.queue.Add(msg) c.queue.Add(msg)
} else if c.AutoPull { } else if c.AutoPull {
// Dispatch pull msg // Dispatch pull msg
msg := c.pullTask.WithArgs(ctx, localRepoLoc, defaultBranch) msg := c.pullTask.WithArgs(context.Background(), localRepoLoc, defaultBranch)
msg.OnceInPeriod(time.Second, rid) msg.OnceInPeriod(time.Second, rid)
c.queue.Add(msg) c.queue.Add(msg)
} }

2
go.mod
View File

@ -5,7 +5,7 @@ go 1.21
require ( require (
code.gitea.io/sdk/gitea v0.19.0 code.gitea.io/sdk/gitea v0.19.0
github.com/google/go-github/v63 v63.0.0 github.com/google/go-github/v63 v63.0.0
github.com/hanwen/go-fuse/v2 v2.7.2 github.com/hanwen/go-fuse/v2 v2.5.1
github.com/vmihailenco/taskq/v3 v3.2.9 github.com/vmihailenco/taskq/v3 v3.2.9
github.com/xanzy/go-gitlab v0.107.0 github.com/xanzy/go-gitlab v0.107.0
gopkg.in/yaml.v2 v2.4.0 gopkg.in/yaml.v2 v2.4.0

2
go.sum
View File

@ -59,8 +59,6 @@ github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I=
github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/hanwen/go-fuse/v2 v2.5.1 h1:OQBE8zVemSocRxA4OaFJbjJ5hlpCmIWbGr7r0M4uoQQ= github.com/hanwen/go-fuse/v2 v2.5.1 h1:OQBE8zVemSocRxA4OaFJbjJ5hlpCmIWbGr7r0M4uoQQ=
github.com/hanwen/go-fuse/v2 v2.5.1/go.mod h1:xKwi1cF7nXAOBCXujD5ie0ZKsxc8GGSA1rlMJc+8IJs= github.com/hanwen/go-fuse/v2 v2.5.1/go.mod h1:xKwi1cF7nXAOBCXujD5ie0ZKsxc8GGSA1rlMJc+8IJs=
github.com/hanwen/go-fuse/v2 v2.7.2 h1:SbJP1sUP+n1UF8NXBA14BuojmTez+mDgOk0bC057HQw=
github.com/hanwen/go-fuse/v2 v2.7.2/go.mod h1:ugNaD/iv5JYyS1Rcvi57Wz7/vrLQJo10mmketmoef48=
github.com/hashicorp/go-cleanhttp v0.5.2 h1:035FKYIWjmULyFRBKPs8TBQoi0x6d9G4xc9neXJWAZQ= github.com/hashicorp/go-cleanhttp v0.5.2 h1:035FKYIWjmULyFRBKPs8TBQoi0x6d9G4xc9neXJWAZQ=
github.com/hashicorp/go-cleanhttp v0.5.2/go.mod h1:kO/YDlP8L1346E6Sodw+PrpBSV4/SoxCXGY6BqNFT48= github.com/hashicorp/go-cleanhttp v0.5.2/go.mod h1:kO/YDlP8L1346E6Sodw+PrpBSV4/SoxCXGY6BqNFT48=
github.com/hashicorp/go-hclog v1.6.3 h1:Qr2kF+eVWjTiYmU7Y31tYlP1h0q/X3Nl3tPGdaB11/k= github.com/hashicorp/go-hclog v1.6.3 h1:Qr2kF+eVWjTiYmU7Y31tYlP1h0q/X3Nl3tPGdaB11/k=

View File

@ -98,11 +98,7 @@ func main() {
logger, logger,
mountpoint, mountpoint,
parsedMountoptions, parsedMountoptions,
&fstree.FSParam{ &fstree.FSParam{GitClient: gitClient, GitForge: gitForgeClient},
UseSymlinks: loadedConfig.FS.UseSymlinks,
GitClient: gitClient,
GitForge: gitForgeClient,
},
*debug, *debug,
) )
if err != nil { if err != nil {