@[email protected]

Mid 2022, a friend of mine helped me set up a selfhosted Vaultwarden instance. Since then, my “infrastructure” has not stopped growing, and I’ve been learning each and every day about how services work, how they communicate and how I can move data from one place to another. It’s truly incredible, and my favorite hobby by a long shot.

Here’s a map of what I’ve built so far. Right now, I’m mostly done, but surely time will bring more ideas. I’ve also left out a bunch of “technically revelant” connections like DNS resolution through the AdGuard instance, firewalls and CrowdSec on the main VPS.

Looking at the setups that others have posted, I don’t think this is super incredible - but if you have input or questions about the setup, I’ll do my best to explain it all. None of my peers really understand what it takes to construct something like this, so I am in need of people who understand my excitement and proudness :)

Edit: the image was compressed a bit too much, so here’s the full res image for the curious: https://files.catbox.moe/iyq5vx.png And a dark version for the night owls: https://files.catbox.moe/hy713z.png

  • empireOfLove2@lemmy.dbzer0.com
    link
    fedilink
    English
    arrow-up
    22
    ·
    edit-2
    10 months ago

    me after 15 years of intermittent learning self hosting:

    i have the one random office PC that runs minecraft

    …yeah that’s it

    • RiderExMachina@lemmy.ml
      link
      fedilink
      English
      arrow-up
      5
      ·
      edit-2
      10 months ago

      With the enshittification of streaming platforms, a Kodi or Jellyfin server would be a great starting point. In my case, I have both, and the Kodi machine gets the files from the Jellyfin machine through NFS.

      Or Home Assistant to help keep IOT devices that tend to be more IoS. Or a Nextcloud server to try to degoogle at least a little bit.

      Maybe a personal Friendica instance for your LAN so your family can get their Facebook addiction without giving their data to Meta?

      • Specal@lemmy.world
        link
        fedilink
        English
        arrow-up
        3
        ·
        10 months ago

        Additionally, using jottacloud with 2 VPS’s (one of them being built on epyc like from OVH cloud) can get you a really good download server and streaming server for about £30 a month, which is the same as having netflix and Disney plus, except now you can have anything you want.

        I have a contabo 4core 8gb ram VPS that handles downloading content.

        A OVH 4core 8gb VPS that handles emby (I keep trying to go back to jellyfin but it’s just slightly slower than emby at transcoding and I need to squeeze as much performance out of my VPS as possible so… Maybe one day jelly)

        And I have a really good streaming experience with subtitles that don’t put big black boxes on the screen making 1/8th of the screen non viewable.

  • ginko@lemmy.world
    link
    fedilink
    English
    arrow-up
    9
    ·
    10 months ago

    Architecture looks dope

    Hope you’ve safeguarded your setup by writing a provisoning script in case anything goes south.

    I had to reinstall my server from scratch twice and can’t fathom having to reconfigure everything manually anymore

    • 7Sea_Sailor@lemmy.dbzer0.comOP
      link
      fedilink
      English
      arrow-up
      6
      ·
      10 months ago

      Nope, don’t have that yet. But since all my compose and config files are neatly organized on the file system, by domain and then by service, I tar up that entire docker dir once a week and pull it to the homelab, just in case.

      How have you setup your provisioning script? Any special services or just some clever batch scripting?

      • ginko@lemmy.world
        link
        fedilink
        English
        arrow-up
        3
        ·
        10 months ago

        Old school ansible at first, then I ditched it for Cloudbox (an OSS provisioning script for media server)

        Works wonders for me but I believe it’s currently stuck on a deprecated Ubuntu release

  • krash@lemmy.ml
    link
    fedilink
    English
    arrow-up
    8
    ·
    10 months ago

    How do you like crowdsec? I’ve used it on a tiny VPS (2 vcpu / 1 GB RAM) and it hogs my poor machine. I also found it to have a bit of learning curve, compared to fail2ban (which is much simpler, but dosen’t play well with Caddy by default).

    Would be happy to see your Caddy / Crowdsec configuration.

    • 7Sea_Sailor@lemmy.dbzer0.comOP
      link
      fedilink
      English
      arrow-up
      4
      ·
      edit-2
      10 months ago

      The crowdsec agent running on my homelab (8 Cores, 16GB RAM) is currently sitting idle at 96.86MiB RAM and between 0.4 and 1.5% CPU usage. I have a separate crowdsec agent running on the Main VPS, which is a 2 vCPU 4GB RAM machine. There, it’s using 1.3% CPU and around 2.5% RAM. All in all, very manageable.

      There is definitely a learning curve to it. When I first dove into the docs, I was overwhelmed by all the new terminology, and wrapping my head around it was not super straightforward. Now that I’ve had some time with it though, it’s become more and more clear. I’ve even written my own simple parsers for apps that aren’t on the hub!

      What I find especially helpful are features like explain, which allow me to pass in logs and simulate which step of the process picks that up and how the logs are processed, which is great when trying to diagnose why something is or isn’t happening.

      The crowdsec agent running on my homelab is running from the docker container, and uses pretty much exactly the stock configuration. This is how the docker container is launched:

        crowdsec:
          image: crowdsecurity/crowdsec
          container_name: crowdsec
          restart: always
          networks:
            socket-proxy:
          ports:
            - "8080:8080"
          environment:
            DOCKER_HOST: tcp://socketproxy:2375
            COLLECTIONS: "schiz0phr3ne/radarr schiz0phr3ne/sonarr"
            BOUNCER_KEY_caddy: as8d0h109das9d0
            USE_WAL: true
          volumes:
            - /mnt/user/appdata/crowdsec/db:/var/lib/crowdsec/data
            - /mnt/user/appdata/crowdsec/acquis:/etc/crowdsec/acquis.d
            - /mnt/user/appdata/crowdsec/config:/etc/crowdsec
      

      Then there’s the Caddyfile on the LabProxy, which is where I handle banned IPs so that their traffic doesn’t even hit my homelab. This is the file:

      {
      	crowdsec {
      		api_url http://homelab:8080
      		api_key as8d0h109das9d0
      		ticker_interval 10s
      	}
      }
      
      *.mydomain.com {
      	tls {
      		dns cloudflare skPTIe-qA_9H2_QnpFYaashud0as8d012qdißRwCq
      	}
      	encode gzip
      	route {
      		crowdsec
      		reverse_proxy homelab:8443
      	}
      }
      

      Keep in mind that the two machines are connected via tailscale, which is why I can pass in the crowdsec agent with its local hostname. If the two machines were physically separated, you’d need to expose the REST API of the agent over the web.

      I hope this helps clear up some of your confusion! Let me know if you need any further help with understanding it. It only gets easier the more you interact with it!

      don’t worry, all credentials in the two files are randomized, never the actual tokens

      • krash@lemmy.ml
        link
        fedilink
        English
        arrow-up
        1
        ·
        10 months ago

        Thanks for the offer! I might take you up on that :-) If you have a Matrix handle and hang out in certain rooms, please DM me and I’ll harass reach out to you there.

        • 7Sea_Sailor@lemmy.dbzer0.comOP
          link
          fedilink
          English
          arrow-up
          1
          ·
          edit-2
          10 months ago

          Hm, I have yet to mess around with matrix. As anything fediverse, the increased complexity is a little overwhelming for me, and since I am not pulled to matrix by any communities im a part of, I wasn’t yet forced to make any decisions. I mainly hang out on discord, if that’s something you use.

    • Sekki@lemmy.ml
      link
      fedilink
      English
      arrow-up
      2
      ·
      10 months ago

      Somehow I only had issues with CrowdSec. I used it with Traefik but it would ban me and my family every time they used my selhosted matrix instance. I could not figure out why and it even did that when I tried it on OPNSense without the Traefik bouncer…

  • thantik@lemmy.world
    link
    fedilink
    English
    arrow-up
    6
    ·
    edit-2
    10 months ago

    Very nice setup imho. Quite a bit more complicated than mine - mine is basically just the left box without being behind a VPS or anything. I don’t expose anything through Caddy except Jellyfin. I’m also running fail2ban in front of my services, so that if it gets hit with too many 404s because someone is poking around, they get IP banned for 30d

    • 7Sea_Sailor@lemmy.dbzer0.comOP
      link
      fedilink
      English
      arrow-up
      2
      ·
      10 months ago

      I’m still on the fence if I want to expose Jellyfin publicly or not. On the one hand, I never really want to stream movies or shows from abroad, so there’s no real need. And in desperate times I can always connect to Tailscale and watch that way. But on the other, it’s really cool to simply have a web accessible Netflix. Idk.

      • thantik@lemmy.world
        link
        fedilink
        English
        arrow-up
        2
        ·
        edit-2
        10 months ago

        Honestly, I installed Ombi, so friends can request movies - and gave them all jellyfin logins as well. I’m not running any kind of pay-for service, I’m just giving them access to my library. Additionally, my kids will sometimes spend the night at friends, etc - and their friend won’t have an anime, or a crunchyroll subscription, so they’ll pull it up on jellyfin. It’s easy to remember for them because it’s just jellyfin.mydomain.com

        They don’t know anything about how the backend gets the movies/tv shows, just that they go to ombi, and it shows up on jellyfin if they want something ;)

        • 7Sea_Sailor@lemmy.dbzer0.comOP
          link
          fedilink
          English
          arrow-up
          1
          ·
          10 months ago

          Gosh, that’s cute. Probably how I’ll end up too. Right now I’m not ready to let friends use my services. I already have friends and family on adguard and vaultwarden, that’s enough responsibility for now.

  • N0x0n@lemmy.ml
    link
    fedilink
    English
    arrow-up
    5
    ·
    edit-2
    10 months ago

    Possible for a dark mode version XD? excalidraw can do that.

  • callcc@lemmy.world
    link
    fedilink
    English
    arrow-up
    5
    arrow-down
    1
    ·
    10 months ago

    Remeber, the more boxes you have, the more advanced you are as an admin! Once you do his job for money, the challenge is the exact opposite. The less parts you have, the better. The more vanilla they are, the better.

    • 7Sea_Sailor@lemmy.dbzer0.comOP
      link
      fedilink
      English
      arrow-up
      3
      ·
      10 months ago

      Absolutely! To be honest, I don’t even want to have countless machines under my umbrella, and constantly have consodilation in mind - but right now, each machine fulfills a separate purpose and feels justified in itself (homelab for large data, main VPS for anything thats operation critical and cant afford power/network outages and so on). So unless I find another purpose that none of the current machines can serve, I’ll probably scale vertically instead of horizontally (is that even how you use that expression?)

  • Cyanogenmon@lemmy.world
    link
    fedilink
    English
    arrow-up
    3
    ·
    10 months ago

    This is oddly similar to some informal workups I’ve done for our work network.

    Nice work 👍.

  • jkrtn@lemmy.ml
    link
    fedilink
    English
    arrow-up
    3
    ·
    10 months ago

    I’ve seen Caddy mentioned a few times recently, what do you like about it over other tools?

    • xantoxis@lemmy.world
      link
      fedilink
      English
      arrow-up
      4
      ·
      10 months ago

      I can answer this one, but mainly only in reference to the other popular solutions:

      • nginx. Solid, reliable, uncomplicated, but. Reverse proxy semantics have a weird dependency on manually setting up a dns resolver (why??) and you have to restart the instance if your upstream gets replaced.
      • traefik. I am literally a cloud software engineer, I’ve been doing Linux networking since 1994 and I’ve made 3 separate attempts to configure traefik to work according to its promises. It has never worked correctly. Traefik’s main selling point to me is its automatic docker proxying via labels, but this doesn’t even help you if you also have multiple VMs. Basically a non-starter due to poor docs and complexity.
      • caddy. Solid, reliable, uncomplicated. It will do acme cert provisioning out of the box for you if you want (I don’t use that feature because I have a wildcard cert, but it seems nice). Also doesn’t suffer from the problems I’ve listed above.
      • jkrtn@lemmy.ml
        link
        fedilink
        English
        arrow-up
        2
        ·
        10 months ago

        I feel so relieved reading that about traefik. I briefly set that up as a k8s ingress controller for educational purposes. It’s unnecessarily confusing, brittle, and the documentation didn’t help. If it’s a pain for people in the industry that makes me feel better. My next attempt at trying out k8s I’ll give Kong a shot.

        I really like solid, reliable, and uncomplicated. The fun part is running the containers and VMs, not spending hours on a config to make them accessible.

        • notfromhere@lemmy.ml
          link
          fedilink
          English
          arrow-up
          2
          ·
          10 months ago

          I have traefik running on my kubernetes cluster as an ingress controller and it works well enough for me after finagling it a bit. Fully automated through ansible and templated manifests.

          • xantoxis@lemmy.world
            link
            fedilink
            English
            arrow-up
            1
            ·
            10 months ago

            Heh. I am, as I said, a cloud sw eng, which is why I would never touch any solution that mentioned ansible, outside of the work I am required to do professionally. Too many scars. It’s like owning a pet raccoon, you can maybe get it to do clever things if you give it enough treats, but it will eventually kill your dog.

            • notfromhere@lemmy.ml
              link
              fedilink
              English
              arrow-up
              1
              ·
              10 months ago

              Care to share some war stories? I have it set up where I can completely destroy and rebuild my bare metal k3s cluster. If I start with configured hosts, it takes about 10 minutes to install k3s and get all my services back up.

              • xantoxis@lemmy.world
                link
                fedilink
                English
                arrow-up
                2
                ·
                edit-2
                10 months ago

                Sure, I mean, we could talk about

                • dynamic inventory on AWS means the ansible interpreter will end up with three completely separate sets of hostnames for your architecture, not even including the actual DNS name. if you also need dynamic inventory on GCP, that’s three completely different sets of hostnames, i.e. they are derived from different properties of the instances than the AWS names.
                • btw, those names are exposed to the ansible runtime graph via different names i.e. ansible_inventory vs some other thing, based on who even fuckin knows, but sometimes the way you access the name will completely change from one role to the next.
                • ansible-vault’s semantics for when things can be decrypted and when they can’t leads to completely nonsense solutions like a yaml file with normal contents where individual strings are encrypted and base64-encoded inline within the yaml, and others are not. This syntax doesn’t work everywhere. The opaque contents of the encrypted strings can sometimes be treated as traversible yaml and sometimes cannot be.
                • ansible uses the system python interpreter, so if you need it to do anything that uses a different Python interpreter (because that’s where your apps are installed), you have to force it to switch back and forth between interpreters. Also, the python setting in ansible is global to the interpreter meaning you could end up leaking the wrong interpreter into the role that follows the one you were trying to tweak, causing almost invisible problems.
                • ansible output and error reporting is just a goddamn mess. I mean look at this shit. Care to guess which one of those gives you a stream which is parseable as json? Just kidding, none of them do, because ansible always prefixes each line.
                • tags are a joke. do you want to run just part of a playbook? --start-at. But oops, because not every single task in your playbook is idempotent, that will not work, ever, because something was supposed to happen earlier on that didn’t. So if you start at a particular tag, or run only the tasks that have a particular tag, your playbook will fail. Or worse, it will work, but it will work completely differently than in production because of some value that leaked into the role you were skipping into.
                • Last but not least, using ansible in production means your engineers will keep building onto it, making it more and more complex, “just one more task bro”. The bigger it gets, the more fragile it gets, and the more all of these problems rears its head.
                • notfromhere@lemmy.ml
                  link
                  fedilink
                  English
                  arrow-up
                  2
                  ·
                  edit-2
                  10 months ago
                  • Dynamic inventory. I haven’t used it on a cloud api before but I have used it against kube API and it was manageable. Are you saying through kubectl the node names are different depending on which cloud and it’s not uniform? Edit: Oh you’re talking about the VMs doh

                  • I’ve tried ansible vault and didn’t make it very far… I agree that thing is a mess.

                  • Thank god I haven’t ran into interpreter issues, that sounds like hell.

                  • Ansible output is terrible, no argument there.

                  • I don’t remember the name for it, but I use parameterized template tasks. That might help with this? Edit: include_tasks.

                  • I think this is due to not a very good IDE for including the whole scope of the playbook, which could be a condemnation of ansible or just needing better abstraction layers for this complex thing we are trying to manage the unmanageable with.

      • OSH@lemmy.ml
        link
        fedilink
        English
        arrow-up
        2
        ·
        10 months ago

        Fully agree to this summary. traefik also gave me a hard time initially, but once you have the quirks worked out, it works as promised.

        Caddy is absolutely on my list as an alternative, but the lack of docker label support is currently the main roadblocker for me.

        • 7Sea_Sailor@lemmy.dbzer0.comOP
          link
          fedilink
          English
          arrow-up
          2
          ·
          edit-2
          10 months ago

          May I present to you: Caddy but for docker and with labels so kind of like traefik but the labels are shorter 👏 https://github.com/lucaslorentz/caddy-docker-proxy

          Jokes aside, I did actually use this for a while and it worked great. The concept of having my reverse proxy config at the same place as my docker container config is intriguing. But managing labels is horrible on unraid, so I moved to classic caddy instead.

          • OSH@lemmy.ml
            link
            fedilink
            English
            arrow-up
            1
            ·
            10 months ago

            Nice catch and thanks for sharing. Will definitely check it out.

    • 7Sea_Sailor@lemmy.dbzer0.comOP
      link
      fedilink
      English
      arrow-up
      4
      ·
      10 months ago

      In addition to the other commenter and their great points, here’s some more things I like:

      • ressource efficient: im running all my stuff on low end servers, and cant afford my reverse proxy to waste gigabytes of RAM (kooking at you, NPM)
      • very easy syntax: the Caddyfile uses a very simple, easy to remember syntax. And the documentation is very precise and quickly tells me what to do to achieve something. I tried traefik and couldn’t handle the long, complicated tag names required to set anything up.
      • plugin ecosystem: caddy is written in go, and very easy to extend. There’s tons of plugins for different functionalities, that are (mostly) well documented and easy to use. Building a custom caddy executable takes one command.
      • jkrtn@lemmy.ml
        link
        fedilink
        English
        arrow-up
        2
        ·
        10 months ago

        I think the two of you have convinced me to check it out! It is sounding pretty great, so thank you in advance.

    • krash@lemmy.ml
      link
      fedilink
      English
      arrow-up
      2
      ·
      10 months ago

      I see everyone else have already chimed in on whats so great about Caddy (because it is!), one thing that has been a thorn in my side though is the lack of integration of fail2ban since Caddy has moved on from the old common log format and moved on to more modern log formats. So if you want to use a IPS/IDS, you’ll have to either find a creative hack to make it work with fail2ban or rely on more modern (and resource heavier) solutions such as crowdsec.

    • 7Sea_Sailor@lemmy.dbzer0.comOP
      link
      fedilink
      English
      arrow-up
      3
      ·
      10 months ago

      its basically a VPS that comes with torrenting software preinstalled. Depending on hoster and package, you’ll be able to install all kinds of webapps on the server. Some even enable Plex/Jellyfin on the more expensive plans.

        • 7Sea_Sailor@lemmy.dbzer0.comOP
          link
          fedilink
          English
          arrow-up
          3
          ·
          edit-2
          10 months ago

          The rclone mount works via SSH credentials. Torrent files and tracker searches run over simple HTTPS, since both my torrent client and jackett expose public APIs for these purposes, so I can just enter the web address of these endpoints into the apps running on my homelab.

          Sidenote, since you said sshfs mount: I tried sshfs, but has significantly lower copy speeds than with rclone mount. Might have been a misconfiguration, but it was more time efficient to use rclone than trying to debug my sshfs connection speed.

          • notfromhere@lemmy.ml
            link
            fedilink
            English
            arrow-up
            2
            ·
            10 months ago

            I have noticed very slow speeds with sshfs as well. I’ll have to give rclone mount over ssh a try. Thanks!

    • 7Sea_Sailor@lemmy.dbzer0.comOP
      link
      fedilink
      English
      arrow-up
      3
      ·
      edit-2
      10 months ago

      If you’re referring to the “LabProxy VPS”: So that I don’t have to point a public domain that I (plan to) use more and more in online spaces to my personal IP address, allowing anyone and everyone to pinpoint my location. Also, I really don’t want to mess with the intricacies of DynDNS. This solution is safer and more reliable than DynDNS and open ports on my router thats not at all equipped to fend off cyberspace attacks.

      If you’re referring to the caddy reverse proxy on the LabProxy VPS: I’m pointing domains that I want to funnel into my homelab at the external IP of the proxy VPS. The caddy server on that VPS reads these requests and reverse-proxies them onto the caddy-port from the homelab, using the hostname of my homelab inside my tailscale network. That’s how I make use of the tunnel. This also allows me to send the crowdsec ban decisions from the homelab to the Proxy VPS, which then denies all incoming requests from that source IP before they ever hit my homelab. Clean and Safe!

        • 7Sea_Sailor@lemmy.dbzer0.comOP
          link
          fedilink
          English
          arrow-up
          2
          ·
          10 months ago

          Oh, that! That app proxies the docker socket connections over a TCP channel. Which provides a more granular control over what app gets what access to specific functionalities of the docker socket. Directly mounting the socket into an app technically grants full root access to the host system in case of a breach, so this is the advised way to do it.

  • thorbot@lemmy.world
    link
    fedilink
    English
    arrow-up
    2
    ·
    10 months ago

    I just have a UniFi firewall, a Synology Diskstation, and a linux server running everything. Provides torrenting, video streaming with plex, file sharing, game server hosting, music hosting, and more, and I don’t ever have to mess with it :). This is impressive but I don’t know if I would want to support it personally

    • 7Sea_Sailor@lemmy.dbzer0.comOP
      link
      fedilink
      English
      arrow-up
      2
      ·
      10 months ago

      I’d love to have everything centralized at home, but my net connection tends to fail a lot and I dont want critical services (AdGuard, Vaultwarden and a bunch of others that arent listed) to be running off of flakey internet, so those will remain in a datacenter. Other stuff might move around, or maybe not. Only time will tell, I’m still at the beginning of my journey after all!

      • thorbot@lemmy.world
        link
        fedilink
        English
        arrow-up
        1
        ·
        10 months ago

        Fair. I’m lucky enough to be able to get business internet at home so I have a static IP and 99.9% uptime. My plex watchers and game hosting players know that sometimes around 3am, they might be booted when my networking gear auto updates itself, haha