How do you monitor your server containers, disks, load…?

Do you use an easy-to-use web interface? Do you do everything via SSH? Or maybe you’ve got a more complicated setup?

I want to change my setup and I’m looking for new ideas, I’ve been using Cockpit for some years and some of the plugins are really outdated (ZFS for example) and others are completely broken (docker-compose).

  • aksdb@lemmy.world
    link
    fedilink
    English
    arrow-up
    18
    ·
    9 months ago

    My own server? YOLO

    At work? Grafana, KOBS, Victoria Metrics, Jaeger, OpsGenie, …

    • AlphaAutist@lemmy.world
      link
      fedilink
      English
      arrow-up
      3
      ·
      9 months ago

      This is the first time I’ve heard of Victoria Metrics. It looks like it has a similar use case as Prometheus, is that correct? If so, what made you or your team choose one over the other?

      • aksdb@lemmy.world
        link
        fedilink
        English
        arrow-up
        2
        ·
        9 months ago

        IIRC it had better performance than Prometheus. We also ditched Elasticsearch in favor of ClickHouse to keep up with log ingestion.

        • scrion@lemmy.world
          link
          fedilink
          English
          arrow-up
          2
          ·
          edit-2
          9 months ago

          I can second that. We had some really good experiences with ClickHouse and its performance. If it fits the bill, it’s a very nice piece of software.

    • cmeu@lemmy.world
      link
      fedilink
      English
      arrow-up
      4
      ·
      9 months ago

      Same been running netdata for years. They’re monetizing now where it used to just be free. Good for them, it’s a great product. And it’s foss

      • ArmoredCavalry@lemmy.world
        link
        fedilink
        English
        arrow-up
        1
        ·
        edit-2
        9 months ago

        As others stated, you can run and access the interface locally (or setup your own reverse proxy) for free. Their Cloud dashboard is also free for up to 5 nodes. They recently added a flat-rate “Homelab” plan as well, if you want to remove the limit. It’s all quite usable for $0 otherwise though!

    • Encrypt-Keeper@lemmy.world
      link
      fedilink
      English
      arrow-up
      1
      ·
      9 months ago

      I love how easy to use NetData is, but when running it on my home servers it destroys their performance lol. Every once in awhile I check in to see if it runs better.

      • ArmoredCavalry@lemmy.world
        link
        fedilink
        English
        arrow-up
        1
        ·
        9 months ago

        That’s strange, I’ve run it fine on some very underpowered hardware. Are you adding a specific monitoring integration with it, or just out of the box settings?

        • Encrypt-Keeper@lemmy.world
          link
          fedilink
          English
          arrow-up
          1
          ·
          9 months ago

          Just out of the box. I am usually running it as a container on UnRAID on an x86 machine. It seems primarily to just be a big memory hog when I’ve tried to use it.

          • ArmoredCavalry@lemmy.world
            link
            fedilink
            English
            arrow-up
            1
            ·
            9 months ago

            Weird! For reference one VM I run on only has 1 GB of memory, and Netdata uses 100-200 MB. Could be something going on with UnRAID though. Definitely some sort of bug I’d think, since normally resource usage should be very low across the board.

    • IHawkMike@lemmy.world
      link
      fedilink
      English
      arrow-up
      1
      ·
      9 months ago

      Adding my vote for Zabbix. It was a bit of a bear to set up and I had to write custom scripts to install the agents with TLS settings that were secure enough for me, but once it’s all set up it’s amazingly easy and intuitive to use and incredibly customizable.

  • Concave1142@lemmy.world
    link
    fedilink
    English
    arrow-up
    4
    ·
    9 months ago

    Zabbix for agent / snmp based statistics.

    Uptime Kuma for up/down states with a webhook notification into Discord so I get instant alerts on my phone when one goes down.

  • Dran@lemmy.world
    link
    fedilink
    English
    arrow-up
    4
    ·
    9 months ago

    How has nobody in this thread said check_mk yet?

    It’s free, you host it yourself. It’s built off of nagios, compatible with nagios plugins, supports snmp or agent based checks. It can email, SMS, slack or discord you when something breaks, you can write your own custom checks in any language that can output to a local console… I could never imagine even looking for something else.

  • sysadmin420@lemmy.world
    link
    fedilink
    English
    arrow-up
    2
    ·
    edit-2
    9 months ago

    I’ve been using uptime Kuma recently and it’s great but works better outside of docker.

    Inside docker I’d get a lot of false down positives from I assume docker throttling the checks.

    Plus it works with email, telegram, and matrix chat alerts. I monitor all my clients sites with it, and it’s bullet proof behind caddy.

    • thirdBreakfast@lemmy.world
      link
      fedilink
      English
      arrow-up
      1
      ·
      9 months ago

      For light touch monitoring this is my approach too. I have one instance in my network, and another on fly.io for the VPSs (my most common outage is my home internet). To make it a tiny bit stronger, I wrote a Go endpoint that exposes the disk and memory usage of a server including with mem_okay and disk_okay keywords, and I have Kuma checking those.

      I even have the two Kuma instances checking each other by making a status page and adding checks for each other’s ‘degraded’ state. I have ntfy set up on both so I get the Kuma change notifications on my iPhone. I love ntfy so much I donate to it.

      For my VPSs, this is probably not enough, so I am considering the more complicated solutions (I’ve started wanting to know things like an influx of fali2ban bans etc.)

      • sysadmin420@lemmy.world
        link
        fedilink
        English
        arrow-up
        2
        ·
        9 months ago

        I just do web hosting for clients sites and use Kuma to monitor uptime and SSL certificates.

        Ive got multiple Kuma’s running as well.

  • qaz@lemmy.world
    link
    fedilink
    English
    arrow-up
    1
    ·
    edit-2
    9 months ago

    Cockpit, Cosmos Cloud, Portainer, Grafana, and a few other things. It’s not the most optimal solution but it kinda of works for now.

  • Ackward@lemmy.world
    link
    fedilink
    English
    arrow-up
    1
    ·
    9 months ago

    Prometheus + node/container exporters. + Grafana for dashboards I haven’t touched zabbix in years but last time it didn’t support very well dynamic scalations. Also all of them are focused on monitoring infrastructure, you need to pay if you want APM or UX.

    Enterprise level, for APM I like datadog, much better than NR. For UX we use acoustic tealeef.

  • keyez@lemmy.world
    link
    fedilink
    English
    arrow-up
    1
    ·
    9 months ago

    Grafana, fronting information from Prometheus, Loki and Telegraf/influxdb since I’m used to that from work and has been a bit more set and forget compared to node_exporter. Easier to add in plugins as well instead of a new container/service to scrape.

  • JoeKrogan@lemmy.world
    link
    fedilink
    English
    arrow-up
    1
    ·
    edit-2
    9 months ago

    Btop and logwatch with logrotate. I use healthchecks to check if the server is unreachable and it notifies me.

  • khorak@lemmy.dbzer0.com
    link
    fedilink
    English
    arrow-up
    1
    ·
    9 months ago

    For the physical hosts / bare metal I use fluentbit, with Loki as the backend. Grafana for visualization and alerts. This gives me utilization metrics and uptime monitoring. The app containers themselves I do not monitor.