strum

#+TITLE: Acksin STRUM
#+AUTHOR: Acksin
#+OPTIONS: html-postamble:nil body-only: t

#+begin_quote
With my supervision,
primordial nature
brings forth the moving
and the nonmoving.

-- Bhagavad Gita
#+end_quote

#+begin_html
<a href="https://travis-ci.org/acksin/strum"><img src="https://travis-ci.org/acksin/strum.svg?branch=master" /></a>
<a href="https://godoc.org/github.com/acksin/strum"><img src="https://godoc.org/github.com/acksin/strum?status.svg" alt="GoDoc"></a>
#+end_html

* tl;dr

While the tool can be used without =sudo= there may be information
that is left out because of a lack of permissions. As such it is
better to run the tool as =sudo=.

#+begin_src sh
sudo strum
#+end_src

If you just want info on a single PID run the following:

#+begin_src sh
sudo strum [pid]
#+end_src

* Introduction

Acksin STRUM is the tool to diagnose Linux issues quickly. It gives
you a complete picture of the CPU, Memory, IO, Networking, Processes,
Kernel Limits, etc. of a system. STRUM was born out of the frustration
that diagnosing an issue was to go through a mental checklist and
dealing with fallible human memory (especially at 3am when the pager
goes off). We hope that this tool will help.

* Use Case

System Admins turn to tools like =vmstat=, =free=, =top=, =ps=,
etc. to quickly figure out what the issues are with a Linux and UNIX
machines. However, as common as these tools are they are still limited
insofar as they need to be strung together to give a complete picture
of what is happening.

If a MySQL machine is high CPU your workflow may be running =top= to
see what the CPUs are like, =free= to see if the machine is swapping,
=ss= to see if there are a lot of connections, etc.

STRUM will give you a birds eye view of the system and a process
quickly.

* Basic Usage
** Syntax

We will reference the different keys of the output with the following
syntax.

=.System.Attribute= refers to

#+begin_src json
  {
      "System": {
          "Attribute": "foobar"
      }
  }
#+end_src

=.Processes.0.PID= refers to the first item in the array Processes.

#+begin_src json
  {
      "Processes": [
          {
              "Exe": "systemd",
              "PID": 1,
              "Memory": {
                  "Swap": {
                      "Size": 0,
                      "Unit": "kb"
                  }
              },
              "IO": {
                  "Limits": {
                      "OpenFiles": 65536,
                      "FileSize": 0,
                      "CPUTime": -1
                  },
                  "FD": null
              }
          }
      ]
  }
#+end_src

** Getting Help

#+begin_src sh :results output example :exports both
./strum -help
#+end_src

** Basic Usage

#+begin_src sh :results output code json :exports both :noweb
./strum
#+end_src

** Information on a Process

#+begin_src sh :results output code json :exports both :noweb
./strum 1505
#+end_src

** Alternate Outputs

#+begin_src sh :results output code json :exports both :noweb
./strum -flat
#+end_src

** STRUM Cloud

STRUM Cloud is a web interface to look at the details of your machines
data.

* Diagnostics

Acksin STRUM is a tool for diagnosing issues quickly. It deals with
things at the system level, the cloud level, and at the process level.
The output of STRUM therefore has the following output:

#+begin_src json
  {
      "System": ...
      "Container": ...
      "Cloud": ...
      "Processes": ...
  }
#+end_src

** System

The output of the =.System= section looks like the following:

#+begin_src sh :results output code :exports both :noweb
strum | jq '.System'
#+end_src

#+RESULTS:
#+BEGIN_SRC sh
{
  "Memory": {
    "Unit": "kb",
    "Physical": {
      "Total": 8173844,
      "Free": 1182716,
      "Used": 6991128,
      "Cached": 4902344,
      "Buffers": 502108,
      "TotalFree": 6587168
    },
    "Swap": {
      "Total": 10485756,
      "Free": 10215728,
      "Used": 270028,
      "Cached": 75100
    },
    "Virtual": {
      "Total": 34359738367,
      "Used": 0,
      "Chunk": 0
    },
    "Dirty": 140,
    "Writeback": 0,
    "Mapped": 97444
  },
  "Network": {
    "Total": 284,
    "RAW": {},
    "UDP": {},
    "TCP": {
      "Total": 0,
      "Established": 14,
      "Closed": 0,
      "Orphaned": 0,
      "Synrecv": 0,
      "Timewait": 0
    }
  },
  "Kernel": {
    "/proc/sys/abi/vsyscall32": "1",
    "/proc/sys/debug/exception-trace": "1",
    "/proc/sys/debug/kprobes-optimization": "1",
    "/proc/sys/dev/cdrom/autoclose": "1",
    "/proc/sys/dev/cdrom/autoeject": "0",
    "/proc/sys/dev/cdrom/check_media": "0",
    "/proc/sys/dev/cdrom/debug": "0",
    ...
  }
}
#+END_SRC

*** Memory

#+begin_src sh :results output code :exports both :noweb
strum | jq '.System.Memory'
#+end_src

#+RESULTS:
#+BEGIN_SRC sh
{
  "Unit": "kb",
  "Physical": {
    "Total": 8173844,
    "Free": 1180308,
    "Used": 6993536,
    "Cached": 4902140,
    "Buffers": 502140,
    "TotalFree": 6584588
  },
  "Swap": {
    "Total": 10485756,
    "Free": 10215728,
    "Used": 270028,
    "Cached": 75104
  },
  "Virtual": {
    "Total": 34359738367,
    "Used": 0,
    "Chunk": 0
  },
  "Dirty": 68,
  "Writeback": 0,
  "Mapped": 97356
}
#+END_SRC

The main thing to worry about when doing diagnostics is to see if the
=.System.Memory.Swap= is being used. A swapping system means that
memory is being moved from disk to memory and back again which can
lead to high CPU usage and poor performance.

*** CPU

NOT IMPLEMENTED

*** Networking

INCOMPLETE

*** IO

NOT IMPLEMENTED

*** Limits

NOT IMPLEMENTED

*** Kernel

The =.System.Kernel= output gives you all the key value information
about the running kernel parameters. It is akin to the =sysctl -a=
output. The key is the full path to the change. On the Linux kernel
that is in =/proc/sys/=.

#+begin_src sh :results output code :exports both :noweb
strum | jq '.System.Kernel'
#+end_src

#+RESULTS:
#+BEGIN_SRC sh
{
  "/proc/sys/abi/vsyscall32": "1",
  "/proc/sys/debug/exception-trace": "1",
  "/proc/sys/debug/kprobes-optimization": "1",
  "/proc/sys/dev/cdrom/autoclose": "1",
  "/proc/sys/dev/cdrom/autoeject": "0",
  "/proc/sys/dev/cdrom/check_media": "0",
  "/proc/sys/dev/cdrom/debug": "0",
  ...
}
#+END_SRC

** Container

The container portion contains various information about the
containers that are running on the machine as well as any quick
diagnostic information that can be immediately useful.

*** Docker

#+begin_src sh :results output code :exports both :noweb
strum | jq '.Container.Docker'
#+end_src

#+RESULTS:
#+BEGIN_SRC sh
  {
      "Containers": [
          {
              "Id": "3364ca00225b54080675e24aea8bf85b2d6a59985ea4c50d7a9390cca92f9d75",
              "Names": [
                  "/naughty_bose"
              ],
              "Image": "busybox",
              "ImageID": "sha256:47bcc53f74dc94b1920f0b34f6036096526296767650f223433fe65c35f149eb",
              "Command": "sh",
              "Created": 1460584455,
              "Ports": [],
              "Labels": {},
              "State": "",
              "Status": "Exited (0) 5 seconds ago",
              "HostConfig": {
                  "NetworkMode": "default"
              },
              "NetworkSettings": {
                  "Networks": {
                      "bridge": {
                          "IPAMConfig": null,
                          "Links": null,
                          "Aliases": null,
                          "NetworkID": "",
                          "EndpointID": "",
                          "Gateway": "",
                          "IPAddress": "",
                          "IPPrefixLen": 0,
                          "IPv6Gateway": "",
                          "GlobalIPv6Address": "",
                          "GlobalIPv6PrefixLen": 0,
                          "MacAddress": ""
                      }
                  }
              },
              "Mounts": null
          }
      ],
      "Images": [
          {
              "Id": "sha256:47bcc53f74dc94b1920f0b34f6036096526296767650f223433fe65c35f149eb",
              "ParentId": "",
              "RepoTags": [
                  "busybox:latest"
              ],
              "RepoDigests": null,
              "Created": 1458325368,
              "Size": 1112820,
              "VirtualSize": 1112820,
              "Labels": {}
          }
      ]
  }
#+END_SRC

#+RESULTS:


** Cloud

The Cloud section gives various information about the cloud provider
and cloud attributes that the machine has. This information can be
useful for quickly figuring out if the machine is on an under powered
machine for the tasks of the application.

*** AWS

The key =.Cloud.AWS= provides the following output.

#+begin_src sh :results output code :exports both :noweb
strum | jq '.Cloud.AWS'
#+end_src

#+RESULTS:
#+BEGIN_SRC sh
{
  "AmiID": "ami-1121ca71",
  "AmiLaunchIndex": "0",
  "AmiManifestPath": "(unknown)",
  "Hostname": "ip-172-31-27-98.us-west-2.compute.internal",
  "InstanceAction": "none",
  "InstanceID": "i-05bcbe3e3563e1039",
  "InstanceType": "t2.micro",
  "LocalHostname": "ip-172-31-27-98.us-west-2.compute.internal",
  "LocalIpv4": "172.31.27.98",
  "MAC": "02:3e:a9:c6:1a:5f",
  "Profile": "default-hvm",
  "PublicHostname": "ec2-52-38-49-127.us-west-2.compute.amazonaws.com",
  "PublicIpv4": "52.38.49.127",
  "ReservationID": "r-0555ad3d4b37c692f",
  "SecurityGroups": "launch-wizard-1"
}
#+END_SRC

*** DigitalOcean

#+begin_src sh :results output code :exports both :noweb
strum | jq '.Cloud.DigitalOcean'
#+end_src

#+RESULTS:
#+begin_src json
  {
      "floating_ip": {
          "ipv4": {}
      },
      "interfaces": {
          "public": [
              {
                  "anchor_ipv4": {
                      "gateway": "10.17.0.1",
                      "netmask": "255.255.0.0",
                      "ip_address": "10.17.0.5"
                  },
                  "ipv4": {
                      "gateway": "104.236.0.1",
                      "netmask": "255.255.192.0",
                      "ip_address": "104.236.17.208"
                  },
                  "type": "public",
                  "mac": "04:01:cb:e3:c3:01"
              }
          ]
      },
      "dns": {
          "nameservers": [
              "8.8.8.8",
              "8.8.4.4"
          ]
      },
      "region": "nyc3",
      "public_keys": [
      ],
      "vendor_data": "#cloud-config\ndisable_root: false\nmanage_etc_hosts: true\n\n# The modules that run in the 'init' stage\ncloud_init_modules:\n - migrator\n - ubuntu-init-switch\n - seed_random\n - bootcmd\n - write-files\n - growpart\n - resizefs\n - set_hostname\n - update_hostname\n - [ update_etc_hosts, once-per-instance ]\n - ca-certs\n - rsyslog\n - users-groups\n - ssh\n\n# The modules that run in the 'config' stage\ncloud_config_modules:\n - disk_setup\n - mounts\n - ssh-import-id\n - locale\n - set-passwords\n - grub-dpkg\n - apt-pipelining\n - apt-configure\n - package-update-upgrade-install\n - landscape\n - timezone\n - puppet\n - chef\n - salt-minion\n - mcollective\n - disable-ec2-metadata\n - runcmd\n - byobu\n\n# The modules that run in the 'final' stage\ncloud_final_modules:\n - rightscale_userdata\n - scripts-vendor\n - scripts-per-once\n - scripts-per-boot\n - scripts-per-instance\n - scripts-user\n - ssh-authkey-fingerprints\n - keys-to-console\n - phone-home\n - final-message\n - power-state-change\n",
      "hostname": "postgres9.4-512mb-nyc3-01",
      "droplet_id": 13362015
  }
#+end_src



** Process

Here we want to get information about the process with the =PID= 2277.

#+begin_src sh :results output code :exports both :noweb
strum 2277 | jq '.Processes[0]'
#+end_src

#+RESULTS:
#+BEGIN_SRC sh
{
  "Exe": "/lib/systemd/systemd",
  "PID": 2277,
  "Memory": {
    "Swap": {
      "Size": 0,
      "Unit": "kb"
    }
  },
  "IO": {
    "Limits": {
      "OpenFiles": 1024,
      "FileSize": 0,
      "CPUTime": -1
    },
    "FD": {
      "0": "/dev/null",
      "1": "socket:[21619]",
      "10": "/proc/2277/mountinfo",
      "11": "anon_inode:inotify",
      "12": "/proc/swaps",
      "13": "socket:[21669]",
      "14": "socket:[21670]",
      "2": "socket:[21619]",
      "3": "socket:[21635]",
      "4": "anon_inode:[eventpoll]",
      "5": "anon_inode:[signalfd]",
      "6": "/sys/fs/cgroup/systemd/user/abhi/2",
      "7": "anon_inode:[timerfd]",
      "8": "socket:[21650]",
      "9": "anon_inode:[eventpoll]"
    }
  }
}
#+END_SRC

*** Memory

We can see the memory usage for the process.

#+begin_src sh :results output code :exports both :noweb
strum 2277 | jq '.Processes[0].Memory'
#+end_src

#+RESULTS:
#+BEGIN_SRC sh
{
  "Swap": {
    "Size": 0,
    "Unit": "kb"
  }
}
#+END_SRC


*** CPU

NOT IMPLEMENTD

*** Networking

NOT IMPLEMENTED

*** IO

We can get information about the IO of a process. We can see what
files it has open as well as the limits that it has.

#+begin_src sh :results output code :exports both :noweb
strum 2277 | jq '.Processes[0].IO'
#+end_src

#+RESULTS:
#+BEGIN_SRC sh
{
  "Limits": {
    "OpenFiles": 1024,
    "FileSize": 0,
    "CPUTime": -1
  },
  "FD": {
    "0": "/dev/null",
    "1": "socket:[21619]",
    "10": "/proc/2277/mountinfo",
    "11": "anon_inode:inotify",
    "12": "/proc/swaps",
    "13": "socket:[21669]",
    "14": "socket:[21670]",
    "2": "socket:[21619]",
    "3": "socket:[21635]",
    "4": "anon_inode:[eventpoll]",
    "5": "anon_inode:[signalfd]",
    "6": "/sys/fs/cgroup/systemd/user/abhi/2",
    "7": "anon_inode:[timerfd]",
    "8": "socket:[21650]",
    "9": "anon_inode:[eventpoll]"
  }
}
#+END_SRC

*** Limits

We can see here what the kernel limits are for the process.

#+begin_src sh :results output code :exports both :noweb
strum 2277 | jq '.Processes[0].IO.Limits'
#+end_src

#+RESULTS:
#+BEGIN_SRC sh
{
  "OpenFiles": 1024,
  "FileSize": 0,
  "CPUTime": -1
}
#+END_SRC

* Release Notes
** 0.4.0

 - =.Cloud.AWS.Spot.Termination= to show when an AWS instance is about
   to go down.
 - New flags to output content. =-cloud=, =-json=, =-flat=.
 - Output to [[https://www.acksin.com/console][Acksin Console]].

** 0.3.0

 - =.System.Kernel=: Get information about =/proc/sys/= the same data that
   you get from =sysctl -a=
 - =.Cloud.DigitalOcean=: Get the Droplet metadata from DigitalOcean now
   along with AWS.
 - =.Container.Docker=: Get information about the Docker containers
   running on your machine as well as the images that are on there.
 - =.Memory.Physical.TotalFree=: This is the real free memory on
   Linux. It is the free memory minus the cached and buffered data
   that Linux uses for files.

* License

Copyright (C) 2016 Acksin <hey@acksin.com>

This Source Code Form is subject to the terms of the Mozilla Public
License, v. 2.0. If a copy of the MPL was not distributed with this
file, You can obtain one at http://mozilla.org/MPL/2.0/.