## Hardware Starting in December the environment will contains a freebsd based router/firwall and a single enterprise class server * sitka -- a RiverBed Stealhead CX-770 * tk2018 -- a HP ProLiant DL380 (g7) . ## Network The network is divided into 3 segments * 192.168.31.0/24 a private administrative lan * 10.0.0.0/24 wireguard lan * 69.41.138.97/27 A public facing lan. The host itself does not have any public facing interfaces. It only accessible though the wireguard protected admin lan. The containers, which handle all public facing work do so via an anonymous bridge configuration, allowing them to access the internet directly without allowing external access to the underlying servers. As we move forward the unfiltered interface used by the public facing containers will eventually be replaced by a filtered interface through the firewall. ### Sitka's Network Config | | | | sitka ports| |------|-----------|-------------------|-----------------------------| | port | Interface | IP Address/mask | purpose | | igb0 | bridge0 | 192.168.31.159/24 | internal / admin lan | | igb1 | bridge0 | | | | igb2 | N/A | N/A | unused | | igb3 | igb3 | ?.?.?.?/?? | TBD | | igb4 | igb4 | 69.41.138.126/27 | external interface | | igb5 | igb5 | 0.0.0.0/32 | firewalled public interface | ### TK2022's Network Config | | | | | tk2022 ports| |------|-----------|-------------------|----------|-------------------------------| | port | Interface | IP Address/mask | device | purpose | | 1 | br0 | 0.0.0.0/32 | enp3s0f0 | unfiltered public interface. | | 2 | br2 | 0.0.0.0/32 | enp3s0f1 | firewalled public interface | | 3 | N/A | ?.?.?.?/?? | enp4s0f0 | TBD | | 4 | br1 | 192.168.31.159/24 | enp4s0f1 |internal / admin lan | | ilo | | 192.168.31.119/24 | | remote console | #### As Drawn ![as drawn](images/as-drawn2.jpeg) #### As Deployed ![as deployed a](images/asdeployed2024a.jpeg) ![as deployed b](images/asdeployed2024b.jpeg) #### As implimented ##### in /etc/network/interfaces ```sh #-------------------------------------------------------------------/etc/network/interfaces # 2: enp3s0f0: mtu 1500 qdisc mq master br0 state UP group default qlen 1000 # 3: enp3s0f1: mtu 1500 qdisc mq master br1 state UP group default qlen 1000 # 4: enp4s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 # 5: enp4s0f1: mtu 9000 qdisc mq master br3 state UP group default qlen 1000 source /etc/network/interfaces.d/* auto lo iface lo inet loopback iface enp3s0f0 inet manual iface enp3s0f1 inet manual iface enp4s0f0 inet manual iface enp4s0f1 inet manual auto br0 iface br0 inet manual bridge_ports enp3s0f0 bridge_stp off # disable Spanning Tree Protocol bridge_waitport 0 # no delay before a port becomes available bridge_fd 0 # no forwarding delay auto br1 iface br1 inet manual bridge_ports enp3s0f1 bridge_stp off # disable Spanning Tree Protocol bridge_waitport 0 # no delay before a port becomes available bridge_fd 0 # no forwarding delay auto br2 iface br2 inet manual bridge_ports enp4s0f0 bridge_stp off # disable Spanning Tree Protocol bridge_waitport 0 # no delay before a port becomes available bridge_fd 0 # no forwarding delay auto br3 iface br1 inet static address 192.168.31.159 network 192.168.31.0 netmask 255.255.255.0 broadcast 192.168.31.255 bridge_ports enp4s0f1 bridge_stp off # disable Spanning Tree Protocol bridge_waitport 0 # no delay before a port becomes available bridge_fd 0 # no forwarding delay``` ``` ##### and in /etc/rc.conf ```sh ifconfig_igb4="69.41.138.126 netmask 255.255.255.224" defaultrouter="69.41.138.97" #ifconfig_igb0="inet 192.168.31.2 netmask 255.255.255.0" cloned_interfaces="bridge0" ifconfig_bridge0="inet 192.168.31.2 netmask 255.255.255.0 addm igb0 addm igb1 up" ifconfig_igb0="up" ifconfig_igb1="up" ``` #### As referenced See: ​[https://bitbucket.org/suspectdevicesadmin/ansible/src/master/hosts](https://bitbucket.org/suspectdevicesadmin/ansible/src/master/hosts) which is built referencing [a google doc with proposed allocations](https://docs.google.com/spreadsheets/d/1KRkqdYvgRtV4vu6AGzdLWJVGTIsV2o2iSSJBEFMZJAw/edit#gid=0) ## Server OS, Filesystems and Disk layout The server runs Debian bookworm along with zabbly supported version of [incus](https://linuxcontainers.org/incus/). Outside of zfs not much is added to the stock installation. This is intentional. The real work is done by the containers the host os is considered disposable. ### Disk Layout The incus server uses hardware raid 1 for the boot disk. The containers and other data are a able to take advantage of zfs mirroring and caching. | |. | | | | kb2018 disks| |------|--------------|--------|------------|-------------------|--------------------:| | disk | device/pool | bay | type | mount point(s) | purpose/notes. | | sdb | /dev/sdb | 2C:1:3 | raid1+0 | /, /var/lib/incus | os and incus data | | | "" "" | 2C:1:4 | raid1+0 | | | | sda | infra, devel | 3C:1:7 | zfs | | incus storage pools | | sdg | "" "" | 3C:1:8 | zfs mirror | | | | sdd | tank | 3C:1:5 | zfs | /tank | space for stuff | | sdc | "" "" | 3C:1:6 | zfs mirror | | | ### Hardware raid on the DL380 The raid controller on the Dell allows a mixing of hardware raid and direct hot swappable connections. The HP 420i does only hardware raid or direct connections (HBA) but not both. Since we use the hardware raid the remaining disks need to be configured using the ssacli or the raid controllers bios. See: [Dude Where Are My Disks](/zeearchive/DL380RaidController) ## Containers Work previously done by standalone servers is now done though incus managed containers. An up to date list of containers is somewhat maintained at ## Ansible Ansible is used to make most tasks reasonable including. * creating containers * updating admin passwords and ssh keys. # Tasks: Accessing Hosts ## tk2022 ssh access The host machines for the containers can be accessed through the admin lan. This is done via wirguard on either [sitka](norouter/using-a-tank-for-crowd-control/) or [virgil](norouter/wireguard-and-tinyproxy/) *note: as of a few updates ago you have to tell apples ssh client to use ssh-dss as below* YOU ARE HERE update the ilo settings so they report the right server. ```sh steve:~ don$ ssh -p22 -oHostKeyAlgorithms=+ssh-dss feurig@tinas-ilo.suspectdevices.com feurig@192.168.31.119's password: User:feurig logged-in to kb2018.suspectdevices.com(192.168.31.119 / FE80::9E8E:99FF:FE0C:BAD8) iLO 3 Advanced for BladeSystem 1.88 at Jul 13 2016 Server Name: kb2018 Server Power: On hpiLO-> vsp Virtual Serial Port Active: COM2 Starting virtual serial port. Press 'ESC (' to return to the CLI Session. tk2022 login: ( hpiLO-> exit _ if the serial port is still in use do the following _ Virtual Serial Port is currently in use by another session. hpiLO-> stop /system1/oemhp_vsp1 ``` See: [ilo 3 notes page](NotesOnILO3) ### ssh access to containers The susdev profile adds ssh keys and sudo passwords for admin users allowing direct ssh access to the container. ```sh steve:~ don$ ssh feurig@ian.suspectdevices.com ... feurig@ian:~$ ``` The containers can be accessed directly from the incus host as root ```sh root@btk2022:~# incus exec harvey bash root@harvey:~# apt-get update&&apt-get -y dist-upgrade&& apt-get -y autoremove ``` ## Updating dns Dns is provided by bind , The zone files have been consolidated into a single directory under /etc/bind/zones on piage (dns.digithink.com). The zones file is maintained in a private git repository on bitbucket. Do not forget to push any changes made to the zones file when the work is complete ```sh root@naomi:/etc/bind/zones# nano suspectdevices.hosts ... @ IN SOA dns1.digithink.com. don.digithink.com ( 2018080300 10800 3600 3600000 86400 ) ; ^^ update ^^ ; .... make some changes .... johnson IN A 69.41.138.103 git IN CNAME johnson ... root@naomi:/etc/bind/zones# systemctl named reload root@naomi:/etc/bind/zones# tail /var/log/messages ... Sep 3 08:10:04 piage named[178]: zone 3dangst.com/IN: loaded serial 2018080300 Sep 3 08:10:04 piage named[178]: zone 3dangst/IN: sending notifies (serial 2018080300) Sep 3 08:10:04 piage named[178]: client 69.41.138.126#56120 (3dangst.com): transfer of '3dangst.com/IN': AXFR-style IXFR started (serial 2018080300) Sep 3 08:10:04 piage named[178]: client 69.41.138.126#56120 (3dangst.com): transfer of '3dangst.com/IN': AXFR-style IXFR ended Sep 3 08:10:04 piaga named[178]: client 69.41.138.126#47381: received notify for zone '3dangst.com' ``` ## Updating Hosts / Containers When updates are available Apticron sends us an email. We prefer this to autoupdating our hosts as it helps us maintain awareness of what issues are being addressed and does not stop working when there are issues. All running containers can be updated using the following update script. ```sh nano /usr/local/bin/update.sh #!/bin/bash # update.sh for debian/ubuntu/centos/suse (copyleft) don@suspecdevices.com echo --------------------- begin updating `uname -n` ---------------------- if [ -x "$(command -v apt-get)" ]; then apt-get update apt-get -y dist-upgrade apt-get -y autoremove fi if [ -x "$(command -v yum)" ]; then echo yum upgrade. yum -y upgrade fi if [ -x "$(command -v zypper)" ]; then echo zypper dist-upgrade. zypper -y dist-upgrade fi echo ========================== done ============================== ^X chmod +x /usr/local/bin/update.sh ``` pushing the update script to containers. ```sh incus file push /usr/local/bin/update.sh virgil/usr/local/bin/ incus exec virgil chmod +x /usr/local/bin/update.sh ``` you can run this against all running containers as follows. ```sh for c in `incus list -cn -f compact|grep -v NAME`; do incus exec $c update.sh; done ; update.sh ``` This could also be used as an ansible ad hoc command. ```sh ansible pets -m raw -a "update.sh" ``` ## Creating containers ```sh cd /etc/ansible nano hosts ... add new host ... ansible-playbook playbooks/create-lxd-containers.yml ``` .....YOU ARE HERE..... *documenting the ansible script to create containers* ## Backing Up Containers YOU ARE HERE RETHINKING THIS # links.... (tbd)