All severs, all the time
2024-06-04
Here are some planned updates for the website
This webpage is all about servers. Home servers, work servers, custom build servers, OEM pre built servers, networking, file systems, containers, virtualization, and all kinds of cool stuff you can do with your computers for work and play.
Servers you love, servers you hate, servers you want to build, servers you want to try out, if its a computer and you do stuff with it, this page will show you things you can do with it.
OLD SERVERS!
NEW SERVERS!
BIG SERVERS!
SMALL SERVERS!
WEIRD SERVERS!
SERVERS FROM AROUND THE WORLD!
Want to build a home media server but dont know where to start? Do you love Linux? Well maybe the "Perfect Media Server" guide will be a good choice for you!
One of the most popular media server applications to watch your legally obtained movies, TV shows, and music, on any device. With Plex, you can share your library with others, and access your library remotely from over the internet easily. A Plex account is free (despite what all the self-ads on their webpage and apps suggest), and you do not need to sign up for any subscriptions to use Plex. The person hosting the Plex Media Server will eventually want to purchase the "Plex Pass" lifetime membership to enable extra features such as better remote library access support and better transcoding support. Users accessing the Plex library (not hosting their own) do not ever need to pay anything. Note that the default Plex app and web dashboards will often be cluttered with self-ads for Plex's own internal subscription services which you do not need, so be sure to un-pin them from the Plex all Home page sidebar.
Plex is not the only game in town, there are a few alternatives, with some feature trade-offs.
"What is the Best CPU for Plex?"tl;dr: get a 7th gen or later Intel CPU if possible in order to utilize Quick Sync (used for transcoding e.g. down-converting media to play on mobile devices, etc.). You can use older CPU's just fine but they will not have as advanced Quick Sync capabilities. If you do not transcode and only play the source media directly as-is, then you should be fine with even less.
While not needed for Plex, if you do wish to encode HEVC 10-bit, then you should look at Intel 11th generation or later. The integrated graphics on those processors are capable of encoding HEVC 10-bit.
Regardless of your requirements, I suggest going with an Intel 7th generation or later, to get the best experience out of your Plex server.Pro Tip: you can find used micro computers such as the Dell Optiplex 3060 and 3070 Micro models with 8th gen Intel CPU's (e.g. i5 8500T) for dirt cheap on eBay which should work great for hosting a Plex Media Server, assuming you store the files on a separate NAS or similar file server. Bonus: since most of these are ex-office computers, they will often have a Windows 10 Pro key embedded in the motherboard. Just make sure to get one includes a power cable!
An Intel NUC with a similar generation CPU also makes a fantastic small Plex Media Server host.
You can of course use a dedicated GPU such as the offerings from Nvidia or AMD, however Intel's Quick Sync is more power-efficient for the price and does not use up extra space in your PC.
source: https://en.wikipedia.org/wiki/Intel_Quick_Sync_Video
"Do I need transcoding?"
Transcoding is the process of converting video files on your computer from one format to another. By default, if you are streaming media on Plex (movies, TV shows) on your own home network, you will usually being using direct-play, where the raw file contents will be streamed directly to your device as-is, without any transcoding.
Transcoding comes into play primarily in two situations;
the device playing the media does not support the format of the original source video
the internet connection between the server and the player is limited so its better to send a lower-quality version of the media to save bandwidth
In both situations, transcoding can be used (and will be chosen automatically by a properly configured Plex Media Server) in order to re-compress and re-format the video in real time while its being servered up to the client device.
If transcoding is performed on the CPU, it can become a very resource intensive process, however GPU's are able to accelerate the process greatly if they offer hardware support to match the requirements. This is the case especially for Intel iGPU's (integrated GPU) which come built-in to many Intel CPUs. So if you choose a system with an Intel CPU with Quick Sync support, you will have access to transcoding capabilities.
Networking can be fun when you use third party router software that you can install and configure yourself! Here are some common and fun software packages to check out.
Homepage: https://www.pfsense.org/
Homepage: https://opnsense.org/
amd64 CPU architecture; as such, it
typically runs on more powerful devices than the standard ARM-based
router hardware.Install: https://docs.opnsense.org/manual/install.html
Download: https://opnsense.org/download/
Great video on installing and setting up OPNSense;
https://www.youtube.com/watch?v=54JWsGedXpo
https://www.youtube.com/watch?v=h2_cQxTkh3Q
How to set a static IP address for a device on your local network
How to set up port forwarding
How to get your current IP address on your internal local network
(Linux, macOS)
ifconfig
# or
ip awill show the details for all network devices, amongst them should be
some devices with names like eth0, with IP addresses such
as 192.168.x.x
Windows: https://www.howtogeek.com/858334/how-to-find-your-ip-address-from-cmd-command-prompt/
How to get your public internet IP address
curl http://checkip.amazonaws.com
# or
curl ifconfig.meHow to get IP address(es) of website
nslookup github.com
# or
dig github.comHow to check that a server (local or remote) can be reached
ping 8.8.8.8How to check what local process is running on a port
sudo lsof -i :1234How to check which local ports are in use
lsof -i TCP | grep LISTENHow to check if a port is open on a server
# netcat
nc -vz 192.168.1.2 7878
# Connection to 192.168.1.2 7878 port [tcp/*] succeeded!
nc -vz 192.168.1.2 7879
# nc: connect to 192.168.1.2 port 7879 (tcp) failed: Connection refusedHow to check if a server has a DHCP server running
sudo nmap -sU -p 67 --script=dhcp-discover 192.168.1.1How to search for connected devices on your local subnet
nmap -sn 192.168.1.0/24Another backup method to search for devices on your subnet
# find the broadcast IP for your subnet ; should end with .255
ifconfig | grep broadcast
# ping that ip
ping 192.168.1.255
# update arp table
arp -aHow to renew your DHCP server lease
# might want to run this in a 'screen' session since the first command may disconnect you
sudo dhclient -r ; sudo dhclientHow to check which TCP and UDP ports are open on your server
netstat -plnutHow to scan all ports on a server
sudo nmap -n -PN -sT -sU -p- 192.168.1.2How to check your internet speed
sudo apt install speedtest-cli
speedtest-cli --secureHere are some tools you can use to monitor your network activity on your server
# top style programs
sudo iptraf
sudo iftop
btop
sudo nethogs
# get the usage of ports
netstat -nlp
sudo lsof -i :51886
# lsof -i tcp:51886
sudo fuser -n tcp 51886Resources;
https://serverfault.com/questions/36586/how-can-i-monitor-network-i-o-usage-per-process-under-linux
https://shop.netgate.com/products/1100-pfsense
Example; https://www.aliexpress.us/item/3256804116114245.html
2.5Gb network switches are now widely available for low prices. See this video for a bunch of examples
https://www.youtube.com/watch?v=-pYQvEX9Ct0
resources:
The best guide around for general purpose building of a basic NAS (network attached storage) server
An alternative to traditional filesystems, object storage can be run at home (or in the cloud) using services such as:
Methods to format, mount, and manage drives in Linux / macOS.
Identify the new drive on the system.
sudo parted -l
# or
lsblk
# or
ls -l /dev/disk/by-uuid/Create a new partition table on a blank drive
# using GPT partition table
# WARNING! put your drive's path here!!
sudo parted /dev/sdc mklabel gptMake a ext4 partition on the drive
sudo parted -a opt /dev/sdc mkpart primary ext4 0% 100%Format the ext4 partition on the drive
sudo mkfs.ext4 -L DRIVELABEL /dev/sdc1Get the UUID of the formatted drive (this can also be set when you format it)
lsblk -fMount the drive interactively. Use the UUID for more stable mount command (the drive letters can change)
# FIRST: make the mount point if it doesnt already exist;
# sudo mkdir -p /mnt/DRIVELABEL
# sudo chown -R username /mnt/DRIVELABEL
# sudo chgrp -R username /mnt/DRIVELABEL
sudo mount UUID=abc123xyz /mnt/DRIVELABELIf it looks good, add it to your /etc/fstab
# put a line like this in your fstab
UUID=abc123xyz /mnt/DRIVELABEL ext4 defaults 0 0You can unmount the drive like this
sudo umount /mnt/DRIVELABELYou can mount all the drives in your /etc/fstab like
this
sudo mount -aCheck the SMART stats on your drives using
smartmontools
sudo smartctl --all /dev/sdcCheck the stats on NVMe SSD using nvme-cli
sudo nvme smart-log /dev/nvme0Run a SMART drive test on every mounted drive (drives a
through f)
for i in {a..f}; do sudo smartctl -t long /dev/sd${i}; doneCheck the status of a running SMART test (output in JSON format so
you can parse with jq)
sudo smartctl -j -l selftest /dev/sdaCheck disk usage per process
sudo iotopResources
So you have your storage on one server (or PC, etc.), and you want to access it from another server (or PC, etc.), on the same network. How do you do that? You can do this with an SMB share.
Resources
Different operating systems will have different methods for creating a SMB share. For example, on macOS you can use the System Preferences > Sharing > File Sharing menu options to create and configure a SMB share (you can also do it from the command line; link). Note that you may or may not also want to set up guest access to the share, and or create a share user just for sharing.
Different operating systems will have different methods for accessing a SMB share.
On Ubuntu 22.04, you first need to install some extra tools
sudo apt install samba cifs-utils smbclientYou can check if a SMB share is available on the network with a command like
smbclient -L 192.168.1.10 -U guestWhere the IP is the address of the computer on your network hosting the share, and "guest" is used as the user assuming you enabled guest access to the share from the share host. If it works, you should see a list of available share points.
You can manually mount a SMB share with a command like this
# need to create the mount point first
mkdir -p /media/share/Media
# try to mount the share
sudo mount -t cifs -o guest,username=guest,vers=3.0,uid=yourlocalusername //192.168.1.10/Media /media/share/MediaIn this command;
//192.168.1.10/Media is the local network address to the
SMB share "Media" on the system at IP address 192.168.1.10.
Note that the share point name "Media" does not include the path to the
directory on the local system in the SMB address.
we are using both the guest mounting option to disable
the password prompt, and the username=guest to mount it
under the remote guest user access available on the remote SMB host ;
you might want to exclude this if you are not using guest access on the
SMB share
vers=3.0 is specifying to use SMB Version 3.0 ; there
have been numerous security vulnerabilities found in old versions of
SMB, so this option forces the local client to try and connect to the
remote host with one of the most modern versions of SMB which is
currently 3.0
uid=yourlocalusername, where
yourlocalusername can be replaced with the username of your
current user, changes the file ownership of the mounted files on the
local system to that of the current user, in order to alleviate some
issues with file permissions when trying to write to the files and
directories on the SMB share
/media/share/Media is the path to the location on the
local system where the SMB should be mounted; make sure this directory
path already exists, and you may need to adjust the ownership of that
location for your current user as well (cant remember if I needed to do
that or not but I usually end up having to do it for other mount points
with sudo chown -R myusername /path/to/dir)
If it does not work, you may need to check the system logs with
sudo dmesgIf it does work, you should not see any error messages, and you
should be able to see the contents of the SMB share at the specified
local path with ls -l.
Once you get it working, you can unmount the share
(sudo umount /media/share/Media) and then add an entry to
your /etc/fstab, it might look like this
# open the fstab file; always a good idea to make a backup copy first!
sudo nano /etc/fstab
...
# edit the file; add a line like this to the file:
//192.168.1.10/Media /media/share/Media cifs guest,username=guest,vers=3.0,uid=yourlocalusername 0 0
# save and close the file
# reload all the mount points from your /etc/fstab
sudo mount -aIf you have issues, troubleshoot based on error messages from
dmesg, and keep in mind that you can copy / paste most
error messages into Google and even ChatGPT to search for some
guidance.
Note also that you may need to allow SMB access through your firewall, if you have it enabled, with a command such as
sudo ufw allow SambaThis requires that you have installed the samba package
from apt. You can check your firewall configuration with
commands like
sudo ufw status
sudo ufw app listResources
Creating a SMB (Samba) Share on Linux (Ubuntu)
Resources:
Creating and configuring a SMB share in Ubuntu is one of the more involved tasks I have done yet. Its not hard, its just a number of steps that need to be completed and are easy to miss or mix up.
The overall process goes like this;
mergerfs or
another disk array file system, you must make sure that your sharing
user has rw access to both the array volume and the underlying
disks!phew that was a lot... lets see if I can recount all the steps correctly to achieve this; I have them in my bash history but the history is a little out of order after messing up and having to repeat a few steps
sudo apt install samba# create the "sharing" user group
sudo addgroup sharing
# check that sharing user group we just created actually exists
getent group
# ...
# sharing:x:1002:
# create a user account just for sharing
sudo adduser shareuser
# follow the prompts to create user account password
# you can skip the input for these;
# Full Name []:
# Room Number []:
# Work Phone []:
# Home Phone []:
# Other []:
# log in as the new user to verify
su - shareuser
# as the new shareuser check your groups;
id
# or
groups
# log out with the 'exit' command or Ctrl-D
# as your primary user account
# add the shareuser to the sharing user group
sudo addgroup shareuser sharing
# also add your primary user account to the sharing group for convenience
sudo addgroup username sharing
# verify all the users in the group
sudo getent group sharing# first make a backup copy of the file!
cp /etc/samba/smb.conf smb.conf.bu
# edit the file
sudo nano /etc/samba/smb.conf
Add a section to the file that looks like this
[Media]
path = /mnt/media
valid users = @sharing
browsable = yes
writable = yes
read only = nowhere "Media" is the name of your share, and /mnt/media
is the path on the host system to the directory to share. The config
@sharing gives all users in the "sharing" user group access
to the share.
Once you have edited the file, you must test it to make sure there are no errors
testparm# change the group of the array disks to the new sharing group
sudo chgrp -R sharing /mnt/disk1
sudo chgrp -R sharing /mnt/disk2
sudo chgrp -R sharing /mnt/disk3
sudo chgrp -R sharing /mnt/disk4
sudo chgrp -R sharing /mnt/media # this one is the mergerfs array volume made from the other disks
# change the access permissions to allow for sharing group users to have read and write access
sudo chmod -R 755 /mnt/disk1
sudo chmod -R 755 /mnt/disk2
sudo chmod -R 755 /mnt/disk3
sudo chmod -R 755 /mnt/disk4
sudo chmod -R 755 /mnt/media
# optional: should probably do a umask setting to make sure that these permissions
# propagate automatically to all newly created subdirs of these volumes
# verify that the sharing user has access to the dirs
su - shareuser
# as shareuser;
touch /mnt/disk1/foo
mkdir /mnt/media/bar
touch /mnt/media/bar/foo
# log out of shareuser
# as primary user accountThe sharing user needs to also be added Samba, and enabled, and supplied with a password.
# add the user
sudo smbpasswd -a shareuser
# enable the user
sudo smbpasswd -e shareuser
# restart the samba daemon
sudo systemctl restart smbd
# check that your shares are active
sudo smbstatus --shares
# verify that your Samba user account is set up
sudo pdbedit -L -v# check that the SMB share is available
# insert the IP address to the other server here;
smbclient -L 192.168.1.2 -U shareuser
# test out the mount manually
# make sure that /mnt/media exists on the local system already with 'sudo mkdir -p' and that you have ownership of the dir first!! Before you run this
sudo mount -t cifs -o username=shareuser //192.168.1.2/Media /mnt/media
# if it worked then unmount it;
sudo umount /mnt/media
# edit your fstab
# good idea to make a backup copy first!
sudo nano /etc/fstabAdd an entry like this to your fstab
//192.168.1.2/Media /mnt/media cifs username=shareuser,password=putpasswordhere,uid=username,gid=username,vers=3.0 0 0shareuser here or you can save it to an external credential
fileuid to your primary
user account and the gid to your primary user account's
user group; this will apply those permissions to the mounted volume,
making it easier to interact with the volume (read and write) from
within your Docker container which are running under your primary user
accountTry to mount the new fstab
sudo mount -a
# if you get errors, check
sudo dmesgFinally, you should verify that it works;
touch /mnt/media/fooIf you start having permissions issues, you need to verify that the permissions are set correctly for all dirs and subdirs and files - from the SMB host "shareuser" on the individual mergerfs array disks - on the mergerfs array volume - on the SMB client's mounted volume
Make sure to propagate the SMB client UID and GID into your Docker containers correctly in the Docker compose file.
Make sure to mount the volume into your Docker containers with
/mnt/media:/mnt/media; preserving the exact same file path
from your SMB host to your SMB client to inside your Docker containers
will save you a lot of path-mapping headaches.
To verify that your docker containers have correct permissions, you can enter them and test it out;
docker exec -ti radarr bash
# inside the docker container
# check that you can see the mounted volume
ls -l /mnt/
# check that you can write to the mounted volume
touch /mnt/media/foo
# potentially repeat ^^^ for other subdirs as neededLighter than a virtual machine, heavier than a virtual environment, containers are a popular way to manage applications in an isolated manner that keeps apps' software installations from colliding with each other, and with the host operating system. This allows for greater portablity, reproducibility, and orchestration of applications on your system. Many types of containers can be shared freely on remote container repositories such as Docker Hub and Quay.io, and containers can usually be created in a scripted manner using a recipe file which lists all the commands needed to install and configure desired packages on top of a base operating system image.
Docker is the de facto standard for portable containers. Note that the term "Docker" can refer both to the container format, and to the engine which runs the containers; some other non-Docker container engines are compatible with Docker containers. Many containers are freely available on Docker Hub, and Docker containers can be (re)built using the Dockerfile recipe format. Multiple independent containers can be orchestrated at the same time with the Docker Compose plugin.
Resources
IMPORTANT NOTE: If you use ufw or firewalld to manage firewall settings, be aware that when you expose container ports using Docker, these ports bypass your firewall rules. For more information, refer to Docker and ufw.
Official docs here; https://docs.docker.com/engine/install/ubuntu/
# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
# Add the repository to Apt sources:
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
# install the latest version of Docker
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
# verify that its running
sudo docker run hello-worldIf you are having trouble getting Docker to run, you can do a couple things to investigate;
# try to start the docker daemon
dockerd
# check the status of the Docker service
systemctl status dockerIf Docker does not start automatically on system boot, you can also configure that like this;
sudo systemctl enable docker.service
sudo systemctl enable containerd.serviceDocker Compose is a plugin for Docker that lets you use a YAML file to define the config for multiple containers, and start and stop them all at once (or individually) as needed.
resources: - https://docs.docker.com/compose/ - https://www.howtogeek.com/devops/what-is-docker-compose-and-how-do-you-use-it/ - https://docs.linuxserver.io/general/docker-compose/
Make sure you installed compose when you installed
Docker
sudo apt-get install docker-compose-plugin
# verify that its installed
docker compose versionEach of the Wiki pages for the -*arr apps includes an example Docker Compose YAML. You can use a single YAML file with all of the services listed for easier management. See the link there for the guide's example YAML configuration.
in this example, the contents of a file named
docker-compose.yml look like this:
---
version: "2.1"
services:
radarr:
image: lscr.io/linuxserver/radarr:latest
container_name: radarr
environment:
- PUID=1000
- PGID=1000
- TZ=EST
volumes:
- /home/username/radarr:/config
- /media/share/Media:/media/share/Media
ports:
- 7878:7878
restart: unless-stopped
bazarr:
image: lscr.io/linuxserver/bazarr:latest
container_name: bazarr
environment:
- PUID=1000
- PGID=1000
- TZ=EST
volumes:
- /home/username/bazarr:/config
- /media/share/Media:/media/share/Media
ports:
- 6767:6767
restart: unless-stopped
lidarr:
image: lscr.io/linuxserver/lidarr:latest
container_name: lidarr
environment:
- PUID=1000
- PGID=1000
- TZ=EST
volumes:
- /home/username/lidarr:/config
- /media/share/Media:/media/share/Media
ports:
- 8686:8686
restart: unless-stopped
sonarr:
image: lscr.io/linuxserver/sonarr:latest
container_name: sonarr
environment:
- PUID=1000
- PGID=1000
- TZ=EST
volumes:
- /home/username/sonarr:/config
- /media/share/Media:/media/share/Media
ports:
- 8989:8989
restart: unless-stoppedPUID and PGID values match the values
given by the id command for my local server user
account/config inside the container, which is the
default location inside the container for configs/media/share/Media to the
same address inside the container, for all containers, for consistency
(optional)ufw) so
these ports will be accessible on the local networkStart the containers with the command
docker compose up -dThis starts all containers and runs them in a detached state in the background.
Check on which containers are running with
docker compose ps(you can also use docker ps but its less
descriptive)
Stop all the containers with
docker compose downYou can also start / stop / restart containers individually
docker compose up lidarr -dOnce the containers are running, you can navigate to your web browser
to access them. Navigate to a URL that looks like
http://192.168.1.123:8989/, inserting the local IP address
of the server running the services.
See the guides at LinuxServerIO and Servarr Wiki and TRaSH Guides and the official docs for each service for further setup instructions for each service.
You can use the command line tool lftp to interact with
FTP servers in order upload and download files, and even mirror entire
directories.
lftp home page is here; https://lftp.yar.ru/ - docs here; https://lftp.yar.ru/lftp-man-alt.html
To install lftp, on macOS you can use Homebrew
(brew install lftp), and on Ubuntu you can use the package
manager (sudo apt install lftp). Once installed, you can
see the tool's internal documentation with man lftp.
Check the man pages for lftp to see the full details of commands, settings, and more.
lftp can be used interactively, or non-interactively,
from the command line. In interactive mode, lftp will start a new
session inside your terminal connecting you to the remote FTP server.
From there you can use a handful of commands to browse the FTP file
system and upload / download files and directories, among other
things.
You can connect to an FTP server with a URL in a format like this:
ftp://username:password@server.comwhere username and password are the
username and password you plan to log in with; you can omit the password
and enter it at the interactive prompt if you like, though it might be a
good idea to keep the username in the URL so it does not try to
automatically log you in as a user you did not intend for. If the
username or password contains special characters that your shell does
not like, you may need to wrap some or all of the URL in single (') or
double (") quotes.
The full command should look like this
lftp ftp://username:password@server.comand if it works you should be presented with an interactive prompt
lftp username@server.com:~> pwd
ftp://username:password@server.com
lftp username@server.com:~> ls
drwxrwxrwx 1 user group 0 Jan 01 1970 ..
drwxrwxrwx 1 user group 0 Jan 01 1970 .
drwxrwxrwx 1 user group 1 Dec 07 20:52 Books
drwxrwxrwx 1 user group 1 Nov 27 22:01 Movies
drwxrwxrwx 1 user group 1 Dec 07 18:04 TV
lftp username@server.com:/>lftp has its own set of commands that mimic (but are NOT
100% equivalent to) your standard POSIX shell commands, such as
ls, pwd, and cd.
Importantly, unlike a normal shell, lftp tracks both
your local pwd and your remote pwd,
and uses both values (often implicitly) in its commands. By default,
your local pwd will be the directory on your local system from where you
started the lftp session. You can check your local pwd by
prefixing the command with the local keyword
lftp username@server.com:/> local pwd
file:/Volumes/Media/TV/
lftp username@server.com:/> local ls
total 38132360
drwxr-xr-x 4 username staff 128 Dec 4 14:23 The Big O
-rw-r--r--@ 1 username staff 1170119875 Oct 31 2018 batman.mkvYou can upload a single file with the put command. The
default behavior is to upload a file from your local pwd to the remote
pwd with the same filename
put batman.mkvThis will upload the file batman.mkv from your
local pwd to the remote pwd. You can
change the filename on the remote side with the -o flag
put batman.mkv -o batman2.mkvThe easiest way to upload an entire directory is with the
mirror command;
mirror sourceRemoteDir targetLocalDir
By default the source is remote and the target is a local
directory. When using -R, the source directory is local and
the target is remote. If the target directory is omitted, base name of
the source directory is used. If both directories are omitted, current
local and remote directories are used.
mirror -R "The Big O"will upload the entire local directory "The Big O" to the remote pwd, with the name "The Big O"
If you instead ran the command
mirror "The Big O"it would attempt to download the entire remote directory "The Big O" to your local pwd
a command like
mirror remoteDir/ /path/to/localDir/would download the contents of remoteDir to the local
path /path/to/localDir/
It can be a little confusing to remember which direction the mirror
will run, so you can preview the transfer without running it by
including the --dry-run argument.
You can run lftp from the command line non-interactively
with the -e and -c flags.
A command like this
lftp -c "pwd; ls" ftp://username:password@server.comwill log into the server, run the pwd and
ls commands, then exit.
You can use -e to instead drop you into an interactive
session after running the commands. This is really useful for example in
situations where you need to pass in some settings in order for your
login to work, such as for disabling SSL verification on servers with
self-signed certificates
lftp -e "set ssl:verify-certificate false; pwd" ftp://username:password@server.comThis will start a session on the server with SSL verification
disabled, print your remote pwd, then enter the interactive
session.
Official Raspberry Pi background docs
Docker Installation guide: https://docs.docker.com/engine/install/debian/ - since Pi is using a Debian based OS the steps are slightly different from the Ubuntu steps I listed previously
Copied verbatim from official docs;
# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
# Add the repository to Apt sources:
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null# Install the Docker packages.
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugindont forget these steps
sudo systemctl status docker
sudo usermod -aG docker ${USER}
sudo reboot
# check that it works
docker run hello-worldSome example Docker compose files
You may need to adjust the router config in a couple places to ensure that the Pi Hole is being used for DNS. You may also need to flush the DNS cache on client devices and/or renew DHCP leases for the new DNS settings to propagate
Dont want a physical server living in your house? Consider using the cloud! You can spin up a virtual private server (VPS) easily with providers such as AWS and Digital Ocean, among others. Cloud is a great option for applications that need high-availability.
Want a general purpose cloud based server for cheap? Try some of these options!
Unlike torrenting, which is peer-to-peer and essentially free, for Usenet you typically pay fees to a provider service and also maybe for your indexer account too. The fees allow you to access and download whatever you want. There is no "seeding" required. You are paying the platform providers directly for access. Many private indexers will have paid memberships, and may or may not have open account registration.
In order to download stuff, you will need accounts with both a provider and an indexer. Then you will get an NZB file (similar to the .torrent file you may be used to) and load that into your NZB downloader app (SABnzbd)
You need three things to use Usenet to pirate shit
indexer (the search engine website you use to find stuff to download)
provider (the service that provides you the file to download)
newsreader application (to download NZB files which point to the files you want to download)
The Indexer is a search engine that you will use to search items to download. Similar to how you use a torrent website to find torrents, you use an indexer's website (or API) to find content.
You can use the indexer website interactively directly and download .nzb files (conceptually similar to .torrent files) which you load into your newsreader app to download the files (over your provider service). If you use services such as Sonarr / Radarr / Lidarr, etc., you do not even need to access the indexer site directly, instead you can configure those services to search (multiple) indexers for you and automatically download and import your desired movies / tv shows / music.
Some popular Indexers include;
Ninja Central ninjacentral.co.za
NZBGeek nzbgeek.info
NZB Planet nzbplanet.net
Drunken Slug drunkenslug.com
AltHub althub.co.za
NZB Finder nzbfinder.ws
Miatrix miatrix.com
prices for private indexers are in the ballpark of $15/year for basic access, some have free accounts (with limitations, example Ninja Central has a login requirement for free accounts), and many also offer lifetime memberships too which can be a great deal.
find info on more indexers here:
The provider is the server which is holding the data you want to download. Providers will attempt to mirror each others' content, with varying levels of success, and varying levels of retention time for old files. In order to download from a provider you generally must pay for access.
Prices are in the ballpark of $3 - $10/month (USD) or less, and you can purchase longer subscriptions for different discounts.
some providers will also let you purchase "blocks", which is basically purchasing a set amount of data (e.g. 500GB or 2TB or w/e) that you can use to download at any point in the future, without a set time-period to use it in.
From: someuser at 2023-11-26 20:14:54
tc block accounts end up being way cheaper than a monthly sub
pick up two different block accounts that are on two different providers, that way you have better completion. (but even with multiple providers, networks like HBO are so good at takedowns that it can be really hard to watch their shows)
unless you're downloading like 300+gb a month you come out way ahead vs $3-6 a month
Some popular Providers (some of these have gimmick pricing deals as well)
Newshosting https://www.newshosting.com/
Eweka https://www.eweka.nl/en
Frugal https://frugalusenet.com/
UseNight https://usenight.com/en/
Maximum Usenet https://www.maximumusenet.com/
more info here; https://www.reddit.com/r/usenet/wiki/faq/#wiki_providers
you can read more about some popular ones that are having Black Friday Deals here as well https://www.reddit.com/r/usenet/comments/17noby8/2023_black_friday_deal_thread_providers_and/
there's a list of some more popular providers here; https://usenetreviewz.com/best-usenet-providers/
some more provider lists; https://www.reddit.com/r/usenet/wiki/providerdeals/
Provider Map - Usenet provider services are dominated by a few large companies and a lot of smaller companies, you might want to select two or more providers on different backbones (or not if your current provider is not giving you issues) - https://www.reddit.com/r/usenet/wiki/providers/
You need a newsreader app to download stuff from Usenet. Just use this one:
You will configure your newsreader app with the login credentials given by your provider after you sign up for a provider plan.
The same way that a torrent client loads a torrent from a .torrent file or magnet URL, the newsreader loads a file for download from the .nzb file provided by the indexer. It then uses that file to direct itself to the data for download from the configured providers of your choice.
If you are use Sonarr / Radarr / Lidarr / etc., you will also be able to link your newsreader (SABnzbd) with these services, in order for the services to automatically search for movies / tv shows / music / etc. for you and queue them up for download by SABnzbd, and then automatically import them to your media library
(manual)
manual method is rarely needed, only need to log in to some indexer websites occasionally to maintain your account. Otherwise use the automatic method
(automatic)
If you want to start automating your downloads, you can connect them to Sonarr / Radarr / Lidarr / etc. too, so you dont actually have to search the indexers interactively and manually download .nzb files, you can connect these services to your Indexers and your download client (SABnzbd)
Super easy, and fast download speeds too. No messing with ratios, seeding, or asshole elitist private torrent tracker admins. Dont need a seedbox either. Just sign up, install a newsreader, and start downloading stuff. Also bonus, a lot of providers and indexers have PayPal so its super easy to pay for subscriptions too.
Usenet is cool!
tl;dr: peer-to-peer file sharing where people sharing pieces of files with each other until everyone has all the pieces needed to assemble the completed file(s)
You need two main things to torrent;
a tracker website to direct you to the torrent peers
a torrent download client
Since your IP address is publicly visible while torrenting, if you are downloading illegal content then you should use a VPN with a reliable killswitch, and/or a dedicated seedbox on someone else's server.
There are lots of resources on the internet for information about torrenting
By default, all of the ports used on your computer for networking are
typically no accessible outside of your local network. For example, if
you have a program on your PC such as a torrent download client that
listens on port 1234 for communication with other programs,
that port will only be reachable by other computers on your network.
Internally, your PC's local IP address may be something like
192.168.1.2, and the port would be accessed at an address such as
http://192.168.1.2:1234.
However, if your public IP address is something like
200.100.9.23, your PC's port would usually by default not
be reachable at an address such as
http://200.100.9.23:1234. This is partially due to the
public IP being assigned at the router-level, not the device level, so
all devices on your network would typically share the same public IP
address.
In order to get a public address such as
http://200.100.9.23:1234 to point back to your PC's
internal port 1234, you use port forwarding. It is a
configuration that you apply inside your router's settings that allow an
internal port on a local device to be reached publicly.
More information;
It is possible to download torrents without configuring port forwarding, but you may experience poor performance for downloads and uploads.
https://www.youtube.com/watch?v=1YDVebJlGbM
Port forwarding on its own is not unsafe. Any potential security risks would be placed on the software listening on the ports. So make sure that any software listening on a publicly accessible is kept up to date. You may even consider using something like a Docker container to try to further isolate it from the host system.
If a computer does not have any application listening on a particular port, then any attempts to communicate on that port will simply be left with no response.
will update with more info later
A lot of information can be found in publicly available FAQ threads and websites;
the private tracker Bible https://wiki.installgentoo.com/wiki/Private_trackers
https://www.reddit.com/r/trackers/comments/tw4ji0/tracker_faq_and_recommended_sites/
Tracker IRC channels https://torrentinvites.org/f96/list-torrent-trackers-irc-channels-65589/
"climbing the ladder" of trying to reach the "holy grail" of PassThePopcorn tracker
how to find trackers with open invites
A "seedbox" is just a virtual server instance that you can rent from a company for the purpose of downloading and seeding torrents.
You can purchase a seedbox with a monthly subscription the same way you would subscribe to any other digital service (e.g. YouTube Premium, etc.).
Once you sign up for a seedbox, you will be provided access to a small server where you can run download clients for torrents, among other things. The "seed" part of the seedbox comes from the fact that these server instances typically have very fast internet connections, making them ideal for seeding torrents.
You may or may not be provided access to log into the seedbox server
directly over ssh ("root access") but most seedbox providers will have
available various apps you can install easily on the seedbox, especially
torrent download clients such as rtorrent +
rutorrent.
A handy list of some seedbox providers can be found here; https://www.reddit.com/r/seedboxes/comments/140pyv8/cheap_seedbox_table/
| Website | Cost £ | Disk Space (GB) | Max Band |
|---|---|---|---|
| https://dediseedbox.com/ | £ 8.60 | 750 | Unlimited |
| https://evoseedbox.com/ | £ 4.02 | 150 | 2 TB |
| https://www.feralhosting.com/pricing | £ 10.00 | 1000 | Unlimited |
| https://giga-rapid.com/products/gigabox | £ 5.98 | 1000 | 6TB |
| https://www.sonicbit.net/ | £ 3.82 | 100 | |
| https://www.seedboxco.net/ | £ 7.87 | 1000 | Unlimited |
| https://www.seedhost.eu/shared-hosting.php | £ 5.16 | 2000 | 3TB |
| https://ultra.cc/ | £ 4.05 | 1000 | 2 TB |
| https://www.seedboxws.com/configurator | £ 1.71 | 100 | Unlimited |
| https://www.seedboxws.com/configurator | £ 13.67 | 1000 | Unlimited |
| https://www.seedboxws.com/configurator | £ 2.99 | 200 |
Reddit also has a nice wiki on seedboxes; https://www.reddit.com/r/seedboxes/wiki/index/what_is_a_seedbox/
Some torrent trackers such as TorrentLeech and Avistaz may also have deals that will allow you to get invites to their trackers if you sign up for a seedbox with their promotional providers such as Seedit4me (https://seedit4.me/) or RapidSeedbox (https://www.rapidseedbox.com/), and then retain the seedbox for at least 1 month. This can be a convenient way to gain access to such trackers when they are not currently holding open registration.
Once you have access to your seedbox, you can load its torrent download client in the web browser and start adding torrents as normal.
Be sure to read the fine print on your seedbox carefully. For example, some seedboxes may or may not allow the use of Public Trackers (such as The Pirate Bay). There may be other restrictions as well.
To download files from your seedbox to your local system, you can use a couple different methods;
some seedboxes will include a web browser based File Manager that allows file downloads
you can use a program such as WinSCP (https://winscp.net/eng/index.php) or CyberDuck (https://cyberduck.io/) to download files from the seedbox over FTP, or SFTP (requires ssh access)
if you have ssh access you can use the standard
rsync program to download from your seedbox with a command
such as
rsync -vrthPz -e 'ssh -p1234' seedboxUsername@seedboxServername.com:/home/seedboxUsername/torrents/ /path/to/local_torrents/where 1234 is the port forwarded for ssh access on your
seedbox and you fill in the seedbox's local username, and either IP
address or hostname. rsync is a good choice here because it
supports resuming partial downloads in case your network connection
breaks, and it also has --bwlimit args to control download
speed (e.g. --bwlimit 1000 limits to 1MB/s download) so you
dont saturate your home network connection if you need to do other work.
Suggested to keep rsync running in a screen
session on your file server to avoid interruptions.
If you are downloading and seeding torrents from public trackers on your home computer, you will likely want to use a VPN.
You can find recommendations on the best VPN's for torrenting from a basic Google search; e.g. https://www.techradar.com/vpn/best-vpn-for-torrenting
A VPN is important for use with public trackers because your IP address will be seen publically by all torrent downloaders including any movie-industry folks trying to send Cease & Desist letters to pirates' ISPs.
If you are using a VPN for torrents, there are some key considersations;
try to get a "no logs" VPN provider
make sure your VPN software has a robust "killswitch" that prevents any network traffic from your device while the VPN is not running. For example, there may be brief periods of time when the VPN software is refreshing its own network connection that could accidentally let some network traffic slip through unprotected
I have used PIA (https://www.privateinternetaccess.com/) in the past with good results. Your mileage may vary.
Its commonly believed that if you use private trackers, you will not need a VPN due to the barrier of entry for DMCA-types to gain access, and many report having torrented for years without VPN using private trackers without issues.
If you have a seedbox, you likely do not need to use a VPN, since most seedboxes are going to be based in countries outside the USA's jurisdiction such as EU countries, and the seedbox may already have its own VPN configured. Check the details on your seedbox to be sure.
VPN's can be useful for other purposes, such as spoofing your global location on the internet so that you can gain access to region-locked content. It can also give you some extra security if you are using a public Wifi or network.