Starbeamrainbowlabs

Stardust
Blog


Archive


Mailing List Articles Atom Feed Comments Atom Feed Twitter Reddit Facebook

Tag Cloud

 3d 3d printing account algorithms android announcement architecture archives arduino artificial intelligence artix assembly async audio automation backups bash batch blender blog bookmarklet booting bug hunting c sharp c++ challenge chrome os cluster code codepen coding conundrums coding conundrums evolved command line compilers compiling compression conference conferences containerisation css dailyprogrammer data analysis debugging defining ai demystification distributed computing dns docker documentation downtime electronics email embedded systems encryption es6 features ethics event experiment external first impressions freeside future game github github gist gitlab graphics guide hardware hardware meetup holiday holidays html html5 html5 canvas infrastructure interfaces internet interoperability io.js jabber jam javascript js bin labs latex learning library linux lora low level lua maintenance manjaro minetest network networking nibriboard node.js open source operating systems optimisation outreach own your code pepperminty wiki performance phd photos php pixelbot portable privacy problem solving programming problems project projects prolog protocol protocols pseudo 3d python reddit redis reference release releases rendering research resource review rust searching secrets security series list server software sorting source code control statistics storage svg systemquery talks technical terminal textures thoughts three thing game three.js tool tutorial twitter ubuntu university update updates upgrade version control virtual reality virtualisation visual web website windows windows 10 worldeditadditions xmpp xslt

Learn your terminal (or command line)

Enter stage left: the terminal (or command line, on windows). That window with strange white text on a black background. You might not see it, but every operating system has one - humming away in the background, just waiting to be used, but epic arcane skills are needed to navigate this bizarre and perhaps dated window into your computer.... or so it seems.

When you think of your computer, you will probably think of a GUI (a.k.a. goo-ey), with windows, a cursor, and perhaps a few buttons. GUIs make it easy for newcomers to easily find their way around a computer by referencing things that exist in the real world (e.g. folders and files, a floppy disk on the save button, etc.), but they can be inherently slower to use - especially for long series of perhaps repetitive tasks that stay essentially the same.

A terminal (linux and friends) or a command line (windows) is another view into your computer. It's a way of controlling your computer with text. Text that follows particular set of rules, that can be saved and repeated at will through the use of scripts. It's built on commands, each of which does one thing and one thing well. On their own they're mildly useful, but together they form a powerful framework that can perform almost any task. It's certainly different (and there's a little bit of learning curve, to be sure), but not as hard or arcane as you might think it currently.

A knowledge of the terminal or command line on your computer can be rather useful - especially so for those involved in computer science or technical support. How long would it take you to flatten a large set of deeply nested folders with a GUI? Or convert and recompress few folders worth of videos? Or even renew all your ssl certificates on your web server? All of these things can be automated through the use of a terminal or command line.

Even if you're just a casual computer user who's not into programming, it's still worth at least looking into. Perhaps it'll save you some time! Perhaps it'll save you from asking your friend where something is on their computer when you can't find it. Maybe it'll even save you if your computer suddenly decides it doesn't want to boot up properly. And you'll look cool doing it too :P (What better reason is there?)

If I've somehow managed to convince you to dive in and take up the challenge learning, then I'll end this somewhat different post with a collection of places you can go to get started.

Running Prolog on Linux

The learning prolog banner. Hello! I hope you had a nice restful Easter. I've been a bit busy this last 6 months, but I've got a holiday at the moment, and I've just received a lovely email about my learning prolog series. It's reminded me about how much I actually rather like (swi) prolog (I'm a bit nuts, I know :P), and I've decided that it would be splendid if I could get back into it a bit.

I'm not sure whether it'll go anywhere, but I'm going to look into web crawling and parsing HTML. Anyway, since I'm re-discovering prolog, I had a teensy bit of trouble getting prolog to run one of my old scripts just now, so I thought I'd blog about it to save me the trouble next time :D

Running prolog scripts on linux isn't actually that hard. First, you need to install SWI-Prolog:

sudo add-apt-repository ppa:swi-prolog/stable
sudo apt install swi-prolog-nox

Then, you run a prolog script like this:

swipl -f ./Awesome-Prolog.pl
Welcome to SWI-Prolog (threaded, 64 bits, version 7.4.1)
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software.
Please run ?- license. for legal details.

For online help and background, visit http://www.swi-prolog.org
For built-in help, use ?- help(Topic). or ?- apropos(Word).

?-

...then you can interact with your Prolog knowledge bases as normal. When you want to exit, the easiest way I've found is if you press CTRL + D.

If, however, you're lazy and don't want to type prolog -f AwesomeSauce.pl out every time you want to run your script, you can prefix your file with the following shebang:

#!/usr/bin/prolog -f

Make sure it's the very first line in the file. Prolog appears to ignore it and carry on regardless. Then you can run it like this (making sure you've chmod +x'd it first:

./CoolBot.pl

Fancy message of the day over SSH

Since my time to sit down for a good chunk of time and write some code has been extremely limited as of late, I've been playing around with a few smaller projects. One of those is a fancy message of the day when you log into a remote machine (in my case the server this website is hosted on!), and I thought I'd share it here.

My take on a fancy SSH message of the day.

The default message shown at the top when you login via ssh is actually generated by something called update-motd, and is generated from a set of scripts in /etc/update-motd.d. By customising these scripts, we can do almost anything we like!

To start off with, I disabled the execution of all the scripts in the directory (sudo chmod -x /etc/update-motd.d/*), and created a subfolder to store the script in that actually generated the system information (sudo mkdir /etc/update-motd.d/parts). Here's the script I wrote to generate the system information:

#!/usr/bin/env bash

. /etc/lsb-release

LOAD=$(cat /proc/loadavg | cut -d' ' -f 2);

CPU_COUNT=$(cat /proc/cpuinfo | grep -i "core id" | uniq | wc -l);
THREAD_COUNT=$(cat /proc/cpuinfo | grep -i "core id" | wc -l);

APT_UPDATE_DETAILS="$(/usr/lib/update-notifier/apt-check --human-readable | fold -w 40 -s)"

IPV4_ADDRESS=$(dig +short myip.opendns.com A @resolver1.opendns.com)
IPV6_ADDRESS=$(dig +short myip.opendns.com AAAA @2620:0:ccc::2);

LAST_LOGIN=$(last -1 | head -n 1 | awk '{ print $1,"at",$4,$5,$6,$7,"from",$3 }');

REBOOT_REQUIRED=$(/usr/lib/update-notifier/update-motd-reboot-required);

echo 
echo Welcome to $(hostname)
echo "  running ${DISTRIB_DESCRIPTION}"
echo 
echo Kernel: $(uname -r)
echo Uptime: $(uptime --pretty | sed -e 's/up //')
echo Load: ${LOAD}
echo 
echo IPs: ${IPV4_ADDRESS}, ${IPV6_ADDRESS}
echo 
echo "${APT_UPDATE_DETAILS}"
echo 
echo "${REBOOT_REQUIRED}"
#echo 
#echo Last login: ${LAST_LOGIN}

exit 0

Basically, I collect a bunch of information from random places on my system (several of which were taken from the existing scripts in /etc/update-motd.d/) and re-output them in a different format.

Then, I converted an image of my favicon logo with the brilliant catimg by posva to a set of unicode characters and sent that to a file (catimg -w 35 image.png >/etc/update-motd.d/sbrl-logo.txt) - you could alternatively use some ascii art from the internet (e.g. this site). Once done, I put the two together with the following script directly in my /etc/update-motd.d/ folder:

#!/usr/bin/env bash

### Settings ###
TMP_FILENAME=/run/sysinfo.txt

#/etc/update-motd.d/parts/sysinfo

################

/etc/update-motd.d/parts/sysinfo >$TMP_FILENAME

### Output ###
echo 
pr -mtJ /etc/update-motd.d/sbrl-logo.txt $TMP_FILENAME

##############

### Cleanup ###
rm $TMP_FILENAME

###############

Finally, I manually cleared and regenerated the message of of the day with sudo update-motd, giving the result you see at the top of this blog post. I also made sure to re-enable the execution of the other scripts I didn't use in my fancy motd so as to not miss out on their notifications.

If you're interested, I've generated an archive of my final /etc/update-motd.d folder (minus my logo in text format), which you can find here: 20170203-Fancy-Motd.7z.

Can you do better? Got a cool enhancement of your own? Post about it below!

Importing your friends' public keys automatically with gpg

Today's post is just a quick one, as I've had a rather busy week that I'm now recovering from :)

Several times over the past few weeks I've been finding myself googling around to try and find a way to download someone's GPG / PGP public key and import it into GPG automatically, so that I can verify their GPG signatures. I'm posting the command here so that I don't have to keep looking for it :D

Here's the command to search for a someone's public key:


gpg --keyserver pgp.mit.edu --search [email protected]

The above asks the key server at pgp.mit.edu for GPG keys associated with the email address [email protected]. You don't actually have to provide the keyserver - GPG will default to searching keys.gnupg.net.

Sources and further reading

A Script to update Node.js to the latest version

As you may be aware, io.js and Node.js have merged into one project once again. While this is good news for the community, it also means that those of us using a bash script to update io.js to the latest version (like myself) need to find another method to stay up to date.

I found nvm, but I found it really didn't work with my current setup on my server. I took a look at the iojs-update script I found online that I've been using, and I found that it wasn't too tough to reconfigure for Node.js. Now that it's working, I thought that I'd post here so that you can use it too. I'm using it with Ubuntu 15.04, but it should work with any Debian based system.

I've forgotton the original author's name, but if you post in the comments, I'll credit you appropriately in this post.

Here's a link to the gist: https://gist.github.com/sbrl/48e6423f2e97462149ef

If you just want to download and run it, here's a command to download it to your ~/bin folder and execute it:

cd ~/bin && curl -OL https://gist.github.com/sbrl/48e6423f2e97462149ef/raw/9bf780f8685634dbaae6e0a229d2ad551f3323ed/node-update && chmod +x node-update && sudo ./node-update

If someone knows of an up to date ppa, I'll gladly use that instead.

I hope this helps someone out!

Custom Brightness Controller for Ubuntu

While I love Ubuntu's desktop, the brightness and volume controls are rather annoying as they don't provide a great degree of control. To fix this, I wrote a bash script to control the brightness. Here's what I came up with:

#!/usr/bin/env bash
increment=10
backlight_prefix=/sys/class/backlight/intel_backlight/

echo $1
if [[ "$1" = "decrease" ]]; then
    echo decreasing
    increment=$(expr -${increment})
fi

cur_brightness=$(cat ${backlight_prefix}brightness)
max_brightness=$(cat ${backlight_prefix}max_brightness)

new_brightness=$(expr ${cur_brightness} + ${increment})

# Permissions changes on brightness: 
## change group to sbrl
## add g+w
# Old command:
#gksudo -- bash -c "echo ${new_brightness} >${backlight_prefix}brightness"
echo ${new_brightness} >${backlight_prefix}brightness

####################
### Notification ###
####################
### uncomment the following line to disable the notification
#exit
# Calculate the percentage
new_percent=$(echo "(${new_brightness} / ${max_brightness}) * 100" | bc -l)
new_percent=$(printf "%.1f" "${new_percent}")

echo new_percent: $new_percent

max_bar_length=100
bar_length=$(echo "(${new_percent} / 100) * ${max_bar_length}" | bc -l)
bar_length=$(printf "%.0f" "${bar_length}")

n_bar=$(head -c $bar_length < /dev/zero | tr '\0' '=')

# Kill the previous notification
killall notify-osd
notify-send "Brightness: ${new_percent}%" "${n_bar} (${new_brightness})"
#notify-send "Brightness" "${new_percent}%"

(Pastebin, Raw)

To use the above, you need to do several things. Firstly, you need to find your screen's brightness settings. Open a terminal, and navigate to /sys/class/backlight, and find your backlight's folder. Mine is intel_backlight, but yours might acpi_video0. Once found, you should have a file called brightness inside it. Change the value of the backlight_prefix variable on line #3 to equal the path to this folder, not forgetting the trailing slash.

You then need to alter the permissions on the brightness file in order to allow your user account to change it - otherwise you will get prompted for your password every time you change your brightness! To do this, open a terminal and navigate to the folder we found earlier that contains the brightness file. Change the user group to be your username with sudo chgrp username brightness, and then allow write access to group members with sudo chmod g+w brightness. If this doesn't persist across reboots, you might need to add these commands to your rc.local or Xsession files.

It should work now. If you don't want the notification to show every time you change your brightness (or if it doesn't actually work), uncomment line #27.

Securing a Linux Server Part 1: Firewall

Welcome to a new tutorial series, where I will show you what I have learnt so far about making sure that your linux server (and desktop too!) are secure so that nobody can get in (easily) and assume control.

Disclaimer: This tutorial series will not cover everything, and should not be taken to. There probably will be some mistakes in this post too. Check other guides online or consult a professional to make sure that your machine is secure. Please suggest improvements or point out mistakes in the comments.

To start this tutorial session off, I will talk about firewalls. Firewalls control how data is allowed to travel in and out of your computer. In Ubuntu, a firewall called ufw, the 'uncomplicated firewall' is already present. It acts as a nice frontend to iptables, which I find to be difficult to understand and use. We will be using that as our firewall.

I have done an asciinema recording on a virtual machine of this whole process:

Enabling the firewall

Ufw by default allows all outgoing connections and denys all incoming connections. This means that if you are using ssh to connect to your server, you will need to open the appropriate ports first before enabling ufw. Do that like this:

~$ sudo ufw allow 22/tcp

Ufw will automatically configure iptables to allow incoming connections on port 22 that use tcp. I will talk more about allowing and denying different connections later.

Just in case ufw blocks your ssh connection and you are unable to get back in, you can use another program called at to schedule the disabling of the ufw so that you can get back in again. If you don't have it installed, you can install it with sudo apt-get install at.

~$ sudo at -vM now +10 minutes
ufw disable
^D

Where ^D stands for CTRL + D. Now that you have it set such that ufw will disable itself in 10 minutes time, we go ahead and turn ufw on:

~$ sudo ufw enable

It will warn you that this may disrupt any existing ssh connections you have open. Reply yes to this. Once it have been enabled successfully, you should check that you can still ssh into your server (if that is the method that you are using to control it). If yes, great! If not, ufw will disable itself in 10 minutes and then you can try again.

Now that we have ufw enabled, we can cancel the at job we created to disable ufw. Type sudo atq to list the jobs you have schedules, and sudo atrm <number> to remove it, where <number> is the number of the jobs that you want to delete.

You may also want to cheeck the status of ufw to make sure that it is enabled, or to get a list of the rules that are currently in force. You can do that like this:

~$ sudo ufw status
Status: active

To                         Action      From
--                         ------      ----
80/tcp                     ALLOW       Anywhere
80/tcp (v6)                ALLOW       Anywhere

Allowing connections

Allowing connections through the firewall is easy. Simply type something like this:

~$ sudo ufw allow 80/tcp

Ufw will automatically configure iptables, in this example, to allow all connections on port 80 that use tcp. It will also configure it appropriately for both ipv4 and ipv6. Replace 80 with the port number you want to allow, and tcp with udp if needed. Ufw also understands several protocol names, and can configure itself accordingly:

~$ sudo ufw allow http
~$ sudo ufw allow imap

Denying connections

Denying all connections on a given port is very similar., Simply type something like this:

~$ sudo ufw deny 4722/tcp

The above would deny all tcp connections on port 4722.

You can also prevent a particular ip from gaining access to your server:

~$ sudo ufw deny from 123.123.123.123

The above would block all packets from the ip address 123.123.123.123. It works with IPv6 addresses too:

~$ sudo ufw deny from 2607:f8b0:4003:c05::65

The above would block all packets from the ip address 2607:f8b0:4003:c05::65, which just happens to belong to Google.

Port Ranges

You can open a range of ports with a colon:

~$ sudo ufw allow 60000:61000/udp

The above will allow udp connections on any port in the range 60,000 - 61,000 (the ports used for mosh).

Deleting Rules

Deleting rules can be done like this:

~$ sudo ufw delete allow 4724/tcp

The above would delete the rule(s) allowing tcp connections on port 4724.

Summary

In this post, I have shown you how to activate and configure a simple firewall that is bundled with Ubuntu. Next time, I will talk about securing you ssh daemon.

If you spotted a mistake in this post, have a suggestion, or are having trouble following along, please leave a comment below.

Other useful posts

These posts helped me to understand and use the uncomplicated firewall:

Sending POST Requests with curl

This is a quick post about sending POST requests via the command line. I have been using curl for a while now - and I find it to be a good alternative to wget. I have just found out how to use it to send a POST request from the command line, and I thought that I would share my findings here.

Sending a POST request is really quite simple:

curl -X POST --data-binary "@$input_filename" -o $output_filename $url
  • $input_filename is the name of the filename that contains the data that you want to send.
    • @- can be used to specify stdin, allowing you to pipe the output of the previous command into curl.
    • You can also drop at @ symbol and hard-wire the data you want to send into the command itself.
  • $output_filename is the name of the file you want to save the response to. You can drop -o $output_filename if you want curl to output he result to he standard output for further processing.

If you find that you get some king of error message from the above, you may need to add --header "Expect: 100-continue" just before the $url - some servers require this.

The --data-binary is rather important, as if you use --data on it's own, line breaks are not preserved for some bizarre reason.

I am finding that curl is much more powerful that I first expected, as it understands just about any protocol you care to name....I need to experiment with it further.

Curl also has a modular structure to it's command line arguments, so you can tack and extra setting on the end and it will work exactly as you would expect it to (most of the time!).

Terminal Reference Sheet

While browsing my stackedit markdown documents, I found a reference sheet for the linux terminal. While originally posted on novitiate, I thought that it might be of use to someone on this website too. The reference sheet was originally designed for the raspberry pi, but it works for other bash like shells too :)

You can find it below.

Basic Linux Terminal Reference Sheet

Command Function
ls List the contents of the current directory
cd orange Change the current directory to the sub-folder orange
cd .. Change the current directory to the parent folder
cd ~ Change the current directory to your home folder
rm banana.txt Delete the file banana.txt
rmdir pineapple Delete the directory pineapple, but only if it is empty
rm -rf mango Delete the directory mango, even if it has files and folders inside it
mkdir cheese Create a new folder called cheese
mv kiwi.py cake.py Rename kiwi.py to cake.py
cp bread.7z cherry.7z Copy bread.7z to cherry.7z
mv cabbage oranges/tangerines Move the folder cabbage to the folder tangerines inside the folder oranges
sudo command Execute command as the root (administrator) user
sudo apt-get update Update the package list
sudo apt-get upgrade Update the currently installed packages
sudo apt-get install foo Install the package foo
rpi-update Update the Raspberry Pi's firmware1
raspi-config Start the Raspberry Pi configuration utility
nano apple.txt Edit the file apple.txt in the nano command line text editor
exit Exit the terminal
sudo shutdown -h now Shutdown
sudo reboot Restart
top View a list of processes using your CPU most intensively
CTRL + C Keyboard shortcut to exit most programs
CTRL + Z Keyboard shortcut to suspend the currently active process and return focus to the terminal
fg Resume the currently suspended process
jobs Display a list of jobs attached to the current terminal
bg 2 Put the job with the id 2 into the background2
disown 2 Disown the job with the id 2 so that it will not be terminated when the terminal is terminated2
help | less Pipe the output of the command help into the command less, which lets you scroll through large amounts of output with the arrow keys (press q to quit).
cat cakes.log | View the contents ofcakes.log. Tip: add lessto pipe it intoless` so that you can browse the file with the arrow keys.
man less Display the manual page about the command less
help less Display some basic help about the command less
ssh [email protected] Connect to the device at the IP 192.168.0.3 with the username pi and start a remote terminal session

Links to other useful lists

If you have any other commands or links to useful lists / references, please leave a comment below.

Have you found a cool bash reference? Post a comment below and I will consider adding it to the list.

  1. This needs installing first. See this page: https://github.com/Hexxeh/rpi-update ↩

  2. Note that to specify job #1, you type the command on it's own without the job id. ↩ ↩

Art by Mythdael