After setting up my Raspberry Pi as a NAS, I wanted to set up backups that are easy to run and check on. Initially, I wanted to set them up to run automatically, but another goal of the Pi set up was for me to turn my PC off more often. I’m currently thinking that if I’m going to turn the PC off, I will just run the backups manually since turning my PC on every Saturday night (or whenever I’d set it to run) isn’t really automated. If I go back on this and set up a cron job I’ll be sure to post about that as well.
One problem I haven’t been able to solve yet is how to backup Windows itself with this set up. Windows 7 backup tool fails, and I can’t see my network drives with the two free backup applications I tried for Windows. I can include specific folders from that drive in my backup script and/or occasionally switch my external drive to my PC to run that full backup, which is probably what I’ll end up doing.
Apart from Windows, I have a few different backups I want to run:
- The SD card for my raspberry pi
- A large hard drive with a lot of media files (movies, music, pictures, etc.)
- A SSD that has all of my games on it.
- Also on that SSD are folders for my personal software and game development projects. I want to also back these up to AWS.
Here is the script as it currently stands. I run it from Windows Subsystem for Linux on Windows 10. I have some notes below to explain the set up and why I chose to use the tools and configurations that I did.
#!/bin/bash today=`date '+%Y_%m_%d'`; #backup raspberry pi ssh username@ipaddress "sudo dd if=/dev/mmcblk0 bs=1M | gzip - | dd of=/media/pi/HDDName/pibackup/pibackup$today.gz" > /mnt/e/rsynclogs/pibackuplog$today.txt #backing up all of my development work including Unity to S3 for offsite backups. Have to add dates to the log files otherwise it overwrites the file aws s3 sync /mnt/e/Development s3://developmentFolder/ --delete > /mnt/e/rsynclogs/S3DevOutput$today.txt aws s3 sync /mnt/e/Unity s3://unityFolder/ --delete > /mnt/e/rsynclogs/S3UnityOutput$today.txt #backup D drive excluding a few folders, and write logs out. rsync -avP --delete --size-only --exclude-from '/mnt/d/rsynclogs/exclude.txt' --log-file=/mnt/d/rsynclogs/rsynclog$today.txt /mnt/d/ username@ipaddress:/media/pi/MediaBackup/ #backup E drive excluding a few folders, and write logs out. rsync -avW --delete --size-only --exclude-from '/mnt/e/rsynclogs/exclude.txt' --log-file=/mnt/e/rsynclogs/rsynclog$today.txt /mnt/e/ username@ipaddress:/media/pi/GamesBackup/
The Raspberry Pi backup is modified from: https://johnatilano.com/2016/11/25/use-ssh-and-dd-to-remotely-backup-a-raspberry-pi/
I don’t have access to the network drives from the terminal (or at least I don’t know how to access them from WSL without ssh ing), so I needed the output to be relative to the Pi. The quotations enclose the commands that get sent to the Pi, so I had to extend them to include the output location. I also changed ‘bs=1m’ to ‘bs=1M’. I believe the lowercase m is expected on Mac, but the uppercase is required on most flavors of Linux.
In order to run it from the script I had to set up my user to not require a password to execute the command, which I did by doing the following:
At a terminal on the Pi, enter ‘sudo visudo’, change the last line to: ‘username ALL = NOPASSWD: ALL’, where username is the username you are using to ssh. If you are doing this as the pi user, I don’t think this will be necessary. I’d kind of like to limit this to just the ‘dd’ command, but I’m not sure how to tell it where dd. I may update this in the future.
For setting up rsync with the correct flags, I used these two links: https://www.howtogeek.com/175008/the-non-beginners-guide-to-syncing-data-with-rsync/ https://www.thegeekstuff.com/2011/01/rsync-exclude-files-and-folders/?utm_source=feedburner
Two notes for the rsync commands:
By default, drives mount with the ‘pi’ user. Since I was setting up my backups to work with a different user, rsync was giving me errors about not being able to set the time on the files when I’d run the command. I’m pretty sure this was because the user didn’t have permissions on the drive. By adding the drives to fstab, it mounts them as root instead, which allows the user to access them since it has root permissions. I should have done this when setting up the drive as a NAS, but I only did it for the initial drive I was testing. See here for instructions on adding drives to fstab: https://www.howtogeek.com/139433/how-to-turn-a-raspberry-pi-into-a-low-power-network-storage-device/
For my E drive rsync, I tried initially with the same settings as the D drive, but the backup kept hanging on different files. I saw several recommendations of different flags that people claimed to be the culprit. I tried turning several off and on, but the one that seemed to fix things was swapping -P for -W (suggested here: https://github.com/Microsoft/WSL/issues/2138), which forces entire files to be transferred instead of partial files. I could probably add –progress back in, but -v for verbose gives me enough output to see where issues arise. I’d advise adding –progress back in if you encounter issues and need to check where things are going wrong..
You can find instructions for setting up the AWS CLI tools and using syncing with S3 in the AWS docs. I couldn’t find anything in there for logging, but StackOverflow had a good solution: https://stackoverflow.com/questions/35075668/output-aws-cli-sync-results-to-a-txt-file
The last thing I added to the script was a variable to grab the current date so I don’t overwrite the pi backup or the log files.
One thing I’d like to add is a way to clean up the pi backups. At ~3GB each, it isn’t a big issue currently, but eventually I’ll want to clean them up.
When my external HDD failed, I debated getting a network attached storage device (NAS) before realizing the price wasn’t worth it for me. I don’t have that much data, and really all I wanted was a way to automate backups and have a plex server that requires less power than my PC (so I could turn it off more often).
While I was looking around at options, I found I could do both of those things with a Raspberry Pi. I’d been wanting to get one for a while, but never had a good project to justify picking one up. I ordered a Raspberry Pi 3, a case with a fan and a power supply (that has a power switch), and a 32GB SD card. That is more storage than I need, but a 16GB card wasn’t much cheaper. I also picked up an 8 TB Seagate external hard drive.
While I think anyone can set this up, I will say that I have a decent working knowledge of Linux, which helped getting started and troubleshooting issues I ran into. If you aren’t very familiar with Linux and the terminal, you can still get all of this set up, but debugging issues and working through it all might be a little more difficult.
First, I would suggest setting up SSH on your pi so you don’t have to go back and forth between working on the pi and another machine: https://www.raspberrypi.org/documentation/remote-access/ssh/ I’d also recommend setting it up with an ssh key so you don’t have to enter your PW every time you log in: https://www.thegeekstuff.com/2008/11/3-steps-to-perform-ssh-login-without-password-using-ssh-keygen-ssh-copy-id/
I didn’t do either of those at first and it became annoying switching back and forth between machines since I only have one keyboard/mouse/display set up. If you have a separate keyboard, mouse, and display for your Pi, it might not be as helpful right away, but I’d still recommend it.
My first step was to try to set up the NAS. I ran through this tutorial: https://www.howtogeek.com/139433/how-to-turn-a-raspberry-pi-into-a-low-power-network-storage-device/, but I hit a wall near the end. I think this was written for Windows 7 and I’m on Windows 10 where mapping a network drive looks different. I also think I may have skipped the last step of setting the samba PW for my user account, which may have been the bigger problem. Here are some notes about where I did things a little differently:
My external HDD had 4 partitions when I started the process and they automatically mounted, so I skipped the parts about mkdir /media/USBHDD1 and mount…USBHDD1
‘security = user’ was not already in the samba config file commented out so I just added it in the authentication section. For the section they tell you to add to the config file, I made four copies of that at the bottom of the file, one for each partition that I have.
For the last part of the tutorial about adding the network drives on my Windows PC I had to follow different directions and go to Windows Explorer -> This PC -> Map network drive (in the file menu) -> then in the file for folder enter ‘\raspberrypi', then I clicked ‘Browse’ which let me select the folder (I divided my external HDD into a few partitions). I was also to manually enter ‘\raspberrypi\nameOfFolder’ to get it to see a drive. I repeated that for each of my drive partitions.
One mistake I made was only setting up one of the partitions in fstab. This caused me some serious issues with permissions when trying to set up backups with rsync.
I’m in the process of automating my backups. When I finish testing my backup scripts, I’ll be sure to post about it.
To set my pi up as a plex server, I followed this tutorial: https://www.codedonut.com/raspberry-pi/raspberry-pi-plex-media-server/
The only problem I ran into was that I had to change permissions on the folder ‘/media/pi’ where it automounted my drives because Plex couldn’t access them. The permissions on the drive folders themselves were fine.
After that small adjustment, I was able to stream a 1080p movie with a bitrate ~10Mb/s over my local network without any trouble, but I tried streaming one closer to 25 Mb/s and the Pi definitely couldn’t handle it. I’m not sure where exactly the limit is, but that is something to note.
Finally, I wanted to test how resilient this set up is so I made sure I could restart while ssh’d and even shut down the pi.
To restart from command line: ‘sudo reboot’. This allowed me log back in after a couple of minutes.
To shutdown then start up the pi: ‘sudo halt’ on the command line. This shuts it down, but the red light stays on (and the fan, so the board is still getting power). I can then use the power button on the AC cable to shut off power, then press it again to turn it back on. When it comes back up, the NAS drives automount and I can see them on my PC and Plex is running.
One last thing I did, for security, was to create a user other than pi, give it sudo permissions (‘sudo usermod -a -G sudo USERNAME’), and give the ‘pi’ user a much more complicated password so it would be more difficult to hack. I saw one tutorial recommend deleting the pi user account, but I decided that was overkill. At the very least, you should change the default password of you default user if you are going to make the Pi visible on a network.
This week for our local Unity meetup group, I presented an intro to some of the new 2D Tools in Unity (There is an intro about more general Unity topics, so for the 2D stuff skip to 17 minutes in): https://www.youtube.com/watch?v=xopzxmzFJUs
Here are the links to things I mentioned were outside of the scope of that talk but might be interesting to learn:
Sprite masks: https://docs.unity3d.com/Manual/class-SpriteMask.html
2D side scrolling brawler style camera (focus on 9-slicing sprites and new features for sorting): https://unity3d.com/learn/tutorials/topics/2d-game-creation/introduction-and-goals
Platformer character controller: https://unity3d.com/learn/tutorials/topics/2d-game-creation/intro-and-session-goals
https://github.com/MelvynMay/UnityPhysics2D - a lot of interesting scenes demoing 2D physics.
Pretty cool topdown game from Unity to show off tilemap and other 2d features (from a talk at Unite Austin, >https://www.youtube.com/watch?v=RkaEh–qUAY>): https://github.com/Unity-Technologies/2d-gamedemo-robodash
2D Game Kit - This is a 2D Game that Unity built to show off 2D features, and what a complete project looks like including tools for designers so that they don’t need to dive into the code to create new puzzles, levels, etc. https://blogs.unity3d.com/2018/02/13/introducing-2d-game-kit-learn-unity-with-drag-and-drop/, https://unity3d.com/learn/tutorials/s/2d-game-kit. Unity also recorded a live training for this recently that I’m assuming they will publish soon, but I can’t find a link to it yet.
Edit: Unity posted the live training for 2D Game Kit here: https://unity3d.com/learn/tutorials/projects/2d-game-kit/overview-and-goals?playlist=49633
What I covered in the video is using the new Tilemaps and associated features for designing levels in 2D. This was also covered by this Unity Learn tutorial: https://unity3d.com/learn/tutorials/topics/2d-game-creation/intro-2d-world-building-w-tilemap, and this blog post: https://blogs.unity3d.com/2018/01/25/2d-tilemap-asset-workflow-from-image-to-level/. This are very thorough and a great reference for these features. I found that there were a couple of things I could talk about not covered in those videos, specifically how to create your own rule and random tiles, and how to create tiles and tilemaps from art that you generate or find yourself.
Finally, here is the collection of brushes and tiles that Unity has coded that cover a huge range of use cases: https://github.com/Unity-Technologies/2d-extras
The ground sprites I used came from here: https://bitcan.itch.io/tileset-simples
And the flower sprites I used came from here: https://onimaru.itch.io/green-platform
subscribe via RSS