ownCluster

Installing ownCloud on a Banana Pi

Jörn • • bananapi, bananian, install, and owncloud

Long time no post ... well, life happened and such. Show must go on ...

Before we can install ownCloud we have to get an OS installed first. How to download and copy the bananian image onto an SD card is described over at the distribution. Now lets see how you configure a few basics after logging in:

[jfd@lance]$ ssh bananian -l root
root@bananian's password: <the default password is "pi">
Linux bananapi 3.4.108-bananian #2 SMP PREEMPT Thu Aug 13 06:08:25 UTC 2015 armv7l

------------------------------------------------------------------------ 
Welcome to Bananian Linux!
For news and updates check: https://www.bananian.org
Any questions? Read the FAQ first: https://www.bananian.org/faq

Run 'bananian-config' to set up Bananian Linux
Run 'bananian-update' to check for distribution updates
------------------------------------------------------------------------

root@bananapi ~ # bananian-config
---------------------------------------------------------------------------------
Welcome to bananian-config! 

This script assists you to set up some basic parameters... 

For news and updates check: http://www.bananian.org 

---------------------------------------------------------------------------------
No keyboard found. Skipping keyboard configuration.

---------------------------------------------------------------------------------
Your current root password is still 'pi'. You have to change it now!

Configuring root password... 

Enter new UNIX password: <choose whatever suits you>
Retype new UNIX password: <choose whatever suits you>
passwd: password updated successfully

---------------------------------------------------------------------------------
Your current timezone is 'Etc/UTC'. Do you want to change it? (y/N) y

Configuring timezone... 


Current default time zone: 'Europe/Berlin'
Local time is now:      Wed Nov 11 20:59:42 CET 2015.
Universal Time is now:  Wed Nov 11 19:59:42 UTC 2015.

---------------------------------------------------------------------------------
Your current locale is 'en_US.UTF-8'. Do you want to change it? (y/N) y

Configuring locales... 

Generating locales (this might take a while)...
  de_DE.UTF-8... done
  en_US.UTF-8... done
Generation complete.

---------------------------------------------------------------------------------
Your current hostname is 'bananapi'. Do you want to change it? (y/N) y

Configuring hostname... (restart required) 

Enter new hostname: bpicln04 <choose whatever suits you>

---------------------------------------------------------------------------------
Your current hardware configuration is: BananaPi
Do you want to change it? (y/N) <enter>
---------------------------------------------------------------------------------
Do you want to expand the root file system (recommended)? (y/N) y

Expanding root file system... (ignore the warnings and reboot immediately) 


Welcome to fdisk (util-linux 2.25.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): Disk /dev/mmcblk0: 3.7 GiB, 3965190144 bytes, 7744512 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x0ccea0b3

Device         Boot Start     End Sectors  Size Id Type
/dev/mmcblk0p1       2048   43007   40960   20M 83 Linux
/dev/mmcblk0p2      43008 3710936 3667929  1.8G 83 Linux


Command (m for help): Partition number (1,2, default 2): 
Partition 2 has been deleted.

Command (m for help): Partition type
   p   primary (1 primary, 0 extended, 3 free)
   e   extended (container for logical partitions)
Select (default p): Partition number (2-4, default 2): First sector (43008-7744511, default 43008): Last sector, +sectors or +size{K,M,G,T,P} (43008-7744511, default 7744511): 
Created a new partition 2 of type 'Linux' and of size 3.7 GiB.

Command (m for help): Disk /dev/mmcblk0: 3.7 GiB, 3965190144 bytes, 7744512 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x0ccea0b3

Device         Boot Start     End Sectors  Size Id Type
/dev/mmcblk0p1       2048   43007   40960   20M 83 Linux
/dev/mmcblk0p2      43008 7744511 7701504  3.7G 83 Linux


Command (m for help): The partition table has been altered.
Calling ioctl() to re-read partition table.
Re-reading the partition table failed.: Device or resource busy

The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8).

Please reboot

---------------------------------------------------------------------------------
done! please reboot your system now! (shutdown -r now) 

bananian-config  14.26s user 1.85s system 13% cpu 2:03.23 total
root@bananapi ~ # shutdown -r now

Broadcast message from root@bpicln04 (pts/0) (Wed Nov 11 21:06:49 2015):

The system is going down for reboot NOW!
root@bananapi ~ # Connection to bananapi closed by remote host.
Connection to bananapi closed.
[jfd@lance]$ ssh bpicln04 -l root
root@bpicln04's password: 
Linux bpicln04 3.4.108-bananian #2 SMP PREEMPT Thu Aug 13 06:08:25 UTC 2015 armv7l

------------------------------------------------------------------------ 
Welcome to Bananian Linux!
For news and updates check: https://www.bananian.org
Any questions? Read the FAQ first: https://www.bananian.org/faq

Run 'bananian-config' to set up Bananian Linux
Run 'bananian-update' to check for distribution updates
------------------------------------------------------------------------
Last login: Wed Nov 11 21:08:25 2015 from 192.168.1.244
root@bpicln04 ~ #

That was not so hard, was it? Now how hard can it be to install ownCloud? This is going to be a command line only install. I will list all commands I used to install and configure owncloud. I tried to comment the steps but ping me on irc or twitter if you have any questions.

# let's see what version we are on
root@bpicln04 ~ # cat /etc/debian_version
8.1
# ok, get the latest upgrades
root@bpicln04 ~ # apt-get update && apt-get upgrade
# now some basic LAMP setup (remember the db password, we need it later)
root@bpicln04 ~ # apt-get install -y apache2 mariadb-server libapache2-mod-php5 \
php5-gd php5-json php5-mysql php5-curl php5-intl php5-mcrypt php5-imagick 
# we should now be able to see debian stuff at http://bpicln04
# but since http is not good enough let's switch to https
root@bpicln04 ~ # apt-get install ssl-cert && a2enmod ssl && a2ensite default-ssl && service apache2 reload
# great, we now can access https://octestshib (after accepting a ton of warnings ... )
# let us start to install a standard oc setup
root@bpicln04 ~ # wget https://download.owncloud.org/community/owncloud-8.2.0.tar.bz2
root@bpicln04 ~ # apt-get install bzip2
root@bpicln04 ~ # tar -xvjf owncloud-8.2.0.tar.bz2
root@bpicln04 ~ # mv owncloud /var/www/
# by default apache on debian serves from /var/www/html. let us change that to /var/www/owncloud
root@bpicln04 ~ # sed -i 's_DocumentRoot /var/www/html_DocumentRoot /var/www/owncloud_' /etc/apache2/sites-enabled/default-ssl.conf
# there are some more apache modules oc needs as documented in https://doc.owncloud.org/server/8.2/admin_manual/installation/source_installation.html#additional-apache-configurations
root@bpicln04 ~ # a2enmod rewrite headers env dir mime
root@bpicln04 ~ # service apache2 restart
# you could now install oc by going to https://octestshib but we will use the `occ` commant to prevent timeouts and typos, see https://doc.owncloud.org/server/8.2/admin_manual/installation/command_line_installation.html
root@bpicln04 ~ # cd /var/www/owncloud
# let us create a datadir outside of our web root
root@bpicln04 ~ # mkdir /mnt/data
root@bpicln04 ~ # chown www-data.www-data /mnt/data
# sudo is not installed, we will just use su and install with occ
root@bpicln04 ~ # su www-data -s /bin/bash -c 'php occ  maintenance:install --database "mysql" --database-name "owncloud"  --database-user "root" --database-pass "password" --admin-user "admin" --admin-pass "password" --data-dir "/mnt/data"'
ownCloud is not installed - only a limited number of commands are available
{"reqId":"ztIEC9R7VwES7AiC+yu4","remoteAddr":"","app":"PHP","message":"Undefined offset: 0 at \/var\/www\/owncloud\/lib\/private\/setup\/mysql.php#45","level":3,"time":"2015-11-11T20:44:02+00:00"}
ownCloud was successfully installed
{"reqId":"ztIEC9R7VwES7AiC+yu4","remoteAddr":"","app":"PHP","message":"chmod(): No such file or directory at \/var\/www\/owncloud\/lib\/private\/log\/owncloud.php#92","level":3,"time":"2015-11-11T20:44:23+00:00"}
# hm the warnings are annoying, but can be ignored
# opening https://bpicln04 gives us a dialog regarding the trusted_domains, so we will add that
# unfortunately the occ commant in 8.2 cannot yet update arrays in the config file correctly, so we have to fallback to sed:
root@bpicln04 ~ # sed -i "s_0 => 'localhost',_0 => 'localhost', 1 => 'bpicln04',_" config/config.php
# we might as well set up the cli interface properly:
root@bpicln04 ~ # su www-data -s /bin/bash -c 'php occ config:system:set overwrite.cli.url --value http://bpicln04'
# we can now login at https://bpicln04 but the admin page still gives us a few warnings
# the HSTS can easily be enabled by adding it to the apache vhost:
root@bpicln04 ~ # sed -i 's|</VirtualHost>|\t<IfModule mod_headers.c>\n\t\t\tHeader always set Strict-Transport-Security "max-age=15768000; includeSubDomains; preload"\n\t\t</IfModule>\n\t</VirtualHost>|' /etc/apache2/sites-enabled/default-ssl.conf
# for transactional filelocking we need to installing redis
root@bpicln04 ~ # apt-get install -y redis-server php5-redis
root@bpicln04 ~ # service apache2 restart
# configure it to use unix socket only
root@bpicln04 ~ # sed -i 's_port 6379_port 0_' /etc/redis/redis.conf
root@bpicln04 ~ # sed -i 's_# unixsocket /tmp/redis.sock_unixsocket /var/run/redis/redis.sock_' /etc/redis/redis.conf
root@bpicln04 ~ # sed -i 's_# unixsocketperm 700_unixsocketperm 0777_' /etc/redis/redis.conf
root@bpicln04 ~ # sed -i 's_# ULIMIT=65536_ULIMIT=65536_' /etc/redis/redis.conf
root@bpicln04 ~ # service redis-server restart
# now setup oc to use it ... requires sed because it uses an array
root@bpicln04 ~ # su www-data -s /bin/bash -c 'php occ config:system:set redis --value FIXME'
root@bpicln04 ~ # sed -i "s_'FIXME'_['host' => '/var/run/redis/redis.sock','port' => 0,'timeout' => 0.0]_" config/config.php
root@bpicln04 ~ # su www-data -s /bin/bash -c 'php occ config:system:set memcache.local --value "\OC\Memcache\Redis"'
root@bpicln04 ~ # su www-data -s /bin/bash -c 'php occ config:system:set memcache.locking --value "\OC\Memcache\Redis"'
root@bpicln04 ~ # su www-data -s /bin/bash -c 'php occ config:system:set filelocking.enabled --value true'
# awesome, all checks on the admin page pass!

Now ... this is an installation with most bells and whistles. We could also configure php to use redis as the session cache. But thats a task when we split up the services and try to scale out.

Looking at the above we are now torturing the small BananaPi with apache, mariadb, redis and the file storage at the same time. The sd card will be the bottleneck ... or is it?

Next up: benchmarking and small optimizations

Hardware for a Banana Pi cluster

Jörn • • cluster, bananapi, hardware, lego, and psu

A new purpose for old Lego bricks

While I like the idea of screwing the Banana Pis to the wall, somewhat like The Beast, I simply wanted to build this thing quickly without having to go into the cellar, sawing and drilling holes. Lucky me had a box of old LEGO bricks, which are in general the ultimate tool for rapid prototyping. And they required only one quick trip to the cellar. I was able to build the Banana Pi cage in one evening, redesigning and attaching it to the other cages over the next two evenings.

The current Banana Pi node 1

With newer or updated versions of SoC boards coming out I had to choose which one would be the foundation of my project. Ultimately, the decision was driven by the idea of using ceph as the storage for owncloud. That quickly brought me to the Bananas because they have a SATA interface and GB ethernet port. Both directly connected to the A20 ARM Cortex-A7 dual-core CPU for ~30€. Neither can be found on the Raspberry Pis. The BPI-M1+ comes with Wifi, which I don't need and the BPI-PRO has no SATA -> meh. The Cubieboard 3 uses the same A20 CPU but brings 2GB of RAM and WiFi for three times the cost of a Banana Pi M1. The pcDuino3B also looks interesting, same A20. However, I still don't need wifi and it is more expensive than the BPI-M1.

It would be great to have a SoC with dual NICs but the Banana Pi R1 router board just uses a switch connected to the internal A20 NIC, so no gain there. Three interfaces would be awesome because then the web tier could separate incoming network traffic from storage as well as database traffic. The two USB ports however are also directly connected to the A20, so there might be some more options with USB NICs here. Whether or not the CPU can handle that remains to be benchmarked.

They do need to boot off something and I don't want to waste precious RAM by booting via TFTP and storing the rootfs in RAM. So I ordered 4GB SD-cards for each of the boards.

Some cheap gigabit switch

When thinking about this cluster I did not have an array of hundreds of SoC boards in mind. To be honest, the dimensions of the initial version was determined by the old 8-port GB ethernet switch that I had lying around. That pretty quickly led to ordering the seven Banana Pi boards. It also gave the initial cost a nice boundary. My hope is that the router is so cheap that we can actually saturate the backplane with the seven Banana Pis. Another benchmark.

DIY Rubber band HDD mount

I am a little proud of having built a HDD mount with rubber band and LEGO. We should all build more things out of LEGO. But I am drifting off. A minimal ceph installation requires 3 OSD nodes to replicate and distribute the data, so I also ordered three SATA adapters that pull the power from the Banana Pi. In the future I plan to have at least ten OSD nodes and hope to see and benchmark the claimed sustained performance under load. I'll need to ask around for old HDDs because I am running out of them.

5V 20A PSU

If you do the math 7 * 2A for BPI = 14A, 3 * 2A for HDD = 6A makes a total of 20A. So I ordered seven micro usb cables and a 20A 5V PSU to power all but the switch. Boring ... nothing to benchmark here. Well maybe if I can read some power usage with lmsensors? Ideas welcome.

If you have any questions or feedback send me an email or ping me on twitter.

Next up: installing owncloud on a single node

Notes on scaling ownCloud on a Banana Pi cluster

Jörn • • owncloud, cluster, and bananapi

Current Hardware

As an ownCloud developer I fairly often hear something like "ownCloud is awesome! But it's so slow on my Raspberry Pi". Let me tell you something: everything is slow on a rpi. ownCloud unfortunately will sooner or later be so convenient for most users that they try to upload their latest holiday pictures. All 3000 of them. While that will already take some time, the next time the gallery is opened the pi will come to a grinding halt because it starts generating thumbnails. A single Raspberry Pi / Banana Pi will run calendar end contacts just fine. Just don't expect it to handle file uploads equally well.

That being said, the upcoming 8.1 has seen a lot of performance improvements. Time to finally get to the task and do some benchmarking. Benchmarking is hard ... and there are many aspects to consider when trying to benchmark owncloud. Furthermore, I really want to play with different deployment scenarios. Given the same hardware, what are the impacts of using mysql vs postgresql, redis vs memcached, apache vs nginx, ceph s3 vs phprados, does a second usb NIC make sense for ceph nodes based on a Banana Pi? Can I scale out ownCloud with Banana Pis? What will become the bottlenecks? When will the network become the bottleneck?

Answering all of these questions will take some time, so I created this blog to note down my findings. If you have any questions or feedback send me an email or ping me on twitter.

Next up: hardware details. Stay tuned!