I have an instance of OwnCloud running from a machine at home that provides file-syncing services for family members. The OwnCloud data is then encrypted and sent on to CrashPlan for backups.
I recently pointed 1.3 TB of data to sync into OwnCloud. These are old home videos in raw format with files sized up to 25 GB. The upload speed was atrocious. The server is connected to my desktop on a gigabit switch and transfer speeds were topping out at 2.0 MB/s.
Most of the issues people have with poor OwnCloud performance are when uploading many small files which is not my scenario. But I followed whatever advice I could find. I modified MariaDB settings and used MySqlTuner to find potential performance gains, which helped a little. I finally found the backported php-apc package I needed for Ubuntu 14.04 to provide php caching, which helped a little. But I was still only up to ~4.5 MB/s.
Then I considered my larger system. The server is on my local gigabit switch, but my desktop is configured using its public domain name, which resolves to my public IP address. This mean every request from my desktop wasn’t just going through the gigabit switch and in to the server. Instead every request was going through the switch to the router, being NAT-translated, back to the switch, and then to the server. Due to an issue with my high-performance EdgeRouter Lite, I’ve been using my old WRT54GL as my router. And that old thing simply can’t handle the load. It’s CPU was maxed out and network throughput was abysmal.
Since I wanted to bypass the router and go directly from desktop, to switch, to server I made an entry in my /etc/hosts file to tell my machine to use the server’s internal IP address instead of the public IP address associated with the domain name. The CPU load on the router is now gone and the OwnCloud throughput increased to ~11 MB/s. Still pretty awful compared to the ~60 MB/s I get using scp, but substantially better than 2 MB/s.
That speed increase was going strong for a while, but after about 20 minutes it slowed back down to ~4.5 MB/s again. The router, however, is no longer in the loop, so at least I’ve removed one potential bottleneck.
I have no idea what the bottleneck is now except for OwnCloud just being abysmally slow. The server is using a fair bit of CPU, but it’s not quite maxed out (usually showing 20% idle overall on a 4-core machine). IO doesn’t seem to be the bottleneck, iotop doesn’t show anything being held up. There’s 2 GB of free RAM available, so that doesn’t seem to be the issue.
I’m running OwnCloud using Apache with mod-php; but I didn’t see anything suggesting that running PHP using fcgi or fastcgi would be better. Using nginx instead of Apache might be better, but I have no experience configuring nginx so it wouldn’t be a short little project to try it.
If anyone has any suggestions on how to get OwnCloud to perform better (particularly when syncing very large files) when connecting over a gigabit local network I’d love to hear them.
Update 4/17/2016
The slowdown from 10MB/s seems to have been the accumulation process running over large files. Files are transferred in small chunks. Once all the chunks have been uploaded, the chunks are accumulated and the original file is reconstituted. During this time the upload speed drops dramatically.
I had a terminal case where a known memory leak in the accumulation process kept causing the reconstitution to fail. Since the chunks are deleted as they’re used during accumulation, the client would then re-upload all the deleted chunks, then the accumulation process would run, fail, and we’d go round and round. This made it look like performance was worse than it truly was (though not uploading files is sort of a bad thing for a file sync tool to do).
In and attempt to get my files finished without waiting until the memory leak fix is released I split some files to be smaller so the accumulation process doesn’t leak as much memory. I also set a rather absurd 4GB memory limit on the PHP process hoping that will be enough to get it through the large files without failing.
Setting aside the accumulation/reconstitution process, I’m getting a consistent 10MB/s transfer on a 100Mbit switch. I had to RMA the gigabit switch because it began misbehaving. I’m hopeful that when the replacement arrives my throughput will increase beyond 10MB/s since that’s about the limit of the 100Mbit switch and the gigabit switch wasn’t working properly.
Update 4/19/2016
My replacement gigabit switch is in and the transfer rate has gone up to ~19MB/s at times (when no accumulation/reconstitution work is occurring). There is still plenty of room for improvement, but at least I’m not stuck at 2MB/s anymore.