Maniatux's Blog

Welcome to the internet

Openbay : full installation guide on FreeBSD

Rédigé par Xavier - - 2 commentaires

Openbay is not easy to install because of the lack of documentation and sphinxsearch which is unknown for many people (like me). I bet there are crazy admins who wants to install Openbay on a FreeBSD server. So I wrote a full tutorial to install it with isohunt's centralized database (Option 1), or with a local one (Option, the hardcore mode).

This tutorial has been tested on FreeBSD 10.1 amd64 with jails. I use mysql56 and php56 because they are available in the ports but older versions should work (php54 or php55). You can use mariadb if you want.

Note: Unfortunately, it's not possible to install the the packages with "pkg install" because some options are missing (for example the php lack the pdo-mysql and mbstring support). You have to work this the ports, that means build everything.

Option 1 : Minimal installation

If you are in a hurry, this is the easiest and fastest way to install openbay. All you need is a web server (nginx) and php. Your openbay installation will use the isohunt's centralized database.

Ports installation

Install these ports :

  • www/nginx
  • lang/php56
  • lang/php56-extensions [select pdo-mysql and mbstring]
  • git

Open /etc/rc.conf.local and add :

nginx_enable="YES"
php_fpm_enable="YES"

Now, fetch openbay and set the permissions :

# mkdir -p /home/htdocs
# cd /home/htdocs
# git clone https://github.com/isohuntto/openbay.git
# chown -R www: /home/htdocs/openbay

Nginx configuration

Edit /usr/local/etc/nginx/nginx.conf :

#user  nobody;
worker_processes  1;
#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;
#pid        logs/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  logs/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    #gzip  on;

server {
   listen 80;
   server_name openbay.dev;

    root /home/htdocs/openbay/src/www;
    index  index.php;
    set $yii_bootstrap "index.php";
    charset utf-8;
    gzip       on;
    gzip_comp_level 3;
    gzip_types text/plain text/css text/javascript application/javascript application/json application/x-java
    script text/xml application/xml application/xml+rss;

location / {
    index  index.html $yii_bootstrap;
    try_files $uri $uri/ /$yii_bootstrap?$args;
}

location ~ \.(js|css|png|jpg|gif|swf|ico|svg)$ {
    fastcgi_hide_header        Set-Cookie;

    open_file_cache            max=10000 inactive=14d;
    open_file_cache_errors     on;
    open_file_cache_min_uses   3;
    open_file_cache_valid      1m;
    output_buffers             1  2m;
    expires                    14d;
}

location ~* \.php$ {
    fastcgi_split_path_info  ^(.+\.php)(.*)$;
    set $fsn /$yii_bootstrap;
    if (-f $document_root$fastcgi_script_name){
        set $fsn $fastcgi_script_name;
    }
    fastcgi_pass unix:/var/run/php-fpm.sock;
            include fastcgi_params;
    fastcgi_param  SCRIPT_FILENAME  $document_root$fsn;
    fastcgi_param  PATH_INFO        $fastcgi_path_info;
    fastcgi_param  PATH_TRANSLATED  $document_root$fsn;
    #fastcgi_param  HTTPS $https_value;
  }
 }
}

Php-fpm configuration

Open /usr/local/etc/php-fpm.conf, locate listen = 127.0.0.1:9000 and modify like this :

#listen = 127.0.0.1:9000
listen = /var/run/php-fpm.sock
listen.owner = www
listen.group = www

At the end of the file, add :

php_admin_value[date.timezone] = Europe/Paris

Now you can start/restart php-fpm and nginx :

# service nginx start
# service php-fpm start

Configuration wizard

Now open http://YOURSERVERIP.

The only thing you have to set is the name for your Openbay instance. Leave all the switches to "ON".

Click on "deploy" !

Option 2 : Full installation

This is a hardcore mode, for true sysadmins. This time you will need your own database (mysql) and sphinxsearch.

Warning : Sphinx uses a lot of memory. Make sure your have at least 4GB available.

Ports installation

Install the following ports :

  • www/nginx
  • databases/mysql56-server
  • lang/php56
  • lang/php56-extensions [select pdo-mysql and mbstring]
  • textproc/sphinxsearch
  • git [defaults]

Open /etc/rc.conf.local and add :

nginx_enable="YES"
php_fpm_enable="YES"
mysql_enable="YES"
sphinxsearch_enable="YES"

Now, fetch openbay :

# mkdir -p /home/htdocs
# cd /home/htdocs
# git clone https://github.com/isohuntto/openbay.git
# chown -R www: /home/htdocs/openbay

MySQL-server configuration

Make sure mysql-server is running and use the mysql_secure_installation script to secure everything.

Then create a user and a database :

# mysql -u root -p
> create database dbopenbay;
> create user 'openbay'@'localhost' identified by 'password';
> grant all privileges on dbopenbay.* to 'openbay'@'localhost' identified by 'password';
> flush privileges;
> exit

Now you must find a valid dump of the piratebay/isohunt database. The official website have a link to download a .csv but I recommend this thread. Download and extract torrents_mini.sql (1,3 Gio) on your server.

Now you have to import the .sql file on your database :

# mysql -u openbay -p dbopenbay < torrents_mini.sql

This may takes up to 1 hour !

Sphinxsearch configuration

Make sure sphinxsearch is stopped. Then edit /usr/local/etc/sphinx.conf :

source opb_dbconnect {
    type = mysql
    sql_host = localhost
    sql_user = openbay
    sql_pass = password
    sql_db = dbopenbay
    sql_port = 3306
    sql_query_pre = SET NAMES utf8
}

index opb_common {
    source = opbtorrents
    path = /var/db/sphinxsearch/data/opb_common
    docinfo = extern
    charset_type = utf-8
    charset_table = 0..9, A..Z->a..z, a..z, U+410..U+42F->U+430..U+44F, U+430..U+44F
    #min_word_len = 2
    enable_star = 1
    #min_infix_len = 3
    #min_prefix_len = 3
    expand_keywords = 1
    index_exact_words = 1
    morphology = stem_enru
    min_stemming_len = 4
    dict = keywords
    ngram_len = 1
    ngram_chars = U+4E00..U+9FBB, U+3400..U+4DB5, U+20000..U+2A6D6, U+FA0E, U+FA0F, U+FA11, U+FA13, U+FA14, U+FA1F, U+FA21, U+FA23, U+FA24, U+FA27, U+FA28, U+FA29, U+3105..U+312C, U+31A0..U+31B7, U+3041, U+3043, U+3045, U+3047, U+3049, U+304B, U+304D, U+304F, U+3051, U+3053, U+3055, U+3057, U+3059, U+305B, U+305D, U+305F, U+3061, U+3063, U+3066, U+3068, U+306A..U+306F, U+3072, U+3075, U+3078, U+307B, U+307E..U+3083, U+3085, U+3087, U+3089..U+308E, U+3090..U+3093, U+30A1, U+30A3, U+30A5, U+30A7, U+30A9, U+30AD, U+30AF, U+30B3, U+30B5, U+30BB, U+30BD, U+30BF, U+30C1, U+30C3, U+30C4, U+30C6, U+30CA, U+30CB, U+30CD, U+30CE, U+30DE, U+30DF, U+30E1, U+30E2, U+30E3, U+30E5, U+30E7, U+30EE, U+30F0..U+30F3, U+30F5, U+30F6, U+31F0, U+31F1, U+31F2, U+31F3, U+31F4, U+31F5, U+31F6, U+31F7, U+31F8, U+31F9, U+31FA, U+31FB, U+31FC, U+31FD, U+31FE, U+31FF, U+AC00..U+D7A3, U+1100..U+1159, U+1161..U+11A2, U+11A8..U+11F9, U+A000..U+A48C, U+A492..U+A4C6
   preopen = 1
}

source opbtorrents : opb_dbconnect {
    sql_query_range = SELECT MIN(id), MAX(id) FROM torrents
    sql_range_step = 100000
    sql_query = SELECT tr.id AS id, tr.hash, tr.name, tr.tags, tr.category_id, UNIX_TIMESTAMP(tr.created_at) AS created_at, tr.size, tr.downloads_count, tr.seeders, tr.leechers, tr.torrent_status, \
    tr.visible_status, \
    tr.files_count AS files_count \
    FROM `torrents` AS tr \
    WHERE tr.id BETWEEN $start AND $end \
    GROUP BY tr.id
    sql_field_string        = name
    sql_field_string        = tags
    sql_attr_uint           = category_id
    sql_attr_timestamp      = created_at
    sql_attr_bigint         = size
    sql_attr_uint           = downloads_count
    sql_attr_uint           = seeders
    sql_attr_uint           = leechers
    sql_attr_uint           = torrent_status
    sql_attr_uint           = visible_status
    sql_attr_uint           = files_count
    sql_attr_string         = hash
}

index opbtorrents : opb_common {
    source = opbtorrents
    path = /var/db/sphinxsearch/data/opbtorrents2
    min_infix_len = 3
    dict = keywords
}

indexer {
    mem_limit = 32M
}

searchd {
    listen = 9312
    listen = 9306:mysql41
    log = /var/log/sphinxsearch/searchd.log
    query_log = /var/log/sphinxsearch/query.log
    read_timeout = 5
    max_children = 0
    pid_file = /var/run/sphinxsearch/searchd.pid
    max_matches = 100000
    seamless_rotate = 0
    preopen_indexes = 1
    unlink_old = 1
    workers = threads
    binlog_path = /var/db/sphinxsearch/data
}

Generate all indexes :

# indexer --all

This may take up to 1 hour. If no error you can now start sphinxsearch :

# service sphinxsearch start

Nginx configuration

Edit /usr/local/etc/nginx/nginx.conf :

#user  nobody;
worker_processes  1;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;

#pid        logs/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  logs/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    #gzip  on;

server {
    listen 80;
    server_name openbay.dev;

    root /home/htdocs/openbay/src/www;
    index  index.php;

    set $yii_bootstrap "index.php";

    charset utf-8;
    gzip       on;
    gzip_comp_level 3;
    gzip_types text/plain text/css text/javascript application/javascript application/json application/x-java
script text/xml application/xml application/xml+rss;

location / {
    index  index.html $yii_bootstrap;
    try_files $uri $uri/ /$yii_bootstrap?$args;
}

location ~ \.(js|css|png|jpg|gif|swf|ico|svg)$ {
    fastcgi_hide_header        Set-Cookie;

    open_file_cache            max=10000 inactive=14d;
    open_file_cache_errors     on;
    open_file_cache_min_uses   3;
    open_file_cache_valid      1m;
    output_buffers             1  2m;

    expires                    14d;
}

location ~* \.php$ {
    fastcgi_split_path_info  ^(.+\.php)(.*)$;
    set $fsn /$yii_bootstrap;
    if (-f $document_root$fastcgi_script_name){
        set $fsn $fastcgi_script_name;
    }
    fastcgi_pass unix:/var/run/php-fpm.sock;
            include fastcgi_params;
    fastcgi_param  SCRIPT_FILENAME  $document_root$fsn;
    fastcgi_param  PATH_INFO        $fastcgi_path_info;
    fastcgi_param  PATH_TRANSLATED  $document_root$fsn;
     #fastcgi_param  HTTPS $https_value;
}
}
}

Php-fpm configuration

Open /usr/local/etc/php-fpm.conf, locate listen = 127.0.0.1:9000 and modify like this :

#listen = 127.0.0.1:9000
listen = /var/run/php-fpm.sock
listen.owner = www
listen.group = www

At the end of the file, add :

php_admin_value[date.timezone] = Europe/Paris

Now you can start/restart php-fpm and nginx :

# service nginx start
# service php-fpm start

Configuration wizard

Open http://YOURSERVERIP.

Set a title for your openbay instance.

On the database section, switch to OFF with these parameters :

  • Host : 127.0.0.1
  • Port : 3306
  • Database : dbopenbay
  • Username : openbay
  • Password : password

On the Sphinx section, switch to OFF with these parameters :

  • Host : 127.0.0.1
  • Port : 9306

Leave Cache and Log to ON.

Click on "deploy" !

Frequent errors / questions

404 Shit happens

Open /home/htdocs/openbay/src/protected/config/debug.php and turn on debugging. Reload your page to get a more explicit error.

Empty results

If you mess with sphinxsearch (running "indexer --all" more than 1 time) some times your database appears "broken" :

  • Stop sphinxsearch
  • Delete everything in /var/db/sphinxsearch/data
  • Connect to mysql, drop the database
  • Re create the database, don't forget to grant all privileges to 'openbay'@'localhost'
  • Import the torrents_mini.sql
  • Run indexer --all
  • Start sphinxsearch
  • Profit

Mysql or sphinxsearch get killed by pid

Yeah, searchd (sphinxsearch daemon) uses a lot of memory. Your server must have at least 4GB of memory.

How can I update my database ?

There are no updates at all. Just use the dump file (torrents_mini.sql) which is static.

I'm a normal person, I use debian or ubuntu

I try to write a similar tutorial for Ubuntu and Debian, wait a few days :)

Should I use the isohunt's centralized database or a local one ?

Well, since there are no updates at all and because of the huge memory consumption of sphinxsearch, I'd say use the centralized database. But there are two flaws :

  • Privacy : all your request goes through Isohunt, this can be problematic
  • Uptime : their database is under heavy load, you may get connection errors

So running a local database and sphinxsearch may be a reasonable choice after all.

2 commentaires

#1 MyTestName a dit :

Very good, very useful. Could you write a tutorial about 'setup openbay on debian with apache2'. Thanks very much.

#2 Xavier a dit :

@MyTestName : Hi,
I'm writing a tutorial debian+nginx :)
But no apache...

Les commentaires sont fermés.