Warning: Can't synchronize with repository "(default)" (/home/git/ome.git does not appear to be a Git repository.). Look in the Trac log for more information.
Notice: In order to edit this ticket you need to be either: a Product Owner, The owner or the reporter of the ticket, or, in case of a Task not yet assigned, a team_member"

Task #3199 (closed)

Opened 13 years ago

Closed 13 years ago

Bug: "fastcgi" (non-tcp) doesn't seem to work

Reported by: jamoore Owned by: cxallan
Priority: minor Milestone: OMERO-Beta4.3
Component: Deployment Version: n.a.
Keywords: n.a. Cc: cneves
Resources: n.a. Referenced By: n.a.
References: n.a. Remaining Time: n.a.
Sprint: 2010-12-23 (2)

Description

bin/omero web server fastcgi-tcp
bin/omero web start
bin/omero web config nginx > tcp.conf
# Manually edit tcp.conf. See #3198
nginx -c `pwd`/tcp.conf
# Access via localhost:8000 works fine

bin/omero web stop
bin/omero web server fastcgi
bin/omero config nginx > socket.conf
nginx -c `pwd`/socket.conf
# localhost:8000 hangs with "Waiting on localhost
$ netstat -na | grep 8000
tcp4       0      0  *.8000                 *.*                    LISTEN

$ pstree
...
 |-+- 15429 moore /opt/local/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python manage.py runfcgi workdir=./ method=prefork
 | |--= 15430 moore /opt/local/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python manage.py runfcgi workdir=./ method=prefo
 | |--= 15431 moore /opt/local/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python manage.py runfcgi workdir=./ method=prefo
 | |--= 15432 moore /opt/local/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python manage.py runfcgi workdir=./ method=prefo
 | |--= 15433 moore /opt/local/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python manage.py runfcgi workdir=./ method=prefo
 | \--= 15434 moore /opt/local/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python manage.py runfcgi workdir=./ method=prefo
 |-+= 15459 moore nginx: master process nginx -c /Users/moore/code/git/dist/socket.conf  
 | |--- 15460 moore nginx: worker process  
 | |--- 15461 moore nginx: worker process  
 | |--- 15462 moore nginx: worker process  
 | |--- 15463 moore nginx: worker process  
 | \--- 15464 moore nginx: worker process  

# And no errors in var/log/nginx_error.log

Change History (11)

comment:1 Changed 13 years ago by jburel

  • Sprint changed from 2010-10-28 (18) to 2010-11-11 (19)

Moved from sprint 2010-10-28 (18)

comment:2 Changed 13 years ago by cxallan

With a socket configuration I have no problems getting it to work.

pid /Users/callan/code/OMERO-TRUNK/dist/var/pid.nginx;
error_log /Users/callan/code/OMERO-TRUNK/dist/var/log/nginx_error.log;
worker_processes  5;
working_directory /Users/callan/code/OMERO-TRUNK/dist/var;

events {
    worker_connections  1024;
}


http {
    access_log    /Users/callan/code/OMERO-TRUNK/dist/var/log/nginx_access.log;
    include       /Users/callan/code/OMERO-TRUNK/dist/etc/mime.types;
    default_type  application/octet-stream;
    client_body_temp_path /Users/callan/code/OMERO-TRUNK/dist/tmp;

    keepalive_timeout  65;

    server {
        listen       8000;
        server_name  _;
        fastcgi_temp_path /Users/callan/code/OMERO-TRUNK/dist/var;
        proxy_temp_path /Users/callan/code/OMERO-TRUNK/dist/var;

         # weblitz django apps serve media from here
        location /appmedia {
            alias /Users/callan/code/OMERO-TRUNK/dist/lib/python/omeroweb/media;
        }

        # django's admin media
        location /media {
            alias /Users/callan/code/OMERO-TRUNK/dist/lib/python/omeroweb/media/django_admin_media;
        }

        location / {
            if (-f /Users/callan/code/OMERO-TRUNK/dist/var/maintenance.html) {
               error_page 503 /maintenance.html;
               return 503;
            }
            fastcgi_pass unix:/Users/callan/code/OMERO-TRUNK/dist/var/django_fcgi.sock;
            fastcgi_param PATH_INFO $fastcgi_script_name;
            fastcgi_param REQUEST_METHOD $request_method;
            fastcgi_param QUERY_STRING $query_string;
            fastcgi_param CONTENT_TYPE $content_type;
            fastcgi_param CONTENT_LENGTH $content_length;
            fastcgi_param SERVER_NAME $server_name;
            fastcgi_param SERVER_PROTOCOL $server_protocol;
            fastcgi_param SERVER_PORT $server_port;
            fastcgi_pass_header Authorization;
            fastcgi_intercept_errors on;
            fastcgi_read_timeout 300;
        }

        location /maintenance.html {
            root /Users/callan/code/OMERO-TRUNK/dist/var;
        }

    }

}
$ nginx -V
nginx version: nginx/0.7.65
configure arguments: --prefix=/opt/local --with-cc-opt='-I/opt/local/include -O2' --with-ld-opt=-L/opt/local/lib --conf-path=/opt/local/etc/nginx/nginx.conf --error-log-path=/opt/local/var/log/nginx/error.log --http-log-path=/opt/local/var/log/nginx/access.log --pid-path=/opt/local/var/run/nginx/nginx.pid --lock-path=/opt/local/var/run/nginx/nginx.lock --http-client-body-temp-path=/opt/local/var/run/nginx/client_body_temp --http-proxy-temp-path=/opt/local/var/run/nginx/proxy_temp --http-fastcgi-temp-path=/opt/local/var/run/nginx/fastcgi_temp

$ dist/bin/omero web server
OMERO.web is configured to be served by 'fastcgi'

$ ps auxww | grep manage.py | head -n 1
callan   27069   0.0  0.0  2477056   1556   ??  S     4:59pm   0:00.00 python manage.py runfcgi workdir=./ method=prefork socket=/Users/callan/code/OMERO-TRUNK/dist/var/django_fcgi.sock pidfile=/Users/callan/code/OMERO-TRUNK/dist/var/django.pid daemonize=true maxchildren=5 minspare=1 maxspare=5 maxrequests=400

comment:3 Changed 13 years ago by jmoore

Same version:

~/code/omero.git/dist $ nginx -V
nginx version: nginx/0.7.65
configure arguments: --prefix=/opt/local --with-cc-opt='-I/opt/local/include -O2' --with-ld-opt=-L/opt/local/lib --conf-path=/opt/local/etc/nginx/nginx.conf --error-log-path=/opt/local/var/log/nginx/error.log --http-log-path=/opt/local/var/log/nginx/access.log --pid-path=/opt/local/var/run/nginx/nginx.pid --lock-path=/opt/local/var/run/nginx/nginx.lock --http-client-body-temp-path=/opt/local/var/run/nginx/client_body_temp --http-proxy-temp-path=/opt/local/var/run/nginx/proxy_temp --http-fastcgi-temp-path=/opt/local/var/run/nginx/fastcgi_temp

Same conf:

~/code/omero.git/dist $ perl -i -pe 's/callan.code.OMERO-TRUNK/moore\/GlencoeSoftware.git\/git\/omero.git/' chris.conf 
~/code/omero.git/dist $ diff socket.conf chris.conf 
61d60
< 

Seems to be running:

~/code/omero.git/dist $ bin/omero web server fastcgi
OMERO.web has been configured to be served by 'fastcgi'
~/code/omero.git/dist $ bin/omero web start
Starting OMERO.web... [OK]
~/code/omero.git/dist $ ps auxww | grep -v grep | grep manage.py | head -n 1

moore    37424   0.0  0.0  2475192   1452   ??  S     6:09PM   0:00.00
/opt/local/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python
manage.py runfcgi workdir=./ method=prefork socket=/Users/moore/GlencoeSoftware.git/git/omero.git/dist/var/django_fcgi.sock
pidfile=/Users/moore/GlencoeSoftware.git/git/omero.git/dist/var/django.pid
daemonize=true maxchildren=5 minspare=1 maxspare=5 maxrequests=400

Any other ideas what I can try? (I'll kick off a clean build now)

comment:4 Changed 13 years ago by jmoore

After a clean build I'm still getting the same hang on http://localhost:8000/webclient (also tried "syncmedia" :

~/code/omero.git $ cd dist
~/code/omero.git/dist $ bin/omero web server 
Please use 'web settings' first
~/code/omero.git/dist $ bin/omero web settings
You just installed OMERO, which means you didn't have settings configured in OMERO.web.
Please enter the domain you want to run OMERO.web on (http://www.domain.com:8000/):^CCancelled
~/code/omero.git/dist $ bin/omero web settings --force --domain=http://localhost:8000 --email=j@gs.com --smtphost=localhost --server=fastcgi --quiet
You just installed OMERO, which means you didn't have settings configured in OMERO.web.
Saved to /Users/moore/GlencoeSoftware.git/git/omero.git/dist/var/lib/custom_settings.py
PYTHONPATH updated.
~/code/omero.git/dist $ bin/omero web server 
OMERO.web is configured to be served by 'fastcgi'
~/code/omero.git/dist $ bin/omero web start
Starting OMERO.web... [OK]
~/code/omero.git/dist $ cp ../dist-old/chris.conf .
~/code/omero.git/dist $ nginx -c `pwd`/chris.conf
[alert]: could not open error log file: open() "/opt/local/var/log/nginx/error.log" failed (13: Permission denied)
~/code/omero.git/dist $ 
~code/omero.git/dist $ pstree
...
 |-+- 74167 moore /opt/local/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python manage.py runfcgi workdir=./ method=prefork
 | |--= 74168 moore /opt/local/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python manage.py runfcgi workdir=./ method=prefo
 | |--= 74169 moore /opt/local/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python manage.py runfcgi workdir=./ method=prefo
 | |--= 74170 moore /opt/local/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python manage.py runfcgi workdir=./ method=prefo
 | |--= 74171 moore /opt/local/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python manage.py runfcgi workdir=./ method=prefo
 | \--= 74172 moore /opt/local/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python manage.py runfcgi workdir=./ method=prefo
 |-+= 74188 moore nginx: master process nginx -c /Users/moore/code/omero.git/dist/chris.conf  
 | |--- 74189 moore nginx: worker process  
 | |--- 74190 moore nginx: worker process  
 | |--- 74191 moore nginx: worker process  
 | |--- 74192 moore nginx: worker process  
 | \--- 74193 moore nginx: worker process  
~/code/omero.git/dist $ file /Users/moore/GlencoeSoftware.git/git/omero.git/dist/var/django_fcgi.sock
/Users/moore/GlencoeSoftware.git/git/omero.git/dist/var/django_fcgi.sock: socket
~/code/omero.git/dist/var/log $ cat OMEROweb.log 
Fri, 29 Oct 2010 09:20:19 root         INFO     Application Starting...
Fri, 29 Oct 2010 08:20:57 views-feedback ERROR    handler404: Page not found
Fri, 29 Oct 2010 08:20:57 views-feedback ERROR    Traceback (most recent call last):

  File "/Users/moore/GlencoeSoftware.git/git/omero.git/dist/lib/python/django/core/handlers/base.py", line 83, in get_response
    request.path_info)

  File "/Users/moore/GlencoeSoftware.git/git/omero.git/dist/lib/python/django/core/urlresolvers.py", line 233, in resolve
    raise Resolver404, {'tried': tried, 'path': new_path}

Resolver404: {'path': u'', 'tried': ['^favicon\\.ico$', '^appmedia/webgateway/(?P<path>.*)$', '^appmedia/webmobile/(?P<path>.*)$', '^appmedia/omeroweb/(?P<path>.*)$', '(?i)^webadmin/', '(?i)^webclient/', '(?i)^feedback/', '(?i)^webgateway/', '(?i)^webtest/', '(?i)^webemdb/', '(?i)^webmobile/']}

Fri, 29 Oct 2010 08:20:59 webemdb      WARNING  Failed to import EMAN2. Some features of webemdb will not be supported.
Fri, 29 Oct 2010 08:20:59 views-admin  INFO     INIT '74168'
Fri, 29 Oct 2010 08:20:59 views-web    INFO     INIT '74168'
Fri, 29 Oct 2010 08:21:02 views-admin  INFO     INIT '74169'
Fri, 29 Oct 2010 08:21:03 views-web    INFO     INIT '74169'
~/code/omero.git/dist/var/log $ cat nginx_error.log 
~/code/omero.git/dist/var/log $ cat nginx_access.log 
127.0.0.1 - - [29/Oct/2010:09:20:59 +0200] "GET / HTTP/1.1" 404 1025 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.10) Gecko/20100914 Firefox/3.6.10"

comment:5 Changed 13 years ago by jmoore

Just reproduced with the latest. Waiting until nginx times out shows this in nginx_error.log:

2010/11/01 13:49:03 [error] 86601#0: *4 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 127.0.0.1, server: _, request: "GET /webclient/ HTTP/1.1", upstream: "fastcgi://unix:/Users/moore/GlencoeSoftware.git/git/omero.git/dist/var/django_fcgi.sock:", host: "localhost:8000", referrer: "http://localhost:8000/webclient/login/?url=/webclient/"

comment:6 Changed 13 years ago by jmoore

When running with debug here's OMEROweb-DEBUG.log on "fastcgi":

Mon, 01 Nov 2010 14:46:29 root         INFO     Application Starting...
Mon, 01 Nov 2010 13:46:40 omero.util.TempFileManager DEBUG    Added file /Users/moore/.lock_testZizd1Q.tmp
Mon, 01 Nov 2010 13:46:40 omero.util.TempFileManager DEBUG    Chose gloabl tmpdir: /Users/moore
Mon, 01 Nov 2010 13:46:40 omero.util.TempFileManager DEBUG    Using temp dir: /Users/moore/omero/tmp/omero_moore/98991
Mon, 01 Nov 2010 13:46:40 webgateway   DEBUG    INIT
Mon, 01 Nov 2010 13:46:40 views-admin  INFO     INIT '98991'
Mon, 01 Nov 2010 13:46:40 views-web    INFO     INIT '98991'
Mon, 01 Nov 2010 13:46:40 webgateway   DEBUG    trying stored connection
Mon, 01 Nov 2010 13:46:40 webgateway   DEBUG    []
Mon, 01 Nov 2010 13:46:40 webgateway   DEBUG    creating new connection with "S:" (False)
Mon, 01 Nov 2010 13:46:40 blitz_gateway DEBUG    Connect attempt, sUuid=b83f7401-726d-4246-a742-2735f2f0fe11, group=None, self.sUuid=None
Mon, 01 Nov 2010 13:46:40 blitz_gateway DEBUG    connected? False
var/log/OMEROweb-DEBUG.log (END) 

and here's "fastcgi-tcp":

Mon, 01 Nov 2010 14:51:49 root         INFO     Application Starting...
Mon, 01 Nov 2010 13:52:02 omero.util.TempFileManager DEBUG    Added file /Users/moore/.lock_test4ErtCb.tmp
Mon, 01 Nov 2010 13:52:02 omero.util.TempFileManager DEBUG    Chose gloabl tmpdir: /Users/moore
Mon, 01 Nov 2010 13:52:02 omero.util.TempFileManager DEBUG    Using temp dir: /Users/moore/omero/tmp/omero_moore/887
Mon, 01 Nov 2010 13:52:02 webgateway   DEBUG    INIT
Mon, 01 Nov 2010 13:52:02 views-admin  INFO     INIT '887'
Mon, 01 Nov 2010 13:52:02 views-web    INFO     INIT '887'
Mon, 01 Nov 2010 13:52:02 omero.util.TempFileManager DEBUG    Added file /Users/moore/.lock_testTQPmjD.tmp
Mon, 01 Nov 2010 13:52:02 omero.util.TempFileManager DEBUG    Chose gloabl tmpdir: /Users/moore
Mon, 01 Nov 2010 13:52:02 omero.util.TempFileManager DEBUG    Using temp dir: /Users/moore/omero/tmp/omero_moore/886
Mon, 01 Nov 2010 13:52:02 webgateway   DEBUG    INIT
Mon, 01 Nov 2010 13:52:02 views-admin  INFO     INIT '886'
Mon, 01 Nov 2010 13:52:02 views-web    INFO     INIT '886'
Mon, 01 Nov 2010 13:52:02 webgateway   DEBUG    trying stored connection
Mon, 01 Nov 2010 13:52:02 webgateway   DEBUG    []
Mon, 01 Nov 2010 13:52:02 webgateway   DEBUG    creating new connection with "S:" (False)
Mon, 01 Nov 2010 13:52:02 blitz_gateway DEBUG    Connect attempt, sUuid=None, group=None, self.sUuid=None
Mon, 01 Nov 2010 13:52:02 blitz_gateway DEBUG    ## Creating proxies
Mon, 01 Nov 2010 13:52:02 blitz_gateway DEBUG    ## Reusing proxies
Mon, 01 Nov 2010 13:52:02 blitz_gateway DEBUG    ## - refreshing getAdminService
Mon, 01 Nov 2010 13:52:02 blitz_gateway INFO     created connection (uuid=b3a75606-3c4a-4375-aa41-0cf3ec576e5c)
Mon, 01 Nov 2010 13:52:02 blitz_gateway DEBUG    .. connected!
Mon, 01 Nov 2010 13:52:07 webgateway   DEBUG    created new connection <webclient.webclient_gateway.OmeroWebGateway object at 0x10372e950>
Mon, 01 Nov 2010 13:52:07 webgateway   DEBUG    [('S:b3a75606-3c4a-4375-aa41-0cf3ec576e5c#None', <webclient.webclient_gateway.OmeroWebGateway object at 0x10372e950>)
]
Mon, 01 Nov 2010 13:52:07 webgateway   DEBUG    blitz session key: b3a75606-3c4a-4375-aa41-0cf3ec576e5c
Mon, 01 Nov 2010 13:52:07 webgateway   DEBUG    stored as session.S:b3a75606-3c4a-4375-aa41-0cf3ec576e5c#None
Mon, 01 Nov 2010 13:52:07 webemdb      WARNING  Failed to import EMAN2. Some features of webemdb will not be supported.
Mon, 01 Nov 2010 13:52:07 webgateway   DEBUG    INIT
Mon, 01 Nov 2010 13:52:07 webgateway   DEBUG    trying stored connection
Mon, 01 Nov 2010 13:52:07 webgateway   DEBUG    []
Mon, 01 Nov 2010 13:52:07 webgateway   DEBUG    creating new connection with "S:" (False)
Mon, 01 Nov 2010 13:52:07 blitz_gateway DEBUG    Connect attempt, sUuid=b3a75606-3c4a-4375-aa41-0cf3ec576e5c, group=None, self.sUuid=None
Mon, 01 Nov 2010 13:52:07 blitz_gateway DEBUG    connected? False
Mon, 01 Nov 2010 13:52:07 webgateway   DEBUG    trying stored connection
Mon, 01 Nov 2010 13:52:07 webgateway   DEBUG    [('S:b3a75606-3c4a-4375-aa41-0cf3ec576e5c#None', <webclient.webclient_gateway.OmeroWebGateway object at 0x10372e950>)
]
Mon, 01 Nov 2010 13:52:07 blitz_gateway DEBUG    joinSession(b3a75606-3c4a-4375-aa41-0cf3ec576e5c)
Mon, 01 Nov 2010 13:52:07 blitz_gateway DEBUG    ## Creating proxies
Mon, 01 Nov 2010 13:52:07 omero.util.TempFileManager DEBUG    Added file /Users/moore/.lock_testnpkkWH.t
...

For some reason, the "fastcgi" case is doing far less. The major difference I can find is that one is on port 8000 and one is on port 8888. Is there some django cache issue going on?

comment:7 Changed 13 years ago by cxallan

Also tested to be working on Howe:

omero@howe:~$ nginx -V
nginx version: nginx/0.7.65
TLS SNI support enabled
configure arguments: --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --pid-path=/var/run/nginx.pid --lock-path=/var/lock/nginx.lock --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/body --http-proxy-temp-path=/var/lib/nginx/proxy --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --with-debug --with-http_stub_status_module --with-http_flv_module --with-http_ssl_module --with-http_dav_module --with-http_gzip_static_module --with-http_realip_module --with-mail --with-mail_ssl_module --with-ipv6 --add-module=/build/buildd/nginx-0.7.65/modules/nginx-upstream-fair

omero@howe:~$ cat nginx.conf
pid /home/omero/OMERO-CURRENT/var/pid.nginx;
error_log /home/omero/OMERO-CURRENT/var/log/nginx_error.log;
worker_processes  5;
working_directory /home/omero/OMERO-CURRENT/var;

events {
    worker_connections  1024;
}


http {
    access_log    /home/omero/OMERO-CURRENT/var/log/nginx_access.log;
    include       /home/omero/OMERO-CURRENT/etc/mime.types;
    default_type  application/octet-stream;
    client_body_temp_path /home/omero/OMERO-CURRENT/tmp;

    keepalive_timeout  65;

    server {
        listen       8080;
        server_name  _;
        fastcgi_temp_path /home/omero/OMERO-CURRENT/var;
        proxy_temp_path /home/omero/OMERO-CURRENT/var;

         # weblitz django apps serve media from here
        location /appmedia {
            alias /home/omero/OMERO-CURRENT/lib/python/omeroweb/media;
        }

        # django's admin media
        location /media {
            alias /home/omero/OMERO-CURRENT/lib/python/omeroweb/media/django_admin_media;
        }

        location / {
            if (-f /home/omero/OMERO-CURRENT/var/maintenance.html) {
               error_page 503 /maintenance.html;
               return 503;
            }
            fastcgi_pass unix:/home/omero/OMERO-CURRENT/var/django_fcgi.sock;
            fastcgi_param PATH_INFO $fastcgi_script_name;
            fastcgi_param REQUEST_METHOD $request_method;
            fastcgi_param QUERY_STRING $query_string;
            fastcgi_param CONTENT_TYPE $content_type;
            fastcgi_param CONTENT_LENGTH $content_length;
            fastcgi_param SERVER_NAME $server_name;
            fastcgi_param SERVER_PROTOCOL $server_protocol;
            fastcgi_param SERVER_PORT $server_port;
            fastcgi_pass_header Authorization;
            fastcgi_intercept_errors on;
            fastcgi_read_timeout 300;
        }

        location /maintenance.html {
            root /home/omero/OMERO-CURRENT/var;
        }

    }

}

comment:8 Changed 13 years ago by jburel

  • Sprint changed from 2010-11-11 (19) to 2020-11-25 (20)

Moved from sprint 2010-11-11 (19)

comment:9 Changed 13 years ago by jburel

  • Milestone changed from OMERO-Beta4.2.1 to OMERO-Beta4.3
  • Sprint changed from 2010-11-25 (20) to 2010-12-09 (21)

Moved from sprint 2010-11-25 (20)

comment:10 Changed 13 years ago by jburel

  • Sprint changed from 2010-12-09 (21) to 2010-12-23 (22)

Moved from sprint 2010-12-09 (21)

comment:11 Changed 13 years ago by jburel

  • Resolution set to fixed
  • Status changed from new to closed

Closing cf. discussion during 09/12/10 meeting

Note: See TracTickets for help on using tickets. You may also have a look at Agilo extensions to the ticket.

1.3.13-PRO © 2008-2011 Agilo Software all rights reserved (this page was served in: 0.67810 sec.)

We're Hiring!