Performance is painful

Functionality is impressive, but G2 is way TOOOOO slow. Uploading pictures is dreadfully slow and retrieval is not great either. Is the team going to work on performance before 2.0 goes final? I sure hope so.
----

Gallery URL (optional):
Gallery version: Core 931
Webserver (with version):
Datatabase (with version):
PHP version (eg 4.2.1):
phpinfo URL (optional):
Graphics Toolkit(s):
Operating system:
Web browser/version:
G1 version (for migration bugs):

Here is the output from System Maintenance/System Information:

Gallery version = 2.0-beta-3+ core 0.9.31
PHP version = 4.4.0 cgi
Webserver = Apache/1.3.33 (Unix) FrontPage/5.0.2.2635 mod_ssl/2.8.22 OpenSSL/0.9.7d PowWeb/1.1
Database = mysql 4.1.9-log
Toolkits = ArchiveUpload, Exif, Gd, ImageMagick, NetPBM, Thumbnail
Operating system = FreeBSD clust06-www03.powweb.com 4.11-RELEASE FreeBSD 4.11-RELEASE #0: Thu Ma i386
Browser = Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.8) Gecko/20050511 Firefox/1.0.4

I am configured as multi-site, so I don't know how to give you the link to the phpinfo.php file. I tried using gallery2/lib/tools/phpinfo.php and I got the following error:

You must create a config.php file in the Gallery directory before you can continue configuring the application. Use the installer to create one.

Thanks for helping with this,
Bobby

My web site and gallery is hosted by PowWeb which I understand has a fair number of users who use Gallery. I would have to contact them for the specs of the server that my account runs on. I don't have time to do that right now.

I did do a comparison between Gv1.5 and Gv2 uploading photos:

Test is 5 photos from 2.1 megapixel camera, total size 4.5MB.

G1.5 took 7 minutes to upload
G2 to 9 minutes to upload and I unchecked the create thumbnails on upload so it would be faster.

9 minutes for 5 photos from a very small megapixel camera is not good. My Nikon D70 pictures are much larger than these Canon S110 pics.

Any thoughts about these numbers? Could you try an upload of 5 similar pics to your host and measure the time just to see if we're in the same ballpark?

Thanks,
Bobby

BobbyCarp, an uploading benchmark? that's interesting, but not too important. because you view much more often than you upload new pictures.
and RalphTheWonderLlama is right, there are a lot of variables in this uploading. you'd have to make sure that your connection and the connection of powweb is in the exact same state (bandwidth, response times). etc. you'd have to run this "benchmark 20+ times and maybe run also a upload only 1 file and upload 50 files etc. cases.

having said that, you are not the only user who is reporting that g2 is slower than g1. that's true for small galleries. as soon as your gallery gets a little larger, g2 will be faster than g1 because g2 won't suffer from the size of your gallery or only minimally.

and we're planning to make g2 faster. we want to introduce more caching (we cache already a lot on different levels).

I have a question.
I would like to benchmark gallery performance with apache 2.0.56 mysql 4.1.10 and PHP5 on fedora core 4 without any specific compiler optmizations. I have a PII-450 with 128Mb ram with 3g harddrive.
Then I would like to add optimizations such as march=pentium2 etc. to see how will this change the performance. Plus, this machine is mine and connected to LAN. I think that this way there are not many things that can alter the consistance of the result.

Thank you

valiant, well, my question was how to benchmark g2.
I do not expect my setup to be particulary fast. I would like to see some numbers so if I get myself a faster machine (or more memory) I can compare not just by feel but by some number.
I had Mandrake 10 before with similar setup and I feel that FC 4 is slower. Maybe its just my feeling ?

I have a little file on my webiste who give me lots of info about my web hoster,

It need some changes for each Webhoster, but i t should at least give som basic info (Server related info)

Together with phpinfo, you should get pretty good ide about the server your site is running on (Linux)

See attachment info.php

Very simple php system commands

it display this info: (and some more) http://gallery.nenni.net/info.php

Uptime:
8:25pm up 1 day, 1:56, 0 users, load average: 2.86, 2.67, 2.78

System Information:
Linux node10.c1 2.4.20-28.8smp #1 SMP Thu Dec 18 12:25:21 EST 2003 i686 i686 i386 GNU/Linux

CPU Information:
processor : 0
model name : Intel(R) Pentium(R) 4 CPU 3.00GHz
processor : 1
model name : Intel(R) Pentium(R) 4 CPU 3.00GHz

Memory Usage (MB):
total used free shared buffers cached
Mem: 1999 1984 15 0 20 1041
-/+ buffers/cache: 922 1077
Swap: 1027 92 934

More memory info (vmstat):
>> Ask google, he will tell...
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
1 3 0 94916 15608 21288 1066536 5 44 7 53 166 126 6 6 89

Disk Usage:
Filesystem Size Used Avail Use% Mounted on
/dev/hda3 35G 1.4G 31G 4% /
/dev/hda1 197M 15M 173M 8% /boot
none 1000M 0 999M 0% /dev/shm
super.c1:/exports 106G 91G 9.8G 91% /imports
storage.c1:/home 458G 302G 133G 70% /mounted-storage/home
storage2.c1:/home 223G 200G 12G 95% /mounted-storage/home2
storage3.c1:/home 362G 330G 14G 96% /mounted-storage/home3
storage4.c1:/home 360G 143G 200G 42% /mounted-storage/home4
storage5.c1:/home 367G 135G 214G 39% /mounted-storage/home5

Ip address:
192.168.10.10

Process info:
>> Change this as you like
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 Jul19 ? 00:00:05 init [3]
root 2 1 0 Jul19 ? 00:00:00 [keventd]

Where is the attached file? I only see a link, and then I cant see the source.

I can be course I did outsource my private home servers to a Web hosting company, and pay only 7,50€ a month... :D I did have Quad CPU Server at home, but the Electricity was to expensive... I save many many $$$$ by outsource this... :D And it does look good at this Webhost, at least my little script show at the CPU and memory usage is very low in average...

don't wonder if it doesn't work on your (freebsd) host.
i've googled a lot a few months ago and the
cat /proc/cpuinfo
or a few others commands / files used here are not available on all systems. it will work on most linux distributions though.

I have many years of experience with performance tuning on software, so I understand the need to benchmark similar test cases over different times of day, etc. with the fluctuations in Internet traffic & other issues.

That being said, I ran 3 tests this morning which I think highlights the performance problem with gallery and isolates it to a specific functional area and removes the concern about my PowWeb web host server performance.

All 3 tests involved the upload of the same 5 pictures which total 4.4MB.

Test 1: Upload 5 pictures using the "From Web Browser" tab in the Add Items page. Turned off the create thumbnails now check box.
Time: 7 min 30 secs

Test 2: Upload 5 pictures to my web server using FTP.
Time 1 min 58 secs

Test 3: Upload 5 pictures using the "Upload Applet" tab in the Add Items page.
Time: 2 min 36 secs

As you can see, the FTP and Upload Applet are very similar in performance, I would expect the applet to take longer as it involves server logic that has to run in addition to the transfer. During the applet upload, the speed shown in the dialog was approximately 38kb/s.

Something is clearly wrong with the upload from web browser functionality. Can a gallery developer help me instrument the code to figure out where the time is being spent?

Thanks,
Bobby

I just did add items from server for the 5 files and left the create thumbnails now on. It took 22 seconds.

For more info, I include my DSL connection speed info below:

I also ran broadbandrepors test on my connection. I get 1.277 Mbps upload and 319 kbps download. 319 kbps is 31 kps which is around what the upload applet told me.

The 4.4MB for 5 files is 4296K / 31 kps = 138 seconds. This is 2 min 18 secs which really validates my connection working good for FTP and the upload applet.

I have to leave for the day, so I can pick this up tomorrow.

It also would help to gain some performance when g2 wouldnt create thumbnails parallel when viewing fresh added pictures without creating thumbnails first. When i do so, G2 starts 12x times "convert" to do the thumbnails and resizes at ones. This really sucks much power and mainly freeze my server. (amd1700+, 256mb ram, turckmmcache 8mb) Would be much nicer to get the thumbs generated in a "serial" way. Maybe this is possible? (Maintenance-Module do it serial not parallel).

cheers

doc

Thats right, but when i use Gallery Remote it wont create the thumbs for me. Does it also create the resizes at upload time? I dont think so at all.

Is it possible to get info from the Developers about what we should expect in performance?

Response time, or some info how we should test (like the info from valiant)

Or some simple performance test we can use to test our G2 install, so we do not need to complain so much... I did try ab from apache project, but I am not so sure if that give correct results, we need some methods we can use as "standard"...

When I say at Gallery is slow, it is when I Display the Gallery, or access for the first time... (My ADSL is 3700/600, but at work I have BIG connection to Internet) and I get very similar results when testing from home and from work.

Things like 5 standard photos in diffrent size, or standard configuration on one test album, so we can test this photos against standard album (using same toolkit, same thumb size, same anything...)

doctormord, you get 12 convert running because your browser requests all 12 images at once. apache will spawn a new thread or instance to serve each request via g2. g2 realizes that it needs to convert the image first so spawns convert. The threads/instances are not aware of the fact that 11 other ones are already running. Your browser wants the image now, so apache/g2 tries to give it to it.

Technically you could configure your apache to not run multiple instances or threads, but that would just kill your performance for everything and potentially cause your browser to time out...

Instead, I would recommend you run site admin -> maintenance -> build thumbs/resizes like valiant suggested above. I usually do this once a day if I have uploaded new pictures on that day.

For people who complain about performance. Think about what you really need and choose the correct Gallery version based on that:

If you just want a gallery with less than 5-10,000 pictures. Gallery 1 may be your choice. It is not database driven and is quick.

If you want more than 10,000 pictures G2 is your choice. G2 scales very nicely to 100,000+ pictures. But there is an overhead for this.

So just because it's version 2 it doesn't mean it's better and it's what you need. I think Gallery 1 & 2 are complimentary tools of each other addressing different needs.

So, if you have an older linux box and want to share your pictures with your family use Gallery1. G2 will have an added db overhead regardless if you're sharing 100 pictures or 100,000.

Valiant,

I have spent a lot of time isolating and further understanding this issue (too much time!). I did what you requested putting an exit; right in main.php after starting the upload and found that all of the time is in the upload. In order to understand if this was a PHP vs. Perl thing, I downloaded and tested both a PHP and a Perl CGI upload script and that they were both as slow as the gallery upload.

I then went up to my notebook upstairs to run the test, which it turns out I should have done first to compare performance with my desktop! Gallery upload tests using Firefox were 2:01 for "From Web Browser" form and 2:10 "Upload Applet" which is dead on with FTP times.

It turns out that the CPU on the notebook is around 2-5% utilization during the upload. On the desktop the CPU is 100% for the first 3 minutes and then 2-5% for the final 1 minute. I turned off ZoneAlarm load on startup and rebooted my desktop to make sure that ZA was not causing this CPU usage and it isn't the cause. iexplorer process in the Task Manager shows to be the one using all the CPU, but now the question is why? What is configured on my TCP/IP on the desktop that is causing IE to peg the CPU? Any ideas for further digging into this problem would be appreciated.

The setup is a DSL modem to a Linksys router. The desktop has a cable connect to the router and the notebook is using wireless.

I hope when we figure this out, others can benefit from our findings...

Cheers,
Bobby