could the G1->G2 migration module save its progress?

timbo

Joined: 2004-08-01
Posts: 10

Posted: Sun, 2005-10-09 22:43

I have a G1 installation with over 40,000 photos that I want to migrate to G2. The migration module is nice, but it always fails part way through. I've tried increasing the memory and execution time settings in PHP. Even when I import albums in small batches, it occasionally dies.

Could the G1 migration module save its progress in a file on the web server so that if the script dies for whatever reason, the user can just hit "reload" and the script continues from where it left off? It could delete the last album (in case it was only partially migrated) before starting again. That would be a huge help to people with large Gallery setups that take a long time and lots of memory to migrate.

Thanks very much,
Tim

Gallery version = 2.0 core 1.0.0 (migrating from 1.5.1)
PHP version = 4.3.10-16 apache
Webserver = Apache/1.3.33 (Debian GNU/Linux) mod_ruby/1.2.4 Ruby/1.8.2(2005-04-11) PHP/4.3.10-16 mod_auth_pam/1.1.1 mod_ssl/2.8.22 OpenSSL/0.9.7e mod_perl/1.29 DAV/1.0.3
Database = mysqlt 4.0.24_Debian-10sarge1-log
Toolkits = ArchiveUpload, ImageMagick, NetPBM
Operating system = Linux okcomputer 2.4.29 #1 SMP Thu Jan 20 20:41:12 MST 2005 i686
Browser = Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.8) Gecko/20050517 Firefox/1.0.4 (Debian package 1.0.4-2)

valiant

Joined: 2003-01-04
Posts: 32509

Posted: Mon, 2005-10-10 07:01

that would be a non-trivial change and we're very busy.

already tried disablign thumbnail creation and disabling the exif module?

what's the memory limit right now? and is memory limit the problem?

timbo

Joined: 2004-08-01
Posts: 10

Posted: Mon, 2005-10-10 08:12

Yep, I disabled exif and unchecked thumbnail creation on import.

memory_limit was set to 512 MB, but I eventually just changed it to -1 (no limit). Didn't make any difference. The counter in the top right of the migration progress page never shows memory consumption going above 30 MB or so anyway.

max_execution_time is set to 60000.

Is there some way to run the migration script from the command line instead of through the web server?

valiant

Joined: 2003-01-04
Posts: 32509

Posted: Mon, 2005-10-10 08:17

@command line:
no, one could rewrite it to make this possible though.

any entry in your apache error log? (not the access log)
and does it always stop at the same item / album?

but the progress bar works?

timbo

Joined: 2004-08-01
Posts: 10

Posted: Mon, 2005-10-10 16:06

Nothing appears in the error log when the import stops. It's not the same item/album that fails each time. I try to import a batch of albums at a time. When it fails, I delete the last that was (partially) imported, then import that one and the next few.

The progress bar works. Firefox stops loading the page when the import stops.

valiant

Joined: 2003-01-04
Posts: 32509

Posted: Mon, 2005-10-10 21:18

well, something must be the reason for this. it doesn't just stop.
is it always after the same time?

jmullan

Joined: 2002-07-28
Posts: 974

Posted: Mon, 2005-10-10 21:31

Try going into the file ConfirmImport.inc and changing all the instances of $gallery->guaranteeTimeLimit(30) to a larger number, like 360.

Did you read this?
http://codex.gallery2.org/index.php/Gallery2:migration

timbo

Joined: 2004-08-01
Posts: 10

Posted: Sun, 2005-10-16 20:56

Sorry for taking so long to respond.

valiant: No, it stopped at random times. When I tried to import the failed albums a second time, they worked fine. So I just kept doing that until I got through all the albums.

jmullan: Yes, I read that thread earlier and changed all instances of $gallery->guaranteeTimeLimit(30) to $gallery->guaranteeTimeLimit(3000) in ConfirmImport.inc. I also inserted that line at about line 1567 in ConfirmImport.inc (right before "while ($albumfields) {" etc.) because it seemed to be stopping there.

At any rate, thanks for the suggestions and all the work on G2. I eventually managed to import all 32 GB.

timbo Joined: 2004-08-01 Posts: 10	Posted: Sun, 2005-10-09 22:43
	I have a G1 installation with over 40,000 photos that I want to migrate to G2. The migration module is nice, but it always fails part way through. I've tried increasing the memory and execution time settings in PHP. Even when I import albums in small batches, it occasionally dies. Could the G1 migration module save its progress in a file on the web server so that if the script dies for whatever reason, the user can just hit "reload" and the script continues from where it left off? It could delete the last album (in case it was only partially migrated) before starting again. That would be a huge help to people with large Gallery setups that take a long time and lots of memory to migrate. Thanks very much, Tim Gallery version = 2.0 core 1.0.0 (migrating from 1.5.1) PHP version = 4.3.10-16 apache Webserver = Apache/1.3.33 (Debian GNU/Linux) mod_ruby/1.2.4 Ruby/1.8.2(2005-04-11) PHP/4.3.10-16 mod_auth_pam/1.1.1 mod_ssl/2.8.22 OpenSSL/0.9.7e mod_perl/1.29 DAV/1.0.3 Database = mysqlt 4.0.24_Debian-10sarge1-log Toolkits = ArchiveUpload, ImageMagick, NetPBM Operating system = Linux okcomputer 2.4.29 #1 SMP Thu Jan 20 20:41:12 MST 2005 i686 Browser = Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.8) Gecko/20050517 Firefox/1.0.4 (Debian package 1.0.4-2)
	XUser login Username: * Password: * Create new account Request new password
valiant Joined: 2003-01-04 Posts: 32509	Posted: Mon, 2005-10-10 07:01
	that would be a non-trivial change and we're very busy. already tried disablign thumbnail creation and disabling the exif module? what's the memory limit right now? and is memory limit the problem?

timbo Joined: 2004-08-01 Posts: 10	Posted: Mon, 2005-10-10 08:12
	Yep, I disabled exif and unchecked thumbnail creation on import. memory_limit was set to 512 MB, but I eventually just changed it to -1 (no limit). Didn't make any difference. The counter in the top right of the migration progress page never shows memory consumption going above 30 MB or so anyway. max_execution_time is set to 60000. Is there some way to run the migration script from the command line instead of through the web server?

valiant Joined: 2003-01-04 Posts: 32509	Posted: Mon, 2005-10-10 08:17
	@command line: no, one could rewrite it to make this possible though. any entry in your apache error log? (not the access log) and does it always stop at the same item / album? but the progress bar works?

timbo Joined: 2004-08-01 Posts: 10	Posted: Mon, 2005-10-10 16:06
	Nothing appears in the error log when the import stops. It's not the same item/album that fails each time. I try to import a batch of albums at a time. When it fails, I delete the last that was (partially) imported, then import that one and the next few. The progress bar works. Firefox stops loading the page when the import stops.

valiant Joined: 2003-01-04 Posts: 32509	Posted: Mon, 2005-10-10 21:18
	well, something must be the reason for this. it doesn't just stop. is it always after the same time?

jmullan Joined: 2002-07-28 Posts: 974	Posted: Mon, 2005-10-10 21:31
	Try going into the file ConfirmImport.inc and changing all the instances of $gallery->guaranteeTimeLimit(30) to a larger number, like 360. Did you read this? http://codex.gallery2.org/index.php/Gallery2:migration

timbo Joined: 2004-08-01 Posts: 10	Posted: Sun, 2005-10-16 20:56
	Sorry for taking so long to respond. valiant: No, it stopped at random times. When I tried to import the failed albums a second time, they worked fine. So I just kept doing that until I got through all the albums. jmullan: Yes, I read that thread earlier and changed all instances of $gallery->guaranteeTimeLimit(30) to $gallery->guaranteeTimeLimit(3000) in ConfirmImport.inc. I also inserted that line at about line 1567 in ConfirmImport.inc (right before "while ($albumfields) {" etc.) because it seemed to be stopping there. At any rate, thanks for the suggestions and all the work on G2. I eventually managed to import all 32 GB.

could the G1->G2 migration module save its progress?

XUser login