Upgrade fail [Function](): Stage2 failed... Exit Code: 254

Ask your questions regarding TimeTrex installation here.
frsanc
Posts: 6
Joined: Thu May 07, 2015 8:47 am

Upgrade fail [Function](): Stage2 failed... Exit Code: 254

Post by frsanc »

Good day,

This morning we can not login to timetrex and it looks like at 4:52 on Nov. 26th an upgrade was ran. Here is the last part of the /var/log/timetrex/timetrex.log file that I found:

DEBUG [L0662] [143875ms]: [Function](): Stage2 failed... Exit Code: 254
DEBUG [L0051] [143879ms] Array: SystemSettingFactory::isUniqueName(): Unique Name: auto_upgrade_failed
string(2) "13"


DEBUG [L0184] [143879ms]: SystemSettingFactory::setSystemSetting(): Key: auto_upgrade_failed Value: 1 isNew: 0
DEBUG [L1864] [143879ms]: Factory::StartTransaction(): StartTransaction(): Transaction: Count: 0 Off: 0 OK: 0
DEBUG [L2002] [143879ms]: Factory::Save(): Calling preSave()
DEBUG [L2060] [143879ms]: Factory::Save(): Updating...
DEBUG [L1833] [143880ms]: Factory::getUpdateQuery(): Data changed, set updated date:
DEBUG [L0072] [143882ms]: Debug::Shutdown(): PHP ERROR - FATAL(1): Class 'TTUUID' not found File: /var/www/html/timetrex/classes/modules/core/TTLog.class.php Line: 72
DEBUG [L0433] [143882ms]: Debug::Shutdown(): Detected PHP errors (1), emailing log...
DEBUG [L0434] [143882ms]: Debug::Shutdown(): ---------------[ 26-Nov-2017 4:53:11 -0600 [1511693591.7496] (PID: 13879) ]---------------
DEBUG [L0281] [144128ms] Array: TTMail::Send(): Attempting to send email To:
string(19) "errors@timetrex.com"


DEBUG [L0293] [144128ms]: TTMail::Send(): Sending Email: Body Size: 172127 Method: soap To:
DEBUG [L0333] [144128ms]: TTMail::Send(): 0. Recipient: errors@timetrex.com
DEBUG [L0399] [144878ms] Array: TTMail::Send(): Send Email Failed To: errors@timetrex.com
bool(false)


DEBUG [L0409] [144878ms] Array: TTMail::Send(): Send Email Failed!
bool(false)


---------------[ 26-Nov-2017 4:53:12 -0600 [1511693592.7466] (PID: 13879) ]---------------

I believe I already fixed the issue that was preventing the server from sending emails but I am not sure what needs to be sent to errors@timetrex.com

Let me know if you need any additional information and thank you for your help,

Franklin.
shaunw
Posts: 7839
Joined: Tue Sep 19, 2006 2:22 pm

Re: Upgrade fail [Function](): Stage2 failed... Exit Code: 254

Post by shaunw »

Please see this FAQ: viewtopic.php?f=6&t=89
frsanc
Posts: 6
Joined: Thu May 07, 2015 8:47 am

Re: Upgrade fail [Function](): Stage2 failed... Exit Code: 254

Post by frsanc »

Hello,

The current version showing on the login screen is 11.0.0-20171020-105207
When entering any username and password I get an error that says "Username or password is incorrect". I am using the local timetrex database for authentication (no AD).
This instance of timetrex was installed manually from the .zip file.
PHP 5.5.26
Mysql 5.5.35
Ubuntu 12.04
Attached the log entries created after turning on debugs according to the post you referred me to.

Thanks.
Attachments
timetrex.log
(55.95 KiB) Downloaded 4218 times
shaunw
Posts: 7839
Joined: Tue Sep 19, 2006 2:22 pm

Re: Upgrade fail [Function](): Stage2 failed... Exit Code: 254

Post by shaunw »

Unfortunately that log only shows this afternoon, could you please attach a copy of the log file that covers the time range of when the upgrade took place? That should hopefully show the actual error that prevented the upgrade from completely successfully.
frsanc
Posts: 6
Joined: Thu May 07, 2015 8:47 am

Re: Upgrade fail [Function](): Stage2 failed... Exit Code: 254

Post by frsanc »

Hello,

Attached is another log file, this one should show everything that happened from Nov 26th during the upgrade process until this afternoon when I turned on debug on timetrex and attempted to login. Thank you.
Attachments
timetrex(3).log
(2.98 MiB) Downloaded 4237 times
shaunw
Posts: 7839
Joined: Tue Sep 19, 2006 2:22 pm

Re: Upgrade fail [Function](): Stage2 failed... Exit Code: 254

Post by shaunw »

So this appears to be the actual error:

Code: Select all

DEBUG [L0273] [36273ms]: InstallSchema_Base::_InstallSchema(): SQL Command failed on line: 821 of: 1100A
DEBUG [L0062] [36273ms]: DBError::__construct(): Begin Exception...
DEBUG [L0063] [36274ms] Array: DBError::__construct():  BackTrace: 
string(917) "#1.Debug::backTrace() /var/www/html/timetrex/classes/modules/core/Exception.class.php:63
#2.DBError->__construct(Object(ADODB_Exception)) /var/www/html/timetrex/classes/modules/install/InstallSchema_Base.class.php:275
#3.InstallSchema_Base->_InstallSchema() /var/www/html/timetrex/classes/modules/install/InstallSchema_Base.class.php:325
#4.InstallSchema_Base->InstallSchema() N/A:N/A
#5.call_user_func_array(Array(2), Array(0)) /var/www/html/timetrex/classes/modules/install/InstallSchema.class.php:131
#6.InstallSchema->__call(InstallSchema, Array(0)) /var/www/html/timetrex/classes/modules/install/Install.class.php:792
#7.InstallSchema->InstallSchema() /var/www/html/timetrex/classes/modules/install/Install.class.php:792
#8.Install->createSchema(1100A) /var/www/html/timetrex/classes/modules/install/Install.class.php:695
#9.Install->createSchemaRange(, ) /var/www/html/timetrex/tools/unattended_upgrade.php:415
"


DEBUG [L0084] [36274ms]: DBError::__construct(): Code: DBError(1054) Message: mysqli error: [1054: Unknown column 'new_user_report_data_id' in 'payroll_remittance_agency_event'] in adodb_throw(
DEBUG [L0084] [36274ms]: DBError::__construct(): 
DEBUG [L0084] [36274ms]: DBError::__construct(): ALTER TABLE `payroll_remittance_agency_event` DROP COLUMN `user_report_data_id`,
DEBUG [L0084] [36274ms]: DBError::__construct():   CHANGE `new_user_report_data_id` `user_report_data_id` char(36),
DEBUG [L0084] [36274ms]: DBError::__construct():   ADD COLUMN `new_user_report_data_id` char(36) AFTER `user_report_data_id`, )
DEBUG [L0091] [36274ms] Array: DBError::__construct(): Exception...
string(1284) "                                                  ADOConnection._Execute(  ALTER TABLE `payroll_remittance_agency_event` DROP COLUMN `user_report_data_id`,   CHANGE `new_user_report_data_id` `user_repo..., false)% line 1229, file
: /var/www/html/timetrex/classes/adodb/adodb.inc.php
                                                ADOConnection.Execute(  ALTER TABLE `payroll_remittance_agency_event` DROP COLUMN `user_report_data_id`,   CHANGE `new_user_report_data_id` `user_repo...)% line  270, file: /var/www/html/timetre
x/classes/modules/install/InstallSchema_Base.class.php
                                        InstallSchema_Base._InstallSchema()% line  325, file: /var/www/html/timetrex/classes/modules/install/InstallSchema_Base.class.php
                                InstallSchema_Base.InstallSchema()% line    0, file: 
                        call_user_func_array(Array[2], Array[0])% line  131, file: /var/www/html/timetrex/classes/modules/install/InstallSchema.class.php
                InstallSchema.__call(InstallSchema, Array[0])% line  792, file: /var/www/html/timetrex/classes/modules/install/Install.class.php
        InstallSchema.InstallSchema()% line  792, file: /var/www/html/timetrex/classes/modules/install/Install.class.php
Install.createSchema(1100A)% line  695, file: /var/www/html/timetrex/classes/modules/install/Install.class.php
Install.createSchemaRange(null, null)% line  415, file: /var/www/html/timetrex/tools/unattended_upgrade.php
"
Which doesn't make a lot of sense, since that query works fine on our test systems here and it just ran about 800 similar queries immediately before that, but I'm not seeing any other error. I wouldn't be surprised if its a bug or some other strange failure in MySQL though, v5.5 of MySQL is past its end-of-life date, as is Ubuntu 12.04.

Either way, because you are using MySQL you will need to restore your database from a backup before the upgrade took place and attempt the upgrade again from there. I would recommend downloading TimeTrex from our website first and using that to perform the upgrade just in case though.
NCrawler
Posts: 3
Joined: Mon Nov 27, 2017 1:09 pm

Re: Upgrade fail [Function](): Stage2 failed... Exit Code: 254

Post by NCrawler »

I have the same issue on our Ubuntu 15.04 server running MySQL 5.6.28. Tomorrow I will reinstall Timetrex on a more current Ubuntu machine. If I export the structure and data from the current installation, import it into the new installation and copy the salt value from the current timetrex.ini.php to the new one is that all that would be required to get it back up and running?
NCrawler
Posts: 3
Joined: Mon Nov 27, 2017 1:09 pm

Re: Upgrade fail [Function](): Stage2 failed... Exit Code: 254

Post by NCrawler »

Further info. I've just moved my installation to an Ubuntu 16.10 system, running MySQL 5.7.18 and this is the error email I got when trying to set it up:

Code: Select all

DEBUG [L0269] [73954ms]: InstallSchema_Base::_InstallSchema():   Executing SQL command: 820 of: 1806
DEBUG [L0269] [74135ms]: InstallSchema_Base::_InstallSchema():   Executing SQL command: 821 of: 1806
DEBUG [L0273] [74136ms]: InstallSchema_Base::_InstallSchema(): SQL Command failed on line: 821 of: 1100A
DEBUG [L0062] [74136ms]: DBError::__construct(): Begin Exception...
DEBUG [L0063] [74136ms] Array: DBError::__construct():  BackTrace: 
<pre class='xdebug-var-dump' dir='ltr'>
<small>/usr/share/timetrex/html/classes/modules/core/Debug.class.php:356:</small><small>string</small> <font color='#cc0000'>'#1.Debug::backTrace() /usr/share/timetrex/html/classes/modules/core/Except
ion.class.php:63&#10;#2.DBError-&gt;__construct(Object(ADODB_Exception)) /usr/share/timetrex/html/classes/modules/install/InstallSchema_Base.class.php:275&#10;#3.InstallSchema_Base-&gt;_InstallSchema(
) /usr/share/timetrex/html/classes/modules/install/InstallSchema_Base.class.php:325&#10;#4.InstallSchema_Base-&gt;InstallSchema() /usr/share/timetrex/html/classes/modules/install/InstallSchema.class.p
hp:131&#10;#5.InstallSchema-&gt;__call(InstallSchema, Array(0)) /u'...</font> <i>(length=1058)</i>
</pre>

DEBUG [L0084] [74136ms]: DBError::__construct(): Code: DBError(1054) Message: mysqli error: [1054: Unknown column 'new_user_report_data_id' in 'payroll_remittance_agency_event'] in adodb_throw(
DEBUG [L0084] [74136ms]: DBError::__construct(): 
DEBUG [L0084] [74136ms]: DBError::__construct(): ALTER TABLE `payroll_remittance_agency_event` DROP COLUMN `user_report_data_id`,
DEBUG [L0084] [74136ms]: DBError::__construct():   CHANGE `new_user_report_data_id` `user_report_data_id` char(36),
DEBUG [L0084] [74136ms]: DBError::__construct():   ADD COLUMN `new_user_report_data_id` char(36) AFTER `user_report_data_id`, )
DEBUG [L0091] [74142ms] Array: DBError::__construct(): Exception...
<pre class='xdebug-var-dump' dir='ltr'>
<small>/usr/share/timetrex/html/classes/modules/core/Debug.class.php:356:</small><small>string</small> <font color='#cc0000'>'							ADOConnection._Execute(  ALTER TABLE `payroll_remittance_agency_eve
nt` DROP COLUMN `user_report_data_id`,   CHANGE `new_user_report_data_id` `user_repo..., false)% line 1229, file: /usr/share/timetrex/html/classes/adodb/adodb.inc.php&#10;						ADOConnection.Execute( 
 ALTER TABLE `payroll_remittance_agency_event` DROP COLUMN `user_report_data_id`,   CHANGE `new_user_report_data_id` `user_repo...)% line  270, file: /usr/share/timetrex/html/classes/modules/install/I
nstallSchema_Base.class.php&#10;					InstallSc'...</font> <i>(length=1428)</i>
</pre>

DEBUG [L0094] [74142ms]: DBError::__construct(): End Exception...
DEBUG [L0281] [74177ms] Array: TTMail::Send(): Attempting to send email To: 
<pre class='xdebug-var-dump' dir='ltr'>
<small>/usr/share/timetrex/html/classes/modules/core/Debug.class.php:356:</small><small>string</small> <font color='#cc0000'>'[Redacted]</font> <i>(length=27)</i>
</pre>

DEBUG [L0293] [74177ms]: TTMail::Send(): Sending Email: Body Size: 10369 Method: soap To: 
DEBUG [L0333] [74177ms]: TTMail::Send(): 0. Recipient: [Redacted]
DEBUG [L0333] [74180ms]: Factory::saveCache(): WARNING: Unable to write cache file, likely due to permissions or locking! Cache ID: 78590976-a7f5-6627-2364-000000000000 Table: company File: /var/cache/timetrex/cache_c/cache_company_78590976-a7f5-6
DEBUG [L0333] [74180ms]: Factory::saveCache(): 627-2364-000000000000
DEBUG [L0474] [74310ms] Array: APIFactory::returnHandler(): returnHandler v2 ERROR: 0
<pre class='xdebug-var-dump' dir='ltr'>
<small>/usr/share/timetrex/html/classes/modules/core/Debug.class.php:356:</small>
<b>array</b> <i>(size=2)</i>
  'api_retval' <font color='#888a85'>=&gt;</font> <small>boolean</small> <font color='#75507b'>false</font>
  'api_details' <font color='#888a85'>=&gt;</font> 
    <b>array</b> <i>(size=7)</i>
      'code' <font color='#888a85'>=&gt;</font> <small>string</small> <font color='#cc0000'>'EXCEPTION'</font> <i>(length=9)</i>
      'description' <font color='#888a85'>=&gt;</font> <small>string</small> <font color='#cc0000'>'TimeTrex is unable to connect to its database, please make sure that the database service on your ow
n local TimeTrex server has been started and is running. If you are unsure, try rebooting your server.'</font> <i>(length=202)</i>
      'record_details' <font color='#888a85'>=&gt;</font> 
        <b>array</b> <i>(size=3)</i>
          'total' <font color='#888a85'>=&gt;</font> <small>int</small> <font color='#4e9a06'>0</font>
          'valid' <font color='#888a85'>=&gt;</font> <small>int</small> <font color='#4e9a06'>0</font>
          'invalid' <font color='#888a85'>=&gt;</font> <small>int</small> <font color='#4e9a06'>0</font>
      'user_generic_status_batch_id' <font color='#888a85'>=&gt;</font> <small>boolean</small> <font color='#75507b'>false</font>
      'request' <font color='#888a85'>=&gt;</font> <small>boolean</small> <font color='#75507b'>false</font>
      'pager' <font color='#888a85'>=&gt;</font> <small>boolean</small> <font color='#75507b'>false</font>
      'details' <font color='#888a85'>=&gt;</font> <small>boolean</small> <font color='#75507b'>false</font>
</pre>

DEBUG [L0126] [74311ms]: ProgressBar::start(): start: '' Iterations: 9999 Update Iterations: 9999 Key: (1511881954.1144) Message: TimeTrex is unable to connect to its database, please make sure that the database service on your own local TimeTrex 
DEBUG [L0126] [74311ms]: ProgressBar::start(): server has been started and is running. If you are unsure, try rebooting your server.
DEBUG [L0599] [74311ms]: Debug::Shutdown(): ERROR: Detected UNCOMMITTED transaction: Count: 1 Off: 2 OK: 0, emailing log...
DEBUG [L0607] [74311ms]: Debug::Shutdown(): ---------------[ 28-Nov-2017 15:12:34 +0000 [1511881954.1146] (PID: 17770) ]---------------
Could it be missing a table in my database or something?
pilot
Posts: 7
Joined: Tue Nov 28, 2017 1:40 pm

Re: Upgrade fail [Function](): Stage2 failed... Exit Code: 254

Post by pilot »

It seems like I am getting the same error. On Tue Nov 28, after running a cron job, TimeTrex (10.6.1) failed to upgrade to the current version. As a result, I am not able to log in, it does not recognize user name/password. An attempt to install the current version manually was unsuccessful. The current version of TimeTrex cannot connect to the database. Looks like a bug. I tried to reinstall the original version 10.6.1, it went OK, but still no luck with log in. I tried to do:

#php tools/set_admin_permissions.php <username>, and use a prefix password to override the actual password. It seems it could find user, but still does not allow to log in:

Found user, apply administrator permissions...
WARNING: Clear TimeTrex cache after running this.

I wonder if there is any way to reset user name and password ? Thanks.

This is the actual error form timetrex.log:
********************************
DEBUG [L0273] [104715ms]: InstallSchema_Base::_InstallSchema(): SQL Command failed on line: 821 of: 1100A
DEBUG [L0062] [104715ms]: DBError::__construct(): Begin Exception...
DEBUG [L0063] [104715ms] Array: DBError::__construct(): BackTrace:
string(917) "#1.Debug::backTrace() /var/www/html/timetrex/classes/modules/core/Exception.class.php:63
#2.DBError->__construct(Object(ADODB_Exception)) /var/www/html/timetrex/classes/modules/install/InstallSchema_Base.class.php:275
#3.InstallSchema_Base->_InstallSchema() /var/www/html/timetrex/classes/modules/install/InstallSchema_Base.class.php:325
#4.InstallSchema_Base->InstallSchema() N/A:N/A
#5.call_user_func_array(Array(2), Array(0)) /var/www/html/timetrex/classes/modules/install/InstallSchema.class.php:131
#6.InstallSchema->__call(InstallSchema, Array(0)) /var/www/html/timetrex/classes/modules/install/Install.class.php:792
#7.InstallSchema->InstallSchema() /var/www/html/timetrex/classes/modules/install/Install.class.php:792
#8.Install->createSchema(1100A) /var/www/html/timetrex/classes/modules/install/Install.class.php:695
#9.Install->createSchemaRange(, ) /var/www/html/timetrex/tools/unattended_upgrade.php:415
"


DEBUG [L0084] [104727ms]: DBError::__construct(): Code: DBError(1054) Message: mysqli error: [1054: Unknown column 'new_user_report_data_id' in 'payroll_remittance_agency_event'] in adodb_throw(
DEBUG [L0084] [104727ms]: DBError::__construct():
DEBUG [L0084] [104727ms]: DBError::__construct(): ALTER TABLE `payroll_remittance_agency_event` DROP COLUMN `user_report_data_id`,
DEBUG [L0084] [104727ms]: DBError::__construct(): CHANGE `new_user_report_data_id` `user_report_data_id` char(36),
DEBUG [L0084] [104727ms]: DBError::__construct(): ADD COLUMN `new_user_report_data_id` char(36) AFTER `user_report_data_id`, )
DEBUG [L0091] [104736ms] Array: DBError::__construct(): Exception...
string(1284) " ADOConnection._Execute( ALTER TABLE `payroll_remittance_agency_event` DROP COLUMN `user_report_data_id`, CHANGE `new_user_report_data_id` `user_repo..., false)% line 1229, file
: /var/www/html/timetrex/classes/adodb/adodb.inc.php
shaunw
Posts: 7839
Joined: Tue Sep 19, 2006 2:22 pm

Re: Upgrade fail [Function](): Stage2 failed... Exit Code: 254

Post by shaunw »

NCrawler wrote: Mon Nov 27, 2017 10:01 pm I have the same issue on our Ubuntu 15.04 server running MySQL 5.6.28. Tomorrow I will reinstall Timetrex on a more current Ubuntu machine. If I export the structure and data from the current installation, import it into the new installation and copy the salt value from the current timetrex.ini.php to the new one is that all that would be required to get it back up and running?
Once an upgrade/install fails with MySQL, you must restore the entire database from a backup prior to the upgrade/install taking place, otherwise its essentially corrupted and in an unknown state. Migrating a corrupted database to any other server will just result in a still corrupted database.

This is just one of the big reasons why we don't recommend using MySQL, as PostgreSQL doesn't have such a huge deficiency and will happily continue from where any possible failure left off without any corruption.
shaunw
Posts: 7839
Joined: Tue Sep 19, 2006 2:22 pm

Re: Upgrade fail [Function](): Stage2 failed... Exit Code: 254

Post by shaunw »

pilot wrote: Tue Nov 28, 2017 9:04 pm It seems like I am getting the same error. On Tue Nov 28, after running a cron job, TimeTrex (10.6.1) failed to upgrade to the current version. As a result, I am not able to log in, it does not recognize user name/password. An attempt to install the current version manually was unsuccessful. The current version of TimeTrex cannot connect to the database. Looks like a bug. I tried to reinstall the original version 10.6.1, it went OK, but still no luck with log in. I tried to do:

#php tools/set_admin_permissions.php <username>, and use a prefix password to override the actual password. It seems it could find user, but still does not allow to log in:

Found user, apply administrator permissions...
WARNING: Clear TimeTrex cache after running this.

I wonder if there is any way to reset user name and password ? Thanks.

This is the actual error form timetrex.log:
********************************
DEBUG [L0273] [104715ms]: InstallSchema_Base::_InstallSchema(): SQL Command failed on line: 821 of: 1100A
DEBUG [L0062] [104715ms]: DBError::__construct(): Begin Exception...
DEBUG [L0063] [104715ms] Array: DBError::__construct(): BackTrace:
string(917) "#1.Debug::backTrace() /var/www/html/timetrex/classes/modules/core/Exception.class.php:63
#2.DBError->__construct(Object(ADODB_Exception)) /var/www/html/timetrex/classes/modules/install/InstallSchema_Base.class.php:275
#3.InstallSchema_Base->_InstallSchema() /var/www/html/timetrex/classes/modules/install/InstallSchema_Base.class.php:325
#4.InstallSchema_Base->InstallSchema() N/A:N/A
#5.call_user_func_array(Array(2), Array(0)) /var/www/html/timetrex/classes/modules/install/InstallSchema.class.php:131
#6.InstallSchema->__call(InstallSchema, Array(0)) /var/www/html/timetrex/classes/modules/install/Install.class.php:792
#7.InstallSchema->InstallSchema() /var/www/html/timetrex/classes/modules/install/Install.class.php:792
#8.Install->createSchema(1100A) /var/www/html/timetrex/classes/modules/install/Install.class.php:695
#9.Install->createSchemaRange(, ) /var/www/html/timetrex/tools/unattended_upgrade.php:415
"


DEBUG [L0084] [104727ms]: DBError::__construct(): Code: DBError(1054) Message: mysqli error: [1054: Unknown column 'new_user_report_data_id' in 'payroll_remittance_agency_event'] in adodb_throw(
DEBUG [L0084] [104727ms]: DBError::__construct():
DEBUG [L0084] [104727ms]: DBError::__construct(): ALTER TABLE `payroll_remittance_agency_event` DROP COLUMN `user_report_data_id`,
DEBUG [L0084] [104727ms]: DBError::__construct(): CHANGE `new_user_report_data_id` `user_report_data_id` char(36),
DEBUG [L0084] [104727ms]: DBError::__construct(): ADD COLUMN `new_user_report_data_id` char(36) AFTER `user_report_data_id`, )
DEBUG [L0091] [104736ms] Array: DBError::__construct(): Exception...
string(1284) " ADOConnection._Execute( ALTER TABLE `payroll_remittance_agency_event` DROP COLUMN `user_report_data_id`, CHANGE `new_user_report_data_id` `user_repo..., false)% line 1229, file
: /var/www/html/timetrex/classes/adodb/adodb.inc.php
The upgrade has failed, so attempting to reset the password or do anything other than restoring from a backup prior to the when the upgrade failed isn't likely to help. What OS and version of MySQL are you using?
shaunw
Posts: 7839
Joined: Tue Sep 19, 2006 2:22 pm

Re: Upgrade fail [Function](): Stage2 failed... Exit Code: 254

Post by shaunw »

So far it seems this error is occurring with versions of MySQL that are past their end-of-life, however we would like to be sure so if any of you are willing to zip and email support@timetrex.com your database dump prior to the upgrade failing we will be happy to investigate the issue further. Please make sure that you reference this thread in your email though.
NCrawler
Posts: 3
Joined: Mon Nov 27, 2017 1:09 pm

Re: Upgrade fail [Function](): Stage2 failed... Exit Code: 254

Post by NCrawler »

shaunw wrote: Tue Nov 28, 2017 11:59 pm
NCrawler wrote: Mon Nov 27, 2017 10:01 pm I have the same issue on our Ubuntu 15.04 server running MySQL 5.6.28. Tomorrow I will reinstall Timetrex on a more current Ubuntu machine. If I export the structure and data from the current installation, import it into the new installation and copy the salt value from the current timetrex.ini.php to the new one is that all that would be required to get it back up and running?
Once an upgrade/install fails with MySQL, you must restore the entire database from a backup prior to the upgrade/install taking place, otherwise its essentially corrupted and in an unknown state. Migrating a corrupted database to any other server will just result in a still corrupted database.

This is just one of the big reasons why we don't recommend using MySQL, as PostgreSQL doesn't have such a huge deficiency and will happily continue from where any possible failure left off without any corruption.
Ok, I understand that. Unfortunately I do not have a backup prior to this failure, so would I be able to reinstall TT using PostgreSQL, change the salt value in my config and then import the data from the failed MySQL database? Were there many/any database structure changes in this latest update?
shaunw
Posts: 7839
Joined: Tue Sep 19, 2006 2:22 pm

Re: Upgrade fail [Function](): Stage2 failed... Exit Code: 254

Post by shaunw »

NCrawler wrote: Mon Nov 27, 2017 10:01 pm Ok, I understand that. Unfortunately I do not have a backup prior to this failure, so would I be able to reinstall TT using PostgreSQL, change the salt value in my config and then import the data from the failed MySQL database? Were there many/any database structure changes in this latest update?
Unfortunately that won't work either, since the database is still corrupted, so converting it to PostgreSQL will result in a still corrupted database.

Its possible that our developers could try to recover the database for you, however this process often requires a fairly substantial amount of manual work, so there would be a fee for that. You can email sales@timetrex.com for more information if you are interested.
poccadmin
Posts: 2
Joined: Tue Nov 28, 2017 9:17 am

Re: Upgrade fail [Function](): Stage2 failed... Exit Code: 254

Post by poccadmin »

Unfortunately, we are in the same situation as you, an automatic upgrade failed, database is now corrupted, no backup to restore from. . . In attempting to fix this issue, I have gone through every route I can find only to fail at every step.

As we're really only using timetrex for it's attendance features, it's not life or death if we start fresh so I'm trying that route now. It seems that the latest version of timetrex really doesn't like mysql one bit and even installing fresh on mysql fails. I'm going a new route and installing the Windows version to try that but if this fails, we may be forced to look at alternatives.
shaunw
Posts: 7839
Joined: Tue Sep 19, 2006 2:22 pm

Re: Upgrade fail [Function](): Stage2 failed... Exit Code: 254

Post by shaunw »

poccadmin wrote: Wed Nov 29, 2017 9:55 am Unfortunately, we are in the same situation as you, an automatic upgrade failed, database is now corrupted, no backup to restore from. . . In attempting to fix this issue, I have gone through every route I can find only to fail at every step.

As we're really only using timetrex for it's attendance features, it's not life or death if we start fresh so I'm trying that route now. It seems that the latest version of timetrex really doesn't like mysql one bit and even installing fresh on mysql fails. I'm going a new route and installing the Windows version to try that but if this fails, we may be forced to look at alternatives.
We run daily tests with MySQL and fresh installs are working on all of our test systems. So there is something similar with all these MySQL failures, and so far the common denominator is a old version of MySQL. Can you please confirm what version of MySQL you are using?
poccadmin
Posts: 2
Joined: Tue Nov 28, 2017 9:17 am

Re: Upgrade fail [Function](): Stage2 failed... Exit Code: 254

Post by poccadmin »

MySQL Server version: 5.7.20-0ubuntu0.16.04.1 - (Ubuntu)
shaunw
Posts: 7839
Joined: Tue Sep 19, 2006 2:22 pm

Re: Upgrade fail [Function](): Stage2 failed... Exit Code: 254

Post by shaunw »

poccadmin wrote: Wed Nov 29, 2017 12:31 pm MySQL Server version: 5.7.20-0ubuntu0.16.04.1 - (Ubuntu)
Can you please post your TimeTrex log after the failure during fresh install?
shaunw
Posts: 7839
Joined: Tue Sep 19, 2006 2:22 pm

Re: Upgrade fail [Function](): Stage2 failed... Exit Code: 254

Post by shaunw »

I think we may have a solution that might work for some of you at least and it should work on already corrupted MySQL databases to recover them:

****************************************************************************************************************************
Please make sure you have a SQL dump backup of your database prior to running this!
****************************************************************************************************************************

1. In your TimeTrex cache directory (specified in the timetrex.ini.php file), you should find a file called: tt_schema_failure.state -- Edit this file, and you should see this line:

Code: Select all

a:1:{s:5:"1100A";i:821;}
Change the "821" to "830" so it looks like this:

Code: Select all

a:1:{s:5:"1100A";i:830;}
If you do not see this file, or do not see "821" in the file, stop now, this procedure will not work for you and will cause further damage to your database.

2. Download the manual (.ZIP) TimeTrex installer from our website and unzip it over top of your existing installation. Do not use an already downloaded .ZIP file, you must download it from our website again.

3. Connect to your MySQL console and run the following queries:

Code: Select all

SET SQL_MODE='ANSI';
UPDATE `payroll_remittance_agency_event` set `new_created_by` = CASE WHEN ( `created_by` = 0 ) THEN '00000000-0000-0000-0000-000000000000' ELSE CASE WHEN ( `created_by` = -1 ) THEN 'ffffffff-ffff-ffff-ffff-ffffffffffff' ELSE CONCAT('#UUID_PREFIX#-', lpad(`created_by`, 12, '0') ) END END;
ALTER TABLE `payroll_remittance_agency_event` DROP COLUMN `created_by`,
  CHANGE `new_created_by` `created_by` char(36),
  ADD COLUMN `new_updated_by` char(36) AFTER `updated_by`;
UPDATE `payroll_remittance_agency_event` set `new_updated_by` = CASE WHEN ( `updated_by` = 0 ) THEN '00000000-0000-0000-0000-000000000000' ELSE CASE WHEN ( `updated_by` = -1 ) THEN 'ffffffff-ffff-ffff-ffff-ffffffffffff' ELSE CONCAT('#UUID_PREFIX#-', lpad(`updated_by`, 12, '0') ) END END;
ALTER TABLE `payroll_remittance_agency_event` DROP COLUMN `updated_by`,
  CHANGE `new_updated_by` `updated_by` char(36),
    ADD COLUMN `new_deleted_by` char(36) AFTER `deleted_by`;
UPDATE `payroll_remittance_agency_event` set `new_deleted_by` = CASE WHEN ( `deleted_by` = 0 ) THEN '00000000-0000-0000-0000-000000000000' ELSE CASE WHEN ( `deleted_by` = -1 ) THEN 'ffffffff-ffff-ffff-ffff-ffffffffffff' ELSE CONCAT('#UUID_PREFIX#-', lpad(`deleted_by`, 12, '0') ) END END;
ALTER TABLE `payroll_remittance_agency_event` DROP COLUMN `deleted_by`,
  CHANGE `new_deleted_by` `deleted_by` char(36);
ALTER TABLE `payroll_remittance_agency_event` ADD COLUMN `new_user_report_data_id` char(36) AFTER `user_report_data_id`;
UPDATE `payroll_remittance_agency_event` set `new_user_report_data_id` = CASE WHEN ( `user_report_data_id` = 0 ) THEN '00000000-0000-0000-0000-000000000000' ELSE CASE WHEN ( `user_report_data_id` = -1 ) THEN 'ffffffff-ffff-ffff-ffff-ffffffffffff' ELSE CONCAT('#UUID_PREFIX#-', lpad(`user_report_data_id`, 12, '0') ) END END;
ALTER TABLE `payroll_remittance_agency_event` DROP COLUMN `user_report_data_id`,
  CHANGE `new_user_report_data_id` `user_report_data_id` char(36);
4. In the TimeTrex "tools" directory, run: php unattended_upgrade.php -f --schema_only

If the last command completes successfully, you should be back up and running.
JamesFromAustin
Posts: 6
Joined: Wed Nov 29, 2017 5:58 pm

Re: Upgrade fail [Function](): Stage2 failed... Exit Code: 254

Post by JamesFromAustin »

Thanks for helping with this. Am experiencing exact same problem - auto upgrade failed, can't log in. Recovered system via backup and attempted manual install. Same error. Long time user of TimeTrex - 10 years or so. Running MySQL version 5.6.12 on Windows WAMP install.

As added note: I did attempt recover recommended procedure (note that my cache was a:1:{s:5:"1100A";i:1;} ) but tried anyway. Got following error message when running SQL:


Error

SQL query:

UPDATE `payroll_remittance_agency_event` SET `new_created_by` = CASE WHEN (
`created_by` =0
) THEN '00000000-0000-0000-0000-000000000000' ELSE CASE WHEN (
`created_by` = -1
) THEN 'ffffffff-ffff-ffff-ffff-ffffffffffff' ELSE CONCAT( '#UUID_PREFIX#-', lpad( `created_by` , 12, '0' ) ) END END ;

MySQL said: Documentation
#1054 - Unknown column 'new_created_by' in 'field list'
shaunw
Posts: 7839
Joined: Tue Sep 19, 2006 2:22 pm

Re: Upgrade fail [Function](): Stage2 failed... Exit Code: 254

Post by shaunw »

JamesFromAustin wrote: Wed Nov 29, 2017 6:04 pm Thanks for helping with this. Am experiencing exact same problem - auto upgrade failed, can't log in. Recovered system via backup and attempted manual install. Same error. Long time user of TimeTrex - 10 years or so. Running MySQL version 5.6.12 on Windows WAMP install.

As added note: I did attempt recover recommended procedure (note that my cache was a:1:{s:5:"1100A";i:1;} ) but tried anyway. Got following error message when running SQL:
This evening we managed to replicate what appears to be a MySQL bug in some older (less than v5.7) versions of MySQL causing failures that may result in a state file similar to what you are seeing. This is a separate issue from what most of the posts are mentioning in this thread though, but obviously still results in the upgrade failing.

It sounds like you at least have backups, so that being the case we can definitely get you up and running again, you will just need to make sure you restore from backup before trying any upgrade procedure so your database is in a consistent state.

Try downloading the TimeTrex .ZIP file from our website again, and confirm that the 2nd line in classes/modules/install/sql/mysql/1100A.sql has a timestamp of "00:00:01", or 1 second after midnight. If it does, you should be able to run the web-based installer again and see if it works for you. If it fails again, please attach your timetrex.log file here and we will happy to investigate it further.
pilot
Posts: 7
Joined: Tue Nov 28, 2017 1:40 pm

Re: Upgrade fail [Function](): Stage2 failed... Exit Code: 254

Post by pilot »

The same thing is here no back up.

Just in case, my info:

MySQL: mysql Ver 14.14 Distrib 5.5.58, for Linux (x86_64) using readline 5.1
OS: CentOS release 6.9 (Final), 2.6.32-696.16.1.el6.x86_64 GNU/Linux

Thanks for your help.
pilot
Posts: 7
Joined: Tue Nov 28, 2017 1:40 pm

Re: Upgrade fail [Function](): Stage2 failed... Exit Code: 254

Post by pilot »

The procedure worked like a charm !

[root@localhost tools]# php unattended_upgrade.php -f --schema_only
Force Mode enabled...
Upgrade successfull!

Thank you again !
JamesFromAustin
Posts: 6
Joined: Wed Nov 29, 2017 5:58 pm

Re: Upgrade fail [Function](): Stage2 failed... Exit Code: 254

Post by JamesFromAustin »

shaunw wrote: Wed Nov 29, 2017 9:42 pm This evening we managed to replicate what appears to be a MySQL bug in some older (less than v5.7) versions of MySQL causing failures that may result in a state file similar to what you are seeing. This is a separate issue from what most of the posts are mentioning in this thread though, but obviously still results in the upgrade failing.
Just to be clear, I am seeing this error in my log - same as original post - pretty much same problem as everyone is reporting.
DEBUG [L0072] [143882ms]: Debug::Shutdown(): PHP ERROR - FATAL(1): Class 'TTUUID' not found File: /var/www/html/timetrex/classes

I did recover and am running with v10.7.2-20170926, Also disabled autoupdate for now so users can clock in/out. Will attempt this fix later this evening when all the users are off the database (other programs using the db server as well, so I can't take it down during biz hours)

Thanks again for help!
shaunw
Posts: 7839
Joined: Tue Sep 19, 2006 2:22 pm

Re: Upgrade fail [Function](): Stage2 failed... Exit Code: 254

Post by shaunw »

JamesFromAustin wrote: Thu Nov 30, 2017 10:46 am DEBUG [L0072] [143882ms]: Debug::Shutdown(): PHP ERROR - FATAL(1): Class 'TTUUID' not found File: /var/www/html/timetrex/classes
That is usually a result of another failure, not the actual cause of the original failure itself. If you look higher up in the log you should see another error from the database that is the actual cause.
Post Reply