Cyclos logs eating up all disk space

Bug and problem reporting on Cyclos 4 version

Moderators: hugo, alexandre, rmvanarkel

Post Reply
atitokor
Posts: 44
Joined: Tue Jan 01, 2019 8:33 pm

Cyclos logs eating up all disk space

Post by atitokor » Mon Jun 24, 2019 6:11 pm

Hello Admins

My logs are filling up so fast consuming the entire disk space that I have to be clearing them every few hours to keep the site running. Already I have doubled my disk space in 24 hours but still that doesn't help. I'm using Cyclos 4.12 on Docker and the issue is related to the Cyclos container.

Below is a most recent segment of the log.

Thanks in advance for your attention.

Art

Code: Select all

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
2019-06-24T20:57:59.209Z WARN PurgeUnconfirmedUsersPollingTask - Error when Expired pending users purging would mark the given entity as processed: User#10: Mubasher M
org.cyclos.model.IllegalActionException: null
        at org.cyclos.impl.users.UserServiceImpl.onBeforeRemove(UserServiceImpl.java:1507) ~[cyclos-impl.jar:4.12]
        at org.cyclos.impl.users.UserServiceImpl.onBeforeRemove(UserServiceImpl.java:1) ~[cyclos-impl.jar:4.12]
        at org.cyclos.impl.CRUDServiceImpl.remove(CRUDServiceImpl.java:908) ~[cyclos-impl.jar:4.12]
        at org.cyclos.impl.CRUDServiceImpl.remove(CRUDServiceImpl.java:264) ~[cyclos-impl.jar:4.12]
        at org.cyclos.impl.users.PurgeUnconfirmedUsersPollingTask.process(PurgeUnconfirmedUsersPollingTask.java:90) ~[cyclos-impl.jar:4.12]
        at org.cyclos.impl.users.PurgeUnconfirmedUsersPollingTask.process(PurgeUnconfirmedUsersPollingTask.java:1) ~[cyclos-impl.jar:4.12]
        at org.cyclos.impl.utils.tasks.EntityBatchPollingTask.lambda$3(EntityBatchPollingTask.java:237) ~[cyclos-impl.jar:4.12]
        at org.cyclos.impl.utils.transaction.TransactionHandlerImpl.runEnsuringInvocationContext(TransactionHandlerImpl.java:183) ~[cyclos-impl.jar:4.12]
        at org.cyclos.impl.utils.transaction.TransactionHandlerImpl.doRun(TransactionHandlerImpl.java:109) ~[cyclos-impl.jar:4.12]
        at org.cyclos.impl.utils.transaction.TransactionHandlerImpl.run(TransactionHandlerImpl.java:159) ~[cyclos-impl.jar:4.12]
        at org.cyclos.impl.utils.transaction.TransactionHandlerImpl.run(TransactionHandlerImpl.java:80) ~[cyclos-impl.jar:4.12]
        at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_212]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_212]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343) ~[spring-aop-5.1.7.RELEASE.jar:5.1.7.RELEASE]
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:206) ~[spring-aop-5.1.7.RELEASE.jar:5.1.7.RELEASE]
        at com.sun.proxy.$Proxy34.run(Unknown Source) ~[?:?]
        at org.cyclos.impl.InvokerHandlerImpl.performInTransaction(InvokerHandlerImpl.java:238) ~[cyclos-impl.jar:4.12]
        at org.cyclos.impl.InvokerHandlerImpl.doRunAsInTransaction(InvokerHandlerImpl.java:187) ~[cyclos-impl.jar:4.12]
        at org.cyclos.impl.InvokerHandlerImpl.runAsInTransaction(InvokerHandlerImpl.java:168) ~[cyclos-impl.jar:4.12]
        at org.cyclos.impl.utils.tasks.EntityBatchPollingTask.doProcessAndFinish(EntityBatchPollingTask.java:214) ~[cyclos-impl.jar:4.12]
        at org.cyclos.impl.utils.tasks.EntityBatchPollingTask.doRun(EntityBatchPollingTask.java:81) ~[cyclos-impl.jar:4.12]
        at org.cyclos.impl.utils.tasks.AbstractPollingTask.run(AbstractPollingTask.java:77) ~[cyclos-impl.jar:4.12]
        at org.cyclos.impl.utils.cluster.AbstractClusterHandlerImpl.runPollingTask(AbstractClusterHandlerImpl.java:153) ~[cyclos-impl.jar:4.12]
        at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_212]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_212]
        at org.cyclos.bootstrap.BeanHandlerImpl$CustomBeanInvocationHandler.invoke(BeanHandlerImpl.java:84) ~[cyclos-impl.jar:4.12]
        at com.sun.proxy.$Proxy32.runPollingTask(Unknown Source) ~[?:?]
        at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_212]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_212]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343) ~[spring-aop-5.1.7.RELEASE.jar:5.1.7.RELEASE]
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:206) ~[spring-aop-5.1.7.RELEASE.jar:5.1.7.RELEASE]
        at com.sun.proxy.$Proxy42.runPollingTask(Unknown Source) ~[?:?]
        at org.cyclos.impl.utils.tasks.PollingTaskHandlerImpl.lambda$1(PollingTaskHandlerImpl.java:184) ~[cyclos-impl.jar:4.12]
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-5.1.7.RELEASE.jar:5.1.7.RELEASE]
        at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:93) [spring-context-5.1.7.RELEASE.jar:5.1.7.RELEASE]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_212]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_212]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_212]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_212]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
2019-06-24T20:57:59.213Z ERROR PurgeUnconfirmedUsersPollingTask - Error on task Expired pending users purging
org.cyclos.model.IllegalActionException: null
        at org.cyclos.impl.users.UserServiceImpl.onBeforeRemove(UserServiceImpl.java:1507) ~[cyclos-impl.jar:4.12]

Art

luis
Posts: 179
Joined: Fri Feb 17, 2006 11:01 am

Re: Cyclos logs eating up all disk space

Post by luis » Tue Jun 25, 2019 7:38 am

For some reason, it seems like you have inconsistent data in the users table.
There is a task that keeps reading users which haven't yet confirmed the registration (via e-mail validation) after some time.
As a quick workaround, disable that by setting the following in cyclos.properties:

Code: Select all

cyclos.purgeUnconfirmedUsers.days = 0
To help us understand the issue, please send here the result of the following SQL query:

Code: Select all

select name, registration_confirmation_date, validation_key_type, status from users where id = 10;
We've made sure to fix this for the next version, 4.12.1.
Luis Fernando Planella Gonzalez
Cyclos development team

atitokor
Posts: 44
Joined: Tue Jan 01, 2019 8:33 pm

Re: Cyclos logs eating up all disk space

Post by atitokor » Tue Jun 25, 2019 8:50 am

Thanks! I updated cyclos.properties and below are the results of the sql query. I do confirm that I removed this user on 6/23 but the issue has been there before then.

Code: Select all

name	Mubasher M
registration_confirmation_date	NULL
validation_key_type	REGISTRATION
status   REMOVED
Sure! Will update to 4.12.1 as soon as it's out

Again, thanks so much for your time and attention.
Art

luis
Posts: 179
Joined: Fri Feb 17, 2006 11:01 am

Re: Cyclos logs eating up all disk space

Post by luis » Tue Jun 25, 2019 11:01 am

Ok, thanks for the info.
If the issue was happening before, it might have other users that were removed before being confirmed, and that is what caused the bug.
After 4.12.1 you can revert the change in cyclos.properties if you wish.
Luis Fernando Planella Gonzalez
Cyclos development team

Post Reply