Huge ammount of error logs

DocSpace
1

Last night, more than 35gb of logs were generated on our Doc Space server. All the logs entries were the same and were saved on the file aws-logger-errors.txt which was saved on 4 different locations, /ASC.Studio.Notify, /ASC.ClearEvents, /ASC.Notify and /ASC.ApiSystem.

I checked the connection to AWS which we use for backups and it works fine.

Has this happened to anyone before? Why would this happen just out of the blue?

System info:

Ubuntu 22.04
Docspace installed with packages via script.

Log Entry :
01/10/2024 5:13:54 AM
  :
  :Amazon.Runtime.AmazonServiceException: Unable to get IAM security credentials from EC2 Instance Metadata Service.
   at Amazon.Runtime.DefaultInstanceProfileAWSCredentials.FetchCredentials()
   at Amazon.Runtime.DefaultInstanceProfileAWSCredentials.GetCredentials()
   at Amazon.Runtime.DefaultInstanceProfileAWSCredentials.GetCredentialsAsync()
   at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)
   at AWS.Logger.Core.AWSLoggerCore.LogEventTransmissionSetup(CancellationToken token)
   at AWS.Logger.Core.AWSLoggerCore.Monitor(CancellationToken token)
3

Hello @irdi

We will run some tests on our side to replicate the issue and gather as much details as possible. I will let you know if we find out anything or in case any additional details from you are required.

1 Like
4

Thank you.

I also tried changing AWS credentials, but that did not change anything. I have configured log rotate to remove the spam logs, but this is just a workaround.

I think this issue is connected to the backup issue i have mentioned earlier.

If I do not restart docspace-backup-background.service docspace-backup.service daily the daily backups fail.

5

We did not manage to reproduce the issue, hence I am kindly asking you to share more information:

  1. In which directory logs are located exactly?
  2. Was there any particular action after which the log has appeared?
  3. Do I understand correctly that S3 is connected for as a storage for automatic backups?
  4. Did you change access keys to the S3? Can be backup performed there now?
  5. Have exactly did you configure DocSpace to print spam logs?
  6. Does DocSpace return any errors if the service is not restarted?

I am looking forward to the feedback.