Logging Best Practices

Logs come in different shapes and sizes and we highly recommend that clients follow standard logging conventions for custom applications. Following standard logging formats would allow teams to have a uniform data structure, use query language efficiently and also benefit from the full power of PacketAI’s log anomaly detection features.

We support custom logs generated using the most common loggers all for Java, Python, Ruby, Node.js and PhP. Clients are requested to send us the technology/framework that they use along with the name of the logger. We may ask you for some samples before the PoC to help better understand your logs.

Below are some of the best practices that developers can follow while logging for custom apps. Overall, for best anomaly detection on logs we ask developers to:

• Make sure that logs have human readable language

• The logs should tell the story of the program execution -> remove duplicated information and please add information that is important to understand the problem

Things to avoid :

• Including Tracebacks/Exceptions/Errors in the logs: It is recommended to catch the exception and log an appropriate message that explains why we have this error. It helps the team better troubleshoot. For instance, avoid broad or vague exceptions like the following :

  try:
  	function()
  except Exception as e:
  	logger.exception(e)

• Keeping the prints/system.console: It is highly recommended to add the header (Timestamp, Loglevel, Source, ...) to every message written in log file.

  # Avoid doing this ->
  for i in range(1, 10):
  	print(i)
  
  # Instead follow this ->
  for i in range(1, 10):
  	logger.info('i')

• Using multiple formats in one service : Whenever possible, standardise the format across the frameworks for one service especially, if the different parts are coded using the same language.

• Keeping development helpers : Progress bar, stage status, ... are things that helps during development, but need to be excluded in production. It's better to replace them with an appropriate human-readable message.

  # Avoid doing this ->
  60%|██████    | 6/10 [00:18<00:12,  0.33 it/s]
  
  # Instead follow this ->
  time="2021-10-05T23:17:02.272085475Z" level=info msg="Processing item 6/10"

• Writing a log inside a log: A log should have only 1 header.

  # Avoid doing this ->
  Oct 5 23:17:02 docker6 dockerd[685]: time="2021-10-05T23:17:02.272085475Z" level=warning msg="got error while decoding json"
  
  # Instead follow this ->
  Oct 5 23:17:02 docker6 dockerd[685]: level=warning msg="got error while decoding json"

• Logging at the wrong level: Using the right log level is important.

  # Avoid doing this ->
  time="2021-10-05T23:17:02.272085475Z" level=info msg="got error while decoding json"
  
  # Instead follow this ->
  time="2021-10-05T23:17:02.272085475Z" level=error msg="got error while decoding json"

• Changing the log formats often : Changing the format implies a change on the parsing and can have many consequences. Please try and limit this and we strongly suggest to use standard logger conventions/formats whenever possible.

• Logging in french : We use NLP algorithms and they work the best on English logs. Also it is best practice to log in English as not all engineers may speak French.

• Very long/short (extreme) logs : The log should not be too short nor too long. Logs are output to communicate a clear message in a human readable language. Please avoid such messages:

  2021-11-09 01:00:11,668 - DEBUG - This is a very very very very very very very very very very very very very very very very very very very long log

PS:

Changing the log levels: If for any reason you are decreasing the level of logs from INFO to DEBUG for instance, please be aware that it would increase the number of log messages and hence this event could be interpreted as anomaly by our models. This can result in some false positives which will smoothen out over a few days.