Log Health Dashboard
Detecting unhealthy stack logs is key to help our clients reduce issues.
Overview
The Log Health Dashboard detects and reports abnormal or suspicious patterns found in your logs. For instance, it can detect if email addresses are present in the logs, and what percentage of log lines contain emails.
It works by sampling and scanning them against a series of Log Health Feature (LHF) detectors. (Note: a default sampling rate of 1:1000 is used). Each LHF is responsible for detecting and counting the presence of a specific pattern or property. There are 2 types of LHFs: standard and custom.
The dashboard reports the scores of each LHF, ranging from 0% (no trace of the detected pattern among the examined log lines) to 100% (presence of pattern on all examined log lines). It also lets you view in detail which log lines present a particular pattern, from which you can plan a remediation strategy.
Standard Log Health Features
The standard Log Health Features (LHF) are the following:
Incoherent severity: Indicates a mismatch in the different severity levels found in the same log line.
Low message size: Indicates that the log’s message size is too small. This can point to a lack of descriptiveness of some log messages. Logs should provide a sufficiently rich yet concise contextual information to help with debugging or troubleshooting.
High message size: Indicates that the log’s message size is too large. This can be due to an excess of verbosity or to large data chunks present in the log line. Logs should provide a sufficiently rich yet concise contextual information to help with debugging or troubleshooting.
Email: Indicates presence of an email address in a log line. Email addresses are considered personally identifiable information (PII) and should not be logged.
Name: Indicates presence of a proper name in a log line. A proper name can be a person, city or country name. Proper names are considered personally identifiable information (PII) and should not be logged.
UUID: Indicates presence of a UUID (Unique User Identifier) in a log line. UUIDs are considered personally identifiable information (PII) and should not be logged.
Postal address: Indicates presence of a street address in a log line. Street addresses are considered personally identifiable information (PII) and should not be logged.
Payment card number: Indicates presence of a credit or debit card number in a log line. Card numbers are considered personally identifiable information (PII) and should not be logged.
Postal code: Indicates presence of a postal code in a log line. Postal codes are considered sensitive information and should not be logged.
Phone number: Indicates presence of a phone number in a log line. Phone numbers are considered sensitive information and should not be logged.
IP address: Indicates presence of an IPv4 or IPV6 address in a log line. IPs are considered sensitive information and should not be logged.
Geo coordinates: Indicates presence of geolocation information in a log line. This comprises longitude and latitude. Geolocation data are considered sensitive information and should not be logged.
IBAN account number: Indicates presence of an International Bank Account Number (IBAN). IBANs are considered personally identifiable information (PII) and should not be logged.
US bank account number: Indicates presence of a US bank account number is between 8 to 17 digits. Bank accounts are considered personally identifiable information (PII) and should not be logged.
US driving licence number: Indicates presence of a US driver license number according to https://ntsi.com/drivers-license-format/. Driver licence numbers are considered personally identifiable information (PII) and should not be logged.
US taxpayer ID: Indicates presence of a US Individual Taxpayer Identification Number (ITIN). ITINs are considered personally identifiable information (PII) and should not be logged.
US social security number: Indicates presence of a US Social Security Number (SSN) with 9 digits. SSNs are considered personally identifiable information (PII) and should not be logged.
Medical license number: Indicates presence of a common medical license number. Medical licence numbers are considered personally identifiable information (PII) and should not be logged.
US passport number: Indicates presence of a US passport number (9 digits) in a log line. Passport numbers are considered personally identifiable information (PII) and should not be logged.
Nationality, religion or politics: Indicates presence of a person’s nationality, religious or political group. Those informations are considered sensitive information and should not be logged.
Multiple languages - multiple lines: Indicates presence of multiple languages in logs. A single language should be favored for logging messages to ease and maximize information recall when searching for logs.
Custom Log Health Features
While standard Log Health features will detect generic properties, users can scan logs for custom properties using Custom Log Health Features (Custom LHF), using a user-defined regexp.
To create a custom LHF, you must first switch the query type from Simple Query to Regexp Query in the top search bar and supply a regexp of your choice (in the example below, we use the following regexp:
timestamp.*path=\/vector. Please note that special characters like/are escaped.
After hitting space, make sure results show lines that correspond to what you expect to detect. Then press Build Log Health. Please supply a name for your customized log health. You can also review the example lines if you wish
Then click Save to save the new Custom LHF. After 1 minute, the new custom LHF is activated : a non-zero score is displayed for this custom LHF (100% in the example above)
Clicking on the 3-dot icon the LHF card shows indeed some matching lines
Working with the Log Health Dashboard
Inspecting lines that contain some pattern
This displays a flyover that show the corresponding lines. Portion of the message that was matched by the LHF is highlighted.
Enabling or disabling Log Health Features
Last updated
