Office 365 Incident Response

What is possible?  What is Reasonable?

Why write this?

After they entered, they may have left all the other windows and doors open

Before working in cyber-security, I once worked at a company, when I was approached to look at another staff member’s email account which was “acting a bit funny”.  When I looked, I found the sent mailbox was filling every 5 seconds with a new sent email, each to a seemingly random recipient, each purporting to be able to help the recipients “Meet girls” or “enlarge” one’s whatnot.  As amusing as the subject lines where it was painfully obvious that the mail account had been hijacked.   We quickly reset the password and the mails abruptly stopped.  After poking around we eventually reset a few other passwords and left it at that.

Looking back now, through the lens of my more recent cyber security knowledge, I am shocked at how lucky we were that the mal-actor/script/bot in question left such a glaring trail for us to stumble across, but also how much better we should have been with our response and remediation.  To this day I have no idea…

1. How the account was breached
2. When the account was breached
3. If it was the only account or instance of accounts being hacked
4. If the attacker caused any damage or stole any sensitive business or personal information
5. If any further attacks were orchestrated off the back of this one
6. And if the attacker left rules in the account which would continue to leave the company exposed even after the account passwords had been changed

Why did I not know these things?  Because such knowledge and response playbooks come at a cost either in terms of time, money or both.  Furthermore, as you will see, many of the measures we should have taken would need to have been taken BEFORE the incident had occurred.

Let’s ask why?  Putting aside that Cyber Consultants are prohibitively expensive for SMEs;  What is it about Office 365, the linchpin of many SMEs, that is lacking in terms of its ability to aid Incident Response?  It turns out that even if we had dug deeper, requiring considerable time and energy, we may have hit a wall because even if we knew how to follow the small scattered breadcrumbs left behind, we would have been blocked because, in the main, the best historical auditing features are NOT turned on by default in Office 365, a fact that could only have been remedied by the things SMEs lack the most:  

1. Incredible Foresight
2. Time machines

Auditing for Incident Response

Done correctly, incident response requires multiple steps in order to properly minimise, contain, eradicate and remediate.  Here we will just focus on the technical bits with respect to Office 365; What can we do, what we can’t, and what we should have done given a time machine.

LETS SET THE SCENE… A GENERIC HACKING SCENARIO

Somehow, either through phishing, complacency, brute force, or a combination of these; an account has been breached such that a malicious actor (mal actor) has

1. Obtained a user’s credentials without their knowledge.  In this case the mal actor is acting deliberately, is usually external and untrusted, however this latter assumption is not always the case.  The actor may also be a program (bot) in this case
2. Accidentally breached the organisations security through carelessness.  In this case the actor would be internal, trusted.  

To make it simple let’s assume the account is a regular user with a mailbox and does not have administrator access.  The mal actor may have had some time to access the mailbox through Outlook Web Access (OWA), has read some emails, used the knowledge to create some forwarding rules, some deletion rules (to cover their tracks) and has left the scene awaiting for the forwarding rules to bear evil fruits. Let’s further assume that some time has passed and through chance, blind luck, or sloppiness on the part of the mal actor, we have a suspicion that the mailbox has been breached.

IMMEDIATE RESPONSE, AUDIT WHAT YOU DO AND SEE

We recommend PowerShell for incident response remediation since we can control and audit every command and its associated response.  This will include some recommended commands that simply show the state of play should we need to later do some later forensics.  The other unavoidable reason is that we shall see that PowerShell is the only interface (including visual user interfaces) that is common for all matters herein.  

 Depending on the scenario we may want to take some actions (business and otherwise) before we contain the breach.  Much literature will tell you to do a full investigation before acting however for simplicity I would favour immediately shutting any mal actor out from further logins.  This is only practical where we know at least one account that has been breached and we know which account it is.  Further One would think that changing the password would suffice, however the user should be logged out of all devices.  In windows Powershell this can be achieved through the command

Revoke-AzureADUserAllRefreshToken

The password can then be changed using

Set-MsolUserPassword –UserPrincipalName <UserPrincipalName> –NewPassword <New Password> -ForceChangePassword $True

The ForceChangePassword parameter will ensure the user changes the password again when user next logs in.  

Remediate the breach

Even if you reset a password and lock out an attacker from further logins, it may not have contained the breach.  The credentials for the account may only have given a mal actor access to the front door.  After they entered, they may have left all the other windows and doors open, so to speak.  These are the next things we need to check

CURRENT MAILBOX RULES

Let’s assume for now that the mal actions have not been cleaned up, since the breach occurred (we can deal with this later should the remnants of the breach have been cleaned up).  It is important to look at all inbox rules.  Inbox rules are used to automatically take actions on email such as auto forwarding, moving to folders, deleting, etc and hence often form the part of a breach.  

We will go so far as to say that searching for harmful rules should be done on all mailboxes if practical and not just the breached mailbox if known.  This is recommended because, in the best case, the breach may be indicative of poor staff hygiene that needs to be weeded out, and in the worst case, the mal actor may have used their entry into your system to gain further credentials to other mailboxes, thereby spreading the contagion.  

The following command will query all mailbox rules across the organisation and show a human readable description generated from the complex conditions therein

$mbox = Get-Mailbox; $mbox | Foreach { Get-InboxRule -Mailbox $_.DistinguishedName | Select-Object -Property Name,Enabled,Identity,Description | convertto-json }

LETS TALK ABOUT OWA

Unfortunately, Microsoft being Microsoft, introduced inconsistencies in auto forwarding where users can set up a global (unconditional) auto forward rule on a mailbox.  This can be done by a user accessing Outlook Web Access (OWA).  For those that don’t know OWA is outlook in the web browser.  While a very useful tool for those of us away from our normal laptops’ Outlook, it is a fantastic tool for mal actors once credentials have been accessed since it allows them to access, given credentials, without needing to be on a business network, a domain, or a configured company laptop etc.  As an aside, it should be noted that should a company not require staff members to use OWA remotely then they should seriously consider turning it off for those users.  Further to this potential risk is the fact that Microsoft do NOT by default audit all things done in OWA by default or on some licences.  Shame on you Microsoft.  We will speak more on this later but for now we should add the following command to find OWA global forwarding on mailboxes.

Get-Mailbox -Filter {Name -notlike “DiscoverySearchMailbox*” -and (ForwardingSmtpAddress -ne $null -or ForwardingAddress -ne $null)} | ft name,*forward* -auto

REMEDIATING MAILBOX FORWARDING BREACHES

There are several options should an unwanted rule be found.  Firstly, if we believe the rule indicates a breached account that we have not locked out in the previous section then this should be done immediately.  After locking out mal actors, the rule can either be disabled or removed completely.  Be careful though removing the rule completely will make it hard to re-audit it later.  I suggest either way getting the full details of the rule via

Get-InboxRule -Identity <identity> -Mailbox <email-address>| convertto-json

Where the identity is the one obtained in the previously when listing all rules above.  Save this somewhere for later.  Now disable or remove the rule using

Disable-InboxRule -Identity <identity> -Mailbox <email-address>    Remove-InboxRule -Identity <identity> -Mailbox <email-address>

In the case of OWA global auto forwarding this may be removed via

Set-Mailbox -Identity <identity> -ForwardingSmtpAddress $null -ForwardingAddress $null

HIDDEN MAILBOX RULES

These are a documented, but not a commonly known, Microsoft security hole called hidden mailbox rules.  They are in effect a corrupted rule, such that it is hidden but operational.  Finding and remediating them is complex so for brevity we shall include only links:

• To find Hidden Inbox Rules:
• The HAWK module discussed later also includes a script to find these rules.  
• To remove Hidden Inbox Rules:

Investigating the Logs – The When, The Where and The How

Looking at inbox rules may have indicated a breach, however in many cases, it may not be clear either because a suspect rule may be ambiguous or worse yet an attacker may have cleaned up the rules after the breach to conceal their actions (which may be ongoing).  We therefore need to look at historical evidence in audit logs.  It should be noted that this kind of data cannot be erased by an attacker.  Furthermore, the historical evidence will lead us later to identify such matters as; The timeline of events, what data that may have been leaked, internal mal-acting persons, weaknesses in policy, future policy changes.  The list goes on.  

This is where things get a bit fluffy because as we discussed earlier where you look, how you look, and what can be seen, is rather dependent on your licence and the settings in your tenant.  Let’s look at the options:

1.      Universal Audit Log (UAL).  As the name suggests this log is almost universal.  It was introduced to feed from the many disparate logs from within the ecosystem which is your 365 tenant. It consists of most log event types.  There is a handy UI in the security and compliance dashboard. But here is the rub:  Its usually not on.  Probably due to the cost to log storage for Microsoft this log must be turned on manually and it will then fill from that date onwards.  

2.      Mailbox Audit Log (MAL).  There is one of these feeding from each mailbox.  It can be searched from Exchange Online PowerShell.   They are retrieved on a mailbox by mailbox basis.  It contains very useful events such as “MailboxItemsAccessed” (when a mail is accessed), “UserLogin” and “UpdateInboxRules” (when a rule is created/updated).  These 3 events cover the vast majority of breach cases.   Since January 2019 Microsoft has been turning it on by default for all tenants assuming one has not fiddled with the “on” setting manually before then.  This setting can be found by querying

Get-OrganizationConfig | Format-List AuditDisabled

Which should return AuditDisabled : False

This all sounds wonderful, however there are multiple catches which may trip you:

• Depending on your tenant licence, not all event types are logged.  Microsoft had began rolling out the “MailItemsAccessed” event to everyone  but ran into trouble in April 2019 (see also, and again here).  They then decided to roll it in September.  Then later again it would seem (it gets a bit unclear at his point) but you will have to pay  
• Update (April 2020):  Microsoft has announced that the MailItemsAccessed Event will basically come as default with E5 licenes and E3 licence with an additional security and compliance add-on.  More info here.  I wish Microsoft had been more forthcoming on this before now.
• Not all access mediums are logged for all event types.  What we mean here is that a mailbox can be accessed in several mediums such as Outlook, in the browser (OWA) and through PowerShell (Mailbox settings ONLY).  Microsoft seems to audit most of what goes on in Outlook but OWA and PowerShell remain unaudited for some event types.  This is hugely disappointing since it is OWA, and to a lesser extent PowerShell, which are leveraged by mal actors working remotely.  
• Not all event are logged for all user access types (see here under Mar 28)  Microsoft as we can see here are improving this however are not there yet and its probable will not be making improvements to tenants mailboxes where the user/admin in that org has manually made changes to these settings themselves

1.      Admin Audit Logs.  These logs are very useful if the breached account/user has administrator privileges.  Here we have been focused on the more common case that they do not, however if they do, you are in a whole lot more trouble, and this audit log is your friend since it records, without licence discrimination I believe, all admin settings changes whether they were invoked through the online admin portals or through PowerShell.   You can find the details to search it here

2.      Azure Active Directory (Azure AD) Sign in Logs Are a great source of information about who is accessing your tenant and from where.  Azure AD is the authentication and Identity management piece in the Microsoft cloud ecosystem.  The sign in logs however come at a premium (literally) since they require the Azure Active Directory premium 1 (P1) licence   Office 365 comes bundled with only the Free version of AzureAD.    Since many SMEs wont purchase the addition I’m going to leave it out of the discussion for now, however you can find information here on how to query the logs in PowerShell using the AzureADPreview module .  It should be noted that there are useful reports in Azure AD Portal built on these records.

3.      Message Trace Logs.  These are very fine-grained logs that track the movement of mail in and out of mailboxes.  They are retrieved on a mailbox by mailbox basis.  Thankfully they are available in most (probably all, I’m not sure) licences and are on by default for all mailboxes.  They come into their own in audit scenarios where a forensic analysis is required that is robust to a user’s (or mal actors) who may be deleting email messages (either manually or through rules) in order to cover their tracks.  While it is hard to infer an attack by using Message Trace Logs alone, they are useful in tandem with other log sources to paint a full picture

4.      HAWK .  This is a PowerShell toolset for extracting logs for the purpose of investigating “O365 intrusions and potential Breaches”.  It was developed by an Engineer at Microsoft and first appeared in 2018 here .  It is not strictly an audit source since I don’t believe it can obtain any data that is not in the other event logs, and it cannot access logs where they are not configured or not in licence as per above, however it is a source in its own right since it makes sense of the data and infers different breach and risk scenarios.

Querying and Analysing

As discussed earlier we favour PowerShell for this work since it is the only interface which is common to all methods above.  Beyond that there is no single source that will form the whole picture.  The UAL logs come closest assuming they were turned on before the incident.  They can be searched back as far as a maximum of 90 days using.  The beauty of UAL is that you can search events attributed to all users at once using for example:

Search-UnifiedAuditLog -StartDate 5/1/2018 -EndDate 5/2/2018

Be careful that you set your dates in accordance with the date format MM/DD or DD/MM as appropriate.  If you are not sure, then try searching for logs on the 13th day of last month using each format to see which returns and which causes an error.  

Without UAL, the Mailbox Audit Logs and the Admin Audit Logs are of varying digress of usefulness depending on the licence and mailbox type.   The mailbox audit logs also store 90 days of historical information.  Unlike UAL these logs are not global and hence if you do not know the mailbox which was breached then your searching must be somewhat exhaustive.  They follow command is an example of the type of search you can do for all stored event types given a user.

Search-MailboxAuditLog -Identity billbill -LogonTypes Owner -ShowDetails -StartDate 01/23/2020 -EndDate 01/25/2020 -ResultSize 2000 | convertto-json | Search.json

Regardless these global audit log source you have at your disposal they cannot form the full picture and when a mailbox breach is suspected then the Message Trace Logs can complete the analysis at fine grained level.  

Towards a proactive solution

Regardless of the mixed effect of the above approach it is clear that one must get your house in order to alleviate further breaches.  The absolute basics you should be considering are:

1. Turn on all logging mechanism that you can in terms of UAL and in the case of MAL ensure it is logging the maximum for all user types
2. Get your policies in order.  Are your users using basic data hygiene and diligence in their work?  Are their passwords good enough and please, please, ensure those passwords are not used elsewhere, passed around, or written down outside of a secure password manager
3. Purchase a password manager licence (LastPass or similar) for each staff member under a corporate licence
4. Use Multi Factor Authentication (MFA).  If even for your Admin accounts

A comprehensive approach.

The above are just the low hanging fruit.  In security you must be broad in your approach since attackers look past your strengths to find the chinks in your armor. Octiga (disclaimer I work for Octiga but wouldn’t be writing this if I didn’t believe in the need and usefulness of our solutions) has created a product to address not only incident response but just as important; security hardening and alerting. Its features prevent, detect and recover offer incident response, monitoring & alerting and remediation respectively. Try it for yourself. Schedule a demo with us today!