Effective Use of Splunk and AWS in the Time of Coronavirus
By: Bruce Johnson | Director, Enterprise Security
Firstly, be safe and be well. The TekStream family has found itself pulling together in ways that transcend remote conference calls and we hope that your respective organizations are able to do the same. We feel very privileged to be in the Splunk ecosystem as uses for Splunk technology are becoming ever more immediate.
To that end, we have seen all of our customers putting emphasis on monitoring remote access. Was any company sizing their network for virtualizing their entire ecosystem overnight? Network access points were sized for pre-determined traffic profiles leveraging pre-determined bandwidth levels for remote access. Those network appliances were configured to support predictable traffic volumes from occasionally remote workers, they weren’t designed to support 100% of all internal access traffic. The impact to operational monitoring of services supporting remote users became the most critical part of your infrastructure overnight.
Likewise, what you were monitoring for security just got hidden in a cloud of chaff. The changes to network traffic have opened you up to new threats that demand immediate attention.
Free 1-Hour Consultation with Our Experts
Don’t let WFH put your network security at risk.
To keep your network secure, TekStream is offering free consultations with our Splunk and AWS experts. Schedule with the link below.
Security Impact
There are several new areas of concern in the context of the current climate:
Your threat surface has changed
Anomalies relative to RDP sessions or escalation of privileges for remotely logged in users used to be a smaller percentage of traffic and might have figured into evaluating potential threat risk. Obviously that is no longer the case. If you’re able to segregate traffic for access to critical systems from traffic that simply needs to be routed or tunneled to other public cloud-provided applications, that would help cut down on the traffic that needs to be monitored but that will require changes to network monitoring and Splunk searches.
Your policies and processes need to be reviewed and revised
Have you published security standards for home networks for remote workers? Do you have policies relative to working in public networks? Do you have adequate personal firewalls in place or standard implementations for users wanting to implement security add-ons for their home networks or work-provided laptops?
Some employees might now be faced with working on home networks which are not adequate to the bandwidth needs of video conferencing and may opt to work from shared public access points (although they might have to make due with working from the Starbucks parking lot as internal access is prohibited). Many do not have secure wireless access points or firewalls on their home networks. Publishing links to your employees on how to implement additional wi-fi security and/or products that are supported for additional security, as well as how to ensure access to critical systems through supported VPN/MFA methods is worth doing even if you have done it before. There is also the potential expansion of access to include personal devices in addition to company-owned devices. They will need to have the same level of security, and you will also need to consider the privacy implications of employee-owned devices connecting to your business network.
Likewise, help desk resources in support of these efforts as well as level1 security analysts monitoring this type of activity might need to be shifted or expanded.
New threats have emerged
Hackers don’t take the day off because they have to work from home and there are several creative threats that take advantage of Coronavirus panic. Hackers are nothing if not nimble. There are several well-publicized attacks which seek to take advantage of users anxious for more information on the progress of the pandemic. The World Health Organization (WHO) and the U.S. Federal Trade Commission (FTC) have publicly warned about impersonators. Thousands of domains are getting registered every day in support of Coronavirus related phishing attacks. Some of them are even targeting hospitals, which takes “unethical” hacking to a brand new low. Additionally, there are new threat lists to consider, for example, RiskIQ is publishing a list of rapidly expanding domains relative to coronavirus.
Stepping up the normal Splunk monitoring for those domains, moving up plans to augment email filtering, setting up a mailbox that Splunk ingests for reported attacks that can be easily forwarded from end-users that suspect a phishing email, or augmenting your Phantom SOAR implementation to highlight automated response to specific phishing attacks are all appropriate in that context.
Operational impact
VPN Monitoring
If you are not currently monitoring VPN usage in Splunk it is relatively straightforward to implement VPN/Firewall data sources and to begin monitoring utilization and health from those appliances. It is useful to monitor network traffic as a whole relative to VPN bandwidth as well as the normal CPU/memory metrics coming from those appliances directly.
If you’re already monitoring VPN traffic and likely you are if you have Splunk, at the very least, you need to alter your thresholds for what constitutes an alert or an anomaly.
The following are examples of dashboards we’ve built to monitor VPN related firewall traffic as well as cpu/memory:
In addition to straightforward monitoring of the environment, expect troubleshooting tickets to increase. Detailed metrics relative to the connectivity errors might need to be monitored more closely or events might be expanded to make troubleshooting more efficient. Below is an example of Palo Alto Splunk dashboards that track VPN errors:
There are several out of the box applications from Splunk for VPN / NGFW sources including but not limited to:
Palo Alto: Includes firewall data that monitors bandwidth across key links. Additionally, Global protect VPN monitoring can help customers with troubleshooting remote access. https://splunkbase.splunk.com/app/491/
Zscaler: Provides visibility into remote access, no matter where the users are connecting from.https://splunkbase.splunk.com/app/3866/
Cisco: Provides equivalent functionality to populate dashboards around remote access and bandwidth on key links. https://splunkbase.splunk.com/app/1620/
Fortinet: Provides ability to ingest Fortigate Fortinet traffic https://splunkbase.splunk.com/app/2846/
Nagios: Monitors the network for problems caused by overloaded data links or network connections, also monitors routers, switches and more. https://splunkbase.splunk.com/app/2703/
One of the techniques to consider in response to this spike in volume is to split network traffic on your VPNs to segregate priority or sensitive traffic from traffic that you can pass through to external applications.
https://www.networkworld.com/article/3532440/coronavirus-challenges-remote-networking.html
Split tunneling can be used to route traffic and it’s being recommended by Microsoft for O365 access. This also effects how VPN traffic and threats are monitored through established tunnels. Obviously, the traffic to internal critical infrastructure and applications would be the priority and all externally routed traffic could be, if not ignored, at least de-prioritized.
Additionally, MFA applications fall into much the same category as monitoring of VPN sources and the same types of use cases apply to monitoring those sources. Below are a subset of relevant application links.
RSA Multifactor Authentication: https://splunkbase.splunk.com/app/2958/
Duo Multifactor Authentication: https://splunkbase.splunk.com/app/3504/
Okta Multifactor Authentication: https://splunkbase.splunk.com/app/2806/
If you’re familiar with Splunk, you already know that it is typically only a few hours of effort to onboard a new data source and begin leveraging it in the context of searches, dashboards, and alerts.
Engaged Workers
There are some people that are focused in an office environment, then there are the people that work at home with one cup of coffee that can fuel them until they are dragged away from their laptop, then there are the people that have way to much to do around the house to bother with work. It’s nice to know whether people are actually plugged in. A whole new demand for monitoring remote productivity fueling new solution offerings from Splunk. They have developed a set of dashboards under the guise of Remote Work Insights.
Of course, monitoring your VPN and conferencing software is just the beginning and there are a plethora of sources that might be monitored to measure productivity. Often those sources vary by team and responsibilities. The power inherent in Splunk is that each team can be monitored individuality with different measures and aggregated into composite team views at multiple levels, similar to ITSI monitoring of infrastructure layers and components. We are finding a great deal of opportunity in this area and it is expected to be a set of techniques and solutions that will persist well beyond the shared immediate challenges of Coronavirus.
A related use case for VPN monitoring is to track login and logout to confirm that people are actually logging in rather than social distancing on the golf course, but this use case has been less common in practice.
Migrate VPN services to the cloud
Ultimately, when faced with dynamic scaling and provisioning problems, the cloud is your answer. If your VPN infrastructure is taxed, the traffic is now completely unpredictable, and there is no way to scale up your network appliances in the short term, consider moving VPN services to cloud connectivity points. You can move network security to the cloud and consume it just like any other SaaS application. This has the advantage of being instantly scalable up and down (once normal operations resume) as well as being secure. Implementation can be done in parallel to your existing VPN network-based solutions. Virtualizing VPN in AWS is relatively straightforward and it’s certainly something TekStream can help you to accomplish in short order. It has the advantage of scaling and doing so temporarily. There are a variety of options to consider.
AWS Marketplace has VPN appliances you can deploy immediately. This is a good approach if you are already using a commercial-grade VPN like a Cisco ASA or Palo Alto. This will have the least impact on existing users since they can continue to use the same client, just point their connection to a new hostname or IP but it can be a bit pricey. Some examples of commercial options from the AWS Marketplace are:
Cisco ASA: https://aws.amazon.com/marketplace/pp/B00WH2LGM0
Barracuda Firewall: https://aws.amazon.com/marketplace/pp/B077G9FKK7
Juniper Networks: https://aws.amazon.com/marketplace/pp/B01LYWCGDX
You can use AWS’s managed VPN service. This is a great “middle of the road” compromise if you don’t currently have a VPN. As a managed service AWS handles a lot of the nuts and bolts and you can get up and connected quickly. Your users will connect to the AWS VPN which connects to your AWS VPC, (which is connected to your datacenter, network, on-prem resources, etc). As a fully managed client-based VPN you can manage and monitor all your connections from a single console. AWS VPN is an elastic solution that leverages the cloud to automatically scale based on user demand, without the limitations of a hardware appliance. It may also allow you to take advantage of additional AWS provided security mechanisms like rotating keys, credentialing, etc. to augment your security practices.
Finally, if you need something quick and have a smaller number of users, you can deploy your own VPN software on an Ec2 instance and “roll-your-own.” While this can be quick and dirty, this can be error-prone, less secure, and introduce a single point of failure, and it has to be manually managed.
Additional Services
There are a whole host of ancillary supporting services which can might need to be expanded for inclusion into Splunk such as Citrix, Webex, Skype, VoiP infrastructure, Teams, etc.. Below is an example of an Australian customer monitoring Video conferencing solutions with Splunk ITSI, but TekStream has been involved to build out monitoring of critical VoiP infrastructure and relate that to multi-channel support mechanisms including web and chat traffic. The point is that all of these channels might have just become critical infrastructure.
Conclusion
Much of the above recommendations can be accomplished in days or weeks. If there is an urgent need to temporarily expand your license to respond to the Coronavirus threat, that might be possible in the short term as well. With uncertainty around the duration of the pandemic, it would seem to warrant an all-out response from infrastructure, to processes and procedures, to operations, and security.
Your business can’t afford to fail. TekStream is here to help if you need us.
Free 1-Hour Consultation with Our Experts
Don’t let WFH put your network security at risk.
To keep your network secure, TekStream is offering free consultations with our Splunk and AWS experts. Schedule with the link below.
Free Offer