Saltar a la navegación Saltar al contenido principal Ir al pie de página

How Remote Testing Infrastructure Can Streamline Network Pen Testing

Our Technical Director Damon Small’s experience finding solutions for remote assessments.

In early March 2020, a colleague and I traveled to visit a client in another city. The engagement lasted a week and wasn’t memorable. Shortly after going back home, though, things became anything but uneventful. COVID gained a foothold in the US, and the first lockdowns took effect.

Many of the staff at NCC Group were already working from home, so shoring up remote access to accommodate everyone at NCC Group was quickly achieved. A more difficult problem was how to adjust to serving our clients when we could no longer travel.

We needed more robust remote assessment capabilities than the ones we already had, and we needed to implement them quickly. However, we needed to find a way that maintained our clients' confidence that performing infrastructure and network penetration tests remotely would not introduce undue risk to their enterprises.

NCC Group learned a lot during this process, and we have gained efficiency when remote assessments are possible. In short, we have realized that remote assessments offer more flexibility and cost benefits than onsite engagements can in almost every scenario.

The Challenge

Not only did our clients' organizations have to provide remote access for their employees, but they needed to be able to support their service providers during an engagement as well.

Previous solutions included VPN access to the remote site. Depending on the assessment requirements, This could work well depending on the requirements, but there was a big catch: VPNs can introduce latency and often include filtering or VLAN assignments that can negatively impact the consultants' activities.

Adding bastion hosts on the remote network helped but required planning to ensure that they were strategically placed on the network to provide the network visibility needed for the scope of the engagement. Shipping and installing such hosts also required logistical coordination, and once the lockdowns took effect, that didn't prove easy. Some clients could spin up a virtual machine (VM), introducing complexity and technical controls that may have skewed results.

NCC Group needed more options that could scale to perform all infrastructure assessments remotely without introducing unnecessary logistical strain overhead on our clients. Such a solution would also address other common issues, such as:

  • Coordination of onsite access for third parties
  • Travel expenses
  • Network Access Controls (NAC) that may hinder unfettered access required by the consultants
  • Realistic testing perspectives that could simulate real-world attacks
  • Tool availability for network penetration testing in restricted environments

The Solutions

There are two types of solutions that we chose discovered to aid in our need for a robust remote testing infrastructure, (1) Onsite Network Implant (ONI) and (2) Firebase. Both solutions present unique advantages and disadvantages based on their usage and implementation.

Onsite Network Implant (ONI)

NCC Group has used this solution since 2016 and is an evolution of Red Team "drop boxes" used for short-term deployments for surreptitious access. This solution derives from a small form factor computer running a Unix-like operating system, typically Kali Linux. To avoid complex firewall configuration changes, the computer logs into a cloud-based host via SSH to establish an encrypted tunnel.

NCC Group consultants can log in to the server and control the ONI device through the tunnel. This allows consultants to have a physical presence on the network without exposing internal systems to the internet. It is also worth pointing out that ONI is platform-agnostic and can support any cloud provider that is convenient for the client.

Much of this functionality is automated so that the onsite personnel need only provide power and network connectivity. If DHCP is not available, we can preconfigure a static IP address. Regardless, NCC Group works closely with the client to ensure that ONI has access to the appropriate networks that are in scope. In the case there are many security boundaries, multiple systems can be deployed. Multiple ONI deployments are also useful for wireless pen testing.

This use case isn't ideal and may be irrelevant now that the lockdown has abated. Still, it's serviceable for clients who need a quick turnaround, tight budgets, or to review a deployment rolled out by a managed service provider.

Advantages:

  • Requires minimal support from the client; custom-built for each deployment to be as frictionless as possible.
  • Hardware-agnostic and can be deployed on hardware ranging from compact desktops to "stick" PCs.

Disadvantages:

  • Virtualized solutions are available, but they require effort on the part of the client.
  • Fewer granular controls for the client to control access locally, although end-user accounts exist should troubleshooting become necessary.
  • Requires one support day for every two-week engagement, so expensive to run.
  • Single point of failure by an Individual rather than a corporately managed solution, including all associated risks.
  • Each engagement incurs the cost of running an AWS host (possible complexity of billing that per job?)

Firebase

Firebase is the chosen solution for delivering remote security consultancy services in the UK, Europe, and APAC, and it has a strong and growing presence within North America. In total, Firebase has enabled NCC Group to deliver over 27,000 days of consultancy.

The Firebase project commenced in 2018 to build a solution based on best security practices, which could be offered to customers at no cost.

Firebase can be built and issued to customers in 30 minutes, requiring no more than 5 minutes of a person's time to do so. Firebase officially launched in March 2020, prior to the pandemic. 

Firebase is offered to customers in two versions, as a virtual machine or physical appliance. Both have exactly the same functionality. The virtual appliance is supported supports on VMware ESXi and Microsoft Hyper-V, and there are supported cloud versions for AWS, Azure, Google Cloud, Nutanix, and Oracle Cloud.

A virtual Firebase appliance can be offered to customers at no cost, and customers can have as many as they need to fulfill their network penetration testing requirements. Physical Firebase appliances get billed at the cost of the hardware, which is minimal as that cost gets based on a small form factor appliance, and each region has the hardware and supporting infrastructure in place now to deploy either.

As Firebase can be deployed in 30 minutes at no cost, it is perfect for quick jobs where you need a disposable appliance and is equally suitable for long-term projects. Multiple customers that have tried Firebase gained confidence that it works, recognized the cost savings from simplifying the setup of jobs and travel and subsistence savings, and have since moved all of their testing remotely via Firebase.  

Firebase is a fully managed appliance that runs on a hardened Ubuntu operating system and uses Docker containers. Regardless of the type of work, NCC Group can quickly upload a container with the tools consultants need. For example, there are traditional containers for pen testing, managed vulnerability scanning services, incident response, and security improvement and remediation, amongst others. There is a tried and tested formal documented process for the delivery of PCI engagements as agreed by NCC Group's QSAs. 

Deploying Firebase is simple. The customer plugs in a pre-built physical appliance or downloads and deploys a virtual Firebase. The image is under 5GB, so it takes less than 5 minutes to download and is unique to that customer. When the customer wants NCC Group to test, the customer logs into an interface and clicks start. Firebase establishes an outbound connection over TCP/443 (HTTPS) to NCC Group's endpoint in each region. Firebase is proxy compliant, supporting both HTTP and SOCKS with authentication. All data resides within NCC Group's on-premise environment, so no customer data is stored in the cloud to satisfy those clients who must ensure their data does not leave their region.

The Firebase solution has been built with security best practices in mind, as it represents NCC Group's capability and must not expose our clients to risk. An extensive architecture document details the solution to provide clients with absolute confidence. Internally, the authentication process for consultants to access a client Firebase is robust and fed from the scheduling system, thereby ensuring that only the named consultants, as agreed with the customer, may access the customer environment.

The consultant assigned to the job must connect to a Firebase consultant VPN by supplying a valid certificate plus credentials + One Time Pin. However, the consultant can only link to the customer Firebase once the client has started the connection, as it is an outbound connection, ensuring the customer is always in control. The global Firebase support team may revoke or control access granularly from a central point.

Annually the Firebase appliance build and supporting infrastructure are pen tested, and a customer-facing report gets produced. Security patches are automatically applied four times a day, and logs get extracted from each customer Firebase, including commands run on the base operating system. SNMP monitors each appliance's health and has anti-virus and file integrity monitoring software installed.

The authentication model design is such that Firebase cannot be connected from the client network, thereby reducing the risk exposure, and instead can only connect once the VPN has been established. Consultants connect with named accounts rather than a shared root account, so we know who connected, when, and what they did in accordance with security best practices.

A regional Firebase support team is located in the UK, Europe, APAC, and North America.                                                

Advantages:

  • Established enterprise-level product that is the default remote consultancy delivery tool in the UK, Europe, and APAC, with a growing presence in North America
  • Can be built and issued to a customer in 30 minutes via an automated build. Requires no more than 5 minutes of an NCC Group employee's time to do so. The customer can deploy and test it in 10 minutes, so it's perfect for quick or long-term engagements.
  • Can deploy as either a virtual machine or a physical appliance. The virtual appliance supports all major hypervisors and cloud platforms.
  • Firebase can be at no cost to the customer (as a virtual machine), and the physical appliance is available at cost (minimal as it's a small form factor appliance).
  • Enables customers to request security testing at short notice with minimal customer setup time required.
  • Firebase uses Docker containers, so the customer can quickly add, no matter the job, a new container or new tools as different teams deliver services for customers.
  • It connects outbound to our VPN endpoint via TCP/443 (HTTPS) and is proxy aware, thereby minimizing any changes the customer may need to make to their environment.
  • Built with security best practices in mind from the ground up. Supported by corporate IT. Granular access control with named accounts and full accountability/audit control.
  • The customer can revert each container, so no test data resides on Firebase at the end of an engagement.
  • Fully managed appliance that receives security patches four times per day.
  • Pen test annually with a customer-facing report available.
  • Supported by regional support teams to provide near 24x7 support. Suitable for protectively marked environments.

Disadvantages:

  • No remote testing capability can replace the value of collaborating, in real-time and onsite, with the client. Remote capabilities will continue to be an important aspect of delivery, but they will not likely completely replace working side-by-side with them.

Lessons Learned

Although NCC Group could perform remote network infrastructure assessments before COVID, the pandemic increased demand for this feature. This demand justified further development of ONI and Firebase and allowed NCC Group to further introduce efficiency during engagements for our clients and us. However, lessons were learned along the way. Namely:

  • The hardware-based ONI solution created logistical issues related to maintaining, shipping, and troubleshooting devices.
  • ONI uses token-based full-disk encryption. Before shipping devices back to NCC Group, we recommend that the client remove the physical USB token and destroy it, making recovering sensitive data virtually impossible.
  • Firebase is more scalable and features a large, global support team. This is evident from the amount of work the system has facilitated, particularly during the pandemic.
  • There are situations where remote testing may not be appropriate and where being onsite is required. They include engagements that require access to:
    • Operational Technology (OT) networks, such as those that contain Industrial Control Systems (ICS).
    • Production environments where testing in non-production is not feasible.
    • Classified systems, such as those found within the US government or UK Ministry of Defense systems.

Conclusion

Although NCC Group already had remote infrastructure assessment capabilities, pandemic and economic pressure incentivized the firm to advance further and market those services.

This resulted in cost benefits for all organizations involved and has allowed work to continue in circumstances where it may not have otherwise. ONI and Firebase are not a panacea, however. In scoping and planning the instrumentation for any engagement, NCC Group must carefully consider what is fit-for-purpose, fit-for-risk, and what will provide the most value and impact for the client.

By continuing to enhance these remote capabilities, NCC Group will remain able to offer our services in an ever-changing threat landscape, up to and including a global pandemic.

Damon Small, MSc.IA, CISSP

Damon Small, MSc.IA, CISSP

Technical Director, NCC Group North America 

Damon Small began his career studying music at Louisiana State University. Leveraging computer skills learned in the LSO recording studio, he became a systems administrator in the mid-1990s. In 2005, he completed a degree in Master of Science in Information Assurance from Norwich University.

Small's 25 years as a security professional have involved initiatives in healthcare, defense, aerospace, and oil & gas industries. As Technical Director at NCC Group, he works closely with consultants and clients delivering complex security assessments that meet varied business requirements.

Secure Your Organization Today with Remote Testing Services

Migrating and leveraging remote services can lead to improved efficiency and cost savings, but only with the right security in place. Our experts here at NCC Group can assess your remote infrastructure, develop a path to fill any gaps, and help you enhance your security over time. Contact us today to speak to an expert about your specific needs.