Reachability analysis cuts through the noise. Instead of treating every flagged dependency as a fire drill, it verifies whether a vulnerable component is actually invoked by your application. The concept is straightforward; the implementation is anything but. This guide bridges the gap between high-level theory and the gritty engineering required to make reachability analysis production-ready.

The Core Pipeline: From SBOM to Call Graph

Production-grade reachability is not a single tool you install. It is a multi-stage pipeline that translates package-level metadata into code-level execution paths. Here is the flow that actually works.

1. SBOM Generation: Know Your Inventory

You cannot analyze what you cannot see. Start by generating a Software Bill of Materials (SBOM) that captures direct and transitive dependencies. Tools like Syft excel here, producing a machine-readable inventory of every package in your build artifact or container image. This SBOM is your ground truth—the foundational list of packages that will enter the vulnerability scanning stage.

2. Vulnerability Mapping: Find the Needles

Feed your SBOM into a vulnerability database. OSV.dev is the pragmatic choice: it aggregates advisories from NVD, GitHub Security Advisories (GHSA), and GitLab, and it accepts package name, version, and ecosystem as query inputs. The output is a list of CVEs affecting your dependencies.

At this stage, you have package-level vulnerability data. You know that log4j-core:2.14.1 is bad. You do not yet know if your code ever calls the vulnerable JndiLookup class. That gap is what the next two stages must close.

3. Symbol Resolution: The Critical Bottleneck

Here is where most reachability initiatives stall. You must map each CVE to the specific vulnerable functions or symbols—not just the affected package. Standard databases rarely provide function-level granularity. OSV might tell you the fix commit, but it will not hand you a list of tainted methods.

To resolve symbols, you often need to:

• Parse security advisories and commit diffs manually.

• Diff the vulnerable version against the patched version to identify changed functions.

• Cross-reference with PoC exploits when available.

Without this step, your call graph analysis has no target. Symbol resolution is the bridge between "we use a bad package" and "we call the exact bad function."

4. Static Call Graph Generation: Pathfinding

With vulnerable symbols identified, you now need to determine if your application reaches them. Build a static call graph from your application's entry points—typically API controllers, message queue handlers, or CLI entry functions.

• Joern offers multi-language support and is robust for polyglot codebases.

• PyCG is purpose-built for Python and handles modern language features well.

Perform a Breadth-First Search (BFS) from each entry point toward the vulnerable symbols. If a path exists, the vulnerability is reachable. If no path exists, the alert is noise, and you can deprioritize it.

Understanding the Dataset Landscape

Your pipeline is only as good as the data feeding it. A crucial distinction exists between datasets that map CVEs to commits and those that map CVEs to functions.

CVE → Commit Datasets: Where the Fix Happened

These datasets identify the commit that patched a vulnerability, but they do not necessarily isolate the root-cause function.

• Project KB (by SAP): A high-quality, industry-vetted dataset with strong verified links for Java commits. Use it when you need reliable ground truth for JVM ecosystems.

• OSV: Superior to raw NVD data for commit links, but coverage gaps remain. Treat it as a curated feed, not an exhaustive source.

• NVD + GHSA: Often noisy. References frequently point to vendor announcements rather than actual code commits. Use these as input feeds; never treat them as ground truth.

CVE → Function Datasets: Where the Bug Lives

Function-level datasets are the holy grail for reachability, but accuracy varies wildly.

• Vul4J & RustXec: The gold standard for reproducible research. Each entry includes a Proof-of-Concept (PoC) exploit and build scripts. The trade-off is scale: both are limited to roughly 80–100 CVEs each.

• MegaVul: Boasts over 6,000 CVEs, but research from the MONO framework indicates that approximately 31% of entries are non-security patches. You must filter aggressively before using it for reachability modeling.

• BigVul / DiverseVul: Large-scale datasets with broad coverage, yet independent audits show label accuracy often drops below 60%. They require heavy cleaning and manual verification before they can drive production decisions.

Why Reachability is an Engineering Nightmare

Even with a clean pipeline and quality datasets, several structural problems make reachability analysis a maintenance burden.

The Bytecode Barrier

In compiled ecosystems like Java, your dependencies arrive as bytecode while your application is analyzed from source. Aligning these into a unified intermediate representation for cross-referencing is non-trivial. Every JDK update, build tool change, or dependency version bump risks breaking the alignment layer.

Inter-Procedural Ambiguity

Research indicates that 89% of real-world vulnerabilities are inter-procedural. The patch might add a safety check in a caller function, while the actual flawed logic remains buried in a callee. If your analysis evaluates functions in isolation, you will miss the context that determines whether the vulnerability is exploitable in your specific call chain.

Incomplete Codebases

Security analysts frequently work with historical vulnerability snapshots that no longer build cleanly. Dependencies have vanished, build scripts have rotted, and the environment has shifted. Tools like JESS address this by using slicing and stub generation to compile methods in isolation, successfully recovering 73% of files from otherwise unbuildable repositories.

Silent Fixes

Not every vulnerability is tracked with a CVE or a clear commit message. Many are patched quietly in routine maintenance releases. Detecting these "silent" fixes requires either advanced graph-based tools like Fixseeker or lightweight statistical filters like Hash4Patch to suppress false positives from already-patched code.

The Bottom Line

Reachability analysis transforms security from a game of whack-a-mole into a prioritized risk roadmap. The engineering hurdles are real—bytecode alignment, inter-procedural reasoning, incomplete snapshots, and silent patches demand significant tooling investment.

Yet the payoff is a pragmatic security posture. When you can demonstrably show that a critical CVE has no call path from your application's surface, you stop burning engineering hours on phantom risks. You focus patching, testing, and incident response on the subset of vulnerabilities that are actually reachable, exploitable, and relevant to your runtime.

That is the difference between being busy and being secure.

1. System Prerequisites

Make sure you meet the following requirements:

• Ubuntu 24.04

• Publicly resolvable domain (e.g., harbor.kcnaiamh.com)

• Access to your Cloudflare account

• Root privileges or sudo access

• Docker engine version 27.5.1

• Docker compose version 1.29.2

Install Required Tools

sudo apt update && sudo apt install -y curl git unzip apt-transport-https ca-certificates gnupg lsb-release jq docker.io docker-compose openssl tar certbot

2. Install Harbor via Online Installer

Download Harbor Installer

cd ~
curl -s https://api.github.com/repos/goharbor/harbor/releases/latest | grep browser_download_url | cut -d '"' -f 4 | grep '\.tgz$' | tail -1 | wget -i -
tar xf harbor-online-installer-v*.tgz
cd harbor

Prepare the harbor.yml Configuration

cp harbor.yml.tmpl harbor.yml

Edit the following fields in harbor.yml:

hostname: harbor.kcnaiamh.com
https:
  port: 443
  certificate: /etc/letsencrypt/live/harbor.kcnaiamh.com/fullchain.pem
  private_key: /etc/letsencrypt/live/harbor.kcnaiamh.com/privkey.pem

You need to specify your own subdomain here.

You can run the following command to change the required filed without opening the file. Also, don't forget to change harbor.kcnaiamh.com to your own subdomain.

sed -i 's/^hostname: .*/hostname: harbor.kcnaiamh.com/g' harbor.yml
sed -i 's/^http:/# http:/g' harbor.yml
sed -i 's/^  port: 80/  # port: 80/g' harbor.yml
sed -i 's|^  certificate: .*|  certificate: /etc/letsencrypt/live/harbor.kcnaiamh.com/fullchain.pem|g' harbor.yml
sed -i 's|^  private_key: .*|  private_key: /etc/letsencrypt/live/harbor.kcnaiamh.com/privkey.pem|g' harbor.yml

⚠️ Don’t run install.sh yet — we need the certs first.

3. Setup Cloudflare Tunnel (For Let's Encrypt HTTP-01 Challenge)

If you have dedicated public IP address from your ISP then you don't need this step.

Create a Cloudflare Tunnel

1. Install cloudflared:

   If you don’t have cloudflared installed on your machine, run the following:

# Add cloudflare gpg key
sudo mkdir -p --mode=0755 /usr/share/keyrings
curl -fsSL https://pkg.cloudflare.com/cloudflare-main.gpg | sudo tee /usr/share/keyrings/cloudflare-main.gpg >/dev/null

# Add this repo to your apt repositories
echo 'deb [signed-by=/usr/share/keyrings/cloudflare-main.gpg] https://pkg.cloudflare.com/cloudflared any main' | sudo tee /etc/apt/sources.list.d/cloudflared.list

# install cloudflared
sudo apt-get update && sudo apt-get install cloudflared

2. Setup cloudflared service:

   After you have installed cloudflared on your machine, you can install a service to automatically run your tunnel whenever your machine starts:

sudo cloudflared service install eyJhIjoiNGM1MTIxNmFhNGU2MTU0NWM4OGUxMjExMDk3YTNjNWYiLCJ0IjoiODIyZjU0ZjYtMmRlNi00NjNiLWEzZDYtYzdkOWUxMDAxZTkzIiwicyI6Ik16aGhOMlpsWmpNdE5UUXpNeTAwWW1ZMUxXSmxPRE10TW1Rd1lqVTRZamxsWVdSaSJ9

You will get the token when doing the tunnel setup.

3. Now setup appropriate subdomain and domain in the cloudflare route tunnel.

4. Get Let's Encrypt Certificate via Certbot

Since HTTP is tunneled through Cloudflare, Certbot's default HTTP challenge will work.

Temporarily Serve HTTP for Challenge

sudo certbot certonly --standalone --preferred-challenges http -d "harbor.kcnaiamh.com" --agree-tos -n -m "naim@gmail.com" --keep-until-expiring

Change subdomain and email address according to yours.

Certs will be saved under:

/etc/letsencrypt/live/harbor.kcnaiamh.com/

5. Start Harbor

🔧 Install Harbor

sudo ./install.sh --with-trivy

Harbor should now be accessible via:

https://harbor.kcnaiamh.com

Default credentials:

• Username: admin

• Password: set in harbor.yml (Default pass: Harbor12345)

Optional: Setup Cert Renewal with Systemd Timer

Create /etc/systemd/system/renew-harbor-cert.service:

[Unit]
Description=Renew Let's Encrypt certs and restart Harbor
[Service]
Type=oneshot
ExecStart=/usr/bin/certbot renew --quiet
ExecStartPost=/opt/harbor/prepare
ExecStartPost=/opt/harbor/install.sh

Create the timer /etc/systemd/system/renew-harbor-cert.timer:

[Unit]
Description=Timer for cert renewal

[Timer]
OnCalendar=weekly
Persistent=true

[Install]
WantedBy=timers.target

Enable it:

sudo systemctl daemon-reexec
sudo systemctl enable --now renew-harbor-cert.timer

Clean Up

To delete the certificate, run the following command:

sudo certbot delete --cert-name harbor.kcnaiamh.com

Final Thoughts

Using Cloudflare Tunnel with Let's Encrypt offers a neat workaround for systems behind NAT or without public IPs. Combined with Harbor, this makes a secure and professional-grade private registry feasible on even home labs.

If you're using Harbor in production, consider:

• Enabling RBAC & LDAP

• Enabling replication for DR

This document provides a step-by-step guide to installing and configuring the AWS CLI and Pulumi on Ubuntu 24.04.

Prerequisites

• Ubuntu 24.04 installed and configured.

• A user account with sudo privileges.

• An active AWS account.

1. Installing AWS CLI using pipx

Step 1.1: Update Package Lists

First, ensure your system's package lists are up to date:

sudo apt update

Step 1.2: Install pipx (if necessary)

Check if pipx is installed:

pipx --version

If pipx is not installed, install it using apt:

sudo apt install pipx
pipx ensurepath
source ~/.bashrc

Step 1.3: Install AWS CLI using pipx

Install the AWS CLI using pipx:

pipx install awscli

Step 1.4: Verify Installation

Verify the installation by checking the AWS CLI version:

aws --version

This command should display the installed AWS CLI version.

Step 1.5: Configure AWS CLI

Configure the AWS CLI with your AWS credentials. You will need your AWS Access Key ID, Secret Access Key, and Session Token (if using temporary credentials). For details on obtaining these credentials, check out this blog.

aws configure

You will be prompted to enter the following information:

• AWS Access Key ID: Your AWS Access Key ID.

• AWS Secret Access Key: Your AWS Secret Access Key.

• Default region name: The AWS region you want to use by default (e.g., ap-southeast-1).

• Default output format: The output format (e.g., json, text, table).

For AWS Session Token, run the following command:

aws configure set aws_session_token <YOUR_SESSION_TOKEN>

2. Installing Pulumi

Step 2.1: Install Pulumi using the Installation Script

Pulumi provides an installation script that simplifies the process. Run the following command:

curl -fsSL https://get.pulumi.com | sh

This script will download and install Pulumi.

Step 2.2: Add Pulumi to Your PATH

The installation script adds Pulumi to your PATH.

However, if it doesn't, you can manually add it by adding the following line to your ~/.bashrc file:

export PATH=$PATH:~/.pulumi/bin

Then, apply the changes by running:

source ~/.bashrc

Step 2.3: Verify Installation

Verify the installation by checking the Pulumi version:

pulumi version

This command should display the installed Pulumi version.

Step 2.4: Configure Pulumi for AWS

Pulumi uses your AWS CLI credentials by default. If you have already configured the AWS CLI, Pulumi will use those credentials.

If not, you can configure Pulumi to use specific AWS credentials by setting environment variables:

export AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="YOUR_SECRET_ACCESS_KEY"
export AWS_SESSION_TOKEN="YOUR_SESSION_TOKEN" # if applicable
export AWS_REGION="YOUR_AWS_REGION"

Replace YOUR_ACCESS_KEY_ID, YOUR_SECRET_ACCESS_KEY, YOUR_SESSION_TOKEN, and YOUR_AWS_REGION with your actual AWS credentials and region.

Step 2.5: Create a Pulumi Account (Optional but Recommended)

While Pulumi can work with local state files, it's highly recommended to create a Pulumi account for state management, collaboration, and other features.

• Visit the Pulumi website (pulumi.com) and create an account.

• Log in to your Pulumi account from the command line:

pulumi login

To run Pulumi locally without login, use --local flag:

pulumi login --local

In this case all the state data will be stored locally under ~/.pulumi/ directory.

Conclusion

You have now successfully installed and set up AWS CLI and Pulumi on your Ubuntu 24.04 system. You can now use these tools to manage your AWS resources efficiently.

While reading the book Container Security, a strange question popped into my head. Since everything inside a container is essentially a process running on the host machine's Linux kernel, what about creating nested containers? From the kernel's perspective, it would be yet another isolated process, right?

I tried running a container inside another container using Docker, but it didn't work. I then discovered that Docker uses a client-server architecture, and everything is managed by a central root process called the Docker daemon.

Later, I found something called Docker in Docker, or DIND for short. It's a special Docker image designed to run Docker inside a DIND Docker container. Cool, right?

This inspired me to explore whether I could create three levels of nested containers using Namespaces, Control Groups, and by directly modifying the file system. It was an amazing learning experience about containers! In this article, I’m sharing how you can do the same, along with the insights I gained.

Let’s Do This!

To conduct the experiment, I'm using AWS EC2, but you can also use a virtual machine. The main requirements are a Linux kernel and a stable internet connection.

1. Setup File System

So, as usual, before installing any package on Linux, update the package list.

sudo apt update

To set up a minimal Debian-based file system with correct permissions, use the debootstrap tool. This utility creates a basic Debian or Ubuntu filesystem by downloading and extracting only the necessary packages.

sudo apt install debootstrap

Once you have debootstrap installed, the next step is to create a directory where the minimal filesystem will be built. You can name the directory anything you like. In this example, we'll create a directory called ubuntufs:

mkdir ubuntufs

Now, use the debootstrap command to create a minimal Ubuntu filesystem. Here, jammy refers to the codename for Ubuntu 22.04 LTS, and ubuntufs is the directory where the filesystem will be set up.

sudo debootstrap jammy ubuntufs

You might need to wait for some minutes to download and unpack the file system by running this command.

Now let’s switch to root user.

sudo su

As we create nested containers, we'll reuse this ubuntufs file system three times. So, we'll copy this directory as shown below.

/home/ubuntu
      └──── /ubuntufs/root
                      └── /lubuntufs/root
                                     └── /llubuntufs

Note that we are using the -a flag in the cp command. Why -a and not -r? Because the -a flag preserves the permissions of all files while copying.

cp -a ubuntufs/ lubuntufs/
mv lubuntufs/ ubuntufs/root/
cd ubuntufs/root/
cp -a lubuntufs/ llubuntufs/
mv llubuntufs/ lubuntufs/root/
exit

2. Create cgroup for 1st Container

Now, to work with cgroups, we will use the cgcreate, cgset, and cgexec commands. To do this, we need to install the cgroup-tools package.

sudo apt install cgroup-tools -y

To create a new cgroup, we are using cgcreate command. Let’s create a cgroup named container1 with the memory, pids, and cpu controllers. These controllers help to restrict memory usage, process IDs, and CPU usage for processes attach within this cgroup.

sudo cgcreate -g memory,pids,cpu:container1

This command sets up the specified controller for container1, enabling resource restrictions for any processes you later assign to it.

Now that the container1 cgroup is created with specified controllers, you can set specific resource limits for memory, process IDs (pids), and CPU usage. Here's how I’m configuring them:

1. Set the memory limit: All the process in container1 can’t allocate more than 256MB of memory.

2. Set the process limit: At most 10 processes can be created under container1.

3. Set CPU weight: This command adjusts the CPU scheduling priority for container1. A higher value means more CPU time for the cgroup. Here, it's set to default value 100.

sudo cgset -r memory.max=256M container1
sudo cgset -r pids.max=10 container1
sudo cgset -r cpu.weight=100 container1

There are more cgroup controllers. Just for learning purpose we are using main three. To know more read the documentation.

2.1 Create cgroup Without Any Tools

(You can ignore this portion of the article if you want.)

Create container1 directory under /sys/fs/cgroup/ path.

sudo mkdir -p /sys/fs/cgroup/container1

Inside the container1 directory, you will notice that necessary folders are automatically created by the kernel.

Now, to enable the memory, pids, and cpu controllers for the container1 cgroup, you need to configure the cgroup.controllers file.

echo "+memory +pids +cpu" | sudo tee /sys/fs/cgroup/container1/cgroup.controllers

These controllers are now active and will manage resources for any processes assigned to this cgroup.

Now, let's assign specific values to the memory.max, pids.max, and cpu.weight files to set the limits for each controller.

sudo sh -c "echo 512M > /sys/fs/cgroup/container1/memory.max"
sudo sh -c "echo 20 > /sys/fs/cgroup/container1/pids.max"
sudo sh -c "echo 100 > /sys/fs/cgroup/container1/cpu.weight"

3. Create our 1st Container

Finally, we will create our first container by running the following command.

sudo cgexec -g memory,pids,cpu:container1 \
unshare --mount --uts --ipc --pid --fork --net \
chroot ubuntufs /bin/bash -c "hostname mycontainer-1 && bash"

This command runs a new process inside a controlled environment with limited resources and isolation.

1. First, cgexec -g memory,pids,cpu:container1 attach the new process with container1 cgroup.

2. Then, unshare --mount --uts --ipc --pid --fork --net creates an isolated space for the process, separating its file system, hostname, inter-process communication, process IDs, and network from the main system.

3. Finally, chroot ubuntufs /bin/bash -c "hostname mycontainer-1 && bash" changes the root directory to ubuntufs, sets the hostname to mycontainer-1, and starts a Bash shell.

The following screenshot shows how new processes are created for the container:

Basically, the container process is a new fork of the previous process in the main system. Everything executed inside that container will be a fork of that process. We will explore this further in the later part of this article.

3.1 Some additional information about Namespaces

Till now, Linux kernel supports 8 different namespaces. These are:

1. Mount Namespace

   Isolates the file system mountpoint

2. UTS Namespace

   Isolates hostname and domain name. Allows processes to have their own hostname and NIS domain name.

3. IPC Namespace

   Isolates inter-process communication (IPC) resources, such as message queues, shared memory, and semaphores. Prevents processes in different IPC namespaces from interacting.

4. PID Namespace

   Isolates process IDs. Processes in a PID namespace have their own independent set of process IDs, and they cannot see or interact with processes outside their namespace.

5. Network Namespace

   Isolates network interfaces, IP addresses, routing tables, and port numbers. Each network namespace has its own virtual network stack.

6. User Namespace

   Isolates user and group IDs. Allows a process to have root privileges inside the namespace while being a non-root user outside the namespace, enhancing security. (rootless container)

7. Cgroup Namespace

   Isolates the view of control groups (cgroups). Processes in a cgroup namespace see only the part of the cgroup hierarchy that they are in.

8. Time Namespace

   Isolates the system clock. Allows processes to have their own view of the system time, which is useful for containers that need to adjust their clocks independently of the host.

In this article, we will not use the last three namespaces as they are relatively new and not all of them support or enabled in a docker container by default.

4. Setup inside the 1st Container

Though you are inside container, you can’t create any process. For that you need to mount the /proc pseudo-filesystem.

mount -t proc proc /proc

This will make the /proc directory available inside the chroot, allowing processes within the container to access vital system information like process IDs, memory stats, and other resources that are normally found in /proc on the host system.

To verify which control groups the current shell (or process) is associated with, you can inspect the /proc/$$/cgroup file. The $$ is a shell variable that represents the process ID of the current shell session. By reading this file, you can see the cgroups and that your process is assigned to.

Run the following command:

cat /proc/$$/cgroup

This command will output the cgroup your current process belongs to.

From the image above, you can see that the container's network stack is isolated and only has a loopback interface. Therefore, we need to configure it further for network connectivity.

4.1 Network setup inside 1st Container

To create a virtual network bridge, you can use the ip command. A bridge allows multiple network interfaces to communicate as if they were on the same physical network, making it useful for connecting virtual machines or containers.

Run the following command on the host machine to create a bridge named v-net-br:

sudo ip link add v-net-br type bridge

To view all active network namespaces on your system, you can use the lsns command with the -t net option. This provides detailed information about network namespaces, including the associated PIDs

sudo lsns -t net

To create a virtual Ethernet (veth) pair and assign one end to a specific network namespace, use the following command. Here, ve1 and ve2 are the names of the veth interfaces, and 16700 is the PID of the target network namespace:

sudo ip link add ve1 netns 16700 type veth peer name ve2

• ip link add: Creates a new network link.

• ve1 netns 16700: Assigns one end of the veth pair (ve1) to the network namespace with PID 16700.

• type veth: Specifies the creation of a virtual Ethernet pair.

• peer name ve2: Names the other end of the veth pair as ve2, which remains in the root namespace.

This setup creates a virtual connection between the two namespaces, allowing them to communicate as if they were connected by a physical link.

To connect one end of a virtual Ethernet (veth) pair to a bridge, you can use the following command. Here, ve2 is the veth interface in the default network namespace, and v-net-br is the bridge created earlier:

sudo ip link set ve2 master v-net-br

• ip link set: Modifies the properties of a network interface.

• ve2: The veth interface you want to attach to the bridge.

• master v-net-br: Specifies that ve2 should become a member of the bridge v-net-br.

This effectively connects ve2 to the bridge, allowing it to communicate with other devices attached to the same bridge.

To assign an IP address to the ve1 interface in the target network namespace, use the following command:

ip addr add 10.0.0.2/24 dev ve1

• ip addr add: Adds an IP address to a network interface.

• 10.0.0.2/24: Specifies the IP address (10.0.0.2) and subnet mask (/24, equivalent to 255.255.255.0).

• dev ve1: Indicates the network interface (ve1) to which the IP address should be assigned.

This command automatically adds a route to the routing table unless specified otherwise.

To assign an IP address to the bridge interface v-net-br, use the following command:

sudo ip addr add 10.0.0.5/24 dev v-net-br

This step allows the bridge to act as a gateway or communicate directly within the 10.0.0.0/24 subnet, making it a central point for routing traffic between connected interfaces.

To assign an IP address to the ve2 interface in the default namespace without modifying the routing table, use the following command:

sudo ip addr add 10.0.0.3/24 dev ve2 noprefixroute

Here, noprefixroute prevents the automatic addition of a route for the specified IP address to the routing table.

This allows ve2 to communicate within the 10.0.0.0/24 subnet without interfering with the system's global routing table.

To bring the ve1, ve2, v-net-br interface up and make it active, use the following command:

ip link set ve1 up

sudo ip link set ve2 up

sudo ip link set v-net-br up

To set a default gateway for the network interface inside container, use the following command:

ip route add default via 10.0.0.5

• ip route add: Adds a new route to the system’s routing table.

• default: Specifies the default route, which is used when no other route matches the destination.

• via 10.0.0.5: Sets 10.0.0.5 as the gateway (next-hop) for the default route.

This command ensures that any traffic destined for an unknown network will be forwarded to the IP address 10.0.0.5, which is the bridge IP that is working as gateway.

To enable NATting for the 10.0.0.0/24 subnet, use the following iptables command. This command allows traffic from the subnet to be masqueraded with the IP address of the internet facing interface’s private IP address.

sudo iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -j MASQUERADE

• iptables -t nat: Specifies the NAT table.

• -A POSTROUTING: Appends a rule to the POSTROUTING chain, which handles traffic leaving the system.

• -s 10.0.0.0/24: Defines the source network (10.0.0.0/24), specifying the traffic that should be subject to NAT.

• -j MASQUERADE: Uses the MASQUERADE target, which dynamically replaces the source IP address of outgoing packets with the IP address of the outgoing network interface.

To enable IP forwarding on host system, which is necessary for routing traffic between different network interfaces, use the following command:

sudo sysctl -w net.ipv4.ip_forward=1

• sysctl -w: Modifies a kernel parameter at runtime.

• net.ipv4.ip_forward=1: Enables IP forwarding for IPv4, allowing the system to forward packets between different network interfaces.

This command is essential when setting up network routing or working with NAT, as it ensures the system can route traffic between networks or namespaces. To make the change persistent across reboots, you can add net.ipv4.ip_forward=1 to /etc/sysctl.conf and run sysctl -p to apply the settings.

To change the DNS nameserver in container to 9.9.9.9, use the following command:

sed -i 's/nameserver .*/nameserver 9.9.9.9/' /etc/resolv.conf

This command will update the DNS resolver to use 9.9.9.9. This change allows your system to resolve domain names using Quad9’s DNS service.

To mount the devpts filesystem, which is used for managing pseudo-terminals, use the following command:

mount -t devpts devpts /dev/pts

• mount -t devpts: Specifies that the filesystem type to mount is devpts, which is responsible for providing pseudo-terminals.

• devpts: The source of the filesystem (in this case, the devpts virtual filesystem).

• /dev/pts: The target directory where the devpts filesystem will be mounted, typically used for controlling terminal sessions.

This command enables the use of terminal devices, which are essential for processes that require interactive shell access, such as when running commands inside containers or chroot environments.

To enable additional software repositories and install the required tools for managing cgroups, use the following commands:

This installs the software-properties-common package, which provides tools for managing repositories and adding PPAs.

This command adds the universe repository, which contains open-source software packages that are not officially supported but are still available for installation.

Finally, this installs the cgroup-tools package, which provides utilities for managing control groups (cgroups) on Linux systems.

These steps set up the system for working with cgroups, which are essential for resource management and isolation in containerized environments.

apt install software-properties-common
add-apt-repository universe
apt install cgroup-tools -y

Running the previous command resulted in this error in my case: the public key is not available: NO_PUBKEY 871920D1991BC93C. To fix this, I added the keyring using the following command:

apt install wget -y
mkdir -p /etc/apt/keyrings
wget -qO /etc/apt/keyrings/ubuntu-archive-keyring.gpg https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x871920D1991BC93C
apt update

To set up the cgroup filesystem and mount it for use, you can run the following commands:

This creates the /sys/fs/cgroup directory if it doesn't already exist. It's the mount point where the cgroup filesystem will be accessed.

mkdir -p /sys/fs/cgroup
mount -t cgroup2 none /sys/fs/cgroup

This command mounts the cgroup version 2 filesystem at /sys/fs/cgroup. The none option indicates that there is no device associated with the mount, and the -t cgroup2 specifies that the mount should be of type cgroup2.

Now we can use cgcreate command like before to create container2 cgroup.

cgcreate -g memory,pids,cpu:container2

cgset -r memory.max=256M container2
cgset -r pids.max=10 container2
cgset -r cpu.weight=100 container2

Now let’s move to root directory inside the container.

cd ~

5. Create our 2nd Container

Now let’s run the following command to create another container inside a container.

cgexec -g memory,pids,cpu:container2 \
unshare --mount --propagation private --uts --ipc --pid --fork --net \
chroot lubuntufs /bin/bash -c "hostname mycontainer-2 && bash"

And here you go. Container inside container.

If we see this from host system, the process looks like the following.

Note that this command is similar to the one we used to create mycontainer-1. However, we are not using the --mount flag for the mount namespace. Why did we do this? Honestly, I don't have a clear idea. I set the propagation type to private on the host machine, but it didn't work. The only time it works is when I'm not using the --mount flag. We will discuss the consequences of not using the mount namespace later in this article.

Now, we will follow similar steps to set up our third container inside the second container. Instead of explaining all the commands again, I'll simply list them below.

mount -t proc proc /proc

sudo lsns -t net

sudo ip link add ve3 netns 19210 type veth peer name ve4

sudo ip link set ve4 master v-net-br

ip link set ve3 up

sudo ip link set ve4 up

ip addr add 10.0.0.20/24 dev ve3

sudo ip addr add 10.0.0.30/24 dev ve4 noprefixroute

ip route add default via 10.0.0.5

sed -i 's/nameserver .*/nameserver 9.9.9.9/' /etc/resolv.conf

mount -t devpts devpts /dev/pts

apt install software-properties-common
add-apt-repository universe
apt install cgroup-tools -y

mkdir -p /sys/fs/cgroup

mount -t cgroup2 none /sys/fs/cgroup

cgcreate -g memory,pids,cpu:container3

cgset -r memory.max=256M container3
cgset -r pids.max=10 container3
cgset -r cpu.weight=100 container3

cd ~

6. Create our 3rd Container

Now we are creating our third container inside second container.

cgexec -g memory,pids,cpu:container3 \
unshare --uts --ipc --pid --fork --net \
chroot llubuntufs /bin/bash -c "hostname mycontainer-3 && bash"

This is the process view from the host machine.

Finally, if we want to view the route table of our host machine, it looks like this:

Although we have only configured the second entry, the others are initially set up by AWS.

Result of not using Mount Namespace

As mentioned earlier, I couldn't create a separate mount namespace in mycontainer-2 and mycontainer-3. As a result, the mount path of mycontainer-3 is accessible from mycontainer-2 and mycontainer-1, but not from the host machine. We can see a similar situation in mycontainer-2.

If you enjoyed this article, you might also like my article on OverlayFS, how it works, and how Docker uses it. Until my next article, As-salamu Alaykum.

This document outlines the procedures for creating AWS Access Keys for both IAM users and IAM Identity Center (Federated) Users. These credentials are used for programmatic access to AWS services, enabling applications and tools to interact with AWS resources.

Creating Access Keys for IAM Users

1. Log in to the AWS Management Console

   • Log in to the AWS Management Console using an account with sufficient permissions to manage IAM users.

2. Navigate to the IAM Service

   • Search for and select "IAM" from the AWS services list.

3. Select Users

   • In the IAM dashboard, select "Users" from the left navigation pane.

4. Choose the IAM User

   • Select the IAM user for whom you want to create access keys.

5. Go to the Security Credentials Tab

   • Click on the "Security credentials" tab.

6. Create Access Key

   • In the "Access keys" section, click "Create access key."

   • Choose the use case for the key. For automated access choose "Command Line Interface (CLI)".

   • Click "Next" then "Create access key".

7. Store the Access Key and Secret Key

   • A dialog box will display the Access Key ID and Secret Access Key.

   • Important: Download the .csv file or copy the keys and store them in a secure location. The Secret Access Key is only displayed once and cannot be retrieved later.

   • Click "Done".

Obtaining Access Keys for AWS IAM Identity Center Users

AWS IAM Identity Center (formerly AWS SSO) allows users to access multiple AWS accounts and applications with a single set of credentials. Here's how to obtain temporary credentials for IAM Identity Center users.

Go to AWS access portal of your AWS organization and follow any of the following option for temporary authentication.

• Option 1: Using AWS CLI

  • To extend the duration of your credentials, it is recommended to configure the AWS CLI to retrieve them automatically using the aws configure sso command.

  • After running the command, you have to follow these steps:

    1. Give SSO start URL

    2. Give SSO Region

    3. SSO authorization page will pop up in the default browser window. If you're in a server then go to the given URL in your browser and enter the code.

    4. Select the AWS account you want to use.

    5. Specify default region.

    6. Specify output format

    7. Specify profile name

  • Use aws sso login --profile <PROFILE_NAME> command when session token expires, and you need to login again.

• Option 2: Set AWS environment variables

  • Export Access Keys as shells environment variables.

    export AWS_ACCESS_KEY_ID="YOUR_AWS_ACCESS_KEY_ID"
    export AWS_SECRET_ACCESS_KEY="YOUR_AWS_SECRET_ACCESS_KEY"
    export AWS_SESSION_TOKEN="YOUR_AWS_SESSION_TOKEN"

• Option 3: Add a profile to your AWS credentials file

  • Copy and paste Access Keys in your AWS credentials file (~/.aws/credentials).

    [PROFILE_NAME]
    aws_access_key_id=YOUR_AWS_ACCESS_KEY_ID
    aws_secret_access_key=YOUR_AWS_SECRET_ACCESS_KEY
    aws_session_token=YOUR_AWS_SESSION_TOKEN

• To use this profile, specify the profile name using --profile

    aws s3 ls --profile <PROFILE_NAME>

• Option 4: Use individual values in your AWS service client

  • Copy and paste Access Keys in your code.

Conclusion

Though getting access keys of IAM user is pretty straight forward, there is a risk of the secrets getting leaked, thus need to rotate regularly. On the other hand, getting access keys of IAM Identity Center user is cumbersome, but it adds security as the credentials are temporary.

Imagine stumbling across a seemingly harmless script while hunting for tools online, only to uncover a sophisticated malware designed to steal passwords, session tokens, and cookies. That’s exactly what happened to me, and it sent me down a rabbit hole of investigation. In this blog post, I’ll take you on that journey.

The Start of the Journey

It all began when I was searching GitHub for a HTTP flooding DoS script for penetration testing. Like everyone else, I was looking for existing code because, well, why reinvent the wheel, right? That’s when I stumbled upon a GitHub repository (Azepofff/Adv-DDOS) on the first page of my search results.

At first glance, the source code looked normal. But do you notice anything suspicious? Probably not, unless you scroll all the way to the right or click the "Raw" button on GitHub—which is how I noticed the malicious code.

Hidden at the end of the script, after a semicolon, was a line of code that installs the fernet module, decrypts an encrypted string, and executes it. This immediately raised a red flag, and I knew I had to dig deeper.

Digging Deeper

I decrypted the encrypted string locally (without executing it) and discovered a URL pointing to a suspicious domain: 1312services[.]ru. Naturally, my next step was to investigate this domain.

I tried accessing it via a browser, but Cloudflare blocked my request.

I attempted to use curl, but the request was still blocked.

Surprisingly, I managed to access the content using the Python requests module.

This revealed that Cloudflare was likely configured to block all requests except those with the default User-Agent header of the requests module. For reference, here’s the default User-Agent of the requests module (version 2.32.3): python-requests/2.32.3

For an added tip, you can send HTTP GET request to http://httpbin.org/headers to see the headers of your request without intercepting it.

Uncovering the Payload

The response contained Python code that created a file named gruppe.py under the C:\Users\<Username>\AppData\Roaming directory, clearly targeting Windows users. The script wrote a string stored in a content variable to this file and then executed it.

To further analyze this, I copied the string into another file. This revealed another Python script, which also decrypted an encrypted string and executed it. Upon decrypting this text, I uncovered 620 lines of Python code, which is the main payload script.

I’ve shared the payload code in this gist: https://gist.github.com/kcnaiamh/ad72bf34957eaf3c144f8a0899233128

Analyzing the Payload

Given the script’s size, I used ChatGPT to get a quick understanding of its functionality. Here’s what I found:

What the Script Does

This malware is designed to steal sensitive data from the victim’s system. It:

• Targets Chromium-Based Browsers:

  • Steals passwords, cookies, and tokens by decrypting browser-stored credentials.

  • Extracts data from popular browsers like Chrome, Edge, Brave, and Opera.

• Targets Cryptocurrency Wallets:

  • Accesses wallets such as MetaMask, Exodus, and Atomic Wallet.

  • Extracts data from browser extensions linked to these wallets.

• Targets Messaging Apps:

  • Extracts authentication tokens and session cookies from Discord and Telegram.

• Searches for Sensitive Files:

  • Scans common directories (e.g., Desktop, Documents) for files containing keywords like "password," "wallet," or "crypto."

• Persistence Mechanism:

  • Saves itself in the Windows Startup folder to ensure it runs on system reboot.

Overall Observation

Remember the domain name? I did a WHOIS lookup to gather more information.

While I didn’t dive too deeply, I suspect the GitHub user “Azepofff” may be linked to Russia. Every repository from this account is designed for deploying the infostealer malware. Groups like these typically steal credentials to further spread malware, infect systems with ransomware if they gain access to critical infrastructure, or sell initial access on hacker forums. They also pilfer cryptocurrency whenever possible.

This is their business model—and unfortunately, it seems to be a profitable one.

If you have the time, please consider reporting this GitHub user.

Final Thoughts

When using code from unknown sources, simply reading it isn’t enough. You need to dig deeper and examine hidden sections to ensure attackers haven’t sneakily inserted malicious code.

This article explores some popular options for those seeking to delve into the world of Android pentesting:

• OVAA (Oversecured Vulnerable Android App)
  This app, created by Oversecured, aggregates various known and popular vulnerabilities within the Android platform. By studying OVAA, you can gain insight into a wide range of security flaws.

• Damn Vulnerable Android Components
  It is an intentionally vulnerable Android Application designed to expose and demonstrate vulnerabilities related to various Android components such as Activities, Intents, Content Providers, and Broadcast Receivers. It is structured as a password manager application to manage and store passwords securely (!).

• Ostrolab Insecure Android App
  This is a vulnerable Android application which contains a number of vulnerable components. It is created and maintained by Ostrolab to demonstrate the effectiveness of Ostolab Security Scanner. We can use it for our learning purpose.

• InsecureShop
  Insecure Android app that has listed vulnerabilities with documentation.

• Free Ram Installer
  It is an Android application designed as a CTF challenge for BSides Algiers 2023. This app is intentionally vulnerable and offers a hands-on experience for security enthusiasts to practice and enhance their Android hacking skills.

• Android SSL Pinning
  This app has almost all types of certificate pinning implementation. See the code how they are implemented. Try to bypass the pinning (without frida).

• Damn Vulnerable Bank
  It focuses on financial app vulnerabilities. Designed with deliberate security weaknesses, this app allows you to explore common banking app exploits in a safe environment.

• Allsafe
  This intentionally vulnerable application serves as a learning ground for a variety of security issues. Developed by t0thkr1s, Allsafe provides a hands-on experience in identifying and understanding different vulnerabilities.

• MAS Crackmes
  A collection of mobile reverse engineering challenges by OWASP MASTG.

• Awesome Mobile CTF
  A curated list of Mobile CTFs. It also has other Mobile hacking resources.

• MobileHackingLab
  It has lots of mobile hacking labs both for iOS and android. Also, the course it offers is great.

After learning the concepts it's time to crack down unintentional vulnerable apps! Download the app of your choice from playstore. Extract APK using adb. Start Hacking!

To understand the journey properly, you need to understand different terminologies so that you can clearly understand everything.

IAM Principals

In AWS, the term "principal" refers to an entity that can make requests to perform actions and access resources. Principals are a fundamental part of IAM and play a crucial role in managing security and access control within your AWS environment.

Types of IAM principals that make requests:

• AWS Account

  A principal can be an entire AWS account identified by its unique AWS account ID.

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "AWS": "123456789012"
          },
          "Action": "s3:GetObject",
          "Resource": "arn:aws:s3:::example-bucket/*"
        }
      ]
    }
  

  The above JSON object is a policy allows any entity/service in the AWS account with ID 123456789012 to perform the s3:GetObject action on all objects within the S3 bucket named example-bucket.

• IAM Role

  Temporarily gives specific permissions to identities. You can create an IAM role with specific permissions and attach it to a service/user to give a certain level of privilege.

  For giving temporary permission to services/users outside the AWS environment (for example, on-prem cloud), an IAM Role Anywhere service can be used.

• IAM User

  Specific individuals or services within an AWS account, identified by their IAM username. It is used for long-term credential access to the account. By default, an IAM user doesn't have any permission. To allocate permission, the best practice is to put users in a group and then attach a policy to that group, or attach a role to a user.

• Federated User

  External users authenticated through identity providers (IdPs) like SAML, OpenID Connect, or AWS Cognito. This principal gets credential by using the sts:GetFederationToken API. This is not a user that is Federated by way of an identity provider. Most customers won't use this type of Federated user often.

• Anonymous User

  Makes unauthenticated requests to AWS. An anonymous principal refers to any entity that is accessing your AWS resources without any authenticated identity. Essentially, this means access is being granted to the general public, without requiring them to sign in or provide any credentials. This is often used in scenarios where you want to make certain resources publicly accessible, such as a public website or shared documents.

• Root User

  User with full access to AWS account.

Every principal has an ARN, which uniquely identifies the principal within AWS. For example, an IAM user ARN might look like arn:aws:iam::123456789012:user/JohnDoe. It is used to specify principal in your policy statement.

AWS Policy

What is AWS Policy?

AWS Policy is a document that defines what actions a principal can perform on which resources within your AWS environment. It is written in JSON using the IAM JSON Policy Language (except Access Control List).

We can devide all the policies into two categories:

1. Policies associated with IAM Entity (IAM Role, User, Group)

2. Policies associated with AWS Resource

Policies that defines the permissions of IAM Entity are:

• Identity-base Policy

  Grants permission to identities (Users, Groups, Roles). This policy can be managed or inline.

• Organization's Service Control Policy (SCP)

  Limits maximum permissions for account members of an organization or organizational unit (OU).

• Permission Boundary

  Limits how much permission Identity-based policy can give to an entity (user/role). It can only be used as managed policy.

• Session Policy

  Limits permissions that role or user's identity-based policies grant to the session.

Policies that defines the permission of AWS Resource are:

• Resource-base Policy

  Grants permission to principals of same account or different account. It is attached with resources like S3, EC2, etc and also the principal is specified in this policy statment. There are some special type of resource-based policies. Some of them we will talk in this blog are:

  • Trust Policy

    The trust policy defines which principals can assume the IAM Role, and under which conditions.

  • VPC Endpoint Policy

    It is used to control which AWS principals can use the endpoint to access an AWS service. An endpoint policy does not override or replace identity-based policies or resource-based policies.

• Access Control List (ACL)

  Grants permission to principals of same or different accounts. It doesn't use JSON format and few AWS service support this policy.

Following two mindmap screenshots are from my notes:

To learn more about AWS Policy, you can have a look in AWS documentation.

Statement: Building block of AWS Policy

An AWS Policy Statement is a component of an AWS Policy that specifies the permissions for a particular action or set of actions on specific resources. Each policy can have multiple statements, each providing detailed control over permissions. Lets analyze the following Identity-based policy.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::example-bucket",
        "arn:aws:s3:::example-bucket/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": "ec2:StartInstances",
      "Resource": "arn:aws:ec2:us-west-2:123456789012:instance/*"
    },
    {
      "Effect": "Deny",
      "Action": "s3:DeleteObject",
      "Resource": "arn:aws:s3:::example-bucket/*"
    }
  ]
}

This policy grants specific permissions and denies one particular action within AWS. It allows the identity to list the contents of and retrieve objects from the S3 bucket named example-bucket, as well as start any EC2 instances in the us-west-2 region under the account 123456789012. However, the policy explicitly denies the ability to delete objects from the example-bucket S3 bucket.

How is a Policy Evaluated?

A policy can have multiple statements. But to pass through the policy, only one allow statement and no deny statement is required. For an incoming request, the request context is evaluated against the policies. To authorize the request against a policy,

1. At least one statement with an effect of allow must match for a request to be allowed.

2. A matching statement with an effect of deny takes precedence over a matching statement with an effect of allow.

This process will be repeated for all the policies in the policy chain. (Be aware that, based on scenario, different set of policies are evaluated in different order. We will learn about this in later part of this blog.)

Based on the policy evaluation, AWS decides whether to allow or deny the request. The decision is logged in AWS CloudTrail for auditing purposes.

Tools and Resources for AWS Policies

• AWS Policy Visualizer
  Using this tool you can visualize AWS Policies and can manually debug the policy to check if has appropriate permissions. Great tool for writing and debugging effective policies.

• Policy Simulator
  Using this you can simulate how policy will work before adding/using the policy. You need an AWS account to use it (because it will use your IAM user to simulate).

• Policy Generator

  Use this website to generate policy in easy way.

• Actions, resources, and condition keys for AWS services

  If you are someone who write AWS policies, then this page should be bookmarked. You will get all everything to write policies for all services in AWS.

Request Journey

Now let’s break down the journey from the moment a request is made by an AWS principal until it’s authorized or denied. For simplicity we will focus on only one principal, AWS Role.

1. Making the API Request

You can only interact with AWS through API. For that, there are 3 ways — Console, CLI, and SDK.

• AWS Management Console: When you perform an action using the AWS Management Console, the console makes API calls on your behalf.

• AWS CLI: You can run commands directly in your terminal or write a script to interact with AWS services.

• AWS SDKs: When you are building a software and need to interact with AWS programmatically, you will use libraries that is build and maintained by AWS. Boto3 for python3 is such an example.

(You can also interact with AWS as an anonymous user. But for this scenario, I am not considering this.)

2. Authentication

Before interacting with AWS, you have to authenticate yourself - so that AWS can recognize who you are.

In case of console, the authentication process complete when you signin using account ID, IAM username, password and MFS.

When you are interacting with AWS programmatically using SDK and CLI, you need to use appropriate access key and secret key to authenticate yourself.

In case of IAM Role, it authenticates with temporary credentials, which are given by AWS Security Token Service (STS). This service by default runs in every AWS region. Its job is to produce short time credentials to use by an IAM Role.

An IAM Role needs two resource-based policies when it is created. Trust Policy and Role Policy.

Trust Relationship Policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

For Lambda functions, you need to grant this permission to the Lambda service itself. The Principal element with Service set to lambda.amazonaws.com indicates that the AWS Lambda service is trusted to assume this role. The Action element with sts:AssumeRole specifies the action that is allowed. This setup ensures that only AWS Lambda can use this role, adding another layer of security by preventing other AWS services or users from assuming this role and accessing its permissions.

IAM Role Policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::example-bucket/*"
    }
  ]
}

The IAM Role Policy defines what actions the Lambda function is allowed to perform. In this example, the policy allows the Lambda function to read from and write to an specific S3 bucket.

In case of anonymous user, no authentication is required.

3. Building Request Context

Every request to AWS contains unique properties that is be provided along with an authorization request to make more granular authorization decisions. Till of this date there are total 18types of request context parameters — account ID, authorizer principal ID, source IP, user aget, etc are such examples. IAM then compares this authorization context with to your IAM policies to determine which policy statements match.

In case you are curious and want to know more, read the AWS documentation on request context.

4. Evaluate Policies

Before understanding evaluaton process of the policies, its crucial to understand what is Root account and Organization Unit in AWS Organization.

AWS Organization

In a large organization, there might be multiple (sometimes hundreds/thousands) of AWS accounts for different use cases. For proper security, resource, and bill management, AWS offers the AWS Organization service. This service allows you to centrally manage and govern multiple AWS accounts. It simplifies billing by consolidating payments for all accounts under a single master account, known as the Root Account. AWS Organizations also enhance security and compliance by enabling you to apply Service Control Policies (SCPs) that define the services and actions each account can use. Additionally, it helps streamline workflows by allowing you to create Organizational Units (OUs), which are groupings of accounts that can be managed collectively based on similar requirements or functions. This hierarchical structure facilitates efficient resource management, policy enforcement, and cost tracking across the entire organization.

For the example shown above, you can attach SCP policies directly to the Root Account, which will apply to all accounts under it. You can also attach SCP policies to a group of accounts by attaching it to OU. Additionally, it is possible to attach SCP policies to individual accounts.

For the simplicity of explanation and demonstration, we will abstract all the SCP policies into one.

Policy Evaluation: IAM Role

Now lets get to our IAM Role. An IAM role by itself does not make requests without a session. A role always has a corresponding session which is the principal that actually makes request.

The request is first evaluated against the SCPs defined at the organizational level. SCPs apply to all principals (users, roles, etc.) within the organization. These policies determine the upper limit of what actions are allowed or denied within the organization. The account context comes into play next, where the request is evaluated based on the policies specific to the account in which the resource resides.

Here when a request is invoked by a AWS Role Session, the role need to be allowed by multiple policies startign from organization's SCPs. If the role need to access a resource inside the same account then one of the following three policy chain will be evaluated accourding to the principal that is specified in the Resource-based Policy statment.

If the resource-based policy statement specifies principal for Accout, Role or Session then the policy evaluation path will be taken the followign 1, 2 or 3 respectively.

If the request doesn't access any resource then the following policy chain will be evaluated.

When the requested Role session originates from one account and the requested resource is in different account, then all the policies need to approved to access the resource.

Policy Evaluation: IAM User

Till now we have understood the policy chain evaluation in case of AWS Role. But for other principals, the evaluation chain is different. For the completeness of the article, let's consider the AWS User principal in the following scenario.

For request approval of an AWS User, all three session need to be evaluated. Note that the order of evaluation chain is different. It is evaluating Permission boundary policies before Identity-based policy. Rest of the structure is same as IAM Role.

Policy Evaluation with VPC Endpoint

In the previous examples, we assumed that there is no VPC endpoint service attached. If there is any VPC endpoint service, then the evaluation process of VPC endpoint policy will be after SCP, before getting access to the Account.

Conclusion

Every API request in AWS is a multi-step journey involving authentication, building request context, collection, and evaluation of various policies. By understanding how each type of policy contributes to the final authorization decision for each type of principal, you can better manage and troubleshoot access controls in your AWS environment.

To do this lets first generate RSA key pairs using the following command. (Note: whatever prompt comes after that, Just press ENTER.)

ssh-keygen -t rsa

You will find private and public key as id_rsa and id_rsa.pub in ~/.ssh/ folder.

Copy the public key and go to cPanel's 'SSH Access'. Click 'Manage SSH Keys' > 'Import Keys' > Paste the public key into public key text box > 'Import'.

After following the steps you will find that Authorization Status is not authorized . To authorize the key, Click 'Manage' buttion in the same row and press 'Authorize'.

Now we are done with cPanle. Lets go to our website's github repo. To automate the deplyment process we need authentication to access the server. To store our authentication credentials and secrets we will use 'Secrets and variables' actions under Settings tab.

We will store our server domain name/IP, Username and Private Key by pressing 'New repository secret' button for each secret.

After that lets pull our website using git pull and create .github/workflows/deploy.yml file in the root directory of the source code. In this file, we will paste the following YAML code. (make change according to your need)

name: Deploy Website

on:
  push:
    branches:
      - master

jobs:
  deploy:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: "22.2.0"

      - name: Build Project
        run: |
          npm ci
          npm run build

      - name: Setup SSH Key
        env:
          SSH_PRIVATE_KEY: ${{ secrets.SSH_PRIVATE_KEY }}
        run: |
          mkdir -p ~/.ssh
          echo "$SSH_PRIVATE_KEY" | tr -d '\r' > ~/.ssh/id_rsa
          chmod 600 ~/.ssh/id_rsa

      - name: Deployment
        env:
          REMOTE_HOST: ${{ secrets.REMOTE_HOST }}
          REMOTE_USERNAME: ${{ secrets.REMOTE_USERNAME }}
        run: |
          zip -r dist.zip ./dist/

          # Remove existing files on remote server
          ssh -o StrictHostKeyChecking=no -i ~/.ssh/id_rsa $REMOTE_USERNAME@$REMOTE_HOST 'rm -rf ~/public_html/* || true'

          # Upload new files
          scp -o StrictHostKeyChecking=no -i ~/.ssh/id_rsa dist.zip $REMOTE_USERNAME@$REMOTE_HOST:~/public_html/

          # Unzip and move files
          ssh -o StrictHostKeyChecking=no -i ~/.ssh/id_rsa $REMOTE_USERNAME@$REMOTE_HOST 'cd ~/public_html && unzip dist.zip && mv dist/* . && rm -rf dist*'

Now, everytime when you will push the source code into github on master branch, an Action will trigger and deploy you code into your server.

To understand how docker utilized overlayFS, we need to first understand what is OverlayFS and how it works.

OverlayFS is a type of filesystem service for Linux that allows for the layering of multiple directories to form a unified filesystem. This functionality is particularly useful in environments where a separation between static, read-only data and dynamic, writable data is beneficial, such as in containerized applications.

Internally, OverlayFS operates by combining two directories called layers: a lower layer and an upper layer. The lower layer is typically read-only and contains the base files and directories. The upper layer is writable and is where changes and new files are stored. When a file or directory is accessed, OverlayFS first checks the upper layer. If the file is present in the upper layer, it is used. If it is not found, the system looks in the lower layer. This way, the upper layer effectively overlays the lower layer.

When a file in the lower layer is modified, OverlayFS does not alter the lower layer directly. Instead, it copies the file to the upper layer, where the changes are then made. This process is known as "copy-up." Subsequent access to this file will read from the modified version in the upper layer. Similarly, when a file is deleted, it is not removed from the lower layer. Instead, a character file is created in the upper layer, indicating that the file should be considered deleted.

This approach provides several advantages. It allows for the preservation of the original data in the lower layer while maintaining a complete view of the filesystem that includes modifications and additions. It also enables efficient use of storage, as only modified and new files consume space in the writable upper layer. Furthermore, OverlayFS simplifies the rollback of changes and updates, as the lower layer can remain untouched and intact.

We will practically see how OverlayFS is used to manage the layered nature of container images and their writable layers later section of this blog.

Now lets create and overlay file system by our selves. For this we have created a dummy folders and files structure as shown below.

We will use the following command to create the overlay layer.

mount -t overlay -o lowerdir=lower_layer/,upperdir=upper_layer/,\
workdir=temp_layer/ none overlay_layer/

By running the df -ah command you can see that our overlay_layer is running as none file system

Using tree command we can see the structure includes several key layers: lower_layer, overlay_layer, temp_layer, and upper_layer.

The lower_layer represents the base image in our scenario, containing the foundational files upon which other layers build. Inside the base_image directory, we have several files named PART1.txt through PART4.txt, as well as common.txt, second_instruction.txt, and third_instruction.txt. These files form the initial content that is common across multiple layers.

The overlay_layer also contains a base_image directory mirroring the lower_layer. This layer includes identical files (PART1.txt through PART4.txt, second_instruction.txt, and third_instruction.txt), indicating that the overlay layer starts as a copy of the lower layer.

The temp_layer contains a work directory, which serves as a temporary workspace for OverlayFS to manage the merging of the lower and upper layers. This workspace is crucial for operations that modify the file system, ensuring consistency and stability.

Finally, the upper_layer contains only one file, common.txt, which contains different content common.txt file than the lower layer. Overlay layer only tracked the upper_layer's common.txt file.

Now lets create a new file in overlay_layer folder. We can see that the newly created file is also inside the upper_layer folder.

Deleting the newly created file also deletes from the upper_layer folder.

If we delete the common.txt file then it deletes the file from the overlay_layer and upper_layer folder but this also creates a character file inside upper_layer folder. This character file indicates that this file is deleted and so the common.txt file from the lower_layer is not copied to overlay_layer folder.

Now, lets update the second_instruction.txt file in overlay_layer folder.

We can see that second_instruction.txt in upper_layer is updated, but lower_layer is as it was before.

So, we have understood how overlayFS works with two layers. How about there is multiple layers? like docker? Lets try to implement the following diagram.

We will use the following command to build the OverlayFS structure like above diagram.

mount -t overlay -o lowerdir=overlay_layer/,upperdir=upper_layer_2/,\
workdir=temp_layer/ none overlay_layer_2/

We can see overlay_layer_2 has everything identical as overlay_layer. So what happens when we update the overlay_layer now?

We can see, after creating a new file a.txt inside overlay_layer, upper_layer and overlay_layer_2 also updated with the new file.

When we create a new file inside overlay_layer_2, it updates the upper_layer_2 only.

The above observation tells us that overlay_layer_2 is treating overlay_layer as lower layer. So, any update inside overlay_layer_2 will not reflect inside overlay_layer, but updating overlay_layer will reflect inside the overlay_layer_2.

Now lets strengthend our understanding on overlayFS by investigating the changes with docker image and containers.

The /var/lib/docker directory is the backbone of Docker's storage and runtime environment. This directory houses everything from images and containers to volumes and network configurations. To identify files larger than 1 megabyte within this directory, we can use the command find . -size +1M. Upon running this command, we discover that all such files are located in the overlay2 directory. This indicates that Docker stores its images and containers within the overlay2 directory.

When you first install Docker on a Linux system, you will notice that the /var/lib/docker/overlay2 directory is initially empty. However, as you start interacting with Docker—pulling images from a registry, creating containers, and committing containers into images—this directory will begin to populate and grow in size.

Now, lets build an image with dockerfile on a fresh machine. We are using the following dockefile to build an image. The Dockerfile sets up a container by first using the lightweight python:3.9-slim image as its base. It then sets the working directory inside the container to /kcnaiamh. The takbir.txt file from the local machine is copied into this directory. The RUN instruction executes the cat takbir.txt command, displaying the contents of the file. Finally, the USER nobody command specifies that the container should run as the nobody user for increased security.

After running the docker build . command, Docker initiates the process of creating an image based on the instructions specified in the Dockerfile. Below, lets outline the detailed steps involved in this process.

The first step involves sending the build context to the Docker daemon. The build context includes all files and directories that will be used during the build process. In this case, the context is quite small, only 3.072kB, as it only contains the Dockerfile and the takbir.txt file.

The build process starts with the instruction FROM python:3.9-slim, which sets the base image for the build. Since this image is not already present on the local machine, Docker pulls it from the Docker Hub repository. The output shows five layers being downloaded (24c63b8dcb66, e7a43413405, cdae7c163d4, cb40e615cf98, 06ff06197436), which are different parts of the base image. After all layers are pulled, the image is verified by its digest (sha256:088d9217202188598aac37f8db0929345e124a82134ac66b8bb50ee9750b045b), ensuring its integrity. This step completes with the base image being ready, indicated by the image ID 182cc99a2af6.

The second instruction is WORKDIR /kcnaiamh, which sets the working directory inside the container to /kcnaiamh. Docker creates a temporary container to execute this instruction and then removes the container once the instruction is successfully applied, resulting in the intermediate image ID 9b8cc55094ae.

The third step is COPY takbir.txt ., which copies the takbir.txt file from the build context to the current working directory inside the image. The resulting intermediate image is identified by 05b93a68ab95.

The fourth instruction, RUN cat takbir.txt, runs the command cat takbir.txt inside the container. This command outputs the contents of the file ("Allahu Akbar") to the console. Docker again creates a temporary container to execute the command, and after successful execution, removes the container. The intermediate image ID at this stage is dfba555c9e5b.

The final instruction is USER nobody, which changes the user context for the container to nobody for security purposes. This instruction is also run in a temporary container, and upon successful completion, the intermediate container is removed. The resulting final image ID is e8020179299.

This final image can now be used to run containers with the specified configuration.

Previously, we observed that the /var/lib/docker/overlay2 directory was empty. Now that we have built the image, we should be able to find the kcnaiamh folder somewhere within this path. To locate it, we can use the command ls -1R | grep "kcnaiamh", which recursively lists directories and files, filtering the output to show only those containing the name "kcnaiamh".

The output indicates that the kcnaiamh directory is found within two different layers of the Docker overlay filesystem, specifically in paths starting with /25d6761609ad... and /9ff6b8df2eb8....

Next, we can use the command grep -ri "Allahu" to recursively search for the string "Allahu" within files. The search finds the string "Allahu Akbar" in the file located at /9ff6b8df2eb8.../kcnaiamh/takbir.txt, confirming the content previously displayed by the RUN cat takbir.txt command during the Docker build process.

Let’s explore the contents of the /var/lib/docker/overlay2/l directory. After executing the ls -l command, we observe several hard links pointing to diff directories, which are located in different subdirectories under the overlay2 folder. This setup raises the question: What is the purpose of this l directory?

This directory is integral to Docker's overlay2 storage driver, which efficiently manages the layering of filesystem changes. The l directory stores hard links to files in the lower layers, optimizing Docker's ability to reference and share files across multiple layers without duplication. By using hard links, Docker ensures quick access to shared files, which enhances performance and reduces storage overhead. The overlay2 driver works by stacking read-only layers on top of each other. The role of the l folder is to maintain these references, making file management more efficient. This setup is crucial for maintaining the integrity and efficiency of Docker images and containers, particularly when dealing with complex layer hierarchies and numerous containers.

By leveraging the l directory and its hard links, Docker can efficiently manage the filesystem layers that comprise an image. This ensures that common files are stored only once on disk, while still being accessible from multiple layers. This architecture not only conserves storage space but also speeds up access to files, as the need for redundant copies is eliminated. Thus, the l directory is a key component in Docker’s strategy to streamline container storage and enhance overall performance.

The WORKDIR /kcnaiamh instruction in dockerfile created SHBBHRCYVAFCF07ZQ3ZXEDJ2RC layer. And COPY takbir.txt . instruction created T3W3LMFZENSZLZTRNYWTYQXWI layer.

Now lets update the Dockerfile by adding one more instruction (COPY whoami.txt .) after third instruction.

Now, lets build a new docker image named new-build using our modified dockerfile. Initially, the Docker engine executes Step 1 up to Step 3 by fetching the layers from the local cache, signified by the "Using cache" message. Notably, their hash values persist unchanged from prior builds.

However, at Step 4, we can see a divergence. Here, a new file, whoami.txt is added into the image, resulting the formation of a new layer. As the build advances, additional directives are executed, such as executing a command to reveal the contents of "takbir.txt" (Step 5) and configuring the user as "nobody" (Step 6). Each of these actions creates new layers, each with its own hash value.

Docker makes the build process faster by using cached layers when possible, avoiding repeated work for unchanged steps. However, any changes in later steps require new layers, showing changes in the image. The final hash value uniquely identifies the complete image, making it easier to recognize and reuse in future Docker tasks.

Even though the instructions in Steps 5 and 6 of the Dockerfile are the same as in the previous build, Docker's caching works step-by-step. This means that even if the instructions haven't changed, they are part of a new build context because of the new layer added in Step 4. Docker checks for changes in each step, including changes to files or directories in the build directory. If any part of the build context changes, Docker doesn't reuse the cache for that step. So, even though the instructions in Steps 5 and 6 haven't changed, Docker treats them as new steps in the current build, requiring them to run again and creating new layers with unique hash values.

After building the image we see another new layer (folder) is created and inside that we see the whoami.txt file.

Now lets build another image, but we will just remove the last command from our previous dockerfile which set the user to nouser. So the container from the new image will run as root user.
Also note that, this time all the layers are fetched from the cache.

Now lets create a new container from the root-build image and created a directory named naim inside root directory. Inside that direcotry lets create a file named new_thing.txt that contains a specific text.

Here we can see 4 new directories, in which most interesting is 6AIKVNU2UWE3G5Q4B6XGEK5R4H directory. It shows the folders that differs from the root-build image. We've created naim folder inside container. .bash_history file is change inside /root folder. ( why usr folder is here? What changed inside usr? I'm not sure (: )

Now lets create a new file inside /kcnaiamh directory named newfile.txt and check if 6AIKVNU2UWE3G5Q4B6XGEK5R4H reflect some change.

Yes, it does. kcnaiamh folder came inside 6AIKVNU2UWE3G5Q4B6XGEK5R4H and it contains the file we just created, not the file it already had.

Interesting. So what happens when we delete a file? Lets figure it out. Lets delete whoami.txt file inside the container.

We can see a new character file in 6AIKVNU2UWE3G5Q4B6XGEK5R4H. But the acutal whoami.txt file still exists in different layer.

Now letes delete the naim directory, which we have created inside the running container (not while building image).

We can see the folder does not exists in 6AIKVNU2UWE3G5Q4B6XGEK5R4H.

Now lets delete kcnaiamh derectory.

We can see the directory is deleted from 6AIKVNU2UWE3G5Q4B6XGEK5R4H and in place of that a character file is created.

From our previous observations, we can conclude that when a container creates or modifies something from the image it is built from, it creates a new layer to store those changes. If we delete something from the container that was created during its runtime, it is removed from this newly created layer. Conversely, if we delete something that was originally part of the image, a new character file is created in the layer to mark this deletion.
This is exactly the way OverlayFS works.

From the above screenshot we can see two containers are running. So two overlay file systems are created. The following screenshot displays all the directories inside/var/lib/docker/overlay2/ directory.

Within this directory, container IDs with the -init suffix represent the initialization layers of the respective containers. By examining the directories with the same name but without the -init suffix, we find a merged directory inside each.

The merged directory contains the entire file system of the running container, providing a complete view of the container’s filesystem at that layer. This directory structure is a result of Docker's use of the overlay filesystem.

Lets change the content inside this folder. If the change reflects inside the container then we can manipulate the files and folders of container from outside!

Woah! the change actually reflected inside container. Do you know what that means? If we have root access then we can change the container without interacting with docker daemon. Also, note that 7adbb45b2373 container is build on new-build image. What we had set as nobody user. So we can not do anything to the container by interacting with the docker daemon. But we can manipulate the container if we have the root access to the host machine.

If you reached till here, you are a different beast man! Congratulations. Hope to see you in another of my technical blog.

I've used VMware workstation for virtualization. You can use Virtual Box and follow this steps. Just interface name will be different.
Also, ubuntu server 22.04 iso is used for creating 3 virtual machines - client, router, server.

Create Network

• vmnet8: NAT - 192.168.22.0/24 - DHCP (for internet connection)

• vmnet2: Host-only - 10.10.1.0/24

• vmnet3: Host-only - 10.10.2.0/24

Building Router

Create a VM using ubuntu server 22.04 iso image.
It will use the following configurations:

• RAM: 2048 MB

• Processors: 1

• Network Adapter 1 → vmnet8

• Network Adapter 2 → vmnet2

• Network Adapter 3 → vmnet3

Setup the configuration of these 3 interface in /etc/netplan/*.yaml file.

network:
    ethernets:
        ens33:
            dhcp4: true
        ens36:
            addresses: [10.10.1.1/24]
        ens37:
            addresses: [10.10.2.1/24]
    version: 2

The above configuration:

1. Enables the DHCP protocol on ens33 interface. IP and gateway address will be set dynamically.

2. Sets IP address of ens36 interface.

3. Sets IP address of ens37 interface.

Change permission: sudo chmod 600 *.yaml
Apply the change: sudo netplan apply

Stop and remove UFW firewall.

sudo systemctl stop ufw
sudo apt purge ufw

Building Client

Clone the router VM. Go to settings and delete adapter 1, 3.

Configure the interface using netplan. in /etc/netplan/*.yaml file.

network:
    ethernets:
        ens36:
            addresses: [10.10.1.10/24]
            routes:
                - to: default
                  via: 10.10.1.1
            nameservers:
        addresses:
            - 8.8.8.8
            - 8.8.4.4
    version: 2

The above configuration:

1. Sets device IP address (10.10.1.10) and Netmask (/24).

2. Sets any destination IP address route except this network will go to specific gateway (10.10.1.1).

Apply the change: sudo netplan apply

Building Server

Clone the router VM. Go to settings and delete adapter 1, 2.

Configure the interface in /etc/netplan/*.yaml file.

network:
    ethernets:
        ens37:
            addresses: [10.10.2.20/24]
            routes:
                - to: default
                  via: 10.10.2.1
            nameservers:
        addresses:
            - 8.8.8.8
            - 8.8.4.4
    version: 2

Apply the change: sudo netplan apply

Now both client and server can communicate with the router.

Client (or Server) VM can't ping the Router VM?

Ping client (or server) from the router.
This will invoke the ARP request to get the MAC address of client (or server).

Enable packet forwarding

Router need to forward packet from one interface to another. Thant's why we need to enable packet forwarding.

sudo echo 1 > /proc/sys/net/ipv4/ip_forward

Source NAT for local network

From the configured infrastructure, we know that two host-only network is in 10.10.1.0/24 & 10.10.2.0/24 range. Devices connected in these network is configured with 10.10.1.1 & 10.10.2.1 IP address respectively as Gateway IP.

(we don't need to configure MAC address manually. All interface will get the MAC address through ARP broadcast, if they don't have it in cache.)

The router has three network interfaces; ens36 & ens37 for two different host-only network and ens33 for internet-facing connection.

Now your packet from client (or server) can go to routers appropriate interface. If the packet goes from one network to another, the gateway would not recognize the source IP. It will send ARP request to the network to get MAC address of the source IP. If it doesn't get response, it will drop it.

So, to send the packet of different network, we are going to change the source IP address with interfaces own IP address.

We will run the following two rules in router VM to do IP Masquerading when the packet is coming form 10.10.1.0/24 and 10.10.2.0/24 network.

sudo iptables -t nat -s 10.10.1.0/24 -A POSTROUTING -j MASQUERADE

sudo iptables -t nat -s 10.10.2.0/24 -A POSTROUTING -j MASQUERADE

Now, say a device in local network with IP address 10.10.1.10 wants to access a website on the internet. When the device sends a packet to the gateway, the iptables rule kicks in.

The source IP address (10.10.1.10) of the outgoing packet is dynamically replaced with the routers private IP address of the ens33 interface.

This is called IP Masquerading. In this process, nothing is changed except the source IP.

Tracing IP on different interface

I have used tcpdump to check the source and destination IP address of a packet in every interface.

The following table shows when a packet is going form client machine to server machine:

The following table shows when a packet is going from server machine to internet:

How packet is moving from one interface to other

When a packet comes, linux kernel checks the routing table to determine the appropriate interface for the packet.

If the packet was destined for the internet, the routing table directs the packet to the ens33 interface. After going to that interface, in POSTROUTING chain, the source IP is changed to current interface's IP.

If the packet was destined for the server VM, the routing table would direct the packet to ens37 interface. Similar to above, the source IP would change.

1. AhmedS Kasmani

2. Amr Thabet - MalTrak

3. ANY.RUN

4. crow

5. cazz

6. CursoReversing

7. Flashback Team

8. Marcus Hutchins

9. MalwareAnalysisForHedgehogs

10. MalwareAficionado

11. MalGamy

12. Make Me Hack

13. Low Level Learning

14. pwn.college

15. OALabs

16. stacksmashing

17. Zer0Mem0ry

18. The PC Security Channel

19. 0x6d696368

20. 0x41414141

21. 0xca7

22. c3rb3ru5d3d53c

23. Duncan Ogilvie

24. Low Byte Productions

25. Guided Hacking

26. Creel

27. Mitch Edwards

28. LaurieWired

29. Path Cybersec

30. All things IDA

31. dist67

32. Anuj Soni

33. cybercdh

34. Boston Cybernetics

35. TactiKoolSec