Table of Contents for
Python: Penetration Testing for Developers

Version ebook / Retour

Cover image for bash Cookbook, 2nd Edition Python: Penetration Testing for Developers by Dave Mound Published by Packt Publishing, 2016
  1. Cover
  2. Table of Contents
  3. Python: Penetration Testing for Developers
  4. Python: Penetration Testing for Developers
  5. Python: Penetration Testing for Developers
  6. Credits
  7. Preface
  8. What you need for this learning path
  9. Who this learning path is for
  10. Reader feedback
  11. Customer support
  12. 1. Module 1
  13. 1. Understanding the Penetration Testing Methodology
  14. Understanding what penetration testing is not
  15. Assessment methodologies
  16. The penetration testing execution standard
  17. Penetration testing tools
  18. Summary
  19. 2. The Basics of Python Scripting
  20. Python – the good and the bad
  21. A Python interactive interpreter versus a script
  22. Environmental variables and PATH
  23. Understanding dynamically typed languages
  24. The first Python script
  25. Developing scripts and identifying errors
  26. Python formatting
  27. Python variables
  28. Operators
  29. Compound statements
  30. Functions
  31. The Python style guide
  32. Arguments and options
  33. Your first assessor script
  34. Summary
  35. 3. Identifying Targets with Nmap, Scapy, and Python
  36. Understanding Nmap
  37. Nmap libraries for Python
  38. The Scapy library for Python
  39. Summary
  40. 4. Executing Credential Attacks with Python
  41. Identifying the target
  42. Creating targeted usernames
  43. Testing for users using SMTP VRFY
  44. Summary
  45. 5. Exploiting Services with Python
  46. Understanding the chaining of exploits
  47. Automating the exploit train with Python
  48. Summary
  49. 6. Assessing Web Applications with Python
  50. Identifying hidden files and directories with Python
  51. Credential attacks with Burp Suite
  52. Using twill to walk through the source
  53. Understanding when to use Python for web assessments
  54. Summary
  55. 7. Cracking the Perimeter with Python
  56. Understanding the link between accounts and services
  57. Cracking inboxes with Burp Suite
  58. Identifying the attack path
  59. Gaining access through websites
  60. Summary
  61. 8. Exploit Development with Python, Metasploit, and Immunity
  62. Understanding the Windows memory structure
  63. Understanding memory addresses and endianness
  64. Understanding the manipulation of the stack
  65. Understanding immunity
  66. Understanding basic buffer overflow
  67. Writing a basic buffer overflow exploit
  68. Understanding stack adjustments
  69. Understanding the purpose of local exploits
  70. Understanding other exploit scripts
  71. Reversing Metasploit modules
  72. Understanding protection mechanisms
  73. Summary
  74. 9. Automating Reports and Tasks with Python
  75. Understanding how to create a Python class
  76. Summary
  77. 10. Adding Permanency to Python Tools
  78. Understanding the difference between multithreading and multiprocessing
  79. Building industry-standard tools
  80. Summary
  81. 2. Module 2
  82. 1. Python with Penetration Testing and Networking
  83. Approaches to pentesting
  84. Introducing Python scripting
  85. Understanding the tests and tools you'll need
  86. Learning the common testing platforms with Python
  87. Network sockets
  88. Server socket methods
  89. Client socket methods
  90. General socket methods
  91. Moving on to the practical
  92. Summary
  93. 2. Scanning Pentesting
  94. What are the services running on the target machine?
  95. Summary
  96. 3. Sniffing and Penetration Testing
  97. Implementing a network sniffer using Python
  98. Learning about packet crafting
  99. Introducing ARP spoofing and implementing it using Python
  100. Testing the security system using custom packet crafting and injection
  101. Summary
  102. 4. Wireless Pentesting
  103. Wireless attacks
  104. Summary
  105. 5. Foot Printing of a Web Server and a Web Application
  106. Introducing information gathering
  107. Information gathering of a website from SmartWhois by the parser BeautifulSoup
  108. Banner grabbing of a website
  109. Hardening of a web server
  110. Summary
  111. 6. Client-side and DDoS Attacks
  112. Tampering with the client-side parameter with Python
  113. Effects of parameter tampering on business
  114. Introducing DoS and DDoS
  115. Summary
  116. 7. Pentesting of SQLI and XSS
  117. Types of SQL injections
  118. Understanding the SQL injection attack by a Python script
  119. Learning about Cross-Site scripting
  120. Summary
  121. 3. Module 3
  122. 1. Gathering Open Source Intelligence
  123. Gathering information using the Shodan API
  124. Scripting a Google+ API search
  125. Downloading profile pictures using the Google+ API
  126. Harvesting additional results from the Google+ API using pagination
  127. Getting screenshots of websites with QtWebKit
  128. Screenshots based on a port list
  129. Spidering websites
  130. 2. Enumeration
  131. Performing a ping sweep with Scapy
  132. Scanning with Scapy
  133. Checking username validity
  134. Brute forcing usernames
  135. Enumerating files
  136. Brute forcing passwords
  137. Generating e-mail addresses from names
  138. Finding e-mail addresses from web pages
  139. Finding comments in source code
  140. 3. Vulnerability Identification
  141. Automated URL-based Directory Traversal
  142. Automated URL-based Cross-site scripting
  143. Automated parameter-based Cross-site scripting
  144. Automated fuzzing
  145. jQuery checking
  146. Header-based Cross-site scripting
  147. Shellshock checking
  148. 4. SQL Injection
  149. Checking jitter
  150. Identifying URL-based SQLi
  151. Exploiting Boolean SQLi
  152. Exploiting Blind SQL Injection
  153. Encoding payloads
  154. 5. Web Header Manipulation
  155. Testing HTTP methods
  156. Fingerprinting servers through HTTP headers
  157. Testing for insecure headers
  158. Brute forcing login through the Authorization header
  159. Testing for clickjacking vulnerabilities
  160. Identifying alternative sites by spoofing user agents
  161. Testing for insecure cookie flags
  162. Session fixation through a cookie injection
  163. 6. Image Analysis and Manipulation
  164. Hiding a message using LSB steganography
  165. Extracting messages hidden in LSB
  166. Hiding text in images
  167. Extracting text from images
  168. Enabling command and control using steganography
  169. 7. Encryption and Encoding
  170. Generating an MD5 hash
  171. Generating an SHA 1/128/256 hash
  172. Implementing SHA and MD5 hashes together
  173. Implementing SHA in a real-world scenario
  174. Generating a Bcrypt hash
  175. Cracking an MD5 hash
  176. Encoding with Base64
  177. Encoding with ROT13
  178. Cracking a substitution cipher
  179. Cracking the Atbash cipher
  180. Attacking one-time pad reuse
  181. Predicting a linear congruential generator
  182. Identifying hashes
  183. 8. Payloads and Shells
  184. Extracting data through HTTP requests
  185. Creating an HTTP C2
  186. Creating an FTP C2
  187. Creating an Twitter C2
  188. Creating a simple Netcat shell
  189. 9. Reporting
  190. Converting Nmap XML to CSV
  191. Extracting links from a URL to Maltego
  192. Extracting e-mails to Maltego
  193. Parsing Sslscan into CSV
  194. Generating graphs using plot.ly
  195. A. Bibliography
  196. Index

What are the services running on the target machine?

Now you are familiar with how to scan the IP address and identify a live host within a subnet. In this section, we will discuss the services that are running on a host. These services are the ones that are using a network connection. The service using a network connection must open a port; from a port number, we can identify which service is running on the target machine. In pentesting, the significance of port scanning is to check whether any illegitimate service is running on the host machine.

Consider a situation where users normally use their computer to download a game, and a Trojan is identified during the installation of the game. The Trojan goes into hidden mode and opens a port and sends all the keystrokes log information to the hacker. In this situation, port scanning helps to identify the unknown services that are running on the victim's computer.

Port numbers range from 0 to 65536.The well-known ports (also known as system ports) are those that range from 0 to 1023, and are reserved for privileged services. Port ranges from 1024 to 49151 are registered port-like vendors used for applications; for example, the port 3306 is reserved for MySQL.

The concept of a port scanner

TCP's three-way handshake serves as logic for the port scanner; in the TCP/IP scanner, you have seen that the port (137 or 135) is one in which IP addresses are in a range. However, in the port scanner, IP is only one port in a range. Take one IP and try to connect each port as a range given by the user; if the connection is successful, the port opens; otherwise, the port remains closed.

I have written a very simple code for port scanning:

import socket, subprocess,sys
from datetime import datetime

subprocess.call('clear',shell=True)
rmip = raw_input("\t Enter the remote host IP to scan:")
r1 = int(raw_input("\t Enter the start port number\t"))
r2 = int (raw_input("\t Enter the last port number\t"))
print "*"*40
print "\n Mohit's Scanner is working on ",rmip
print "*"*40

t1= datetime.now()
try:
  for port in range(r1,r2):
    sock= socket.socket(socket.AF_INET,socket.SOCK_STREAM)
    socket.setdefaulttimeout(1)

    result = sock.connect_ex((rmip,port))
    if result==0:
      print "Port Open:-->\t", port
      # print desc[port]
    sock.close()

except KeyboardInterrupt:
  print "You stop this "
  sys.exit()

except socket.gaierror:
  print "Hostname could not be resolved"
  sys.exit()

except socket.error:
  print "could not connect to server"
  sys.exit()

t2= datetime.now()

total =t2-t1
print "scanning complete in " , total

The main logic has been written in the try block, which denotes the engine of the car. You are familiar with the syntax. Let's do an R&D on the output.

The output of the portsc.py program is as follows:

root@Mohit|Raj:/port#python portsc.py 
	 Enter the remote host IP to scan:192.168.0.3
	 Enter the start port number	1
	 Enter the last port number	4000
****************************************

 Mohit's Scanner is working on  192.168.0.3
****************************************
Port Open:-->	22
Port Open:-->	80
Port Open:-->	111
Port Open:-->	443
Port Open:-->	924
Port Open:-->	3306
scanning complete in  0:00:00.766535

The preceding output shows that the port scanner scanned the 1000 ports in 0.7 seconds; the connectivity was full because the target machine and the scanner machine were in the same subnet.

Let's discuss another output:

  Enter the remote host IP to scan:10.0.0.1
  Enter the start port number	1
  Enter the last port number	4000
****************************************

Mohit's Scanner is working on  10.0.0.1
****************************************
Port Open:-->  23
Port Open:-->  53
Port Open:-->  80
Port Open:-->  1780
scanning complete in  1:06:43.272751

Now, let's analyze the output; to scan 4,000 ports, the scanner took 1:06:43.272751 hours, scanning took lot of time. The topology is:

192.168.0.10 --> 192.168.0.1 --> 10.0.0.16 ---> 10.0.0.1

The 192.168.0.1 and 10.0.0.16 IPs are gateway interfaces. We put 1 second in socket.setdefaulttimeout(1), which means the scanner machine will spend a maximum of 1 second on each port. The total of 4000 ports means that if all ports are closed, then the total time taken will be 4000 seconds; if we convert it into hours, it will become 1.07 hours, which is nearly equal to the output of our program. If we set socket.setdefaulttimeout(.5), the time taken will be reduced to 30 minutes, but nevertheless, it would be still be a long. Nobody will use our scanner. The time taken should be less than 100 seconds for 4000 ports.

How to create an efficient port scanner

I have stated some points that should be taken into account for a good port scanner:

  • Multithreading should be used for high performance
  • The socket.setdefaulttimeout(1) method should be set according to the situation
  • The port scanner should have the capability to take host names as well as domain names
  • The port should provide the service name with the port number
  • The total time should be taken into account for port scanning
  • To scan ports 0 to 65536, the time taken should be around 3 minutes

So now, I have written my port scanner, which I usually use for port scanning:

import threading
import time
import socket, subprocess,sys
from datetime import datetime
import thread
import shelve

'''section 1 '''
subprocess.call('clear',shell=True)
shelf = shelve.open("mohit.raj")
data=(shelf['desc'])

'''section 2 '''
class myThread (threading.Thread):
  def __init__(self, threadName,rmip,r1,r2,c):
    threading.Thread.__init__(self)
    self.threadName = threadName
    self.rmip = rmip
    self.r1 = r1
    self.r2 = r2
    self.c =c
  def run(self):
    scantcp(self.threadName,self.rmip,self.r1,self.r2,self.c)

'''section 3 '''
def scantcp(threadName,rmip,r1,r2,c):
  try:
    for port in range(r1,r2):
      sock= socket.socket(socket.AF_INET,socket.SOCK_STREAM)
      #sock= socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
      socket.setdefaulttimeout(c)
      result = sock.connect_ex((rmip,port))

      if result==0:
        print "Port Open:---->\t", port,"--", data.get(port, "Not in Database")
        sock.close()

    except KeyboardInterrupt:
      print "You stop this "
      sys.exit()

    except socket.gaierror:
      print "Hostname could not be resolved"
      sys.exit()

    except socket.error:
      print "could not connect to server"
      sys.exit()

    shelf.close()
'''section 4 '''
print "*"*60
print " \tWelcome this is the Port scanner of Mohit\n  "

d=raw_input("\ t Press D for Domain Name or Press I for IP Address\t")   

if (d=='D' or d=='d'):
  rmserver = raw_input("\t Enter the Domain Name to scan:\t")
  rmip = socket.gethostbyname(rmserver)
elif(d=='I' or d=='i'):
  rmip = raw_input("\t Enter the IP Address  to scan:  ")

else:
  print "Wrong input"
#rmip = socket.gethostbyname(rmserver)
r11 = int(raw_input("\t Enter the start port number\t"))
r21 = int (raw_input("\t Enter the last port number\t"))

conect=raw_input("For low connectivity press L and High connectivity Press H\t")

if (conect=='L' or conect=='l'):
  c =1.5

elif(conect =='H' or conect=='h'):
  c=0.5

else:
  print "\t wrong Input"

print "\n Mohit's Scanner is working on ",rmip
print "*"*60
t1= datetime.now()
tp=r21-r11

tn =30
# tn number of port handled by one thread
tnum=tp/tn       # tnum number of threads
if (tp%tn != 0):
  tnum= tnum+1

if (tnum > 300):
  tn = tp/300
  tn= tn+1
  tnum=tp/tn
  if (tp%tn != 0):
    tnum= tnum+1

'''section  5'''
threads= []

try:
  for i in range(tnum):
    #print "i is ",i
    k=i
    r2=r11+tn 
    # thread=str(i)
    thread = myThread("T1",rmip,r11,r2,c)
    thread.start()
  threads.append(thread)
  r11=r2

except:
  print "Error: unable to start thread"

print "\t Number of Threads active:", threading.activeCount()

for t in threads:
  t.join()
print "Exiting Main Thread"
t2= datetime.now()

total =t2-t1
print "scanning complete in " , total

Don't be afraid to see the full code; it took me 2 weeks. I will explain to you the full code section-wise. In section1, the subprocess.call('clear',shell=True) statement works in Linux to clear the screen. The next two lines are related to the database file that stores the port information, which will be explained while creating the database file. In section 2, the myThread class extends the threading class, or you could say, inherits the threading class. In the next line, the def __init__(self, threadName,rmip,r1,r2,c): statement takes 5 values; the first one is threadName, which stores the thread name; actually, I have taken it for debugging purposes. If any thread fails to work, we can print the thread name. The rmip argument is a remote IP address; r1 and r2 are the first and last port numbers, and c is the connection mode; section 4 provides all values to section 1. From the run() function, the scantcp() function is called. Section 3 is the engine of the car, which was explained in the Concept of port scanner section. The data.get(port, "Not in Database") statement is new here; it means that if the port key is found in the dictionary database, then it will display the value; otherwise, it will print Not in Database. Section 4 interacts with users. You can give the hostname as well as the IP address, or you can give the domain name too; the if…else statements do this task. The r11 and r21 variables store the first and last port numbers. The next if…else statements define the value of c if you think connectivity with the target machine is poor, but with no loss of packet, then you can press H; if connectivity is just good, then you can press L. The tn=30 variable defines the number of ports handled by a single thread. The tnum variable calculates the total number of threads needed to accomplish the task.

I have written the following code after performing lots of experiments:

if (tnum > 300):
  tn = tp/300
  tn= tn+1
  tnum=tp/tn
  if (tp%tn != 0):
    tnum= tnum+1

When the total number of threads exceeds 300, the threads fail to work. It means the number of threads must be less or equal to 300. The preceding code defines the new values of tn and tnum. In Section 5, nothing is new as you have seen everything before in IP scanners.

Now it's time to see the output of the portsc14.py program:

root@Mohit|Raj:/port# python portsc14.py 

************************************************************
  Welcome this is the Port scanner of Mohit
  
  Press D for Domain Name or Press I for IP Address	i
  Enter the IP Address  to scan:  10.0.0.1
  Enter the start port number	1
  Enter the last port number	4000
For low connectivity press L and High connectivity Press H	l

Mohit's Scanner is working on 10.0.0.1
************************************************************
  Number of Threads active: 135
Port Open:---->	1780 -- Not in Database
Port Open:---->	80 -- HTTP
Port Open:---->	23 -- Telnet
Port Open:---->	53 -- DNS
Exiting Main Thread
scanning complete in  0:00:33.249338

Our efficient port scanner has given the same output as the previous simple scanner, but from the performance point of view, there is a huge difference. The time taken by a simple scanner was 1:06:43.272751, but the new multithreaded scanner took just 33 seconds. It also shows the service name. Let's check another output with ports 1 to 50000:

root@Mohit|Raj:/port# python portsc14.py 

************************************************************
  Welcome this is the Port scanner of Mohit
  
  Press D for Domain Name or Press I for IP Address	i 
  Enter the IP Address  to scan:  10.0.0.1
  Enter the start port number	1
  Enter the last port number	50000
For low connectivity press L and High connectivity Press H	l

Mohit's Scanner is working on  10.0.0.1
************************************************************
  Number of Threads active: 301
Port Open:---->	23 -- Telnet
Port Open:---->	53 -- DNS
Port Open:---->	80 -- HTTP
Port Open:---->	1780 -- Not in Database
Port Open:---->	5000 -- Not in Database
Exiting Main Thread
scanning complete in  0:02:54.283984

The time taken is 2 minutes 54 seconds; I did the same experiment in high connectivity, where the time taken was 0:01:23.819774, which is almost half of the previous one.

Note

In a multithreading experiment, if we produce tn number of threads, then threading.activeCount() always shows tn+1 number of threads, because it counts the main threads too. The main thread is the thread that runs all the threads. As an exercise, use the threading.activeCount() method in the simple scanner program, and then check the output.

Now, I'm going to teach you how to create a database file that contains the description of all the port numbers; here is the code:

import shelve
def create():
  shelf = shelve.open("mohit.raj", writeback=True)
  shelf['desc'] ={}
  shelf.close()
  print "Dictionary is created"


def update():
  shelf = shelve.open("mohit.raj", writeback=True)
  data=(shelf['desc'])
  port =int(raw_input("Enter the Port: "))
  data[port]= raw_input("\n Enter the  description\t")
  shelf.close()

def del1():
  shelf = shelve.open("mohit.raj", writeback=True)
  data=(shelf['desc'])
  port =int(raw_input("Enter the Port: "))
  del data[port]
  shelf.close()
  print "\n Entry is deleted"

def list1():
  print "*"*30
  shelf = shelve.open("mohit.raj", writeback=True)
  data=(shelf['desc'])
  for key, value in data.items():
    print key, ":", value
    print "*"*30
    print "\t Program to update or Add and Delete the port number detail\n"
  while(True):
    print "Press" 
    print "C for create only one time create"
    print "U for Update or Add \nD for delete"
    print "L for list the all values  "
    print "E for Exit  "
    c=raw_input("Enter :  ")

  if (c=='C' or c=='c'):
    create()

  elif (c=='U' or c=='u'):
    update()

  elif(c=='D' or c=='d'):
    del1()

  elif(c=='L' or c=='l'):
    list1()

  elif(c=='E' or c=='e'):
    exit()

  else:
    print "\t Wrong Input"

In the preceding program, we stored only one dictionary that contains the key as the port number and the values as the description of the port number. The dictionary name is desc. So I made desc a key of the shelf to store in a file named mohit.raj.

def create():
  shelf = shelve.open("mohit.raj", writeback=True)
  shelf['desc'] ={}
  shelf.close()

This create() function is just an empty dictionary. The desc dictionary is a dictionary in the program, whereas shelf['desc'] is a dictionary in the file. Use this function only once to create a file.

def update():
  shelf = shelve.open("mohit.raj", writeback=True)
  data=(shelf['desc'])
  port =int(raw_input("Enter the Port: "))
  data[port]= raw_input("\n Enter the  description\t")
  shelf.close()

This update() function updates the dictionary. In the writeback=True statement, the writeback flag shelf remembers all the received values from the files, and each value, which is currently in the cache, is written back to the file. The data=(shelf['desc']) dictionary is the shelf dictionary, which has been assigned to the variable data. The del() function deletes any port number from the dictionary. The list1() function shows the full dictionary. To accomplish this, the for loop is used.

The output of the updatec.py program is as follows:

G:\Project Snake\Chapter 2>python updatec.py
         Program to update or Add and Delete the port number detail

Press
C for create only one time create
U for Update or Add
D for delete
L for list the all values
E for Exit
Enter :  c
Dictionary is created
Press
C for create only one time create
U for Update or Add
D for delete
L for list the all values
E for Exit
Enter :  u
Enter the Port: 80

 Enter the  description HTTP
Press
C for create only one time create
U for Update or Add
D for delete
L for list the all values
E for Exit
Enter :  l
******************************
80 : HTTP
******************************
Press
C for create only one time create
U for Update or Add
D for delete
L for list the all values
E for Exit
Enter :  e

G:\Project Snake\Chapter 2>

I hope you've got a fair idea of the port scanner; in a nutshell, the port scanner comprises three files, the first file is the scanner (portsc14.py), the second file is the database (mohit.raj), and the third one is updatec.py. You just need to upgrade the mohit.raj file to insert a description of the maximum number of ports.