• Pyramid installation – “UnicodeDecodeError: ‘ascii’ codec can’t decode byte” error fix

    Pyramid installation – “UnicodeDecodeError: ‘ascii’ codec can’t decode byte” error fix

    Following pyramid framework installation notes I kept getting ‘ascii’ codec can’t decode byte error message. The whole error output looked like this:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    $ bin/easy_install pyramid
    Searching for pyramid
    Reading http://pypi.python.org/simple/pyramid/
    Reading http://docs.pylonshq.com
    Reading http://pylonsproject.org
    Reading http://docs.pylonsproject.org
    Best match: pyramid 1.4a4
    Downloading http://pypi.python.org/packages/source/p/pyramid/pyramid-1.4a4.tar.gz#md5=0a5da1a25791764cdfefb15b2d55656a
    Processing pyramid-1.4a4.tar.gz
    Writing /tmp/easy_install-hrcuuc/pyramid-1.4a4/setup.cfg
    Running pyramid-1.4a4/setup.py -q bdist_egg --dist-dir /tmp/easy_install-hrcuuc/pyramid-1.4a4/egg-dist-tmp-kbuxa9
    Traceback (most recent call last):
    File "bin/easy_install", line 9, in
    load_entry_point('distribute==0.6.28', 'console_scripts', 'easy_install')()
    File "/home/pootzko/env/lib/python3.3/site-packages/distribute-0.6.28-py3.3.egg/setuptools/command/easy_install.py", line 1929, in main
    with_ei_usage(lambda:
    File "/home/pootzko/env/lib/python3.3/site-packages/distribute-0.6.28-py3.3.egg/setuptools/command/easy_install.py", line 1910, in with_ei_usage
    return f()
    File "/home/pootzko/env/lib/python3.3/site-packages/distribute-0.6.28-py3.3.egg/setuptools/command/easy_install.py", line 1933, in
    distclass=DistributionWithoutHelpCommands, **kw
    File "/usr/lib/python3.3/distutils/core.py", line 148, in setup
    dist.run_commands()
    File "/usr/lib/python3.3/distutils/dist.py", line 917, in run_commands
    self.run_command(cmd)
    File "/usr/lib/python3.3/distutils/dist.py", line 936, in run_command
    cmd_obj.run()
    File "/home/pootzko/env/lib/python3.3/site-packages/distribute-0.6.28-py3.3.egg/setuptools/command/easy_install.py", line 358, in run
    self.easy_install(spec, not self.no_deps)
    File "/home/pootzko/env/lib/python3.3/site-packages/distribute-0.6.28-py3.3.egg/setuptools/command/easy_install.py", line 598, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
    File "/home/pootzko/env/lib/python3.3/site-packages/distribute-0.6.28-py3.3.egg/setuptools/command/easy_install.py", line 628, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
    File "/home/pootzko/env/lib/python3.3/site-packages/distribute-0.6.28-py3.3.egg/setuptools/command/easy_install.py", line 823, in install_eggs
    return self.build_and_install(setup_script, setup_base)
    File "/home/pootzko/env/lib/python3.3/site-packages/distribute-0.6.28-py3.3.egg/setuptools/command/easy_install.py", line 1103, in build_and_install
    self.run_setup(setup_script, setup_base, args)
    File "/home/pootzko/env/lib/python3.3/site-packages/distribute-0.6.28-py3.3.egg/setuptools/command/easy_install.py", line 1089, in run_setup
    run_setup(setup_script, args)
    File "/home/pootzko/env/lib/python3.3/site-packages/distribute-0.6.28-py3.3.egg/setuptools/sandbox.py", line 31, in run_setup
    lambda: exec(compile(open(
    File "/home/pootzko/env/lib/python3.3/site-packages/distribute-0.6.28-py3.3.egg/setuptools/sandbox.py", line 79, in run
    return func()
    File "/home/pootzko/env/lib/python3.3/site-packages/distribute-0.6.28-py3.3.egg/setuptools/sandbox.py", line 34, in
    {'__file__':setup_script, '__name__':'__main__'})
    File "setup.py", line 122, in
    File "/usr/lib/python3.3/distutils/core.py", line 148, in setup
    dist.run_commands()
    File "/usr/lib/python3.3/distutils/dist.py", line 917, in run_commands
    self.run_command(cmd)
    File "/usr/lib/python3.3/distutils/dist.py", line 936, in run_command
    cmd_obj.run()
    File "/home/pootzko/env/lib/python3.3/site-packages/distribute-0.6.28-py3.3.egg/setuptools/command/bdist_egg.py", line 172, in run
    self.run_command("egg_info")
    File "/usr/lib/python3.3/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
    File "/usr/lib/python3.3/distutils/dist.py", line 936, in run_command
    cmd_obj.run()
    File "/home/pootzko/env/lib/python3.3/site-packages/distribute-0.6.28-py3.3.egg/setuptools/command/egg_info.py", line 179, in run
    self.find_sources()
    File "/home/pootzko/env/lib/python3.3/site-packages/distribute-0.6.28-py3.3.egg/setuptools/command/egg_info.py", line 254, in find_sources
    mm.run()
    File "/home/pootzko/env/lib/python3.3/site-packages/distribute-0.6.28-py3.3.egg/setuptools/command/egg_info.py", line 308, in run
    self.add_defaults()
    File "/home/pootzko/env/lib/python3.3/site-packages/distribute-0.6.28-py3.3.egg/setuptools/command/egg_info.py", line 339, in add_defaults
    self.read_manifest()
    File "/usr/lib/python3.3/distutils/command/sdist.py", line 377, in read_manifest
    for line in manifest:
    File "/home/pootzko/env/lib/python3.3/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 7896: ordinal not in range(128)

    …aaaaand it kept driving me nuts for a couple of hours until it finally came to me that the problem lied my locale setup and not in python setuptools. Simply typing:

    1
    export LANG='en_US.UTF-8'

    in terminal to change the encoding fixed the problem (keep in mind that this changed the locale temporarily only for the current session).

    Hope it saved you some time. Cheers!

  • My first open-source code contribution – spilp

    My first open-source code contribution – spilp

    Spilp is a simple Python script that takes IIS logs, parses them and creates statistical reports which can be used to discover unusual IP activity more easily.”

    ‘Been working on this one for some time now and finally everything I wanted to implement is in place. The whole thing is released under GNU GPL v3 license for everybody to enjoy. You can download the script here. Below is a list of spilp features with links to sample reports.

    Features

    • extracts a list of IP addresses with number of hits they made sorted by number of hits
    • extracts a list of “close” IP addresses that made a certain number of hits
    • extracts a list of user agents sorted by number of hits
    • extracts a list of cs-method hits (GET method excluded)
    • extracts a list of file hits sorted by number of hits
    • extracts extended information for document and web file hits
      • includes timestamps, client IP addresses, methods, ports, user agent details and http status codes
    • extracts a list of “unusual” http status code hits sorted by number of hits
      • client IP address list
      • a list of files hit by an IP and number of hits for that file
    • filtering results (include or exclude filtering – works in “either-or” way)
      • ability to auto-generate an IP range list as a filter
    • reverse DNS country lookup using MaxMinds? GeoIP country downloadable database
      • additional info in certain reports
      • filtering results by country of origin (as a separate filtering option using spilpconf.py file)
    • ability to process large amount of IIS log files
    • CONFIG file for performance and output tweaking
    Rejoice!
  • How to generate an IP range list in Python?

    How to generate an IP range list in Python?

    Here’s a short Python snippet that generates a list of IP addresses based on IP range. Nothing fancy, but could come in handy if you ever need it somewhere in your code. Enjoy!

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    def ipRange(start_ip, end_ip):
       start = list(map(int, start_ip.split(".")))
       end = list(map(int, end_ip.split(".")))
       temp = start
       ip_range = []
       
       ip_range.append(start_ip)
       while temp != end:
          start[3] += 1
          for i in (3, 2, 1):
             if temp[i] == 256:
                temp[i] = 0
                temp[i-1] += 1
          ip_range.append(".".join(map(str, temp)))    
         
       return ip_range
       
       
    # sample usage
    ip_range = ipRange("192.168.1.0", "192.171.3.25")
    for ip in ip_range:
       print(ip)
  • Merging multiple Nessus scans (Python script)

    nessus

    If you for any reason have the need to merge / combine a few Nessus scans into a single *.nessus file, you can do so using this simple Python script. Since *.nessus files are basically just XML files with a different extension, what this script does is it finds all the *.nessus files in the current folder, finds all the “ReportHost” XML nodes accumulating them into a single report.nessus file which is then exported to nss_report folder.

    Note that scans must be of the same type (same plugins must be used), but they can be from different subnets or different parts of the same subnet.

    How to use it? – Put the script and all your *.nessus files into a same folder, run the script, import nss_report/report.nessus into Nessus – and there you have it, all the hosts are in one place..

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    #! /usr/bin/env python3.2

    import xml.etree.ElementTree as etree
    import shutil
    import os

    first = 1
    for fileName in os.listdir("."):
       if ".nessus" in fileName:
          print(":: Parsing", fileName)
          if first:
             mainTree = etree.parse(fileName)
             report = mainTree.find('Report')
             first = 0
          else:
             tree = etree.parse(fileName)
             for element in tree.findall('.//ReportHost'):
                report.append(element)
          print(":: => done.")

    if "nss_report" in os.listdir("."):
       shutil.rmtree("nss_report")

    os.mkdir("nss_report")
    mainTree.write("nss_report/report.nessus", encoding="utf-8", xml_declaration=True)

    If you have any questions, just drop a comment bellow..

    edit: mastahyeti made some improvements to this script, you can get it at his github

  • Removing multiple elements from a list in Python

    Removing multiple elements from a list in Python

    Let’s say you need to filter a Python list and remove all elements that match a given criteria. If you wanted to remove a single element, you could just use ‘del list[i]’ (example). But if you wanted to remove multiple elements this might be a problem since you would be modifying and iterating over the list at the same time (keeping track of list indexes can become very confusing, very fast).

    A simple solution to this problem would be to keep record of all list elements (indexes) that need to be removed, and to remove tham afterwards. Also, one other thing to be wary about is that the removal process should be done in reverse because otherwise you would shift all the elements to the ‘left’ every time you removed an element from a list and you would (again) have to keep track of list indexes.

    A simple function example:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    matchingFilter = ['criteria1', 'criteria2']

    def criteriaFiltering(aList, matchingFilter):
       deletionIndexes = []
       i = 0

       for listLine in aList:
          for match in matchingFilter:
             if match in str(listLine):
                continue             
             else:
                deletionIndexes.append(i)
                break
          i += 1

       for number in reversed(deletionIndexes):
          del listLines[number]

       return aList

    This might not be the fastest ‘algorithm’ to do the job, but it works well enough for me..

  • How to remove ‘lastsaid’ history and chat logs from Pidgin

    Turning on History plugin in Pidgin will insert the last conversation into your current conversation. All this lastsaid entries can be found somewhere inside of blist.xml file which is located in your Pidgin conf directory (/home/username/.purple directory in GNU/Linux (not sure about where it is in MS Windows though)).

    This is convinient if you want to view what was written before you accidentialy closed the chat window or if you forgot what were you last talking about with someone and if you want that information quickly. But, it is not convinient at all if you want to wipe it along with chat logs since lastsaid entries are at a different location and are not the only entries that can be found in blist.xml so you can’t just remove that file either. If you use a couple of protocols and have more than just a few buddies, removing lastsaid lines from blist.xml manually would probably prove to be pain in the ass.

    So, what every curious programmer does in case he doesn’t have a feature he wants/needs? – He just makes it. :D Since I started playing with Python, I decided to create a small py3k script I could use every once in a while to completely wipe both lastsaid history and chat logs from Pidgin.

    What you need to do to make this script work is – you need to put it into your Pidgin conf directory (just put it into a file called something.py) and after that just run the script (you need to have python installed on your machine to run this script).

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    #! /usr/bin/env python3
    import xml.etree.ElementTree as etree
    import shutil
    import os

    # deletes 'lastsaid' blist.xml entries
    if os.path.exists("blist.xml"):
       tree = etree.parse("blist.xml")

       buddies = tree.findall(".//buddy")      
       for buddy in buddies:
          nodes = buddy.findall("setting")    
          for node in nodes:
             if node.attrib == {"type": "string","name": "lastsaid"}:
                buddy.remove(node)

    tree.write("blist.xml")


    # deletes chat log files if they exist
    if os.path.exists("logs"):
       shutil.rmtree("logs")

    I hope this helps. Also, if you have any questions/problems with the script, feel free to drop a comment..

Back to top