Differences between revisions 17 and 18
Revision 17 as of 2018-07-17 14:45:55
Size: 9263
Editor: mdeslaur
Comment:
Revision 18 as of 2018-07-27 03:46:04
Size: 9197
Editor: sbeattie
Comment: start updating the CVE tracker documentation
Deletions are marked like this. Additions are marked like this.
Line 2: Line 2:
Much of this can be found in [[http://bazaar.launchpad.net/~ubuntu-security/ubuntu-cve-tracker/master/annotate/head%3A/README|ubuntu-cve/README]]. Much of this can be found in [[https://git.launchpad.net/ubuntu-cve-tracker/tree/README|ubuntu-cve/README]].
Line 9: Line 9:
$ mkdir ~/bzr-pulls
$ cd ~/bzr-pulls
$ mkdir ~/git-pulls
$ cd ~/git-pulls
Line 12: Line 12:
$ bzr checkout lp:usn-tool
Line 16: Line 15:
export UCT="$HOME/bzr-pulls/ubuntu-cve-tracker" export UCT="$HOME/git-pulls/ubuntu-cve-tracker"

Setup

Much of this can be found in ubuntu-cve/README.

  1. Install the necessary software:

    $ sudo apt-get install python-configobj python-yaml python-genshi python-progressbar git rsync libfile-rsyncp-perl w3m
  2. Download the ubuntu-cve-tracker branch:

    $ mkdir ~/git-pulls
    $ cd ~/git-pulls
    $ git clone git+ssh://<user>@git.launchpad.net/ubuntu-cve-tracker (replace <user> with your launchpad user).
  3. Add the UCT environment variable to your startup scripts (eg ~/.bashrc) and have it point to the ubuntu-cve-tracker branch:

    export UCT="$HOME/git-pulls/ubuntu-cve-tracker"
  4. Create $HOME/.ubuntu-cve-tracker.conf to have something like:

    # python-launchpad-bugs authentication cookies file (sis-changes)
    plb_authentication="/home/<username>/.mozilla/firefox/work.profile/cookies.sqlite"
     
    # path to Debian "security-tracker" GIT tree (check-cves)
    # get from https://salsa.debian.org/security-tracker-team/security-tracker
    secure_testing_path='/home/<username>/bzr-pulls/security-tracker'
     
    # path to archive-layout mirror of supported architectures
    # sis-generate-usn, packages-mirror)
    packages_mirror='<path>/ubuntu-archive-metadata'
    
    # same as packages_mirror, but for Debian testing repository
    debian_mirror='<path>/debian-testing-archive-metadata'
    
    # same as packages_mirror, but for the partner repository
    partner_mirror='<path>/partner-archive-metadata'
    
    # path to usn-tool bzr tree, used to manipulate USN databases
    # and for templates (sis-generate-usn)
    usn_tool='/home/<username>/bzr-pulls/usn-tool-trunk'
    
    # path to copy of master USN database, used when creating
    # a template for a USN (sis-generate-usn)
    usn_db_copy='/home/<username>/bzr-pulls/ubuntu-cve/database.pickle'
    
    # path to individual USN pickle database output when generating a single USN
    # database before merging into the master USN database (sis-generate-usn)
    usn_storage='/tmp/pending-usn'
    
    # path to git tree for the USN website which will be updated with new markdown
    # as USNs are published
    usn_website='/home/<username>/bzr-pulls/usn.ubuntu.com'
    
    # where to get the MITRE CVE and NVD databases (check-cves)
    mitre_loc="lillypilly.canonical.com:/home/ubuntu-security/cache"
    nvd_loc="lillypilly.canonical.com:/home/ubuntu-security/cache"
  5. Populate the required data into the packages_mirror and partner_mirror directories:

    cd $UCT;./scripts/packages_mirror
  6. Populate the required data into the secure_testing directory:

    $ cd ~/bzr_pulls
    $ git clone https://salsa.debian.org/security-tracker-team/security-tracker.git
  7. Pull down an initial copy of the database.pickle database:

    cd $UCT; ./scripts/fetch-db database.pickle.bz2

Usage

check-cves

check-cves is the tool to update the CVE database with new entries.

Prequisites

check-cves needs the archive layout mirrors to contain up-to-date information; the ${UCT}/scripts/packages-mirror script will do this for you.

Usage

To use manually, run like this:

$ cd $UCT ; bzr up
$ ./scripts/check-cves
$ ./scripts/check-cves http://cve.mitre.org/data/downloads/allitems.xml
$ ./scripts/check-cves <file>

Eg:

$ wget -N http://cve.mitre.org/data/downloads/allitems.xml
$ bzr up
$ ./scripts/check-cves ./allitems.xml
$ for i in $(seq 2004 $(date +%Y)) recent; do wget --quiet -N http://nvd.nist.gov/download/nvdcve-$i.xml; done
$ ./scripts/check-cves ./nvdcve-2*.xml

The Ubuntu Security team does certain triage actions depending on the day of the week, and the process_cves script will guide you through these and call check-cves with appropriate arguments. Eg:

$ bzr up
$ ./scripts/process_cves
$ ./scripts/process_cves Mon
$ ./scripts/process_cves Tue

Please see the TRIAGING section in ubuntu-cve/README for more information.

Embargoed Items

Embargoed items are supported in the following scripts in all the scripts except sync-from-usns.py. If no CVE has been assigned yet, an embargoed item should be prefixed with 'EMB-', followed by any combination of alphanumerics and dashes. Eg: EMB-xorg-2007-0001 EMB-foo

To include embargoed items simply create a symlink from 'embargoed' to the directory holding embargoed items. ubuntu-cve-tracker will not use 'embargoed' unless it is a symlink.

Pre-commit Syntax Checking

To perform pre-commit syntax checking, this little hack should allow for it:

$ mkdir -p ~/.bazaar/plugins/hooks
$ cat > ~/.bazaar/plugins/hooks/__init__.py <<EOM
#!/usr/bin/python
from bzrlib.branch import Branch

def run_tests(local, master, old_revno, old_revid, new_revno, new_revid, seven, eight):
    #print local, master, old_revno, old_revid, new_revno, new_revid, seven, eight
    import os
    if 'ubuntu-cve-tracker' in master.base and not os.environ.has_key('UCT_IGNORE_CHECK_SYNTAX'):
        import subprocess
        print ''
        rc = subprocess.call(['./scripts/check-syntax','--verbose'])
        if rc != 0:
            import sys
            sys.exit(1)

Branch.hooks.install_named_hook('pre_commit', run_tests, 'CVE Tracker tests')
EOM

With the above, check-syntax is always run before a commmit, or to avoid check-syntax, you can run: UCT_IGNORE_CHECK_SYNTAX=1 bzr ci

Other commands

Below are various tools to help maintain and manipulate UCT. All commands assume you are in the top-level directory of UCT (ie cd $UCT).

  • Verify syntax of CVE-* files:

    $ ./scripts/check-syntax
  • To commit, use this command:

    $ ./scripts/check-syntax && bzr ci
    NOTE: bzr branch creates a disconnected branch. To commit to a disconnected branch, use bzr ci followed by bzr push. bzr co creates a connected branch (bzr branch + bzr bind). To commit to a connected branch, use only bzr ci. To convert a disconnected branch to a connected branch, use bzr bind.
  • Listings of active CVEs:

    $ ./scripts/ubuntu-table
    $ ./scripts/ubuntu-table --supported 2>/dev/null | grep SUPPORTED   # main only
    $ ./scripts/ubuntu-table --supported 2>/dev/null | grep PARTNER     # partner only
    $ ./scripts/ubuntu-table --untriaged                                # only untriaged
    $ ./scripts/ubuntu-table --supported | egrep -v '[[:space:]]+(untriaged|negligible|low)$' # medium and higher
  • Package status:

    $ ./scripts/pkg_status pkgname1 pkgname2
    $ ./scripts/pkg_status -f pkgname1 pkgname 2 ...                    # full listing
    $ ./scripts/ubuntu-table -p linux -P linux -S | cut -c1-14,36-95    # kernel report
  • CVE status:

    $ ./scripts/cve_status CVE-2006-4519 CVE-2007-2949 CVE-2007-3741
    $ ./scripts/cve_status -s CVE-2006-4519 CVE-2007-2949 CVE-2007-3741 # short listing
  • Create/edit a CVE without check-cves:

    $ ./scripts/active_edit -p package -c CVE-YYYY-XXXX
    $ ./scripts/active_edit -e -p package -c CVE-YYYY-XXXX              # embargoed (private)
    $ ./scripts/active_edit -p package -c CVE-2010-NNN1                 # new CVE without a CVE identifier
  • Mass editing of CVEs:

    $ ./scripts/mass-cve-edit -p <package> -r <release> -v <version> -s <state> CVE...

    Eg:

    $ ./scripts/mass-cve-edit -p seamonkey -r karmic -s released -v 1.1.17+nobinonly-0ubuntu1 CVE-2009-1841 CVE-2009-1838 CVE-2009-1836 CVE-2009-1835 CVE-2009-1392 CVE-2009-1832 CVE-2009-1833 CVE-2009-1311 CVE-2009-1307
  • Listing of packages with number of CVEs attached to them, weighted by CVE priority:

    $ ./scripts/cve_packages
    $ ./scripts/cve_packages -m                                         # main, restricted and partner
    $ ./scripts/cve_packages -u                                         # universe and multiverse
    $ ./scripts/cve_packages -t                                         # just totals
    $ ./scripts/cve_packages -m | grep '^[0-9]' | sort -n               # sorted by weight
    $ ./scripts/cve_packages -m -S | grep '^[0-9]' | sort -n            # sorted by weight without devel
  • CVEs needing retiring:

    $ ./scripts/cve_need_retire
    $ bzr mv $(./scripts/cve_need_retire -p) ./retired/                 # move needed to retired
  • Marking CVEs as 'released':

    $ cd $UCT
    $ rsync -v --progress -e ssh people.canonical.com:~ubuntu-security/public_html/usn/database.pickle ./database.pickle
    $ ./scripts/sync-from-usns.py database.pickle -u

Please see the UBUNTU-CVE Commands section in ubuntu-cve/README for more information.

Reports

  • By date, sorted by priority:

    $ ./scripts/report-date.py -S -m
  • TODO list:

    $ ./scripts/report-todo
    $ ./scripts/report-todo [-S]
  • SUmmary list:

    $ ./scripts/report-todo-numbers                                     # summary report
    $ ./scripts/report-todo-numbers [-S]
  • Monthly report:

    $ ./scripts/monthly-report
  • HTML reports:

    $ ./scripts/html-report [-S]

SecurityTeam/UbuntuCVETracker (last edited 2018-07-27 03:46:04 by sbeattie)