UbuntuCVETracker
141
Comment:
|
9197
start updating the CVE tracker documentation
|
Deletions are marked like this. | Additions are marked like this. |
Line 1: | Line 1: |
'''PLACEHOLDER''' Download the ubuntu-cve-tracker branch:{{{ $ bzr branch lp:ubuntu-cve-tracker }}} Read ubuntu-cve-tracker/README. |
== Setup == Much of this can be found in [[https://git.launchpad.net/ubuntu-cve-tracker/tree/README|ubuntu-cve/README]]. 0. Install the necessary software:{{{ $ sudo apt-get install python-configobj python-yaml python-genshi python-progressbar git rsync libfile-rsyncp-perl w3m }}} 0. Download the ubuntu-cve-tracker branch:{{{ $ mkdir ~/git-pulls $ cd ~/git-pulls $ git clone git+ssh://<user>@git.launchpad.net/ubuntu-cve-tracker (replace <user> with your launchpad user). }}} 0. Add the UCT environment variable to your startup scripts (eg ~/.bashrc) and have it point to the ubuntu-cve-tracker branch:{{{ export UCT="$HOME/git-pulls/ubuntu-cve-tracker" }}} 0. Create $HOME/.ubuntu-cve-tracker.conf to have something like:{{{ # python-launchpad-bugs authentication cookies file (sis-changes) plb_authentication="/home/<username>/.mozilla/firefox/work.profile/cookies.sqlite" # path to Debian "security-tracker" GIT tree (check-cves) # get from https://salsa.debian.org/security-tracker-team/security-tracker secure_testing_path='/home/<username>/bzr-pulls/security-tracker' # path to archive-layout mirror of supported architectures # sis-generate-usn, packages-mirror) packages_mirror='<path>/ubuntu-archive-metadata' # same as packages_mirror, but for Debian testing repository debian_mirror='<path>/debian-testing-archive-metadata' # same as packages_mirror, but for the partner repository partner_mirror='<path>/partner-archive-metadata' # path to usn-tool bzr tree, used to manipulate USN databases # and for templates (sis-generate-usn) usn_tool='/home/<username>/bzr-pulls/usn-tool-trunk' # path to copy of master USN database, used when creating # a template for a USN (sis-generate-usn) usn_db_copy='/home/<username>/bzr-pulls/ubuntu-cve/database.pickle' # path to individual USN pickle database output when generating a single USN # database before merging into the master USN database (sis-generate-usn) usn_storage='/tmp/pending-usn' # path to git tree for the USN website which will be updated with new markdown # as USNs are published usn_website='/home/<username>/bzr-pulls/usn.ubuntu.com' # where to get the MITRE CVE and NVD databases (check-cves) mitre_loc="lillypilly.canonical.com:/home/ubuntu-security/cache" nvd_loc="lillypilly.canonical.com:/home/ubuntu-security/cache" }}} 0. Populate the required data into the packages_mirror and partner_mirror directories:{{{ cd $UCT;./scripts/packages_mirror }}} 0. Populate the required data into the secure_testing directory:{{{ $ cd ~/bzr_pulls $ git clone https://salsa.debian.org/security-tracker-team/security-tracker.git }}} 0. Pull down an initial copy of the database.pickle database:{{{ cd $UCT; ./scripts/fetch-db database.pickle.bz2 }}} == Usage == === check-cves === `check-cves` is the tool to update the CVE database with new entries. ==== Prequisites ==== `check-cves` needs the archive layout mirrors to contain up-to-date information; the `${UCT}/scripts/packages-mirror` script will do this for you. ==== Usage ==== To use manually, run like this:{{{ $ cd $UCT ; bzr up $ ./scripts/check-cves $ ./scripts/check-cves http://cve.mitre.org/data/downloads/allitems.xml $ ./scripts/check-cves <file> }}} Eg:{{{ $ wget -N http://cve.mitre.org/data/downloads/allitems.xml $ bzr up $ ./scripts/check-cves ./allitems.xml $ for i in $(seq 2004 $(date +%Y)) recent; do wget --quiet -N http://nvd.nist.gov/download/nvdcve-$i.xml; done $ ./scripts/check-cves ./nvdcve-2*.xml }}} The Ubuntu Security team does certain triage actions depending on the day of the week, and the `process_cves` script will guide you through these and call check-cves with appropriate arguments. Eg:{{{ $ bzr up $ ./scripts/process_cves $ ./scripts/process_cves Mon $ ./scripts/process_cves Tue }}} Please see the TRIAGING section in [[http://bazaar.launchpad.net/~ubuntu-security/ubuntu-cve-tracker/master/annotate/head%3A/README|ubuntu-cve/README]] for more information. ==== Embargoed Items ==== Embargoed items are supported in the following scripts in all the scripts except sync-from-usns.py. If no CVE has been assigned yet, an embargoed item should be prefixed with 'EMB-', followed by any combination of alphanumerics and dashes. Eg: EMB-xorg-2007-0001 EMB-foo To include embargoed items simply create a symlink from 'embargoed' to the directory holding embargoed items. ubuntu-cve-tracker will not use 'embargoed' unless it is a symlink. ==== Pre-commit Syntax Checking ==== To perform pre-commit syntax checking, this little hack should allow for it:{{{ $ mkdir -p ~/.bazaar/plugins/hooks $ cat > ~/.bazaar/plugins/hooks/__init__.py <<EOM #!/usr/bin/python from bzrlib.branch import Branch def run_tests(local, master, old_revno, old_revid, new_revno, new_revid, seven, eight): #print local, master, old_revno, old_revid, new_revno, new_revid, seven, eight import os if 'ubuntu-cve-tracker' in master.base and not os.environ.has_key('UCT_IGNORE_CHECK_SYNTAX'): import subprocess print '' rc = subprocess.call(['./scripts/check-syntax','--verbose']) if rc != 0: import sys sys.exit(1) Branch.hooks.install_named_hook('pre_commit', run_tests, 'CVE Tracker tests') EOM }}} With the above, check-syntax is always run before a commmit, or to avoid check-syntax, you can run: UCT_IGNORE_CHECK_SYNTAX=1 bzr ci === Other commands === Below are various tools to help maintain and manipulate UCT. All commands assume you are in the top-level directory of UCT (ie `cd $UCT`). * Verify syntax of CVE-* files:{{{ $ ./scripts/check-syntax }}} * To commit, use this command:{{{ $ ./scripts/check-syntax && bzr ci }}} NOTE: bzr branch creates a disconnected branch. To commit to a disconnected branch, use bzr ci followed by bzr push. bzr co creates a connected branch (bzr branch + bzr bind). To commit to a connected branch, use only bzr ci. To convert a disconnected branch to a connected branch, use bzr bind. * Listings of active CVEs:{{{ $ ./scripts/ubuntu-table $ ./scripts/ubuntu-table --supported 2>/dev/null | grep SUPPORTED # main only $ ./scripts/ubuntu-table --supported 2>/dev/null | grep PARTNER # partner only $ ./scripts/ubuntu-table --untriaged # only untriaged $ ./scripts/ubuntu-table --supported | egrep -v '[[:space:]]+(untriaged|negligible|low)$' # medium and higher }}} * Package status:{{{ $ ./scripts/pkg_status pkgname1 pkgname2 $ ./scripts/pkg_status -f pkgname1 pkgname 2 ... # full listing $ ./scripts/ubuntu-table -p linux -P linux -S | cut -c1-14,36-95 # kernel report }}} * CVE status:{{{ $ ./scripts/cve_status CVE-2006-4519 CVE-2007-2949 CVE-2007-3741 $ ./scripts/cve_status -s CVE-2006-4519 CVE-2007-2949 CVE-2007-3741 # short listing }}} * Create/edit a CVE without check-cves:{{{ $ ./scripts/active_edit -p package -c CVE-YYYY-XXXX $ ./scripts/active_edit -e -p package -c CVE-YYYY-XXXX # embargoed (private) $ ./scripts/active_edit -p package -c CVE-2010-NNN1 # new CVE without a CVE identifier }}} * Mass editing of CVEs:{{{ $ ./scripts/mass-cve-edit -p <package> -r <release> -v <version> -s <state> CVE... }}} Eg:{{{ $ ./scripts/mass-cve-edit -p seamonkey -r karmic -s released -v 1.1.17+nobinonly-0ubuntu1 CVE-2009-1841 CVE-2009-1838 CVE-2009-1836 CVE-2009-1835 CVE-2009-1392 CVE-2009-1832 CVE-2009-1833 CVE-2009-1311 CVE-2009-1307 }}} * Listing of packages with number of CVEs attached to them, weighted by CVE priority:{{{ $ ./scripts/cve_packages $ ./scripts/cve_packages -m # main, restricted and partner $ ./scripts/cve_packages -u # universe and multiverse $ ./scripts/cve_packages -t # just totals $ ./scripts/cve_packages -m | grep '^[0-9]' | sort -n # sorted by weight $ ./scripts/cve_packages -m -S | grep '^[0-9]' | sort -n # sorted by weight without devel }}} * CVEs needing retiring:{{{ $ ./scripts/cve_need_retire $ bzr mv $(./scripts/cve_need_retire -p) ./retired/ # move needed to retired }}} * Marking CVEs as 'released':{{{ $ cd $UCT $ rsync -v --progress -e ssh people.canonical.com:~ubuntu-security/public_html/usn/database.pickle ./database.pickle $ ./scripts/sync-from-usns.py database.pickle -u }}} Please see the UBUNTU-CVE Commands section in [[http://bazaar.launchpad.net/~ubuntu-security/ubuntu-cve-tracker/master/annotate/head%3A/README|ubuntu-cve/README]] for more information. === Reports === * By date, sorted by priority:{{{ $ ./scripts/report-date.py -S -m }}} * TODO list:{{{ $ ./scripts/report-todo $ ./scripts/report-todo [-S] }}} * SUmmary list:{{{ $ ./scripts/report-todo-numbers # summary report $ ./scripts/report-todo-numbers [-S] }}} * Monthly report:{{{ $ ./scripts/monthly-report }}} * HTML reports:{{{ $ ./scripts/html-report [-S] }}} |
Setup
Much of this can be found in ubuntu-cve/README.
Install the necessary software:
$ sudo apt-get install python-configobj python-yaml python-genshi python-progressbar git rsync libfile-rsyncp-perl w3m
Download the ubuntu-cve-tracker branch:
$ mkdir ~/git-pulls $ cd ~/git-pulls $ git clone git+ssh://<user>@git.launchpad.net/ubuntu-cve-tracker (replace <user> with your launchpad user).
Add the UCT environment variable to your startup scripts (eg ~/.bashrc) and have it point to the ubuntu-cve-tracker branch:
export UCT="$HOME/git-pulls/ubuntu-cve-tracker"
Create $HOME/.ubuntu-cve-tracker.conf to have something like:
# python-launchpad-bugs authentication cookies file (sis-changes) plb_authentication="/home/<username>/.mozilla/firefox/work.profile/cookies.sqlite" # path to Debian "security-tracker" GIT tree (check-cves) # get from https://salsa.debian.org/security-tracker-team/security-tracker secure_testing_path='/home/<username>/bzr-pulls/security-tracker' # path to archive-layout mirror of supported architectures # sis-generate-usn, packages-mirror) packages_mirror='<path>/ubuntu-archive-metadata' # same as packages_mirror, but for Debian testing repository debian_mirror='<path>/debian-testing-archive-metadata' # same as packages_mirror, but for the partner repository partner_mirror='<path>/partner-archive-metadata' # path to usn-tool bzr tree, used to manipulate USN databases # and for templates (sis-generate-usn) usn_tool='/home/<username>/bzr-pulls/usn-tool-trunk' # path to copy of master USN database, used when creating # a template for a USN (sis-generate-usn) usn_db_copy='/home/<username>/bzr-pulls/ubuntu-cve/database.pickle' # path to individual USN pickle database output when generating a single USN # database before merging into the master USN database (sis-generate-usn) usn_storage='/tmp/pending-usn' # path to git tree for the USN website which will be updated with new markdown # as USNs are published usn_website='/home/<username>/bzr-pulls/usn.ubuntu.com' # where to get the MITRE CVE and NVD databases (check-cves) mitre_loc="lillypilly.canonical.com:/home/ubuntu-security/cache" nvd_loc="lillypilly.canonical.com:/home/ubuntu-security/cache"
Populate the required data into the packages_mirror and partner_mirror directories:
cd $UCT;./scripts/packages_mirror
Populate the required data into the secure_testing directory:
$ cd ~/bzr_pulls $ git clone https://salsa.debian.org/security-tracker-team/security-tracker.git
Pull down an initial copy of the database.pickle database:
cd $UCT; ./scripts/fetch-db database.pickle.bz2
Usage
check-cves
check-cves is the tool to update the CVE database with new entries.
Prequisites
check-cves needs the archive layout mirrors to contain up-to-date information; the ${UCT}/scripts/packages-mirror script will do this for you.
Usage
To use manually, run like this:
$ cd $UCT ; bzr up $ ./scripts/check-cves $ ./scripts/check-cves http://cve.mitre.org/data/downloads/allitems.xml $ ./scripts/check-cves <file>
Eg:
$ wget -N http://cve.mitre.org/data/downloads/allitems.xml $ bzr up $ ./scripts/check-cves ./allitems.xml $ for i in $(seq 2004 $(date +%Y)) recent; do wget --quiet -N http://nvd.nist.gov/download/nvdcve-$i.xml; done $ ./scripts/check-cves ./nvdcve-2*.xml
The Ubuntu Security team does certain triage actions depending on the day of the week, and the process_cves script will guide you through these and call check-cves with appropriate arguments. Eg:
$ bzr up $ ./scripts/process_cves $ ./scripts/process_cves Mon $ ./scripts/process_cves Tue
Please see the TRIAGING section in ubuntu-cve/README for more information.
Embargoed Items
Embargoed items are supported in the following scripts in all the scripts except sync-from-usns.py. If no CVE has been assigned yet, an embargoed item should be prefixed with 'EMB-', followed by any combination of alphanumerics and dashes. Eg: EMB-xorg-2007-0001 EMB-foo
To include embargoed items simply create a symlink from 'embargoed' to the directory holding embargoed items. ubuntu-cve-tracker will not use 'embargoed' unless it is a symlink.
Pre-commit Syntax Checking
To perform pre-commit syntax checking, this little hack should allow for it:
$ mkdir -p ~/.bazaar/plugins/hooks $ cat > ~/.bazaar/plugins/hooks/__init__.py <<EOM #!/usr/bin/python from bzrlib.branch import Branch def run_tests(local, master, old_revno, old_revid, new_revno, new_revid, seven, eight): #print local, master, old_revno, old_revid, new_revno, new_revid, seven, eight import os if 'ubuntu-cve-tracker' in master.base and not os.environ.has_key('UCT_IGNORE_CHECK_SYNTAX'): import subprocess print '' rc = subprocess.call(['./scripts/check-syntax','--verbose']) if rc != 0: import sys sys.exit(1) Branch.hooks.install_named_hook('pre_commit', run_tests, 'CVE Tracker tests') EOM
With the above, check-syntax is always run before a commmit, or to avoid check-syntax, you can run: UCT_IGNORE_CHECK_SYNTAX=1 bzr ci
Other commands
Below are various tools to help maintain and manipulate UCT. All commands assume you are in the top-level directory of UCT (ie cd $UCT).
Verify syntax of CVE-* files:
$ ./scripts/check-syntax
To commit, use this command:
$ ./scripts/check-syntax && bzr ci
NOTE: bzr branch creates a disconnected branch. To commit to a disconnected branch, use bzr ci followed by bzr push. bzr co creates a connected branch (bzr branch + bzr bind). To commit to a connected branch, use only bzr ci. To convert a disconnected branch to a connected branch, use bzr bind.Listings of active CVEs:
$ ./scripts/ubuntu-table $ ./scripts/ubuntu-table --supported 2>/dev/null | grep SUPPORTED # main only $ ./scripts/ubuntu-table --supported 2>/dev/null | grep PARTNER # partner only $ ./scripts/ubuntu-table --untriaged # only untriaged $ ./scripts/ubuntu-table --supported | egrep -v '[[:space:]]+(untriaged|negligible|low)$' # medium and higher
Package status:
$ ./scripts/pkg_status pkgname1 pkgname2 $ ./scripts/pkg_status -f pkgname1 pkgname 2 ... # full listing $ ./scripts/ubuntu-table -p linux -P linux -S | cut -c1-14,36-95 # kernel report
CVE status:
$ ./scripts/cve_status CVE-2006-4519 CVE-2007-2949 CVE-2007-3741 $ ./scripts/cve_status -s CVE-2006-4519 CVE-2007-2949 CVE-2007-3741 # short listing
Create/edit a CVE without check-cves:
$ ./scripts/active_edit -p package -c CVE-YYYY-XXXX $ ./scripts/active_edit -e -p package -c CVE-YYYY-XXXX # embargoed (private) $ ./scripts/active_edit -p package -c CVE-2010-NNN1 # new CVE without a CVE identifier
Mass editing of CVEs:
$ ./scripts/mass-cve-edit -p <package> -r <release> -v <version> -s <state> CVE...
Eg:
$ ./scripts/mass-cve-edit -p seamonkey -r karmic -s released -v 1.1.17+nobinonly-0ubuntu1 CVE-2009-1841 CVE-2009-1838 CVE-2009-1836 CVE-2009-1835 CVE-2009-1392 CVE-2009-1832 CVE-2009-1833 CVE-2009-1311 CVE-2009-1307
Listing of packages with number of CVEs attached to them, weighted by CVE priority:
$ ./scripts/cve_packages $ ./scripts/cve_packages -m # main, restricted and partner $ ./scripts/cve_packages -u # universe and multiverse $ ./scripts/cve_packages -t # just totals $ ./scripts/cve_packages -m | grep '^[0-9]' | sort -n # sorted by weight $ ./scripts/cve_packages -m -S | grep '^[0-9]' | sort -n # sorted by weight without devel
CVEs needing retiring:
$ ./scripts/cve_need_retire $ bzr mv $(./scripts/cve_need_retire -p) ./retired/ # move needed to retired
Marking CVEs as 'released':
$ cd $UCT $ rsync -v --progress -e ssh people.canonical.com:~ubuntu-security/public_html/usn/database.pickle ./database.pickle $ ./scripts/sync-from-usns.py database.pickle -u
Please see the UBUNTU-CVE Commands section in ubuntu-cve/README for more information.
Reports
By date, sorted by priority:
$ ./scripts/report-date.py -S -m
TODO list:
$ ./scripts/report-todo $ ./scripts/report-todo [-S]
SUmmary list:
$ ./scripts/report-todo-numbers # summary report $ ./scripts/report-todo-numbers [-S]
Monthly report:
$ ./scripts/monthly-report
HTML reports:
$ ./scripts/html-report [-S]
SecurityTeam/UbuntuCVETracker (last edited 2023-11-28 22:07:17 by eslerm)