• Launchpad Entry: server-lucid-apt-mirror-ec2

• Created: NickBarcet

• Contributors: JamesTroup

• Packages affected: -

Summary

Improve the way our mirrors works on EC2. This is mainly an IS job.

Release Note

Package mirrors on EC2 have been improve to provide more robustness and better service.

Rationale

The current mirrors have the following problems:

• We assume that the mirror are not going to crash
• Long time to recover from a crash
• We need proper HA mirrors
• From AWS SLA, you can loose a complete availability zone without a problem but mirrors withtin a round robin need to be completely in sync

  • ==> we currently cannot guarantee this

User stories

Assumptions

Design

Identical to what is offered today, just more robust

Implementation

3 possible implementations to chose from:

1. sync proxy type setup
   • a proxy within each zone, mirror syncing from it

     ==> has to be tested need 6 boxes total instead of 4

2. LVM snapshots to make them sync transactionally
3. Using squid (similar to 1) Peer squid in diffrent regions, upstream being archive.u.c

UI Changes

No UI

Code Changes

Migration

No user migration required

Test/Demo Plan

Unresolved issues

BoF agenda and discussion

How does it work?

• Can it work in S3 directly?
  • Problem with symlinks
  • never been looked at by IS
  • unappropriate packages for a ppa
  • would have been elegant to have a repo be directly in S3
  • would need to improve many things
    • need to add a methd to apt
    • need tools to update the mirror

• plans for supporting fuse S3? -> scott moser [I tried creating an apt mirror in S3 using an rsync to an s3fs file system (fuse based). It was taking days to do the initial copy so I abandoned it, but would be willing to provide steps to whoever is investigating this -- Eric Hammond]

=== Problems ====

• WE assume that the mirror are not going to crash
• Long time to recover from a crash
• We need proper HA mirrors
• From AWS SLA, you can loose a complete region without a problem but mirrors withtin a round robin need to be completely in sync

  • ==> we cannot guarantee this

  • [Correction: The EC2 SLA assumes that an entire *availability zones* in a region can fail, but not an entire region --Eric Hammond]

Improvements

• sync proxy type setup
  • a proxy within each zone, mirror syncing from it

    ==> has to be tested need 6 boxes total instead of 4

• LVM snapshots to make them sync transactionally
• Using squid (similar to 1) Peer squid in diffrent regions, upstream being archive.u.c
  • Should we talk with amazon ? Simon/Nick to try establish communitcation
  • Documentation ?? Not right now, but once it is in good shape, yes, it should

  • Would be usefull to have a recommandation for uec, but not basedon this work (not the same constraints -> not te same solution)

    • Good candidate for a puppet reciepe

---

CategorySpec